Large language models - Mathematical Modeling in Biotechnology

What is an LLM?¶

A large language model (LLM) is a neural network trained to predict the next token in a sequence of text.

For example, given the prompt

The capital of France is

the model assigns probabilities to many possible next tokens and may generate

Paris

By repeatedly predicting one token at a time, an LLM can generate complete sentences, answer questions, summarize documents, write code, and perform many other language-based tasks.

Despite these impressive capabilities, an LLM fundamentally performs a simple task: it predicts the most likely continuation of a text sequence based on patterns learned from large amounts of training data.

Tokens and tokenization¶

LLMs do not process text character-by-character or word-by-word. Instead, they operate on units called tokens.

Examples:

biology → one token
machine learning → several tokens
punctuation marks may form separate tokens

Before text is processed by the model, it is converted into a sequence of tokens by a tokenizer.

The number of tokens in the prompt and response determines both memory usage and computational cost.

Next-token prediction¶

An LLM generates text one token at a time.

Given a sequence of previous tokens

t_1, t_2, \ldots, t_n

(1)

the model predicts a probability distribution for the next token

P(t_{n+1}\mid t_1,t_2,\ldots,t_n)

(2)

A response is then generated by repeatedly selecting and appending new tokens.

Consequently, every capability of an LLM — question answering, translation, summarization, and code generation — ultimately emerges from next-token prediction.

Running a local language model¶

In this notebook we use a local language model executed through the llama_cpp library.

Unlike cloud-based services, local inference offers several advantages:

no internet connection is required
data remains on the local computer
no external API costs
complete control over model selection

The downside is that local models are typically smaller and slower than the largest commercial systems.

import llama_cpp

# https://huggingface.co/Qwen/Qwen2.5-3B-Instruct-GGUF/blob/main/qwen2.5-3b-instruct-q4_k_m.gguf
model = r"C:\Users\kthpa\Downloads\qwen2.5-3b-instruct-q4_k_m.gguf"

llm = llama_cpp.Llama(
    model_path=model,
    n_ctx=1024,  # Context window size
    verbose=False,
)

The model can only consider a limited number of tokens at a time. This limit is called the context window.

Prompting¶

The text supplied to the model is called the prompt.

Small changes in wording can lead to substantially different responses.

For example:

“Explain support vector machines.”
“Explain support vector machines to a biotechnology student.”
“Explain support vector machines in one sentence.”

Prompt design is often referred to as prompt engineering.

Define the system prompt¶

system_prompt = """
You are a teaching assistant for the KTH course CB1020.

Your audience consists of undergraduate biotechnology students.

When answering:
- use clear language
- explain technical terms
- give short examples when helpful
- avoid unnecessary jargon
- admit uncertainty when appropriate

Prefer concise answers unless the user explicitly requests details.
"""

Add user input in an interactive chat loop¶

import textwrap

print("🤖 Qwen Chat Session Started! Type 'exit' or 'quit' to stop.\n")

while True:

    # Get user input
    user_query = input("You: ")

    # Exit condition
    if user_query.strip().lower() in ["exit", "quit"]:
        print("Ending chat session. Goodbye!")
        break

    # Skip empty input
    if not user_query.strip():
        continue

    print("\nQwen: ", end="", flush=True)

    # Generate response using token streaming
    response = llm.create_chat_completion(
        messages=[
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": user_query},
        ],
        temperature=0.2,
        max_tokens=512,
        stream=False,
    )

    # Extract the generated text
    response_text = response["choices"][0]["message"]["content"]

    print(textwrap.fill(response_text, width=60, replace_whitespace=False))
    print("\n" + "-" * 50 + "\n")

🤖 Qwen Chat Session Started! Type 'exit' or 'quit' to stop.

You:  What is aspirin


Qwen: Aspirin, also known as acetylsalicylic acid, is a medication
that is commonly used to relieve pain, reduce inflammation,
and lower fever. It is derived from the bark of the willow
tree, which contains a natural compound called salicylic
acid. Aspirin is available over-the-counter and is often
used to treat conditions like headaches, muscle aches, and
minor injuries. It works by inhibiting the production of
prostaglandins, which are substances that cause pain,
inflammation, and fever.

--------------------------------------------------

You:  quit

Ending chat session. Goodbye!

Temperature and randomness¶

LLMs assign probabilities to many possible next tokens.

The temperature parameter controls how strongly the model prefers the most likely tokens.

Low temperature: more deterministic responses
High temperature: more diverse and creative responses

For scientific applications, low temperatures are often preferred because they tend to produce more stable and factual responses.

def llm_query(user_query, temperature):
        
    response = llm.create_chat_completion(
        messages=[
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": user_query},
        ],
        temperature=temperature,
        max_tokens=512,
        stream=False,
    )

    return response["choices"][0]["message"]["content"]

for temperature in [0.0, 2.0]:

    print("temperature:", temperature)

    for repeat in range(2):
        
        print("query:", repeat)
        response_text = llm_query("what is aspirin", temperature)
        print(textwrap.fill(response_text, width=60, replace_whitespace=False))

temperature: 0.0
query: 0
Aspirin, also known as acetylsalicylic acid, is a medication
that is commonly used to relieve pain, reduce inflammation,
and lower fever. It is also used to prevent blood clots and
reduce the risk of heart attacks and strokes. Aspirin works
by blocking the production of certain chemicals in the body
that cause pain, inflammation, and fever. It is available
over the counter in many countries and is one of the most
widely used medications globally.
query: 1
Aspirin, also known as acetylsalicylic acid, is a medication
that is commonly used to relieve pain, reduce inflammation,
and lower fever. It is also used to prevent blood clots and
reduce the risk of heart attacks and strokes. Aspirin works
by blocking the production of certain chemicals in the body
that cause pain, inflammation, and fever. It is available
over the counter in many countries and is one of the most
widely used medications globally.
temperature: 2.0
query: 0
Aspirin, also known as acetylsalicylic acid, is a common
medication used to relieve pain, reduce inflammation, and
lower body temperature. It is derived from the bark of
willow trees, where it naturally occurs as part of the
willow tree's salicylic acid compound. Aspirin is easily
available as a common over-the-counter medication.
query: 1
Aspirin, also known as acetylsalicylic acid, is a non-
steroidal anti-inflammatory drug (NSAID) that is commonly
used to relieve pain, reduce inflammation, and lower fever.
It is also effective in lowering the risk of heart attacks
and strokes in people with certain conditions.

As a
chemical compound, aspirin is a white, crystalline solid and
has a strong odor. It is made from the bark of the willow
tree, which contains salicin, a compound that has a similar
structure to aspirin and provides its anti-inflammatory and
pain-relieving properties. The acetylation step in aspirin's
synthesis prevents the formation of salicylpiridine, a toxic
substance.

Aspirin is sold as an over-the-counter drug in
many countries, often in combination with other pain
relievers, or for heart attack and stroke prevention.

Limitations and hallucinations¶

LLMs do not possess understanding in the human sense.

They generate responses by extending text sequences according to patterns learned during training.

As a result, models may produce statements that sound plausible but are factually incorrect. Such fabricated information is known as a hallucination.

LLM-generated information should therefore be checked carefully, especially in scientific and medical contexts.

response_text = llm_query("what is biozolumab", 0.0)
print(textwrap.fill(response_text, width=60, replace_whitespace=False))

Biozolumab is a monoclonal antibody drug used in the
treatment of certain types of cancer. It is designed to
target and destroy cancer cells while minimizing damage to
healthy cells. This drug is part of a class of treatments
called immunotherapy, which helps the body's immune system
fight cancer.

In this example biozolumab is a fake name, but the LLM confidently tells us something else.

Domain-specific language models¶

General-purpose language models are trained on broad collections of text from many domains.

Specialized models can be further trained on domain-specific literature.

Examples include:

chemistry language models
biomedical language models
legal language models

In this course we compare a general-purpose model with a chemistry-focused model. Specialization often improves performance on technical terminology and scientific concepts.

Take-away¶

LLMs are neural networks trained to predict the next token in a text sequence.
Complex abilities such as question answering and code generation emerge from repeated next-token prediction.
Prompt wording and temperature can strongly influence the generated response.
Language models may produce incorrect information (hallucinations).
Domain-specific language models can improve performance for specialized tasks such as chemistry and biotechnology.