This repository is an educational, pure C# implementation of the main steps needed to run Qwen 2.5 0.5B Instruct from an FP16 GGUF model file.

It is written for regular software developers who want to understand what an LLM engine actually does. The code favors ordinary arrays, loops, descriptive names, and comments over speed.

Goal: make the path from "What is the capital of Ukraine?" to "Kyiv" visible and understandable.

By following one generated token through the project, you will see how an inference engine:

1. Reads model settings and learned numbers from a GGUF file.
2. Converts text into token IDs.
3. Converts a token ID into a meaning vector.
4. Uses self-attention to look at earlier tokens.
5. Processes information through a small neural network.
6. Produces one score for every possible next token.
7. Selects a token, appends it, and repeats.

This project does not train a model. Training creates the learned weights. Inference loads those existing weights and uses them to answer questions.

• .NET 10 SDK
• Several gigabytes of free RAM
• The Qwen model: qwen2.5-0.5b-instruct-fp16.gguf

The default model path is:

D:\LLMModels\qwen2.5-0.5b-instruct-fp16.gguf

Change defaultModelPath in Program.cs if your file is stored elsewhere.

dotnet run --project src\SimpleLlmInference.Console

Run with a different model path and question:

dotnet run --project src\SimpleLlmInference.Console -- `
  "D:\LLMModels\qwen2.5-0.5b-instruct-fp16.gguf" `
  "What is the capital of France?"

dotnet test

The integration test loads the real model, asks for the capital of Ukraine, and checks that the answer contains Kyiv.

Open the test: SimpleInferenceEngineTests.cs

flowchart LR
    A["Human question<br/>What is the capital of Ukraine?"]
    B["Tokenizer<br/>text -> token IDs"]
    C["Embedding<br/>token ID -> meaning vector"]
    D["24 transformer layers<br/>attention + feed-forward"]
    E["Output projection<br/>hidden vector -> logits"]
    F["Greedy selection<br/>largest logit wins"]
    G["Generated token<br/>Kyiv"]
    H["KV cache<br/>memory of earlier tokens"]

    A --> B --> C --> D --> E --> F --> G
    D <--> H
    G -. "feed token back into model" .-> C

The engine predicts only one token at a time. A full answer appears because the selected token is fed back into the model and the process repeats.

Start at SimpleInferenceEngine. Its constructor creates:

• a GgufReader, which reads the file;
• a QwenTokenizer, which understands Qwen's vocabulary;
• a QwenModel, which owns the transformer weights and equations.

A model file mostly contains huge tables of numbers. Those numbers are called weights. Training adjusted them until the model became good at predicting text. Inference does not know facts through if statements; knowledge is distributed across these learned numbers.

Open GgufReader.cs.

GGUF is a model container format. Think of it as a ZIP-like box designed for LLMs. It contains:

• metadata such as architecture name and layer count;
• tokenizer vocabulary and merge rules;
• tensor names and shapes;
• the actual learned weight values.

flowchart TB
    GGUF["GGUF file"]
    Header["Header<br/>version and counts"]
    Metadata["Metadata<br/>layers, heads, tokenizer"]
    Directory["Tensor directory<br/>name, shape, type, offset"]
    Data["Tensor data<br/>hundreds of millions of learned numbers"]

    GGUF --> Header
    GGUF --> Metadata
    GGUF --> Directory
    GGUF --> Data
    Directory -. "tells reader where to seek" .-> Data

The reader first loads the table of contents. When Tensor(name) is called, it jumps to the tensor's byte offset and reads its values.

Open QwenTokenizer.cs.

The model cannot read C# strings. It reads integer token IDs. A token may be a word, part of a word, punctuation, or whitespace.

Example:

"Ukraine" -> ["Uk", "raine"] -> [31231, 1472]

The exact split depends on Qwen's vocabulary.

This tokenizer uses byte-pair encoding, usually called BPE:

1. Convert text to UTF-8 bytes.
2. Give every byte a safe character representation.
3. Start with tiny pieces.
4. Repeatedly join neighboring pieces using Qwen's learned merge priority.
5. Look up the final pieces in the vocabulary.

The question is wrapped in Qwen's chat template:

<|im_start|>system
You are a helpful assistant...
<|im_end|>
<|im_start|>user
What is the capital of Ukraine?
<|im_end|>
<|im_start|>assistant

The final open assistant message tells Qwen that its next tokens should be the answer.

Open Forward(...) in QwenModel.cs.

A token ID is just an integer. The first model operation looks up its embedding: a row of learned decimal numbers representing the token.

token ID 42
    |
    v
embedding table row 42
    |
    v
[0.13, -0.44, 0.08, ...]

No individual number means "capital" or "Ukraine". Meaning is represented by the overall direction and pattern of the vector.

Every transformer layer has two major parts:

1. Attention: gather useful information from earlier tokens.
2. Feed-forward network: process the gathered information.

flowchart TB
    Input["Input hidden vector"]
    Norm1["RMSNorm"]
    QKV["Create Query, Key, Value"]
    Rope["RoPE adds position"]
    Cache["Store Key and Value"]
    Attention["Causal self-attention"]
    Project["Output projection"]
    Residual1["Add original input<br/>residual connection"]
    Norm2["RMSNorm"]
    FFN["SwiGLU feed-forward network"]
    Residual2["Add previous state<br/>residual connection"]
    Output["Layer output"]

    Input --> Norm1 --> QKV --> Rope --> Cache --> Attention --> Project
    Input --> Residual1
    Project --> Residual1 --> Norm2 --> FFN --> Residual2 --> Output
    Residual1 --> Residual2

This sequence is repeated for every Qwen layer.

Attention creates three vectors:

The query is compared with every earlier key using a dot product. A higher score means the vectors point in similar directions and are probably relevant. Softmax changes the scores into shares that add up to one. The engine then mixes the value vectors using those shares.

Example attention chart:

The percentages above are only an illustration, not values captured from this model.

Open KvCache in QwenModel.cs.

Earlier keys and values do not change while generating later tokens. The KV cache stores them instead of recalculating them.

sequenceDiagram
    participant Engine
    participant Model
    participant Cache as KV Cache

    Engine->>Model: Prompt token 1
    Model->>Cache: Store K1 and V1
    Engine->>Model: Prompt token 2
    Model->>Cache: Read K1/V1, store K2/V2
    Engine->>Model: Generated token 1
    Model->>Cache: Read earlier K/V, store new K/V

Think of it as keeping notes from previous pages instead of rereading the whole book before writing every next word.

Attention alone sees vectors, but word order matters:

dog bites man
man bites dog

RoPE, or Rotary Position Embedding, rotates pairs of query and key numbers by angles based on token position. It is similar to turning clock hands by a different amount at each position. Because queries and keys are rotated, attention can detect order and relative distance.

Open ApplyRope(...) in QwenModel.cs.

After attention, each token passes through a wider feed-forward network.

Qwen uses SwiGLU:

output = SiLU(gate) * up

In plain language:

• up creates many possible features;
• gate decides which features should pass;
• SiLU makes the gate smooth instead of simply on or off;
• down compresses the wider result back to the normal hidden size.

Open the feed-forward section inside Forward(...) in QwenModel.cs.

After the final layer, the model creates one logit for every vocabulary token.

A logit is a raw preference score, not a percentage. This project uses ArgMax(...): select the token with the largest logit.

That is called greedy decoding. It is simple and repeatable, but less creative than probabilistic sampling.

Open AnswerAsync(...) and ArgMax(...) in SimpleInferenceEngine.cs.

LLM inference has two phases.

flowchart LR
    Prompt["Complete prompt"]
    Prefill["Prefill<br/>process every prompt token"]
    Cache["Filled KV cache"]
    Decode1["Decode step<br/>choose next token"]
    Decode2["Feed chosen token back"]
    Stop{"Stop token or limit?"}
    Answer["Final answer"]

    Prompt --> Prefill --> Cache --> Decode1 --> Decode2 --> Stop
    Stop -- "No" --> Decode1
    Stop -- "Yes" --> Answer

The engine processes all prompt tokens to build context and fill the KV cache. No answer is shown yet.

The engine repeatedly:

1. Reads the latest logits.
2. Chooses the largest logit.
3. Converts that token to text.
4. Passes the token back through the model.
5. Stops at Qwen's end token or the configured token limit.

Open TransformerMath.cs beside this section. Each video below was checked to be shorter than five minutes.

Suppose the input vector is:

[2, 3]

and one matrix row is:

[4, 5]

The result for that row is:

2 * 4 + 3 * 5 = 23

The row is a learned recipe. A full matrix contains many recipes and therefore produces many output numbers.

RMSNorm calculates the typical magnitude of a vector:

root mean square = sqrt(mean(each value squared) + tiny safety value)

Each input is divided by that magnitude and multiplied by a learned weight. The pattern remains, but its scale becomes predictable.

Given scores:

[1, 2, 4]

softmax produces approximate shares:

[4%, 11%, 84%]

They are all positive and total 100%. The largest score receives most of the attention.

Instead of replacing the old state:

state = newInformation

a transformer adds a correction:

state = state + newInformation

This creates a clear path for existing information to continue through deep networks.

These terms often appear in LLM documentation. This project avoids them so the core algorithm remains readable.

This project accepts FP16 and FP32 GGUF tensors. It intentionally rejects quantized tensor types with a clear error.

The result is slow, but every important transformation can be followed in a debugger.

Set breakpoints in this order:

1. SimpleInferenceEngine.AnswerAsync
2. QwenTokenizer.EncodeChat
3. QwenModel.Forward
4. TransformerMath.RmsNorm
5. QwenModel.Attention
6. TransformerMath.SoftmaxInPlace
7. SimpleInferenceEngine.ArgMax

Useful values to inspect:

• prompt: the token IDs representing the chat;
• hidden: the current token representation;
• query, key, and value: attention vectors;
• scores: relevance scores for earlier positions;
• logits: one next-token score per vocabulary item;
• nextToken: the selected vocabulary ID.

• CPU-only and single-threaded.
• Intentionally slow matrix operations.
• Loads FP16 weights into larger FP32 arrays.
• Supports the tensor names and architecture used by Qwen 2.5.
• No quantized Q4/Q8 tensors.
• No random sampling, temperature, top-k, or top-p.
• No HTTP server, streaming API, batching, or concurrent requests.
• Designed for learning, not production deployment.

src/
  SimpleLlmInference/
    GgufReader.cs             Reads the GGUF model file
    Tensor.cs                 Stores shaped numeric data
    QwenTokenizer.cs          Converts text and token IDs
    TransformerMath.cs        Basic mathematical operations
    QwenModel.cs              Qwen transformer and KV cache
    SimpleInferenceEngine.cs  Autoregressive generation loop

  SimpleLlmInference.Console/
    Program.cs                Small command-line example

tests/
  SimpleLlmInference.Tests/
    SimpleInferenceEngineTests.cs

An LLM inference engine is a loop around a large mathematical function:

question
  -> tokens
  -> vectors
  -> repeated attention and feed-forward layers
  -> scores for possible next tokens
  -> choose one token
  -> repeat until finished

The model's intelligence is in the learned weights. The engine's job is to faithfully load those weights, perform the transformer equations, remember past tokens, and repeat the next-token prediction loop.