Inside The Forward Pass
Moving data through the neural network. Math degree not required.
Introduction
This article explores what happens during a Large Language Model's (LLM) forward pass without diving too deeply into mathematics. Instead, we'll focus on the individual components, what they do, and how they work together to generate text. Some details are intentionally simplified to make it easier to follow.
For those interested, optional deep dives explore how these components work inside Brewference, my Java-based inference engine.
Tokens and their Embedding Table
Every word (technically text, since punctuation and whitespace are considered) can either be a token or a combination of tokens. These tokens are built using a handful of techniques, such as Byte Pair Encoding and Unigram.1
The selected word or partial word is determined over the entire text corpus for the sake of compression.2 Once the vocabulary is set, which varies by model, it is frozen and used for training.
During training, the tokens are used to build an embedding table, which is essentially a dictionary of each token to a set of numbers (vector or one-dimensional tensor).
The table is used during the forward pass by taking the currently selected token (part of the prompt or previously generated token), retrieving the vectors, applying calculations and producing logits.
The logits are scores for every word3 in the vocabulary. So if the vocabulary contains 4,096 tokens, you get 4,096 logits. If we use deterministic processing (known as argMax), we loop through these logits and find the biggest number. That number's index (or position in the logit list) is the token index in our dictionary.
In Brewference, the TransformerGraph interface is the blueprint for all transformers
within the project. Within the forward pass, we pull and embedding table entry for the current token
and store its data in the runState.x array.
Since the table is contiguous and all token embeddings are the same byte-size, we can navigate through the data by using the token's index in the dictionary.
In Listing 01, we can see this copy of data and at the same time dequantizing the
data back into 32-bit single-precision floating-point value for Java-native calculations.
This is partly a consequence of originally targeting Java 8, other engines perform calculations directly on the quantized data.
The Layer Loop
Most modern LLMs are built with surprisingly similar components. While the details vary between models, the general process is largely the same.
- Normalization
- RoPE
- Residual Addition
- Attention
- Feed-Forward Network
Each layer takes information from the previous one, performs a series of operations, and passes the result along to the next.
This overall process may differ between model families. Some may normalize more or less often, use a variation of Rotary Position Encodings, or use alternative implementations of the Feed-Forward Network. But the underlying principles stay the same.
The total number of layers is baked into each model and differ not only between families but also sizes. For instance, Qwen3 0.6B has 28 layers, while Qwen3 4B has 36.
The same is true with embedding lengths, which is the size of the vector for each token. Qwen3 0.6B's embedding length is 1024, while 4B is 2560. These size differences aren't a coincidence, there is a strong relationship between the vector size and the overall size of the model.
Attention Heads - Multiple Perspectives on Language
Think of attention heads like having multiple specialists working together, each examining the same sentence but focusing on different aspects. Just as any team may include different experts, analysts, and other specialists, an LLM uses its heads to look at language from multiple angles.
One head might specialize in identifying grammatical relationships, while another focuses on tracking pronouns ("Kelly" is also "she"). Through training, different heads can develop different patterns of attention, allowing the model to examine relationships between words in multiple ways.
These heads don't simply examine a sentence once and remember what they found. As each new token is processed, they calculate new attention patterns using information retained from previous tokens. This allows the model to continually reconsider which parts of the context are more relevant.
Models have varying numbers of heads. For example, Qwen3 models range from 16 heads in smaller models to 64 in the large 32-billion-parameter model. These heads operate in parallel, giving the model multiple ways to examine relationships within the context.
Remembering What Came Before
Consider the sentence "Kelly picked up her book at the library." When processing "her," the model needs information from earlier to establish the relationship with "Kelly."
Each head can look back at previously processed tokens, deciding which ones are most relevant. Different heads may focus on different parts of that history, even though they are all working with the same context.
This is accomplished using three components: Query, Key, and Value. The Query represents what the current token is looking for, the Keys help determine which tokens are relevant, and the Values contain the information gathered from those tokens.
Repeatedly calculating everything about the previous tokens would be computationally intensive. So instead, the model maintains a collection of previous calculations so that they can be reused. This cache is known as the Key-Value (KV) cache.
The cache grows as the context expands with every generated token.
In Brewference (and other inference engines), the model consumes a large amount of memory, but additional headroom is needed for the KV cache, which directly affects the length of a conversation (context).
The amount of available memory, along with the limitations of the model itself, determines how much cache can be allocated and, therefore, how long a conversation can last.
Once the cache is full, the model can no longer store additional calculations, preventing further token generation (generally speaking). However, techniques can be used to keep the conversation going. Some techniques are built into the model itself, while others are implemented as part of the toolchain.
Real-world example: A 24 billion parameter 8-bit model will weigh in at around 24 GB. That might fit on an RTX 4090 which has 24GB of VRAM, but that would leave very little room for the KV cache, making it practical only for shorter conversations. You could spill into the system RAM, but that causes slower token generation.
Note: While the KV cache and context window aren't technically the same mechanism, they are directly related. The context window determines how much information the model can consider, while the KV cache stores representations of previously processed tokens. As the context grows, so does the cache, requiring additional memory to accommodate the longer context.
Brewference supports KV cache quantization to 16-bit values and sliding window (a function of the Phi-3 model.) Many other engines feature more aggressive quantization with much better performance.
A Note About Normalization
Normalization is fairly straightforward. It adjusts the scale of numerical values to keep them within a manageable range. This helps stabilize data as it passes through the model.
Think of it as making very small numbers larger, or very large numbers smaller, while generally preserving their relationships to one another.
Typical normalization for this topic is Root Mean Square Normalization (RMSNorm), which adjusts values based on their overall magnitude.
A Briefing on Residual Addition
This is an important step in the overall forward pass, but fortunately like normalization, it's pretty straight-forward.
Think of it as an arithmetic accumulator. It preserves the information the model already has while adding the changes produced by the attention or feed-forward processes.
Rotary Position Encoding (RoPE)
As we all know, the order of words can change the meaning of a sentence. "He sat on the chair" means something different from "The chair sat on he." At this point in the pass, the model has numerical representations of the tokens but no idea where they lie within the sentence.
RoPE introduces positional information by rotating pairs of numbers within the Query and Key data used by the attention process.
That might be hard to visualize, but think back to high school math, where a pair of numbers can represent coordinates on a graph. Now draw an arrow from the center of the graph to that point. That arrow will have a certain length. Now imagine that arrow and the dot rotating around the center point, like a clock hand. RoPE works on a similar principle. It rotates the arrow, which changes the coordinates, but the length stays the same.
So how does this determine position? To put it simply, if one clock hand points to 1 and another points to 2, you know they are 1 step apart or a 30 degree difference. There is a small "problem," though. Once a clock completes a full rotation, the hand returns to its original direction. This means a single clock cannot distinguish between certain positions. RoPE addresses this by using multiple clocks, each rotating at a different frequency.
Feed-Forward Network (FFN)
After attention gathers information from the context, the Feed-Forward Network processes that information using patterns learned during training.
Unlike attention, which looks at relationships between tokens, the FFN processes each token's information individually.
Different models use variations of this process, such as SiLU, SwiGLU, and GeGLU. These are mathematical functions that transform the data in different ways.
Once the FFN finishes, its results are added back to the existing information through our friend, Residual Addition, and the model moves on to the next layer.
Brewference supports SwiGLU and GeGLU activations. I'm not going to pretend I fully understand the mathematics behind these functions. I understand how they're implemented, but the cleverness is in how the model learns to use them.
Logits and Token Sampling
After the token (if you got this far, the token being analyzed is either part of a prompt or a previously generated token) has passed through every layer, the model is ready to make a prediction.
The resulting numerical data is converted into a list of scores called logits. Each logit corresponds to a token in the model's vocabulary, meaning every single token in the vocabulary has a value. Higher scores generally indicate a more likely continuation of the text.
The next step is to reduce that set down to a single token. That token will be selected, added to the output, and sent back through the forward pass process to be repeated until generation is complete.
The simplest way to select a token is to choose the one with the highest score. This is called Argmax. The issue with doing this is that the highest-scoring token can produce both predictable and repetitive results. Not ideal for most cases where an LLM is useful.
Instead, many inference engines use sampling techniques that convert these scores into probabilities and select a token based on those probabilities. If you have ever modified settings when using a local LLM setup, you definitely noticed Temperature, Top-K, Top-P and others. These settings influence how a token is ultimately selected.
You may wonder: What ends the generation? The models are trained to identify when a sentence should end. They mark these endings with a special token, EOS or end of sentence. This token informs the inference engine that generation is complete and returns control to the application or user.
Argmax is a simple, deterministic operation, meaning that the same prompt on the same model will produce the same logits, resulting in the same generated token. This is great for benchmarking and testing because the output can be verified (with caveats). However, in practice, this is not the best approach for getting great results.
Alternatives that sample from the logits along with using settings like Temperature, Top-K, and Top-P introduce random variation to the mix. This type of method is considered stochastic, meaning it is governed by a probability distribution.
The code to the right shows some of the methods used. Most are simple in nature, but when combined, they add that bit of randomness needed.
Temperature adjusts the scale of the logits by division. Low values (smaller than 1.0) increase the distance between logits, making it more likely that higher probability logits will be selected, whereas higher values smooth things out. Imagine the bell curve rising sharply like a mountain versus flattening out like a small hill. (Important note, the logits are not sorted in memory like a bell curve, illustrative purposes only.)
Top-K, where K is not an acronym, simply limits the number of logits considered. It reduces the set to the specified value. If there are 4,096 logits and the Top-K is 60, only the 60 largest logits from the group will be considered.
Top-P sets a threshold for total combined probabilities. These probabilities are generated by a mathematical function called softmax, using the logits after the Top-K and Temperature have already been applied. Essentially, if Top-P is set to 0.9, the probabilities are added up until they meet or slightly exceed 0.9. Note that it is possible for the threshold to never be reached during this operation, in which case the Top-P has no effect.
None of the above functions truly introduce randomness. Instead, they prepare and tweak the data for the final sampling. This is performed by a weighted random sampler that typically uses a pseudo-random number generator applied to a sum of the probabilities. The sampler then loops through those probabilities, adding them together until the sum meets (or exceeds) the random threshold. (This describes cumulative probability sampling; other methods exist.)