← Back to BLACKWIRE EMBER BUREAU MODEL ARCHITECTURE Diagram comparing absolute positional vectors in GPT-2 with RoPE integration in MiniMax-M3.

Figure 2 shows token‑embedding layers in GPT‑2 and MiniMax‑M3, highlighting where each model adds positional information.

POSitional EMBEDDINGS EXPLAINED: HOW MODERN LLMs MAP TOKEN POSITIONS

*Sebastian Raschka breaks down the hidden math that lets large language models understand word order. The piece maps classic absolute encodings to the newer RoPE technique used in 2026 open‑weight models.*

By EMBER Bureau - BLACKWIRE  |  November 15, 2026, 17:00 CET  |  positional embeddings, LLMs, RoPE, GPT-2, MiniMax-M3

Sebastian Raschka, PhD, released a deep‑dive on November 15 2026 that finally tackles the “minor” building block many have ignored: positional embeddings. The piece arrives as Raschka prepares a broader LLM architecture trends article, yet he pauses to map how token order is encoded from GPT‑2’s additive vectors to MiniMax‑M3’s RoPE integration. Readers have repeatedly asked for a tutorial on relative positional embeddings, prompting Raschka to revisit old drafts and deliver a walkthrough packed with visual aids, math snippets, and a PyTorch demo. The article is technical, aimed at those familiar with basic transformer pipelines, and it promises to become a reference for anyone dissecting 2026 open‑weight models.

How does token position enter a transformer?

Raschka outlines the pipeline: a tokenizer turns text into token IDs, the token‑embedding layer maps each ID to a feature vector, then positional data is added. In GPT‑2 the position vector is summed with the token embedding before the first transformer block. MiniMax‑M3, a model released in 2026, injects position inside every transformer block using Rotary Positional Embedding (RoPE) within the attention mechanism. The article visualises this contrast in Figure 2, highlighting the orange token‑embedding blocks. The distinction matters because it changes where and how the model learns order, not the downstream feed‑forward layers. Raschka notes that MiniMax‑M3’s first three of its 60 feed‑forward layers are dense, not MoEs, underscoring that positional handling is a core architectural decision.

Why is positional information critical?

Raschka uses a toy example: “The red fox jumps over the gray fox.” versus “The gray fox jumps over the red fox.” Swapping adjectives flips which fox acts. Both sentences contain eight token positions (0‑7) and the word “fox” appears twice with the same token ID (2). Without position cues the model would treat both foxes identically, losing meaning. The article stresses that real tokenizers split words further, but the principle remains: order determines semantics. Figure 3 illustrates the meaning shift, while Figure 4 shows token IDs aligned with positions. The need for explicit positional encoding is therefore not a cosmetic detail but a functional requirement for accurate language understanding.

"I finally revisited my old draft and put this walkthrough together" – Raschka on delivering the positional embedding guide.

Which modern models rely on RoPE versus absolute encodings?

The 2026 open‑weight model gallery, previewed in Figure 1, groups architectures by their positional strategy. Raschka points out that classic models like GPT‑2 still use absolute position vectors added pre‑transformer. In contrast, newer releases such as MiniMax‑M3 adopt RoPE, embedding position directly into the attention computation. The article promises a deeper dive in Section 6, where the author lists the exact choices used across contemporary open models. This shift reflects a broader trend toward relative encodings that can generalise across sequence lengths without fixed vectors.

What extra resources does the author provide?

Raschka supplies visual examples, a modest amount of math, and a small PyTorch implementation to let readers reproduce the embeddings. He links to his earlier self‑attention walkthrough (January 14 2024) for readers fuzzy on query, key, and value vectors. Substack’s new “perks” system now offers image exports and metadata from the architecture gallery for independent analysis. Raschka also notes the article reads best in a web browser and that a more relaxed style will return in his next piece. These supplemental tools aim to cement the positional embedding discussion as a reference point for future LLM architecture analyses.

Frequently asked questions

What is RoPE and how does MiniMax-M3 use it?

RoPE stands for Rotary Positional Embedding, a relative positional method. MiniMax-M3 adds positional information inside each transformer block using RoPE within the attention mechanism.

How does GPT-2 incorporate token positions?

GPT-2 adds a position vector to each token embedding before the modified embedding enters the first transformer block.

Why are positional embeddings necessary for language models?

Changing word order changes meaning, as shown by the example sentences about foxes. Without position cues, identical token IDs would be treated the same, losing semantic distinction.

Raschka’s exposition strips away the veneer of novelty to expose the mechanics that let LLMs respect word order. By contrasting GPT‑2’s absolute vectors with MiniMax‑M3’s RoPE, he highlights a decisive architectural shift that will shape future model design. As open‑weight models proliferate, understanding positional encoding will be as essential as mastering attention itself.

Sources: Hacker News (https://magazine.sebastianraschka.com/p/understanding-positional-embeddings)

More coverageOpenAI