The Causal Grassmann layer replaces the dense attention matrix with geometric operations on a Grassmann manifold.
GRASSMANN-BASED MODELS CHALLENGE ATTENTION IN NLP
*Zhang Chong's Dec 2025 paper argues that self‑attention is not essential for strong language models. An attention‑free architecture built on Grassmann flows reaches near‑Transformer performance on key benchmarks.*
Zhang Chong’s paper, submitted to arXiv on 22 Dec 2025, asks a blunt question: is explicit self‑attention truly required for high‑performing sequence models? The author reframes multi‑head attention as a tensor‑lifting operation—powerful but mathematically opaque—and offers an attention‑free alternative built on Grassmann flows. The proposed Causal Grassmann layer replaces the dense attention matrix with low‑rank geometric operations on a manifold, promising both interpretability and competitive performance. Early results on Wikitext‑2 and SNLI suggest the approach narrows the gap with Transformers while exposing a more structured computational substrate. If the community embraces these findings, the next wave of language models could be defined by geometry rather than brute‑force attention.
Why attention has dominated NLP
Standard multi‑head attention is described by the authors as a form of tensor lifting: hidden vectors are projected into a high‑dimensional space of pairwise interactions. Learning proceeds by constraining this lifted tensor through gradient descent. The mechanism is extremely expressive, yet mathematically opaque; after many layers the model resists description by a compact set of invariants. This opacity has kept attention at the core of most sequence‑modeling advances despite its computational cost.
Can Grassmann flows replace self‑attention?
The paper proposes a Causal Grassmann layer that eliminates the L × L attention matrix. First, token states are linearly reduced. Next, local token pairs are encoded as two‑dimensional subspaces on a Grassmann manifold using Plücker coordinates. Finally, these geometric features are fused back into hidden states through gated mixing. Information propagates via controlled deformations of low‑rank subspaces over multi‑scale local windows, keeping core computation on a finite‑dimensional manifold rather than an unstructured tensor space.
How does the Grassmann approach perform?
On the Wikitext‑2 language‑modeling benchmark, purely Grassmann‑based models with 13 – 18 million parameters achieve validation perplexities within about 10 – 15 percent of size‑matched Transformers. On the SNLI natural‑language inference task, a Grassmann‑Plücker head atop DistilBERT records validation and test accuracies of 0.8550 and 0.8538, marginally beating a Transformer head that scores 0.8545 and 0.8511 respectively. These figures show that the attention‑free design can compete with established attention mechanisms.
What does this mean for future AI research?
The authors analyze Grassmann mixing complexity and demonstrate linear scaling in sequence length for fixed rank. By grounding computation in a structured manifold, the design offers a clearer path toward geometric and invariant‑based interpretations of neural reasoning. This suggests a shift from opaque tensor lifting toward models whose internal dynamics can be mathematically characterized, potentially easing analysis, debugging, and theoretical guarantees in large‑scale language systems.
Frequently asked questions
What is the main claim of the paper "Attention Is Not What You Need"?
The paper claims that explicit self‑attention is not necessary for strong performance and reasoning, and proposes an attention‑free architecture based on Grassmann flows.
How do Grassmann‑based models compare to Transformers on the Wikitext‑2 benchmark?
Grassmann‑based models with 13 to 18 million parameters achieve validation perplexities within about 10 to 15 percent of size‑matched Transformers on Wikitext‑2.
Do Grassmann models outperform Transformers on the SNLI task?
A Grassmann‑Plücker head on DistilBERT records validation and test accuracies of 0.8550 and 0.8538, slightly higher than the Transformer head's 0.8545 and 0.8511.
The Grassmann proposal does not dethrone attention overnight, but it cracks open a door to models that are both efficient and mathematically tractable. As researchers chase interpretability and scaling alike, the community will watch whether manifold‑based designs can sustain the performance edge that attention has long monopolized. The coming months will reveal if geometry becomes the new engine of neural reasoning.
Sources: arXiv:2512.19428 (Attention Is Not What You Need) by Zhang Chong, submitted 22 Dec 2025.