Support Tokens, Stability Margins, and a New Foundation for Robust LLMs

Wed, 25 Feb 2026 00:00:00 +0000

45 pages, 9 figures. Subject classes cs.LG, math.PR, math.ST; ACM classes I.2.7, G.3, G.4. Latest revision v3 (21 March 2026).

Treating causal self-attention rigorously as a probabilistic model over embeddings — the way PCA extends to probabilistic PCA — makes a barrier constraint fall out of the change of variables. The geometry that follows has a degeneracy boundary where the attention map goes locally ill-conditioned, which reads as a stability margin in the same sense an SVM has one, and support tokens are the tokens sitting on it.

The practical consequence is small on purpose: add a smooth log-barrier penalty to ordinary cross-entropy and you get better robustness to input perturbations and a sharper margin geometry, without giving up out-of-sample accuracy.

Probabilistic Models | Dhyey Mavani

Support Tokens, Stability Margins, and a New Foundation for Robust LLMs