What Does an Observability Foundation Model Know?

Sun, 04 Oct 2026 00:00:00 +0000

Accepted to the NeurIPS 2026 main conference (25 pages). Subject classes cs.LG, cs.AI.

The short version: a linear probe showing a label is recoverable from a model’s hidden states tells you neither that the information goes beyond what the raw input already exposes, nor that the model actually uses it. So we audit against both — the strongest raw-window baseline, and the same architecture stripped of its trained configuration — and then test use directly by swapping residuals from high-burst donors. Recoverability and use come apart.

Support Tokens, Stability Margins, and a New Foundation for Robust LLMs

Wed, 25 Feb 2026 00:00:00 +0000

45 pages, 9 figures. Subject classes cs.LG, math.PR, math.ST; ACM classes I.2.7, G.3, G.4. Latest revision v3 (21 March 2026).

Treating causal self-attention rigorously as a probabilistic model over embeddings — the way PCA extends to probabilistic PCA — makes a barrier constraint fall out of the change of variables. The geometry that follows has a degeneracy boundary where the attention map goes locally ill-conditioned, which reads as a stability margin in the same sense an SVM has one, and support tokens are the tokens sitting on it.

The practical consequence is small on purpose: add a smooth log-barrier penalty to ordinary cross-entropy and you get better robustness to input perturbations and a sharper margin geometry, without giving up out-of-sample accuracy.

Interpretability | Dhyey Mavani

What Does an Observability Foundation Model Know?

Support Tokens, Stability Margins, and a New Foundation for Robust LLMs