Mechanistic interpretability
Reverse-engineering the learned features inside a model rather than inferring them from outputs.
What it commits you to
Mechanistic interpretability treats model internals as the object of study, on the model of how a cell biologist takes a cell apart. Sparse autoencoders supply the features and activation steering supplies the experiments. The bet is that a feature you can name is a feature you can audit; the unresolved question is whether naming is explanation.
Which realms lean on this
Every realm has to keep a system honest. These are the terms that say out loud what the other kinds keep implicit.
alignment · 5 realms — The Benthamite Foundry, The Existential Wilds, The Longview Foundry, The Stoic Ramparts, The Ubuntu Weave.
See also
Sparse autoencoder
technicalAn overcomplete dictionary trained to reconstruct activations from few features.
Activation steering
technicalEditing behaviour by adding a direction to the residual stream at inference time.
Interpretability
alignmentThe effort to make what a model represents and computes legible to people.
Deceptive alignment
alignmentAn internal objective that is optimised to look aligned in the places anyone is checking.