Leaderboard
Abilities on the logit scale
Each model's ability is estimated jointly with every item's difficulty. Zero is the average item: a model at 0 passes it with probability one half. Whiskers are one standard error; the bar colour is the model family.
Click a model to see its ability on every panel or screen. misfit marks an outfit above 2: the model passes and fails items its ability does not predict, so its rank is not a reliable summary.
How it is counted
A pass is one item, one model
Representation: the frozen probe names the cell's annotated type from the model's embedding. Perturbation: the model's predicted response correlates with the measured one better than the context's mean shift does.
One scale per setting
Representation panels pool into one scale. Perturbation has two pooled scales, unseen perturbation and unseen combination, because an item means something different under each split. Each panel and screen also has its own fit.
The same items for everyone
Every model on a scale answered every item of that scale. Abilities are only compared within a scale; the pooled representation and perturbation scales are not on the same axis.
Standard errors come with the number
Fisher information from the joint fit gives each ability a standard error. With hundreds of thousands of items it is small; on a single screen of 87 conditions it is a quarter logit.
Floors are ranked too
Random embeddings, random shifts, no change and the depth-only embedding are fit as models. They anchor the bottom of each scale and must come out last.
Items nobody passes stay out of the fit's summaries
They are flagged as degenerate: they carry no information about the current pool. The share of such items is reported per panel and screen, and they are the first candidates for retirement.