The rules in full

Items

Every test case is an item. An item carries a difficulty, a model carries an ability, and both are estimated from which items the model passes. The bank has two components with different items but the same response format.

Representation: one cell is one item

A representation model maps a cell's expression profile to a vector. The response is whether a multinomial logistic regression, trained once on the vectors of a separate training panel with fixed settings, assigns the cell its annotated type. The classifier is identical for every model, so a difference in responses is a difference in embeddings. Models never see the test cells during fitting; UMAP and t-SNE are excluded for that reason.

Each annotation source becomes a panel by one procedure. The unit of the split is chosen first: whole donors, studies or tissues are held out, so that nothing is shared between training cells and test items and the probe cannot learn donor identity as a stand-in for cell type. A stratum is a context, cell type and donor combination; each keeps at most 200 training and 50 to 100 test cells, and strata under 20 cells are dropped. Test cells whose type is absent from training are kept in the panel but not scored.

Perturbation: one condition is one item

An item is a perturbation applied in a context: a knockdown, knockout or activation target, a compound at a dose, or a cytokine, in a cell line, donor or cell type. The cells of a condition are replicates, not items. The measured response of a condition is the mean log-normalised expression of its cells minus that of its matched control cells. Items need at least 50 quality-passing cells and a matched control.

A model predicts the response for conditions it was not trained on. Two splits exist. Unseen perturbation: five folds over perturbation identity, every perturbation held out once. Unseen combination: for multi-context screens, (perturbation, context) pairs are held out while the same perturbation stays visible in other contexts. Folds are deterministic, and the same folds serve the classical pool and the learned models.

Responses

The frozen probe

One multinomial logistic regression with one regularisation setting, standardised inputs and a fixed iteration budget. It is fit once per model on the training panel and scored on the test cells. A pass is the predicted label equalling the annotated one.

The mean-shift pass rule

A perturbation prediction is scored by its Pearson correlation with the measured response over the 2,000 most variable genes, chosen on control cells. The item is passed when that correlation exceeds the one of the context's mean shift, the average response over the training conditions of that context. Recent benchmarks found that this average matches deep models on standard metrics, because much of any response is shared across perturbations. A pass therefore credits the perturbation-specific part of a prediction and not the part every perturbation shares. Difficulty reads as how far past the shared shift a model must be on that condition.

The model

The Rasch model: the probability that model j passes item i is σ(θ_j − b_i), the logistic function of ability minus difficulty. A model at the item's difficulty passes it half the time; one logit above passes it 73% of the time; two logits above, 88%.

All abilities and difficulties are estimated at once by joint maximum likelihood, with a Gaussian prior of standard deviation 3 on abilities and a weaker one of standard deviation 4 on difficulties. The prior on difficulties fixes the location of the scale, which the likelihood alone leaves free, and keeps items that every model passes or fails at a finite value. Such items are flagged as degenerate and excluded from every summary; they carry no information about the current pool. Fitting alternates Newton steps on abilities and difficulties until the largest change is below 10⁻⁶, a few hundred sweeps and eight minutes for the 424,478 representation items.

Standard errors and fit

Standard errors come from the observed Fisher information. Each model and item also has an outfit statistic, the mean squared standardised residual over its responses. A value near 1 means the responses are as the model predicts; a model with outfit above 2 passes and fails items its ability does not explain and is flagged as misfit on the board. Random embeddings misfit by construction, as a pure guesser should.

Pooled scales

Each component is fit on one pooled scale, all fourteen panels together and all fifteen screens together, linked through the models, which answer every panel and every screen under the same name. Item identifiers are namespaced by panel, because the same cell can be a test item under two splits of one atlas. The perturbation component keeps two pooled scales, one per split, so that every item has one response per model. Every panel and screen also gets its own fit; those abilities are comparable within the fit but not across fits.

Two checks validate every fit. On synthetic data with 80 models and 2,000 items at 30% missingness the recovered abilities correlate 0.998 with the truth and the difficulties 0.969. On real data the floors must come out at the bottom and every capacity ladder in order before any other reading of the scale is made.

What the scale does and does not say

Ability is relative to the pool that defined the difficulties. A value of 0.69 for PCA-128D means it passes a typical item of this bank with probability σ(0.69 − b), not that it is good in an absolute sense. Difficulty is relative to the models: an item is hard because these methods fail it. Both are comparable within a component across models and, with care, across datasets; they are not comparable between the two components, which are fit separately.

Items passed by nobody are reported as a share per panel and screen. On curated atlases they are a few percent; on scBaseCount panels, whose labels come from many submissions with different vocabularies, up to a third. Those items are label-noise floors rather than hard biology, and they are the first candidates for retirement.

New models and versions

A new model is placed on the current scale with item difficulties held fixed, so its ability is comparable to everything already on the board. When enough new models have answered, abilities and difficulties are refit together and the scale version advances; every version stays available and citable. The classical pool anchors every version to methods that cannot have memorised the test data. A contamination register, which records for each learned model whether a source was in its pretraining corpus, qualifies the ability of any foundation model placed on the scale.

Scale versions → · Submit a model →