How it works

Why a scale, not a leaderboard

An accuracy score is a property of the test as much as of the model. Virtual Cell Bench fits item response theory instead: every item gets a difficulty, every model an ability, on one scale, with error bars.

What an accuracy leaderboard cannot do

The score belongs to the test

Accuracy one model curated atlas 0.89 scBaseCount lung 0.39 Two tests, two numbers, one model. The scale difficulty → atlas items lung items one ability Items spread out; the model does not move.

Accuracy averages items as if they were interchangeable, so the number follows the panel. The scale gives the items a difficulty and the model one ability.

Every change resets the history

Accuracy 0.80 0.62 0.71 0.55 moves at every change Ability on anchored scales stays comparable v0 v1: +8 panels v2: +scBaseCount v3: +hard model

Add a panel or a stronger model and every accuracy moves. Anchored difficulties keep earlier abilities valid while the bank grows.

Its error bar is not an error bar

item difficulty → a model information an item carries everyone passes: tells nothing nobody passes: tells nothing A few hundred items near a model place it; far-off items only add noise.

Only items near a model's ability can tell it from its neighbours. The fit knows which those are, and a few hundred of them are enough to place a model.

It cannot say why

Accuracy 0.61 low. why? The scale one hard item difficulty +2.4 shallow cell noisy label sibling type rare in training One number per item, correlated with a reason.

A low score hides whether the items were unanswerable, the labels wrong, or the model weak. A difficulty per item can be correlated with a reason.

The same model, fourteen tests

What the fit estimates

A pass is a logistic function of ability minus difficulty: a coin flip at equal values, 73% one logit above, 88% two logits above. Every item and every model sits on that one curve.

model ability (logits) 1½0 P(pass) easy itemhard item difficulty = where the curve crosses ½ a model

What the scale gives instead

-4 -2 0 +2 +4 +6 one logit axis: item difficulty above, model ability below ② one number per item, with a reason difficulty +3.0 · shallow cells, sibling type ④ retired passed by nobody: no information ④ retired passed by everyone: no information ① every panel and screen on one scale items namespaced by source, fit jointly, no averaging random floor depth floor PCA-16 PCA-64 PCA-256 scVI misfit ⚑ new model ③ floors and ladders as evidence floors must come last, capacity ladders in order, before anything else is read ⑤ misfit is reported: outfit above 2, passes that do not follow the scale ⑥ anchored placement new models join with difficulties held fixed; refits advance the version, every version stays citable ability ± SE
① every panel and screen on one axis; ② a difficulty per item that can be explained; ③ floors and ladders as a check on every fit; ④ items that carry no information retired; ⑤ misfit flagged instead of averaged away; ⑥ new models anchored to the current scale, with every version kept.

The rules in brief

One item, one response

A cell passes when the frozen probe names its type; a condition passes when the prediction beats the context's mean shift.

Held-out units

Whole donors, studies or tissues; unseen perturbations or unseen perturbation and context pairs. Nothing is shared between training and test.

The same test for everyone

One probe, one setting, one pass rule, the same folds, for every model on every item.

One joint fit

Rasch by joint maximum likelihood with weak priors; standard errors from the Fisher information; outfit per item and per model.

Pooled scales

All panels on one representation scale, all screens on one perturbation scale per split. Items nobody passes are reported, not counted.

Versions

New models anchor to the current scale; refits advance the version; every version stays citable.

The rules in full → · Scale versions → · Submit a model →