Virtual Bench 0.1 · a living item-response benchmark

The exam for virtual cell models.A benchmark that evolves with the field.

Hundreds of thousands of single cells and perturbation conditions, each with an estimated difficulty. Model abilities are estimated via the IRT framework, on the same scale as the items.

See the leaderboard → Browse the items How it is scored
Pooled representation scale: 424,478 cells from 14 panels on the left, 30 embedding models on the right.

How it is done

From cells to one scale

Every cell and every perturbation is a question. Every model takes the same test. One IRT fit over all the answers gives each model an ability and each item a difficulty, on one scale.

Items cells and perturbations cells perturbations The test the same read-out for every model PCA → ✓ ✓ ✗ ✓ scVI → ✓ ✓ ✓ ✗ Nyström → ✓ ✗ ✓ ✓ random → ✗ ✗ ✓ ✗ one frozen probe, one pass rule Answers every model on every item one row per model one column per item pass fail IRT fit one model of all the answers IRT model ability per model difficulty per item with an error bar on each One scale models and items on the same axis items models new models and items join the scale
The dashed loop is the living part: new models and new items are anchored to the current scale, items that stop discriminating are retired, and every version stays citable.

Leaderboard

Ability, not accuracy

An ability of 0 means a coin flip on the average item. Bars run from the average item to the model's ability; whiskers are one standard error. Random floors pin the bottom of each scale.

Full board, every panel and screen, and how it is counted →

What makes an item hard

Difficulty is a property of the item

A cell is hard when it is shallow, when its type is rare in training, or when its label is a sibling of another type: activated against naive T cells, mural against smooth muscle. A perturbation condition is hard when the shared response already explains it, so beating the mean shift leaves nothing to win.

Difficulty against depth, support and effect size →

Hardest cell types on the pooled scale

Mean difficulty over the cells of each type, types with at least 20 items.

The maintenance loop

A benchmark that stays informative

Items that every model passes or fails carry no information. The bank is refit, pruned and extended on a schedule, and every scale version stays citable.

Anchor

A new model is placed on the current scale with item difficulties held fixed, so its ability is comparable to everything already on the board.

Recalibrate

When enough new models have answered, all abilities and difficulties are refit together and the version number advances.

Retire

Items that no model passes, or that every model passes with high probability, leave the active bank but stay in the archive.

Acquire

New panels and screens are added where the scale is thin: harder items near the top, new sources, new perturbation types.

What keeps the scale honest

The same test for everyone

  • One frozen probeEvery embedding is read by the same multinomial logistic regression with the same settings. A difference in responses is a difference in embeddings.
  • Whole donors, studies or tissues held outNothing is shared between training and test cells, so a probe cannot learn donor identity as a stand-in for cell type.
  • Passing means beating the mean shiftA perturbation prediction passes an item only when it beats the average response of that context, so credit goes to the perturbation-specific part.
  • Floors on every scaleRandom embeddings, random shifts and predicting no change are scored like any model. They must land at the bottom, and every capacity ladder must come out in order.
  • Misfit is reportedEach model's outfit says whether it passes and fails items its ability predicts. A model that only ties the baseline is flagged, not ranked.
  • Versioned scalesEvery release of the scale is kept and cited by version. A model placed on v3 is never silently compared to one placed on v1.

Every rule, step by step →

Contribute

Put a model on the scale

Send embeddings for the representation panels or pseudobulk predictions for the perturbation screens. We run the frozen probe and the fixed pass rule, place the model with item difficulties anchored, and publish its ability with a standard error.

How to contribute →

Cite

Citation

Maintaining a Living Virtual Cell Benchmark with Item Response Theory (PDF), technical report, 2026.

@misc{vcbench2026,
  title  = {Virtual Bench: a living item-response benchmark for virtual cell models},
  year   = {2026},
  note   = {Version 0.1, scale v3. Preprint in preparation.}
}

Author list and preprint link are placeholders until the manuscript is posted.