Contribute

Put a model on the scale

Three ways in, one response format. For version 0.1 the maintainers run the pipeline: open a pull request against the repository or write to us, and the model is placed with item difficulties anchored and published with its ability and standard error.

Route 1 · embeddings cells × dims per panel .npy or obsm, panel cell order frozen probe one multinomial read-out, fixed settings, run by us Route 2 · predictions held-out conditions × genes per screen and split, fold_group pass rule Pearson@HVG beats the context's mean shift Route 3 · response table you ran the probe or rule yourself response table item_id · respondent_id · k · n k passes out of n trials checked against held-out items anchored placement difficulties held fixed takes seconds no published ability moves on the board ability ± SE outfit, pass rate scale version contamination register next recalibration: every model refit together, version advances Three ways in, one response format, one placement step.
Placement holds every item difficulty at its current value, so it takes seconds and moves no published ability. At the next recalibration every model is refit together and the version advances.

Route 1 · embeddings

  • One matrix per panel, cells × dimensions, every cell (training and test) in the panel's cell order.
  • .npy, or .h5ad with the embedding in obsm.
  • Panels ship as .h5ad with raw counts, the split column and strata. Test labels are included: the probe is ours and the split is by whole donors, studies or tissues.
  • State which cells the model saw in pretraining.

The scVI respondents took this route: scripts/extract_scvi_embedding.py, then scripts/append_repr_responses.py.

Route 2 · predictions

  • One matrix per screen and split, held-out conditions × genes, in the screen's gene order.
  • Rows follow the published fold assignment (fold_group in the items table).
  • Train on the training folds and all control cells; never on the held-out conditions.
  • Scored with the fixed metric and pass rule.

scGen and the scVI counterfactual took this route through scripts/append_pert_scores.py.

Route 3 · response table

If you reproduce the probe or the pass rule yourself, send the table both components share:

item_id                  respondent_id  k  n
repr_pool_v0::cell_0001  my_model_v1    1  1
tahoe100m::drugX::ctx=A549::dose=0.05  my_model_v1  0  1
  • k passes out of n trials: cell items have n = 1, stratum items the stratum size.
  • Item identifiers must match the published items tables exactly.
  • Tables are checked against the items we hold out before placement.

Contamination register

For every learned model, declare per source whether it was in pretraining, in fine-tuning, or clean. Most entries for current foundation models will not be clean, and that is fine: the register is published next to the ability and is what makes it interpretable.

House rules

  • Respondent names are stable across every run and version.
  • Published items are never modified; a fix produces a new panel or screen and a new version.
  • Classical respondents and floors are refit alongside every submission.
  • Abilities from different versions are never compared without saying so.
  • Submissions that only tie the baseline are flagged as misfit, not ranked.

Where a new source helps most

  • Representation items above +2 logits that are not label noise: treatment or disease state, at pseudobulk resolution.
  • Perturbation screens with chemical descriptors, so unseen-compound prediction becomes learnable.
  • A cytokine to gene mapping for the Parse screen.
  • Foundation-model embeddings and predictions with a complete contamination declaration.