Analysis
What the scale shows
Four readings that a pass-rate leaderboard cannot give: what makes an item hard, where the bank can still measure, where two models disagree, and how few items an adaptive exam needs.
What makes an item hard
Median difficulty by decile of the covariate, with the interquartile range shaded. Representation items on the pooled scale of 14 panels; perturbation items on the pooled unseen-perturbation scale.
Cells: sequencing depth
Cells: training support of the type
Conditions: effect size
Conditions: cells per condition
Spearman correlation with difficulty, by panel
Spearman correlation with difficulty, by screen
Effect size is the L2 norm of the measured delta. Under the mean-shift pass rule a large effect is not an easy item: the shared shift already captures it.
Difficulty by panel and screen
Boxes span the interquartile range, whiskers the 5th to 95th percentile, over informative items. The shaded band is the ability range of the classical pool; the dashed line is the best model.
Representation panels, pooled scale
Perturbation screens, pooled unseen-perturbation scale
Where the bank can measure
An item is uninformative for the current pool when every model passes it, or fails it, with probability at least t. Test information is the sum of p(1−p) over informative items at each ability; the scale measures best where it peaks.
Uninformative items against the threshold
Test information along the ability axis
Dashed lines mark the best model of each component. Information to the right of them is what a stronger model will be measured with.
Head to head
Two models of similar ability can pass different items. For each panel or screen: the share of items both pass, only the first passes, only the second passes, and neither passes.
An adaptive exam
With item difficulties known, a model's ability can be estimated from a few hundred well-chosen items instead of the whole bank. Each step picks the unanswered item whose difficulty is nearest the running estimate, then takes a Newton step on the Rasch likelihood. Random order is shown for comparison.