Joi's Index
Joi's Index

How far up the ladder does each model get?

Where each one falls off

One strip per model and task family. Each square is a difficulty level, easiest on the left; the darker it is, the more of that level's tasks the model solved. The figure at the end of the strip is the level where it stops solving half of them.

Count a task as solved if
Rows
Order

The wider field, on the first battery

Select a column heading to sort the table. Select it again to reverse the order.

Compare

Pick a battery and up to four setups, then choose what to look at. Every bar is drawn to the largest value among the ones you picked. On B2 you can only pick setups that ran under the same limits.

Battery
Limits

B2 results, grouped by limits

This table ranks served setups (model, route, effort, agent, limits) on two families of sealed puzzles. It is not a ranking of bare models and not a forecast of open-ended work.

Select a column heading to sort a group. Sorting never mixes groups.

The level each one reaches

Filled marks show where the model solves at least half the tasks every time; hollow marks show where it solves at least half at least once. Circles sit at a single level; rounded bars ("between 3 and 4") span the range between two levels, and arrows ("7 or above") mean the model cleared the hardest level on the test. Thin lines show uncertainty; faded marks and grey labels are not yet settled.

Same task, run twice

solved every timesolved once, failed oncenever solved

Head to head

What it took to get that score

Up is better, left is cheaper. Bars show the uncertainty in the score. Tokens are counts and are not equal in price across providers.

Battery
Across

Level by level

Solved tries out of completed tries at each level. Darker means a higher pass rate.

How this was run

What a row is

    What this does not measure

      The bill

        Not yet tested

          Corrections