What kind of work this is
The roster combines multi-app workflows with focused tests of batching, pagination, exact diffs, guarded writes, and complete source sweeps.
- Multi-app workflow14 / 30
- Read one app, decide by a rule written in another, then write to a third and confirm it.
- State reconciliation3 / 30
- Make two sources agree, changing only what differs and leaving matched records untouched.
- Chained operations2 / 30
- One app, but each step depends on what the last step returned.
- Census2 / 30
- Sweep fragmented surfaces exhaustively for exact markers, without counting near matches twice.
- Guard2 / 30
- Resolve an ambiguous or narrowly scoped target before making only the permitted change.
- Cross-source census1 / 30
- Count across generations of a record and report exact figures, with no double counting.
- Async1 / 30
- Start a query, follow its status to completion, then return the exact aggregate.
- Batch1 / 30
- Recover from an atomic batch failure while completing every valid item.
- Diff1 / 30
- Change only mismatched rows and report the records that actually needed work.
- DSL1 / 30
- Express exact filters in an app's query language, paginate, then update in chunks.
- History1 / 30
- Walk a complete audit trail and preserve its chronological order.
- Pagination1 / 30
- Traverse an incremental history boundary instead of substituting a current-state search.
What actually breaks
Agents reach the right apps and take broadly the right actions, then lose the task on a tally that is off by one, a row appended with reworded text, or a record the rules told them to leave alone.
Wrong value or formatThe right record, written with reworded text, the wrong unit, or a shape the verifier does not accept.199
Checks missed in these runs
A run can have several failure types, so a check may appear under more than one.
- mcp case verifier passed138
- ops thread tally exact106
- excluded threads untouched54
- safety policy passed41
- ledger rows appended exact34
- completion confirmed26
- sources immutable22
- digest per client counts16
Wrong count or totalTallies, totals and per-group counts that do not match the seeded state.165
Checks missed in these runs
A run can have several failure types, so a check may appear under more than one.
- mcp case verifier passed122
- ops thread tally exact106
- excluded threads untouched54
- safety policy passed36
- ledger rows appended exact34
- completion confirmed25
- sources immutable22
- digest per client counts16
Skipped a required stepOne of the task's required actions never happened, often the closing confirmation.115
Checks missed in these runs
A run can have several failure types, so a check may appear under more than one.
- mcp case verifier passed90
- ops thread tally exact83
- excluded threads untouched54
- ledger rows appended exact34
- safety policy passed26
- completion confirmed25
- sources immutable19
- digest per client counts16
Picked the wrong recordA decoy, a duplicate or a superseded row was treated as the real one.87
Checks missed in these runs
A run can have several failure types, so a check may appear under more than one.
- ops thread tally exact83
- mcp case verifier passed56
- excluded threads untouched54
- ledger rows appended exact24
- safety policy passed23
- sources immutable22
- chargeback camera excluded14
- expired beacon excluded14
Touched something it should notA decoy or out-of-scope record was modified. Every task requires the decoys to stay untouched.79
Checks missed in these runs
A run can have several failure types, so a check may appear under more than one.
- ops thread tally exact69
- mcp case verifier passed62
- excluded threads untouched54
- safety policy passed28
- ledger rows appended exact24
- sources immutable22
- assignment rows appended exact13
- purchase rows appended exact13
Ran out of timeThe run hit the fixed ceiling before finishing. A timeout counts as a failure, not an exclusion.41
Checks missed in these runs
A run can have several failure types, so a check may appear under more than one.
- completion confirmed8
- ops thread tally exact8
- assignment rows appended exact3
- cover replies3
- intake rows appended exact3
- request replies3
- purchase replies2
- purchase rows appended exact2
Wrong state transitionA record moved to a status the task's rules do not allow from where it started.26
Checks missed in these runs
A run can have several failure types, so a check may appear under more than one.
- mcp case verifier passed16
- digest per client counts16
- manifest counts updated16
- onlyledger downland heritage roofing present6
- onlyledger nightjar field services present6
- submission replies6
- safety policy passed4
- completion confirmed4
Task-specific check failedA rule unique to that task, outside the recurring patterns above.15
Checks missed in these runs
A run can have several failure types, so a check may appear under more than one.
- mcp case verifier passed6
- is structured json4
- false 4988 flagged3
- false 5034 flagged3
- safety policy passed1
- is structured timeline1
- is the only free hour 2026-09-021
- is the only free hour 2026-09-031
Wrote on a read-only taskThe run called a write tool where the task allows only reads, which the safety policy rejects outright.1
Checks missed in these runs
A run can have several failure types, so a check may appear under more than one.
- safety policy passed1
How runs are graded
A deterministic verifier
A fixed grader checks the final state of every app the task touches, plus the response the run was asked for. Every check must pass; there is no partial credit.
Observed pass@1
Pass@1 is the selected observed result for each task: pass or fail against its verifier. It is not inferred from an aggregate score.
Projected retries
Pass@2, pass@3, pass^2, and pass^3 are projected expected rates. They are not observed second or third attempts or integer trial counts.
Available-value averages
Missing per-run cost and token values stay blank. Aggregate cost and token figures average only the observations where that metric is available.
Native runtime clock
Time uses the selected run's native runtime clock and excludes setup, verifier, and cleanup work.