What kind of work this is
The roster combines multi-app workflows with focused tests of batching, pagination, exact diffs, guarded writes, and complete source sweeps.
- Multi-app workflow14 / 30
- Read one app, decide by a rule written in another, then write to a third and confirm it.
- State reconciliation3 / 30
- Make two sources agree, changing only what differs and leaving matched records untouched.
- Chained operations2 / 30
- One app, but each step depends on what the last step returned.
- Census2 / 30
- Sweep fragmented surfaces exhaustively for exact markers, without counting near matches twice.
- Guard2 / 30
- Resolve an ambiguous or narrowly scoped target before making only the permitted change.
- Cross-source census1 / 30
- Count across generations of a record and report exact figures, with no double counting.
- Async1 / 30
- Start a query, follow its status to completion, then return the exact aggregate.
- Batch1 / 30
- Recover from an atomic batch failure while completing every valid item.
- Diff1 / 30
- Change only mismatched rows and report the records that actually needed work.
- DSL1 / 30
- Express exact filters in an app's query language, paginate, then update in chunks.
- History1 / 30
- Walk a complete audit trail and preserve its chronological order.
- Pagination1 / 30
- Traverse an incremental history boundary instead of substituting a current-state search.
What actually breaks
Agents reach the right apps and take broadly the right actions, then lose the task on a tally that is off by one, a row appended with reworded text, or a record the rules told them to leave alone.
Wrong value or formatThe right record, written with reworded text, the wrong unit, or a shape the verifier does not accept.181
Checks missed in these runs
A run can have several failure types, so a check may appear under more than one.
- mcp case verifier passed120
- ops thread tally exact98
- excluded threads untouched48
- safety policy passed30
- ledger rows appended exact30
- completion confirmed23
- sources immutable22
- digest per client counts15
Wrong count or totalTallies, totals and per-group counts that do not match the seeded state.149
Checks missed in these runs
A run can have several failure types, so a check may appear under more than one.
- mcp case verifier passed106
- ops thread tally exact98
- excluded threads untouched48
- ledger rows appended exact30
- safety policy passed27
- completion confirmed22
- sources immutable22
- digest per client counts15
Skipped a required stepOne of the task's required actions never happened, often the closing confirmation.104
Checks missed in these runs
A run can have several failure types, so a check may appear under more than one.
- mcp case verifier passed79
- ops thread tally exact75
- excluded threads untouched48
- ledger rows appended exact30
- completion confirmed22
- sources immutable19
- safety policy passed18
- digest per client counts15
Picked the wrong recordA decoy, a duplicate or a superseded row was treated as the real one.80
Checks missed in these runs
A run can have several failure types, so a check may appear under more than one.
- ops thread tally exact77
- mcp case verifier passed49
- excluded threads untouched48
- sources immutable22
- ledger rows appended exact21
- safety policy passed17
- chargeback camera excluded14
- expired beacon excluded13
Touched something it should notA decoy or out-of-scope record was modified. Every task requires the decoys to stay untouched.72
Checks missed in these runs
A run can have several failure types, so a check may appear under more than one.
- ops thread tally exact63
- mcp case verifier passed55
- excluded threads untouched48
- safety policy passed22
- sources immutable22
- ledger rows appended exact21
- purchase rows appended exact13
- assignment rows appended exact12
Uncategorized failuresFailed or timed-out runs without a recorded failure category.67
Checks missed in these runs
A run can have several failure types, so a check may appear under more than one.
- mcp case verifier passed60
- ops thread tally exact45
- excluded threads untouched26
- sources immutable24
- ledger rows appended exact16
- assignment rows appended exact7
- authority blockwork excluded7
- authority safety gear excluded7
Ran out of timeThe run hit the fixed ceiling before finishing. A timeout counts as a failure, not an exclusion.41
Checks missed in these runs
A run can have several failure types, so a check may appear under more than one.
- completion confirmed8
- ops thread tally exact8
- assignment rows appended exact3
- cover replies3
- intake rows appended exact3
- request replies3
- purchase replies2
- purchase rows appended exact2
Wrong state transitionA record moved to a status the task's rules do not allow from where it started.23
Checks missed in these runs
A run can have several failure types, so a check may appear under more than one.
- digest per client counts15
- manifest counts updated15
- mcp case verifier passed13
- onlyledger downland heritage roofing present6
- onlyledger nightjar field services present6
- submission replies5
- safety policy passed4
- completion confirmed3
Task-specific check failedA rule unique to that task, outside the recurring patterns above.13
Checks missed in these runs
A run can have several failure types, so a check may appear under more than one.
- mcp case verifier passed4
- false 4988 flagged3
- false 5034 flagged3
- is structured json2
- safety policy passed1
- is structured timeline1
- is the only free hour 2026-09-021
- is the only free hour 2026-09-031
Wrote on a read-only taskThe run called a write tool where the task allows only reads, which the safety policy rejects outright.1
Checks missed in these runs
A run can have several failure types, so a check may appear under more than one.
- safety policy passed1
How runs are graded
A deterministic verifier
A fixed grader checks the final state of every app the task touches, plus the response the run was asked for. Every check must pass; there is no partial credit.
Available-value averages
Missing per-run token values stay blank. Aggregate token figures average only the observations where that metric is available.