8 Best AI Agent Harnesses in 2026: Performance, Cost, and Speed Compared

The model gets most of the attention when people compare AI agents. However, the harness also controls how the model receives context, selects tools, handles results, and continues its work.

We wanted to measure the effect of this layer. We ran the same Kimi K3 model through eight agent harnesses and gave each one the same 25 tasks. The model, provider, reasoning level, and tools stayed the same.

The results changed more than we expected. The highest observed pass rate was 88%, while the lowest was 68%. We also found large differences in speed, token use, tool calls, and estimated cost.

Key takeaways

• Oh My Pi had the highest observed pass rate. It completed 22 of 25 tasks, or 88%.

• Kimi Code finished second with 21 passes, or 84%.

• Hermes Agent passed 20 tasks and had the lowest estimated cost for each successful task.

• Pi Agent had the fastest median runtime. It also used the fewest tokens and made the fewest tool calls.

• Codex passed 17 tasks and had the lowest observed pass rate at 68%.

• Claude Code had the highest estimated API cost in this Kimi K3 setup.

• More tool calls did not always improve the result. Grok Build made 402 tool calls, while Pi made 223. Both passed 18 tasks.

• All eight harnesses passed 12 of the 25 tasks. They produced different results on 10 tasks, and every harness failed three tasks.

The 20-point difference between the highest and lowest pass rates shows that the harness can have a large effect, even when the model and tools stay the same.

How we evaluated each harness

We tested these eight harnesses:

1. Kimi Code

2. Hermes Agent

3. Claude Code

4. Pi Agent

5. OpenCode

6. Codex

7. Oh My Pi

8. Grok Build

Each harness used the following shared setup:

• Model: moonshotai/kimi-k3

• Provider: OpenRouter

• Reasoning level: Maximum

• Tools: Hosted Composio MCP

• Valid tasks: 25

• Total scored results: 200

We wanted to measure the effect of the harness, so we kept the main test conditions the same.

Each harness used:

• moonshotai/kimi-k3

• OpenRouter as the model provider

• Maximum reasoning

• The same hosted Composio MCP tools

• The same task instructions

• The same connected application data

• The same automated scoring rules

We evaluated eight harnesses across 25 valid tasks. This produced 200 scored results.

The tasks focused on tool use across business applications. They covered Gmail, Google Calendar, Google Sheets, Airtable, GitHub, Slack, Notion, Linear, and PagerDuty.

What the tasks tested

The tasks included simple audits to complex long workflows that changed data in several applications.

Example 1: CRM migration archive

The CRM migration task asked the agent to review historical subscription data. It had to calculate quarterly subscription counts, quarterly recurring revenue, and totals for each current plan name.

The automated checks looked for:

• The exact count for each quarter

• Revenue values within the allowed range

• The correct plan names

• The correct total for each plan

• The plan with the largest revenue decline

Four harnesses passed this task, while four failed. This made it a useful example of a task where the harness changed the result.

Example 2: GitHub repository sweep

This task asked the agent to inspect a set of GitHub repositories and return the required findings.

All eight harnesses passed. However, they used different amounts of time, tokens, and tool calls. Codex took 777 seconds and made 18 tool calls, while Hermes took 143 seconds and made seven calls.

This example shows that two harnesses can reach the same correct result with very different resource use.

Example 3: Invoice sync

The invoice workflow required the agent to find eligible invoices, add the correct rows to a payment ledger, and reply in the correct operations thread.

It also had to:

• Add exactly 10 eligible invoices

• Exclude expired invoices

• Leave excluded email threads unchanged

• Use the correct payment reference

• Report the correct final count

Every harness failed this task. Most completed a large part of the workflow, but they included two invoices that the task rules required them to exclude.

This shared failure suggests that some task patterns remained difficult for Kimi K3 across all eight harnesses.

How we scored a task

Automated checks inspected the final state of each connected application. They checked the actions taken by the agent and the data left after the run.

The checks included:

• Exact record counts

• Exact sets of names or identifiers

• Numeric values within a set tolerance

• Required rows added to a table

• Required messages or replies

• Records that had to remain unchanged

• Invalid or unsafe records that had to remain excluded

The evaluator combined these checks into the final pass or fail result. Some reports also included non-blocking checks, so the displayed check count could be lower than the total while the task still passed.

A confident final response could not make an incorrect task pass. The final state of the applications had to meet the scoring conditions.

Handling invalid runs

Five tasks were removed because they did not provide a valid shared comparison. We removed them from every harness, which left the same 25-task set for all eight.

We also removed attempts with provider, runtime, or evaluator problems. We replaced them with clean attempts that used the same prompt and configuration.

We kept normal task failures in the results. This included timeouts when the harness used the full fixed time budget without completing the work.

Measuring tokens and cost

The token and cost comparison used a shared set of 24 tasks. One task was removed from this part because complete usage data was missing for at least one harness.

Token use included:

• New input tokens

• Cache reads

• Cache writes

• Output tokens

• Reasoning tokens counted once

The cost estimate used OpenRouter’s Kimi K3 list prices from July 30, 2026. These numbers show estimated model cost, while actual invoices can differ.

This evaluation measures how each harness operated Kimi K3 in the shared setup. A harness can produce different results with another model, tool provider, or task set.

Overall results

The same Kimi K3 model produced different results across the eight harnesses. The observed pass rates ranged from 68% to 88%, which created a gap of 20 percentage points.

Oh My Pi had the highest pass rate. It completed 22 of the 25 tasks. Kimi Code followed with 21 passes, and Hermes Agent completed 20.

Oh My Pi gave Kimi K3 its best result in this test. It also kept its tool use and estimated cost at a reasonable level. This gave it the strongest balance between task completion and resource use.

Kimi Code finished one task behind Oh My Pi. Its 84% pass rate suggests that Kimi K3 works well inside its native coding harness. However, Kimi Code used more tokens than every other harness in the shared usage comparison.

Hermes Agent had the third-highest pass rate and the lowest estimated cost for each successful task. It was also the second-fastest harness by median runtime. These results made Hermes one of the strongest choices for value.

Pi Agent showed a different strength. It had the fastest median runtime, used the fewest tokens, and made the fewest tool calls. Its pass rate was lower at 72%, so its efficiency came with a reduction in observed task completion.

Claude Code passed 19 tasks, but it had the slowest median runtime and the highest estimated API cost. Its estimated cost per successful task was almost four times higher than the next most expensive result. This test used Kimi K3 through OpenRouter, so this result measures how Claude Code handled an external model. It does not measure Claude Code with a native Claude model.

Codex had the lowest observed pass rate at 68%. As with Claude Code, the test used an external model instead of the harness’s native model family. Its result shows how Codex handled Kimi K3 and the shared MCP setup.

Related: Claude code vs Codex

The number of tool calls had no clear link to success. Grok Build made 402 tool calls, which was the highest total, while Pi Agent made only 223. Both harnesses passed 18 tasks.

The task results also show where the harness had the most effect:

• All eight harnesses passed 12 tasks.

• The harnesses produced mixed results on 10 tasks.

• All eight harnesses failed three tasks.

The 10 mixed-result tasks provide the clearest evidence of the harness effect. The model and tools stayed the same, while differences in context, planning, tool handling, and stopping behaviour led to different outcomes.

Total token use in millions

Estimated API cost in USD

Cost per success uses the shared 24-task cost slice. One task was removed from the cost comparison because complete usage data was not available for every harness. The values use OpenRouter list prices and are estimates.

Why did the same model produce different results?

Every harness used Kimi K3, but each harness controlled how the model received information and took action. These differences affected the model’s work during long and complex tasks.

System instructions

A harness adds its own instructions before it sends the user’s task to the model. These instructions can tell the model how to plan, use tools, check its work, and decide when it is finished.

Some harnesses use detailed instructions, while others give the model more freedom. This can change how the model interprets the same request.

Tool descriptions

The harness tells the model which tools are available and how to call them. It can rename tools, change their descriptions, adjust their input structure, or limit which tools the model can see.

Clear tool descriptions help the model select the correct action. A large or unclear tool list can make this decision more difficult.

The harness also decides how it sends tool results back to the model. A short and structured result can be easier to use than a large raw response.

Context management

Long tasks can create a large amount of context. The agent must keep the original request, tool results, intermediate findings, and task rules available during the run.

Each harness manages this context in a different way. It can remove older messages, compress them into a summary, save information in files, or keep the full history.

This choice affects whether the model remembers an important rule near the end of a task. It also affects token use and cost.

Kimi Code and Claude Code processed the most tokens in the shared comparison. Pi Agent processed the fewest. These differences show that each harness built and maintained its context in a different way.

Planning and tool use

Some harnesses encourage the model to make a plan before it starts. Others let the model select the next action during each step.

The harness can also control parallel tool calls, retries, and the number of actions allowed in one run. These settings affect speed and reliability.

More actions did not always improve the result in our test. Grok Build made 402 tool calls, while Pi Agent made 223. Both completed 18 tasks.

Error recovery

Tools can return incomplete data, invalid arguments, timeouts, or large result sets. The harness decides how to present these problems to the model and whether it should retry.

A useful error message can help the model correct its next action. A weak error message can cause repeated calls or an early stop.

This became important during workflows that required several searches and updates across different services.

Verification and stopping

The harness also affects when the agent decides that its work is complete. Some harnesses encourage a final review, while others stop after the main action succeeds.

This difference can be small during a simple search. It becomes more important when the task includes exclusions, exact counts, or several dependent updates.

For example, the handover audit produced only three passes across the eight harnesses. The successful harnesses completed every required part of the workflow, while the others stopped with missing or incorrect results.

Where the harness made the biggest difference?

All eight harnesses passed 12 tasks, and all eight failed three tasks. The remaining 10 tasks produced mixed results.

These mixed tasks give us the clearest view of the harness effect. The model, provider, tools, and task instructions stayed the same, while the results changed across the harnesses.

Three tasks show this effect especially well.

CRM migration archive

The CRM migration task required the agent to review historical subscription data and calculate:

• New subscriptions for each quarter

• New recurring revenue for each quarter

• Subscription totals for each current plan

• The plan with the largest revenue decline

Four harnesses passed:

• Hermes Agent

• Claude Code

• OpenCode

• Oh My Pi

The other four failed:

• Kimi Code

• Pi Agent

• Codex

• Grok Build

The successful harnesses spent more time and used more tools in this case. Hermes made 27 tool calls and finished in 545 seconds. Claude Code made 18 calls and finished in 383 seconds. OpenCode also made 18 calls and finished in 421 seconds. Oh My Pi made 14 calls and finished in 403 seconds.

The failed harnesses stopped much earlier. Kimi Code made five tool calls, Codex made five, Pi made seven, and Grok Build made 11.

Kimi Code’s result found only 222 new subscriptions, while the correct total was 354. It also selected the wrong plan for the largest revenue decline. Its early stop left part of the source data outside the calculation.

This task suggests that additional exploration helped the successful harnesses. However, more work did not always improve results across the full test.

Handover audit

The handover audit created one of the largest differences between the harnesses.

Only three passed:

• Kimi Code

• OpenCode

• Oh My Pi

The other five completed only 6 of the 11 reported checks:

• Hermes Agent

• Claude Code

• Pi Agent

• Codex

• Grok Build

Kimi Code passed all 11 checks after 25 tool calls. OpenCode also passed all 11 after 15 calls. Oh My Pi passed the scoring conditions after 16 calls.

The failed harnesses used between 11 and 27 tool calls. Grok Build made more calls than every successful harness, but it still failed.

This result shows that the number of actions alone cannot explain success. The order of the actions and the information kept in context can also affect the final state.

Reimbursement audit

The reimbursement audit was a long workflow with several dependent steps. Five harnesses passed:

• Kimi Code

• Claude Code

• Pi Agent

• OpenCode

• Oh My Pi

Hermes Agent and Grok Build failed. Codex reached the fixed 900-second time limit and ended with an error.

Pi Agent produced the most efficient successful run in this task. It finished in 446 seconds with 16 tool calls and about 995,000 tokens.

Kimi Code also passed, but it took almost twice as long and processed about 2.93 million tokens. This difference shows that two harnesses can reach the same correct outcome with very different resource use.

Hermes finished faster than every successful harness, but it missed four scoring conditions. Grok Build made the most tool calls and still completed only six checks.

What these tasks tell

The mixed tasks reveal three useful patterns:

• Some harnesses stopped before they collected enough information.

• Some continued for longer but still followed an incorrect path.

• Some completed the task with far fewer tokens and tool calls.

The evaluation cannot identify one internal cause for every result. However, the traces show that each harness guided Kimi K3 through the tasks in a different way.

These differences affected how much data the model collected, which tools it selected, how long it continued, and whether it checked the final result.

Best Harness profiles in descending order

1. Oh My Pi

Result: 22/25 passed, 88%

Median time: 231.7 seconds

Tool calls: 248

Cost per success: $0.52

Oh My Pi had the highest observed pass rate. It passed difficult tasks such as the CRM migration archive, handover audit, lead intake, and reimbursement audit.

It also used fewer tool calls than most of the other harnesses. Its token use was close to OpenCode and lower than Hermes, Grok Build, Claude Code, and Kimi Code.

Oh My Pi gave us the strongest overall result because it combined high task completion with reasonable speed and cost. It still failed the three tasks that every harness failed.

Best for: Users who want the highest observed reliability in this Kimi K3 test.

Main tradeoff: Pi Agent and Hermes completed tasks faster.

2. Kimi Code

Result: 21/25 passed, 84%

Median time: 281.9 seconds

Tool calls: 301

Cost per success: $0.64

Kimi Code had the second-highest pass rate. It performed well on long workflows and was one of only three harnesses to pass the handover audit.

It also used the most tokens. Its shared 24-task total reached 15.27 million tokens, which was about twice the Pi Agent total.

This result suggests that Kimi Code gave Kimi K3 enough context and room to complete difficult work. The added context also increased resource use.

Best for: Users who want to run Kimi K3 in its native coding harness.

Main tradeoff: High token use and a slower median runtime.

3. Hermes Agent

Result: 20/25 passed, 80%

Median time: 164.0 seconds

Tool calls: 262

Cost per success: $0.46

Hermes had the lowest cost per successful task. It also had the second-fastest median runtime and the third-highest pass rate.

It performed especially well on the CRM migration task, where it passed all 10 checks. However, it failed the handover and reimbursement audits.

Hermes is an open-source, self-hosted agent with persistent memory, tools, skills, scheduled tasks, and messaging connections. These features make it useful for work that continues across several sessions. Hermes Agent documentation.

Best for: Users who want a fast and cost-efficient general agent.

Main tradeoff: Its reliability was lower than Oh My Pi and Kimi Code.

4. Claude Code

Result: 19/25 passed, 76%

Median time: 330.5 seconds

Tool calls: 293

Cost per success: $1.96

Claude Code passed difficult tasks such as the CRM migration archive and reimbursement audit. It failed the handover audit, lead intake, and calendar repair.

It had the slowest median runtime and the highest estimated cost. Its total token use was also close to the highest result.

Claude Code provides a mature agent loop with file tools, shell access, context management, skills, hooks, MCP, and subagents. Anthropic describes it as the agentic harness around Claude. How Claude Code works.

This test used Kimi K3, so the result measures Claude Code as an external-model harness. A test with Claude can produce a different result.

Best for: Teams that want a mature coding harness and plan to test several model configurations.

Main tradeoff: High cost and slow runtime in this Kimi K3 setup.

5. Pi Agent

Result: 18/25 passed, 72%

Median time: 156.2 seconds

Tool calls: 223

Cost per success: $0.57

Pi Agent led every main efficiency measure. It had the fastest median runtime, the lowest token use, and the fewest tool calls.

It also passed the reimbursement audit with 16 tool calls and about 995,000 tokens. This was the most efficient successful run for that task.

Pi failed some tasks that needed broader data collection, including the CRM migration archive and handover audit.

Pi is a small and extensible harness with a multi-provider model API, tool calling, and state management. Its documentation states that it uses the access rights of the current process. Users who need stronger isolation must add a container or sandbox. Pi Agent Harness.

Best for: Users who value speed, low token use, and a small system that they can change.

Main tradeoff: Lower observed reliability and no built-in permission boundary.

6. OpenCode

Result: 18/25 passed, 72%

Median time: 270.5 seconds

Tool calls: 299

Cost per success: $0.72

OpenCode passed several tasks that separated the harnesses. It passed the CRM migration archive, handover audit, lead intake, and reimbursement audit.

It failed some smaller guardrail and audit tasks, including calendar repair, the PagerDuty timeline audit, and the Drive sharing audit.

OpenCode is an open-source coding agent with support for several model providers. It includes separate build and plan agents, and the plan agent uses read-only access by default. OpenCode repository.

Its cost per success was the second-highest result, although its token total stayed near the middle of the group.

Best for: Users who want an open-source coding harness with model choice.

Main tradeoff: Mixed reliability and a higher cost per successful task.

7. Grok Build

Result: 18/25 passed, 72%

Median time: 196.2 seconds

Tool calls: 402

Cost per success: $0.66

Grok Build had a faster median runtime than most of the harnesses. It also made far more tool calls than every other system.

The extra calls did not increase its overall pass rate. It tied Pi Agent and OpenCode with 18 passes, although Pi used 179 fewer calls.

Grok Build passed lead intake, but it failed the CRM migration, handover, and reimbursement tasks.

Its result shows the cost of an active tool loop that does not always move toward the correct final state.

Best for: Users who want a fast and active tool-use loop and can monitor its actions.

Main tradeoff: High tool use without a matching increase in task completion.

8. Codex

Result: 17/25 passed, 68%

Median time: 233.4 seconds

Tool calls: 297

Cost per success: $0.66

Codex had the lowest observed pass rate. It failed the CRM migration, errand intake, handover audit, and lead intake. It also reached the 900-second limit during the reimbursement audit.

Its token use was the second-lowest total, and its cost stayed near the middle of the group.

Codex provides file access, command execution, approval controls, sandboxing, MCP, skills, and support for local and remote work. Codex CLI overview.

This evaluation used Kimi K3 through OpenRouter. Codex is designed mainly for OpenAI models, so this result describes its Kimi K3 setup.

Best for: Teams that want Codex features and need to test external-model support.

Main tradeoff: The lowest pass rate in this specific comparison.

Which harness should you choose?

The answer depends on which result matters most to you.

Oh My Pi is the best starting point when task completion is the main goal. Hermes is a strong choice when cost and speed matter together. Pi is a good fit when you need a small and efficient harness.

Kimi Code is a natural choice for Kimi K3 users who accept higher token use. OpenCode gives users more model choice, while Claude Code and Codex offer mature product features.

A team should test its final two or three options with its own tasks. Tool design, data size, permissions, and task length can change the result.

Limitations

This evaluation gives us a controlled comparison, but it has limits.

We used one model

Every harness used Kimi K3 through OpenRouter. The results do not show how each harness performs with other models.

Claude Code can perform differently with Claude, and Codex can perform differently with an OpenAI model. The same rule applies to other harnesses and their preferred models.

We used one valid scored run

Each harness received one scored attempt for each valid task. Invalid attempts caused by provider, runtime, or evaluator problems were replaced with clean runs.

We did not run each valid task several times. A repeated test could produce different pass rates and show how stable each harness is.

Small differences need care. A difference of one task equals four percentage points in this 25-task set.

The tasks focused on business applications

The evaluation used Gmail, Calendar, Sheets, Airtable, GitHub, Slack, Notion, Linear, and PagerDuty.

These tasks measured tool use, data filtering, audits, and cross-application workflows. Coding ability was outside the main scope.

A software-development benchmark can produce a different ranking.

The tools used one shared MCP setup

Every harness used the same hosted Composio MCP tools. This gave us a shared tool layer, but each harness could present tool definitions and results to the model in a different format.

Another MCP server or a native tool connection can change the results.

Runtime includes external delays

The runtime measured the full agent run. It included model response time, tool execution, network delay, retries, and harness processing.

A later run can be faster or slower because external services can change.

Cost values are estimates

The cost table uses a shared 24-task set and OpenRouter list prices from July 30, 2026.

The estimates do not include subscriptions, tool-provider charges, local compute, storage, or human review. Actual invoices can also include discounts or different cache charges.

Five tasks were excluded

Five tasks were removed from every harness because they did not provide a valid shared comparison. This kept the denominator equal, but it reduced the size of the final test set.

Automated checks measured the final state

The evaluator checked records, messages, values, exclusions, and application state. It did not score the quality of the user interface, the clarity of progress updates, or the ease of installation.

These product features still matter when a team selects a harness.

Conclusion

The harness changed how Kimi K3 performed.

Oh My Pi passed 88% of the tasks, while Codex passed 68%. Pi Agent used the fewest tokens and finished fastest. Hermes produced the lowest estimated cost for each successful task. Claude Code used the most money in this external-model setup.

The task-level results gave us the clearest lesson. All eight harnesses handled many simple tasks, while complex workflows created large differences. Some harnesses stopped too early, some used many tools without reaching the correct state, and some completed the same work with far fewer resources.

The model defines much of an agent’s capability. The harness controls how that capability reaches the task. It shapes the context, tools, retries, memory, and stopping behavior.

Teams should evaluate the model and harness as one system. A strong model can still produce weak results inside the wrong harness, and a well-designed harness can help the same model work with greater reliability and efficiency.