Most agent benchmarks focus heavily on the model. They measure reasoning, coding ability, and the model's ability to follow instructions.
In real-world workflows, however, the model is only one part of the overall picture. The tool layer determines which actions the model can discover, how those actions are described, whether batch operations are available, and how much state is returned from each call.
The Setup
We wanted to measure the value of that tool layer, so we ran two complete agent stacks against the same 31 tasks spanning nine SaaS tools: Slack, GitHub, Datadog, LaunchDarkly, Airtable, Linear, PostHog, PagerDuty, and Google Workspace.
Stack A paired MiniMax M3, an open-weight model priced at $0.30 per million input tokens, with Composio as a unified tool layer.
Stack B paired Claude Opus 4.8, a frontier model priced at $5 per million input tokens, with Claude's first-party integrations tooling.
Every task has a deterministic verifier that reads the actual end state of the SaaS account after the agent finishes.
An Overview of the Results
Here is the scoreboard:
* A "turn" is one full round-trip in the agent loop: the model produces a response (usually containing one or more tool calls), the harness executes those calls and feeds the results back, and then the model gets to think again with the new information in context.
Surprisingly, the frontier model combined with first-party tooling did not beat the open-weights model with a third-party tool layer. The results were a tie.
Let's dive deeper and discover the reasons for this.
How Composio helped MiniMax M3 tie with Opus 4.8
To answer this question, we need to look at a few statistics and anecdotes.
Take a look at this table:
Composio averaged 10.6 turns and 9.3 recorded tool calls per case, while the native stack averaged 19 turns and 19.8 recorded tool calls. Composio finished in fewer turns in 23 of the 30 comparable cases. However, native integrations won out in token efficiency, which we'll explore later.
The important thing to note is that for most tasks, the open-weight model won because the tool layer gave it a shorter path through the task.
Take the "slack_thread_harvest" task, where the job was to find every message whose text contains an exact phrase. Both stacks passed the test, but Stack B (Opus + native integrations) required almost 10 times as many tool calls.
This was an interesting case because the native Slack MCP did expose message search, but Opus avoided it. The reason was that the task required exact, whole-word, case-sensitive matches, including thread replies. Slack search is keyword-based, so its results could not be trusted against a deterministic verifier.
Opus chose the safest path: read the full channel history, then open all 57 threads one by one, parallelised across four subagents. MiniMax handled the same issue differently because its tool layer allowed verification. It ran Slack search once, then used Python in Composio’s workbench to clean the results and regex-check the exact phrase.
The power of multi-execute: One task where you could clearly see how Composio helped was linear_backlog_estimates, which requires updating estimates on a large set of Linear issues.
The native Linear MCP only exposes single-record updates, and a tool that only exposes single-record updates can force the agent into a long loop: inspect an issue, save it, inspect the next one, save it, and repeat until done. Opus paginated through list_issues ten times, wrote 18 small Python scripts to track its own progress, and called save_issue 41 separate times. MiniMax, using Composio's multi-execute tool, compressed the whole workflow into batched calls.
Why Composio is better than native integrations
First, the Sandbox. Our Sandbox has helpers like run_composio_tool and proxy_execute that let the model call any Composio tool, or any endpoint of the app's full API, straight from code. For harder cases, Composio hands the agent a snippet to help it out. That's why slack_thread_harvest took two calls, for example: search once, then verify with regex in code.
Plans are the other half of the equation. When an agent searches Composio for a use case, the result includes execution guidance: the steps to reach the target state, plus pitfalls along the way, such as pagination or making the agent more aware of a tool with an awkward input schema. Smaller models benefit the most here, since they don't have to reason their way to the right call sequence
These plans work because of the data behind them. Composio processes about 9 million tool calls per day and learns from patterns while preserving your privacy. For example, we can see that an agent searched for a task, then clicked Gmail, followed by Salesforce, and we can learn from that particular pattern.
Where native integrations won
First, token efficiency. On average, Opus + native integrations spent fewer tokens per case.
However, if you consider that MiniMax M3 is a much cheaper model, the cost per case is still lower.
Here's a "failure map" showing the cases where MiniMax and Opus each failed:
The pattern: When a vendor has a good, purpose-built tool for exactly your operation, the native stack usually wins or things end in a tie. However, when the operation falls outside of what the MCP vendor shipped (whether that means batch writes or anything requiring the full API), the native stack either fails or grinds through dozens of turns to compensate.
In other words, native tooling usually fails on coverage. A unified tool layer combined with an open-weight model currently fails on precision, where you could be a HogQL typo away from a wrong answer.
Conclusion
Frontier models still hold an advantage when it comes to pure reasoning, and nothing in this eval contradicts that. But real agentic work is not only pure reasoning. It is reasoning plus tool discovery, permission handling, pagination, batching, schema understanding, retries, and verification, and the tool layer owns most of this list.
This eval shows that the quality of the tool layer can matter as much as the underlying model for agentic performance. A well-designed unified tool platform like Composio gives the model batch execution and cleaner orchestration that fragmented native integrations often lack, and when the tool layer exposes the right abstraction, an open-weights model performs at frontier-model levels. A $0.30-per-million open-weights model, given a layer that let it batch operations and reach full API surfaces, matched a $5-per-million frontier model on first-party tooling: 26 to 26, two minutes per case, at a one-twelfth of the cost.
One year ago, the model was the bottleneck in agentic workflows. Right now, the bottleneck is a good integration layer.