SIGN UPLOG IN
X (Twitter)LinkedInGitHubDiscordYouTube

How to connect Scorecard MCP to Claude

Connect Claude Web, Desktop and Cowork to Scorecard MCP. Create a new testset for model evaluation, run evaluation on your latest model, and more using natural language, with authentication handled for you.

Get started for freeGet a demo
Scorecard MCP logoScorecard MCP
Dcr Oauth

Scorecard MCP is an evaluation platform for managing testsets, metrics, and runs. Use it to centralize model evaluations, scores, annotations, and docs search.

2 Tools

Introduction

Claude is Anthropic's AI assistant, available on the web, in the desktop app, and on mobile. Cowork is its agent for knowledge work, and it runs on all of those surfaces too. Connected to your apps, Claude can work with your files and services to accomplish complex tasks on your behalf.

This guide walks you through the easiest and most secure way to connect your Scorecard account to Claude via Composio Connect, enabling it to create a new testset for model evaluation, run evaluation on your latest model, list failing testcases in project Alpha, and more such actions on your behalf without compromising your account security.

Setup is the same on Claude Web, Desktop, and Cowork, and you only need to do it once. The connector is tied to your account, so it's available in all three.

Also integrate Scorecard MCP with

ChatGPTHermesAtomic Agent

Connecting Scorecard to Claude Web, Desktop, and Cowork

Two ways to add the Composio connector. Pick whichever you prefer.

1. Add with one click

Click the button. Claude opens with the Composio connector filled in. Click Add, then go to step 2.

Add to Claude

Add manually

In Claude Desktop, click Customize in the left sidebar, then select Connectors and click the + icon at the top.

Claude Desktop connectors screen with Add custom connector selected

Click Add custom connector and paste in the Composio MCP server URL:

bash
https://connect.composio.dev/mcp
Add custom connector dialog with Composio MCP server URL

2. Authorize in your browser

Click Connect. You'll be redirected to a browser window where you can authorize Composio to continue.

Composio authorization screen for Claude Web, Desktop, and Cowork

3. Connect your Scorecard account

Back in Claude, ask it to connect to Scorecard or give it any Scorecard-related task.

For example, ask Claude to:

  • "Create a new testset for model evaluation"
  • "Run evaluation on your latest model"
  • "List failing testcases in project Alpha"

It will prompt you to authenticate and authorize access.

That's it. Composio's tools are now available in Claude, and your Scorecard account is ready to use.

What is the Scorecard Claude Connector, and what's possible with it?

The Scorecard MCP server is an implementation of the Model Context Protocol that connects your AI agent and assistants like Claude, Cursor, etc directly to your Scorecard account. It provides structured and secure access so your agent can perform Scorecard operations on your behalf.

Troubleshooting

1. Why does Claude not use the Composio connector for Scorecard?

The connector can show as connected, but Claude must use it in each Scorecard conversation.

  • Click + at the lower left of the chat, or type /.
  • Hover Connectors.
  • Turn on Composio.
  • Ask Claude to connect to Scorecard again.

2. Why does Claude show an MCP server error before Scorecard connects?

Claude can show Couldn't reach the MCP server or Authorization with the MCP server failed when you add the connector for Scorecard.

  • Check that the connector URL is https://connect.composio.dev/mcp.
  • Connectors run from Anthropic's cloud, so your device network is usually not the problem.
  • Add the connector again from Customize > Connectors, then ask Claude to connect Scorecard.
  • If it fails again, send the error and the ofid_ reference id to support@composio.dev.

3. Why does the Scorecard authorization link not work?

The Composio connector is one server. Your Scorecard account connects separately when Claude asks for it.

  • If the browser does not open, or the link expires, ask Claude to retry the Scorecard connection.
  • Composio creates a fresh Scorecard authorization link.
  • Open that link in your browser and finish the Scorecard sign-in.

4. Why do Scorecard actions fail after they worked before?

The Scorecard connection can expire or lose permission.

  • Ask Claude to inspect the Scorecard connection.
  • If the connection is unhealthy, disconnect it when Claude prompts you.
  • Reconnect the Scorecard account and retry the action.
  • If Claude shows a permission prompt, approve it before Claude changes data in Scorecard.
  • If the action still fails with a permission error, check that the connected account has that permission in Scorecard itself.

5. How do I switch to a different Scorecard account?

Claude can manage the Scorecard connection for you.

  • Ask Claude to disconnect or delete the current Scorecard account.
  • Ask Claude to connect Scorecard again.
  • Open the new Composio authorization link for Scorecard.
  • Sign in with the other Scorecard account in the browser.
  • After the account connects, ask Claude to retry the Scorecard task.

6. Why is my Scorecard connection missing in Claude Desktop or on a Team plan?

The connector belongs to your Claude account, so your Scorecard connection works in Claude Web, Desktop, and Cowork.

  • In Claude Desktop, open Customize > Connectors.
  • Next to Composio, click ⋮ and click Clear cache.
  • If Scorecard tools still do not appear, click ⋮, disconnect, remove the connector, and add it again.
  • On Team or Enterprise, an Owner or Primary Owner must enable the connector before you can connect Scorecard.
TOOLS

Supported Tools

Every Scorecard MCP action and event your agent gets out of the box.

Execute

Runs JavaScript code to interact with the Scorecard API.

Search docs

Search SDK documentation to find methods, parameters, and usage examples for interacting with the API.

FRAMEWORKS

How to build Scorecard MCP Agent with another framework

ChatGPT logo

ChatGPT

Use Scorecard MCP with ChatGPT

Hermes logo

Hermes

Use Scorecard MCP with Hermes

Atomic Agent logo

Atomic Agent

Use Scorecard MCP with Atomic Agent

FAQ

Frequently asked questions

With a standalone Scorecard MCP server, the agents and LLMs can only access a fixed set of Scorecard tools tied to that server. However, with the Composio Tool Router, agents can dynamically load tools from Scorecard and many other apps based on the task at hand, all through a single MCP endpoint.

Yes, you can. Claude Cowork fully supports MCP integration. You get structured tool calling, message history handling, and model orchestration while Tool Router takes care of discovering and serving the right Scorecard tools.

Yes, absolutely. You can configure which Scorecard scopes and actions are allowed when connecting your account to Composio. You can also bring your own OAuth credentials or API configuration so you keep full control over what the agent can do.

All sensitive data such as tokens, keys, and configuration is fully encrypted at rest and in transit. Composio is SOC 2 Type 2 compliant and follows strict security practices so your Scorecard data and credentials are handled as safely as possible.

Start with Scorecard MCP.It takes 30 seconds.

Managed auth, hosted MCP servers, and every Scorecard MCP tool your agent needs.Free to start.

Start building