How to integrate Spider MCP with Hermes

Connect Hermes to Spider MCP. Crawl a product site and summarize pricing, search the web for competitor product updates, and more using natural language, with authentication handled for you.

Spider logoSpider
Api Key

Spider is a web data platform for crawling, scraping, search, screenshots, and AI extraction. It helps teams collect and transform web content at high concurrency.

15 Tools

Introduction

Hermes is a 24/7 autonomous agent that lives on your computer or server — it remembers what it learns and evolves as your usage grows.

This guide explains the easiest and most robust way to connect your Spider account to Hermes. You can do this through either Composio Connect CLI or Composio Connect MCP. For personal use we recommend the CLI, but you won't go wrong with MCP either.

For exact tool slugs and parameter tables, see the tool catalog in the Composio docs.

Also integrate Spider with

What is Composio Connect?

Composio Connect is a consumer offering that lets anyone plug 1,700+ applications directly into their agent harness — including Hermes. It can:

  • Search and load tools from relevant toolkits on-demand, reducing context usage.
  • Chain multiple tools to accomplish complex workflows via a remote workbench, without excessive back-and-forth with the LLM.
  • Manage app authentication end-to-end with zero manual overhead.

Integrating Spider with Hermes

Using Composio Connect CLI

1. Install the Composio CLI

Run the install script directly, or paste https://composio.dev/hermes into your Hermes chat box to have it installed for you.

bash
curl -fsSL https://composio.dev/install | bash
Hermes authenticating with Composio

2. Authenticate

Once the CLI is installed, ask Hermes to authenticate with Composio.

3. Connect to Spider

Ask your agent to connect to Spider, or simply request any Spider-related task. Hermes will prompt you to authenticate and authorize access.

4. Done. You're all set with a new Spider connection.


Using Composio Connect MCP

1. Get your MCP URL and API Key

Go to dashboard.composio.dev and copy your Connect MCP URL and API key.

Copy MCP URL and API key from Composio dashboard

2. Open the Hermes config file

bash
nano ~/.hermes/config.yaml

3. Add the Composio Connect MCP server

bash
mcp_servers:
  composio:
    url: "https://connect.composio.dev/mcp"
    headers:
      x-consumer-api-key: "YOUR_COMPOSIO_API_KEY"
    connect_timeout: 60
    timeout: 180

Save with Ctrl + O, Enter, then exit with Ctrl + X.

4. Restart your Hermes agent

Once restarted, ask your agent to connect to Spider or request any Spider-related task. It will prompt you to authenticate and authorize access.

5. Done!

What is the Spider MCP server, and what's possible with it?

The Spider MCP server is an implementation of the Model Context Protocol that connects your AI agent and assistants like Claude, Cursor, etc directly to your Spider account. It provides structured and secure access to web crawling, scraping, search, screenshots, content transformation, and AI extraction tools, so your agent can crawl sites, scrape pages, search the web, extract structured data, and transform or capture web content on your behalf.

  • AI-powered website extraction: Have your agent crawl websites or scrape specific pages, then extract structured information from them using natural-language instructions.
  • Web search and discovery: Direct your agent to search the web, fetch relevant result pages, or discover links that match a specific description across a site.
  • Browser-based research: Let your agent use the AI browser to navigate websites from a natural-language prompt and return the requested results.
  • Page screenshots: Instruct your agent to capture a rendered webpage as a PNG, JPEG, or WebP image.
  • Content transformation: Have your agent convert supplied HTML into markdown, text, or cleaned output without fetching a URL.

Way Forward

With Spider connected, Hermes can now act on your behalf whenever it detects a relevant task or you ask it to.

From here, you can extend Hermes further:

  • Connect more apps: Calendar, Slack, Notion, Linear, and hundreds of others are available through the same Composio Connect setup. Each new integration compounds what Hermes can do for you.
  • Build workflows across tools: Once multiple apps are connected, Hermes can chain actions together — turn an email into a calendar invite, a Slack message into a Linear ticket, or a meeting note into a follow-up draft.
  • Let it learn your patterns: The more you use Hermes, the better it gets at anticipating how you'd handle recurring tasks. Give it feedback on drafts and decisions, and it will adapt.

If you run into trouble or want to share what you've built, join the community or check out the Docs for deeper configuration options.

TOOLS

Supported Tools

Every Spider action and event your agent gets out of the box.

AI Browser

Drive a browser from a natural-language prompt and extract results.

AI Crawl Website

Crawl pages and use an AI prompt to extract structured data.

AI Discover Links

Collect links matching a natural-language prompt and return structured context.

AI Scrape Page

Fetch exact pages and use an AI prompt to extract structured data.

AI Search Web

Run prompt-driven web search with optional page fetching and structured extraction.

Capture Screenshot

Render one page and capture a PNG, JPEG, or WebP screenshot with a provider-side response-content cap.

Crawl Website

Crawl pages from one or more starting URLs.

Discover Links

Discover URLs from a site without returning full page content.

Fetch Configured Page

Fetch a domain and path with Spider's saved scraper configuration, discovering and storing one on first use.

Get Credit Balance

Get the connected Spider account's remaining credit balance and documented record metadata.

List Crawl Logs

Return the connected account's crawl and scrape activity from the last 24 hours.

List Scraper Directory

Browse public curated scraper configurations with domain, category, text, and page-number filters.

Scrape Pages

Fetch and extract exact pages without following links.

Search Web

Search the web and optionally fetch result pages.

Transform Content

Convert supplied HTML to markdown, text, or cleaned output without fetching a URL; the total serialized JSON request must not exceed 10 MB.

FAQ

Frequently asked questions

Start with Spider.It takes 30 seconds.

Managed auth, hosted MCP servers, and every Spider tool your agent needs.Free to start.

Start building