How to integrate Spider MCP with Atomic Agent

Connect Atomic Agent to Spider MCP. Crawl a product site and summarize pricing, search the web for competitor product updates, and more using natural language, with authentication handled for you.

Spider logoSpider
Api Key

Spider is a web data platform for crawling, scraping, search, screenshots, and AI extraction. It helps teams collect and transform web content at high concurrency.

15 Tools

Introduction

Atomic Agent is an open-source AI agent that runs on your machine and supports local models through llama.cpp. Connect it to 1,700+ apps through Composio, including Spider.

This guide uses Atomic Agent's built-in Composio integration. You need Atomic Agent v0.5.6 or later, your own Composio API key, and a Spider account. See the Atomic Agent documentation for installation instructions.

For exact tool slugs and parameter tables, see the tool catalog in the Composio docs.

Also integrate Spider with

How Atomic Agent connects to Spider

Atomic Agent connects to Composio's Tool Router over Streamable HTTP MCP. It discovers the tools needed for your request, and Composio handles account authentication.

Execution and account-connection tools follow Atomic Agent's configured approval policy. Review any approval prompt before allowing an action on your connected account.

Connect Spider to Atomic Agent

  1. Sign up or sign in to the Composio dashboard. Copy your API key.
  2. In Atomic Agent, open the Integrations tab and select Composio. Select API key, press e, paste the key, and press Enter. The tools become available immediately, without a restart.
  3. Ask Atomic Agent to do something in Spider. It searches Composio for the right tool.
  4. On first use, open the sign-in link returned in the chat. Connect your Spider account, then return to Atomic Agent to continue.

Headless setup

Add the following line to ~/.atomic-agent/.env, replacing YOUR_COMPOSIO_API_KEY with your key. Start Atomic Agent after saving the file.

bash
COMPOSIO_API_KEY=YOUR_COMPOSIO_API_KEY

What is the Spider MCP server, and what's possible with it?

The Spider MCP server is an implementation of the Model Context Protocol that connects your AI agent and assistants like Claude, Cursor, etc directly to your Spider account. It provides structured and secure access to web crawling, scraping, search, screenshots, content transformation, and AI extraction tools, so your agent can crawl sites, scrape pages, search the web, extract structured data, and transform or capture web content on your behalf.

  • AI-powered website extraction: Have your agent crawl websites or scrape specific pages, then extract structured information from them using natural-language instructions.
  • Web search and discovery: Direct your agent to search the web, fetch relevant result pages, or discover links that match a specific description across a site.
  • Browser-based research: Let your agent use the AI browser to navigate websites from a natural-language prompt and return the requested results.
  • Page screenshots: Instruct your agent to capture a rendered webpage as a PNG, JPEG, or WebP image.
  • Content transformation: Have your agent convert supplied HTML into markdown, text, or cleaned output without fetching a URL.

Use Spider from Atomic Agent

With your account connected, ask Atomic Agent to work with Spider in natural language. You can connect more apps through the same Composio integration as you need them.

Visit the Atomic Agent website or inspect the Atomic Agent source code.

Troubleshooting

Why is my API key rejected?

Copy the key directly from the Composio dashboard. Atomic Agent requires ASCII characters because it sends the key in an HTTP header. Remove any extra characters introduced when copying.

Why are Composio tools unavailable after saving a key?

Check that you use Atomic Agent v0.5.6 or later and that Composio is enabled in your configuration. A "composio": { "enabled": false } setting disables the integration while keeping the saved key on disk.

Does Composio have a free plan?

The Hobby plan includes 100,000 tool calls per month. Composio-managed OAuth apps have a lower included allowance. See Composio pricing for the current limits that apply to your connection.

TOOLS

Supported Tools

Every Spider action and event your agent gets out of the box.

AI Browser

Drive a browser from a natural-language prompt and extract results.

AI Crawl Website

Crawl pages and use an AI prompt to extract structured data.

AI Discover Links

Collect links matching a natural-language prompt and return structured context.

AI Scrape Page

Fetch exact pages and use an AI prompt to extract structured data.

AI Search Web

Run prompt-driven web search with optional page fetching and structured extraction.

Capture Screenshot

Render one page and capture a PNG, JPEG, or WebP screenshot with a provider-side response-content cap.

Crawl Website

Crawl pages from one or more starting URLs.

Discover Links

Discover URLs from a site without returning full page content.

Fetch Configured Page

Fetch a domain and path with Spider's saved scraper configuration, discovering and storing one on first use.

Get Credit Balance

Get the connected Spider account's remaining credit balance and documented record metadata.

List Crawl Logs

Return the connected account's crawl and scrape activity from the last 24 hours.

List Scraper Directory

Browse public curated scraper configurations with domain, category, text, and page-number filters.

Scrape Pages

Fetch and extract exact pages without following links.

Search Web

Search the web and optionally fetch result pages.

Transform Content

Convert supplied HTML to markdown, text, or cleaned output without fetching a URL; the total serialized JSON request must not exceed 10 MB.

FAQ

Frequently asked questions

Start with Spider.It takes 30 seconds.

Managed auth, hosted MCP servers, and every Spider tool your agent needs.Free to start.

Start building