Spider MCP server for AI agents and assistants

Securely connect your AI agents and chatbots (Claude, ChatGPT, Cursor, etc) with Spider MCP or direct API to crawl websites, scrape pages, search the web, and extract structured data through natural language.

Spider logoSpider
Api Key

Spider is a web data platform for crawling, scraping, search, screenshots, and AI extraction. It helps teams collect and transform web content at high concurrency.

15 Tools

Try Spider now

Type what you want done — sign in and watch it run live in the Tool Router playground.

TOOL ROUTER PLAYGROUND
Spider
Try asking
TOOLS

Supported Tools

Every Spider action and event your agent gets out of the box.

AI Browser

Drive a browser from a natural-language prompt and extract results.

AI Crawl Website

Crawl pages and use an AI prompt to extract structured data.

AI Discover Links

Collect links matching a natural-language prompt and return structured context.

AI Scrape Page

Fetch exact pages and use an AI prompt to extract structured data.

AI Search Web

Run prompt-driven web search with optional page fetching and structured extraction.

Capture Screenshot

Render one page and capture a PNG, JPEG, or WebP screenshot with a provider-side response-content cap.

Crawl Website

Crawl pages from one or more starting URLs.

Discover Links

Discover URLs from a site without returning full page content.

Fetch Configured Page

Fetch a domain and path with Spider's saved scraper configuration, discovering and storing one on first use.

Get Credit Balance

Get the connected Spider account's remaining credit balance and documented record metadata.

List Crawl Logs

Return the connected account's crawl and scrape activity from the last 24 hours.

List Scraper Directory

Browse public curated scraper configurations with domain, category, text, and page-number filters.

Scrape Pages

Fetch and extract exact pages without following links.

Search Web

Search the web and optionally fetch result pages.

Transform Content

Convert supplied HTML to markdown, text, or cleaned output without fetching a URL; the total serialized JSON request must not exceed 10 MB.

SETUP GUIDE

Connect Spider MCP Tool with your Agent

1

Install Composio

typescript
npm install @composio/core ai @ai-sdk/mcp @ai-sdk/openai
Install the Composio SDK and your agent framework
2

Create a session with MCP enabled

typescript
import { Composio } from "@composio/core";

const composio = new Composio();
const { mcp } = await composio.create("your-user-id", {
  toolkits: ["spider"],
  mcp: true,
});
Create a session scoped to Spider and read its MCP URL and headers
3

Connect your agent to the MCP server

typescript
import { createMCPClient } from "@ai-sdk/mcp";
import { openai } from "@ai-sdk/openai";
import { generateText, stepCountIs } from "ai";

const client = await createMCPClient({
  transport: { type: "http", url: mcp.url, headers: mcp.headers },
});

const { text } = await generateText({
  model: openai("gpt-5.6-sol"),
  tools: await client.tools(),
  prompt: "Crawl https://example.com and summarize the main page content",
  stopWhen: stepCountIs(10),
});

console.log(text);
await client.close();
Pass the session's MCP URL and headers to your agent and run a Spider request
SETUP GUIDE

Connect Spider API Tool with your Agent

1

Install Composio

typescript
npm install @composio/core @composio/openai openai
Install the Composio SDK, the OpenAI provider, and the OpenAI SDK
2

Create a Composio session

typescript
import OpenAI from "openai";
import { Composio } from "@composio/core";
import { OpenAIResponsesProvider } from "@composio/openai";

const composio = new Composio({ provider: new OpenAIResponsesProvider() });
const client = new OpenAI();

const session = await composio.create("your-user-id", { toolkits: ["spider"] });
const tools = await session.tools();
Initialize Composio with the OpenAI Responses provider and create a session scoped to Spider
3

Run Spider tools with your agent

typescript
let response = await client.responses.create({
  model: "gpt-5.6-sol",
  tools,
  input: [{ role: "user", content: "Crawl https://example.com and summarize the main page content" }],
});

while (response.output.some((o) => o.type === "function_call")) {
  const outputs = await composio.provider.handleToolCalls(session, response.output);
  response = await client.responses.create({
    model: "gpt-5.6-sol",
    tools,
    previous_response_id: response.id,
    input: outputs,
  });
}

console.log(response.output_text);
Send a request, execute the Spider tool calls through the session, and print the final answer

Why Use Composio?

AI Native Spider Integration

  • Supports both Spider MCP and direct API based integrations
  • Structured, LLM-friendly schemas for reliable crawling, scraping, search, and extraction
  • Rich coverage for collecting, transforming, and querying web content

Managed Auth

  • Connect Spider with secure API key handling through Composio
  • Central place to manage, scope, and revoke Spider access
  • Per user and per environment credentials instead of hard-coded keys

Agent Optimized Design

  • Tools are tuned using real error and success rates to improve reliability over time
  • Comprehensive execution logs so you always know what ran, when, and on whose behalf

Enterprise Grade Security

  • Fine-grained RBAC so you control which agents and users can access Spider
  • Scoped, least privilege access to Spider resources
  • Full audit trail of agent actions to support review and compliance

Rolling this out across your team?

Give your team centralized access control across every framework with Composio’s MCP Gateway.

EXPLORE MCP GATEWAY
FAQ

Frequently asked questions

Start with Spider.It takes 30 seconds.

Managed auth, hosted MCP servers, and every Spider tool your agent needs.Free to start.

Start building