# Spider

```json
{
  "name": "Spider",
  "slug": "spider",
  "url": "https://composio.dev/toolkits/spider",
  "markdown_url": "https://composio.dev/toolkits/spider.md",
  "logo_url": "https://logos.composio.dev/api/spider",
  "categories": [
    "developer tools & devops"
  ],
  "is_composio_managed": false,
  "updated_at": "2026-11-18T05:41:33.290Z"
}
```

![Spider logo](https://logos.composio.dev/api/spider)

## Description

Securely connect your AI agents and chatbots (Claude, ChatGPT, Cursor, etc) with Spider MCP or direct API to crawl websites, scrape pages, search the web, and extract structured data through natural language.

## Summary

Spider is a web data platform for crawling, scraping, search, screenshots, and AI extraction.
It helps teams collect and transform web content at high concurrency.

## Categories

- developer tools & devops

## Toolkit Details

- Tools: 15

## Images

- Logo: https://logos.composio.dev/api/spider

## Authentication

- **Api Key**
  - Type: `api_key`
  - Description: Api Key authentication for Spider.
  - Setup:
    - Configure Api Key credentials for Spider.
    - Use the credentials when creating an auth config in Composio.

## Suggested Prompts

- Crawl a product site and summarize pricing
- Search the web for competitor product updates
- Extract contact details from company websites

## Supported Tools

| Tool slug | Name | Description |
|---|---|---|
| `SPIDER_AI_BROWSER` | AI Browser | Drive a browser from a natural-language prompt and extract results. Requires AI Studio. Prompts can click, navigate, submit forms, purchase, message, or change accounts, so request those external effects only when intended and authorized. |
| `SPIDER_AI_CRAWL` | AI Crawl Website | Crawl pages and use an AI prompt to extract structured data. Requires AI Studio. limit defaults to 1 to bound crawl and model costs; depth 0 removes the depth limit. |
| `SPIDER_AI_DISCOVER_LINKS` | AI Discover Links | Collect links matching a natural-language prompt and return structured context. Requires AI Studio. limit defaults to 1 to bound crawl and model costs; depth 0 removes the depth limit. |
| `SPIDER_AI_SCRAPE` | AI Scrape Page | Fetch exact pages and use an AI prompt to extract structured data. Requires AI Studio; batched or complex extraction increases model cost. |
| `SPIDER_AI_SEARCH` | AI Search Web | Run prompt-driven web search with optional page fetching and structured extraction. Requires AI Studio and defaults to one result without page fetching. |
| `SPIDER_CAPTURE_SCREENSHOT` | Capture Screenshot | Render one page and capture a PNG, JPEG, or WebP screenshot with a provider-side response-content cap. Full-page capture is the documented default; returns compact metadata by default, optional base64 content, or an offloaded file in binary mode. |
| `SPIDER_CRAWL` | Crawl Website | Crawl pages from one or more starting URLs. Defaults to at most 10 pages and depth 1; raising breadth increases cost and response size. Set billing_mode='unlimited' only for an active Unlimited plan with purchased concurrency. |
| `SPIDER_DISCOVER_LINKS` | Discover Links | Discover URLs from a site without returning full page content. Defaults to 25 pages and depth 1; larger traversal costs more. Set billing_mode='unlimited' only for an active Unlimited plan with purchased concurrency. |
| `SPIDER_FETCH_CONFIGURED_PAGE` | Fetch Configured Page | Fetch a domain and path with Spider's saved scraper configuration, discovering and storing one on first use. Defaults to one page; first-time discovery can take 30-120 seconds and incur AI cost. |
| `SPIDER_GET_CREDIT_BALANCE` | Get Credit Balance | Get the connected Spider account's remaining credit balance and documented record metadata. |
| `SPIDER_LIST_CRAWL_LOGS` | List Crawl Logs | Return the connected account's crawl and scrape activity from the last 24 hours. |
| `SPIDER_LIST_SCRAPER_DIRECTORY` | List Scraper Directory | Browse public curated scraper configurations with domain, category, text, and page-number filters. |
| `SPIDER_SCRAPE` | Scrape Pages | Fetch and extract exact pages without following links. Use crawl only for traversal; set billing_mode='unlimited' only for an active Unlimited plan with purchased concurrency. |
| `SPIDER_SEARCH_WEB` | Search Web | Search the web and optionally fetch result pages. Defaults to 10 results without page fetching or automatic multi-page expansion. |
| `SPIDER_TRANSFORM_CONTENT` | Transform Content | Convert supplied HTML to markdown, text, or cleaned output without fetching a URL; the total serialized JSON request must not exceed 10 MB. |

## Supported Triggers

None listed.

## Installation and MCP Setup

### Path 1: SDK Installation

#### Path 1, Step 1: Install Composio

Install the Composio SDK, the OpenAI provider, and the OpenAI SDK
```python
pip install composio composio_openai openai
```

```typescript
npm install @composio/core @composio/openai openai
```

#### Path 1, Step 2: Create a Composio session

Initialize Composio with the OpenAI Responses provider and create a session scoped to Spider
```python
import json
from openai import OpenAI
from composio import Composio
from composio_openai import OpenAIResponsesProvider

composio = Composio(provider=OpenAIResponsesProvider())
client = OpenAI()

session = composio.create(user_id="your-user-id", toolkits=["spider"])
tools = session.tools()
```

```typescript
import OpenAI from "openai";
import { Composio } from "@composio/core";
import { OpenAIResponsesProvider } from "@composio/openai";

const composio = new Composio({ provider: new OpenAIResponsesProvider() });
const client = new OpenAI();

const session = await composio.create("your-user-id", { toolkits: ["spider"] });
const tools = await session.tools();
```

#### Path 1, Step 3: Run Spider tools with your agent

Send a request, execute the Spider tool calls through the session, and print the final answer
```python
response = client.responses.create(
    model="gpt-5.6-sol",
    tools=tools,
    input=[{"role": "user", "content": "Crawl https://example.com and summarize the main page content"}],
)

while True:
    tool_calls = [o for o in response.output if o.type == "function_call"]
    if not tool_calls:
        break
    results = composio.provider.handle_tool_calls(response=response, session=session)
    response = client.responses.create(
        model="gpt-5.6-sol",
        tools=tools,
        previous_response_id=response.id,
        input=[
            {"type": "function_call_output", "call_id": call.call_id, "output": json.dumps(results[i])}
            for i, call in enumerate(tool_calls)
        ],
    )

print(response.output_text)
```

```typescript
let response = await client.responses.create({
  model: "gpt-5.6-sol",
  tools,
  input: [{ role: "user", content: "Crawl https://example.com and summarize the main page content" }],
});

while (response.output.some((o) => o.type === "function_call")) {
  const outputs = await composio.provider.handleToolCalls(session, response.output);
  response = await client.responses.create({
    model: "gpt-5.6-sol",
    tools,
    previous_response_id: response.id,
    input: outputs,
  });
}

console.log(response.output_text);
```

### Path 2: MCP Server Setup

#### Path 2, Step 1: Install Composio

Install the Composio SDK and your agent framework
```python
pip install composio claude-agent-sdk
```

```typescript
npm install @composio/core ai @ai-sdk/mcp @ai-sdk/openai
```

#### Path 2, Step 2: Create a session with MCP enabled

Create a session scoped to Spider and read its MCP URL and headers
```python
from composio import Composio

composio = Composio()
session = composio.create(user_id="your-user-id", toolkits=["spider"], mcp=True)
```

```typescript
import { Composio } from "@composio/core";

const composio = new Composio();
const { mcp } = await composio.create("your-user-id", {
  toolkits: ["spider"],
  mcp: true,
});
```

#### Path 2, Step 3: Connect your agent to the MCP server

Pass the session's MCP URL and headers to your agent and run a Spider request
```python
import asyncio
from claude_agent_sdk import ClaudeSDKClient, ClaudeAgentOptions, AssistantMessage, TextBlock

options = ClaudeAgentOptions(
    mcp_servers={
        "composio": {
            "type": "http",
            "url": session.mcp.url,
            "headers": session.mcp.headers,
        }
    },
    system_prompt="You are a helpful assistant with access to Spider tools.",
    tools=[],
    allowed_tools=["mcp__composio__*"],
)

async def main():
    async with ClaudeSDKClient(options=options) as client:
        await client.query("Crawl https://example.com and summarize the main page content")
        async for message in client.receive_response():
            if isinstance(message, AssistantMessage):
                for block in message.content:
                    if isinstance(block, TextBlock):
                        print(block.text)

asyncio.run(main())
```

```typescript
import { createMCPClient } from "@ai-sdk/mcp";
import { openai } from "@ai-sdk/openai";
import { generateText, stepCountIs } from "ai";

const client = await createMCPClient({
  transport: { type: "http", url: mcp.url, headers: mcp.headers },
});

const { text } = await generateText({
  model: openai("gpt-5.6-sol"),
  tools: await client.tools(),
  prompt: "Crawl https://example.com and summarize the main page content",
  stopWhen: stepCountIs(10),
});

console.log(text);
await client.close();
```

## Why Use Composio?

### 1. AI Native Spider Integration

- Supports both Spider MCP and direct API based integrations
- Structured, LLM-friendly schemas for reliable crawling, scraping, search, and extraction
- Rich coverage for collecting, transforming, and querying web content

### 2. Managed Auth

- Connect Spider with secure API key handling through Composio
- Central place to manage, scope, and revoke Spider access
- Per user and per environment credentials instead of hard-coded keys

### 3. Agent Optimized Design

- Tools are tuned using real error and success rates to improve reliability over time
- Comprehensive execution logs so you always know what ran, when, and on whose behalf

### 4. Enterprise Grade Security

- Fine-grained RBAC so you control which agents and users can access Spider
- Scoped, least privilege access to Spider resources
- Full audit trail of agent actions to support review and compliance

## Use Spider with any AI Agent Framework

Choose a framework you want to connect Spider with:

- [ChatGPT](https://composio.dev/toolkits/spider/framework/chatgpt)
- [Claude Cowork](https://composio.dev/toolkits/spider/framework/claude-cowork)
- [Hermes](https://composio.dev/toolkits/spider/framework/hermes-agent)
- [Atomic Agent](https://composio.dev/toolkits/spider/framework/atomic-agent)

## Related Toolkits

- [Supabase](https://composio.dev/toolkits/supabase) - Supabase is an open-source backend platform offering scalable Postgres databases, authentication, storage, and real-time APIs. It lets developers build modern apps without managing infrastructure.
- [Codeinterpreter](https://composio.dev/toolkits/codeinterpreter) - Codeinterpreter is a Python-based coding environment with built-in data analysis and visualization. It lets you instantly run scripts, plot results, and prototype solutions inside supported platforms.
- [GitHub](https://composio.dev/toolkits/github) - GitHub is a code hosting platform for version control and collaborative software development. It streamlines project management, code review, and team workflows in one place.
- [1password](https://composio.dev/toolkits/_1password) - 1Password is a password manager and digital vault for storing logins, secrets, notes, and secure documents. It helps individuals and teams protect credentials, share access safely, and reduce password risk.
- [Ably](https://composio.dev/toolkits/ably) - Ably is a real-time messaging platform for live chat and data sync in modern apps. It offers global scale and rock-solid reliability for seamless, instant experiences.
- [Abuselpdb](https://composio.dev/toolkits/abuselpdb) - Abuselpdb is a central database for reporting and checking IPs linked to malicious online activity. Use it to quickly identify and report suspicious or abusive IP addresses.
- [Alchemy](https://composio.dev/toolkits/alchemy) - Alchemy is a blockchain development platform offering APIs and tools for Ethereum apps. It simplifies building and scaling Web3 projects with robust infrastructure.
- [Algolia](https://composio.dev/toolkits/algolia) - Algolia is a hosted search API that powers lightning-fast, relevant search experiences for web and mobile apps. It helps developers deliver instant, typo-tolerant, and scalable search without complex infrastructure.
- [Anakin](https://composio.dev/toolkits/anakin_io) - Anakin is a web automation platform offering scraping, crawling, search, and browser automation. Use it to extract structured data, automate site tasks, and monitor web changes reliably.
- [Anchor browser](https://composio.dev/toolkits/anchor_browser) - Anchor browser is a developer platform for AI-powered web automation. It transforms complex browser actions into easy API endpoints for streamlined web interaction.
- [AnyAPI](https://composio.dev/toolkits/anyapi) - AnyAPI is a marketplace of utility APIs for validation, currency conversion, holidays, documents, and web data. It brings common application workflows together through one practical API platform.
- [Apiflash](https://composio.dev/toolkits/apiflash) - Apiflash is a website screenshot API for programmatically capturing web pages. It delivers high-quality screenshots on demand for automation, monitoring, or reporting.
- [Apiverve](https://composio.dev/toolkits/apiverve) - Apiverve delivers a suite of powerful APIs that simplify integration for developers. It's designed for reliability and scalability so you can build faster, smarter applications without the integration headache.
- [Appcircle](https://composio.dev/toolkits/appcircle) - Appcircle is an enterprise-grade mobile CI/CD platform for building, testing, and publishing mobile apps. It streamlines mobile DevOps so teams ship faster and with more confidence.
- [Appdrag](https://composio.dev/toolkits/appdrag) - Appdrag is a cloud platform for building websites, APIs, and databases with drag-and-drop tools and code editing. It accelerates development and iteration by combining hosting, database management, and low-code features in one place.
- [Appveyor](https://composio.dev/toolkits/appveyor) - AppVeyor is a cloud-based continuous integration service for building, testing, and deploying applications. It helps developers automate and streamline their software delivery pipelines.
- [Authyo](https://composio.dev/toolkits/authyo) - Authyo is a REST API service for passwordless authentication, OTP verification, session management, and transactional notifications. Use it to add secure login flows and user verification without building auth infrastructure from scratch.
- [AWS Marketplace MCP](https://composio.dev/toolkits/aws_marketplace_mcp) - AWS Marketplace MCP provides MCP access to AWS Marketplace's cloud software, data, and services catalog. Use it to discover, compare, and evaluate 30K+ AWS Marketplace listings faster.
- [Azure Monitor Activity Log](https://composio.dev/toolkits/azure_monitor_activity_log) - Azure Monitor Activity Log is Azure's service that records management and control-plane events for a subscription. Use it to audit changes, troubleshoot issues, and track resource operations across your Azure resources.
- [Backendless](https://composio.dev/toolkits/backendless) - Backendless is a backend-as-a-service platform for mobile and web apps, offering database, file storage, user authentication, and APIs. It helps developers ship scalable applications faster without managing server infrastructure.

## Frequently Asked Questions

### Do I need my own developer credentials to use Spider with Composio?

Yes, Spider requires you to configure your own API key credentials. Once set up, Composio handles secure credential storage and API request handling for you.

### Can I use multiple toolkits together?

Yes! Composio's Tool Router enables agents to use multiple toolkits. [Learn more](https://docs.composio.dev/tool-router/overview).

### Is Composio secure?

Composio is SOC 2 and ISO 27001 compliant with all data encrypted in transit and at rest. [Learn more](https://trust.composio.dev).

### What if the API changes?

Composio maintains and updates all toolkit integrations automatically, so your agents always work with the latest API versions.

---
[See all toolkits](https://composio.dev/toolkits) · [Composio docs](https://docs.composio.dev/llms.txt)
