# Crawl4AI

```json
{
  "name": "Crawl4AI",
  "slug": "crawl4ai",
  "url": "https://composio.dev/toolkits/crawl4ai",
  "markdown_url": "https://composio.dev/toolkits/crawl4ai.md",
  "logo_url": "https://logos.composio.dev/api/crawl4ai",
  "categories": [
    "developer tools & devops"
  ],
  "is_composio_managed": false,
  "updated_at": "2026-11-17T05:35:02.235Z"
}
```

![Crawl4AI logo](https://logos.composio.dev/api/crawl4ai)

## Description

Securely connect your AI agents and chatbots (Claude, ChatGPT, Cursor, etc) with Crawl4AI MCP or direct API to search the web, crawl pages, extract structured data, and monitor websites through natural language.

## Summary

Crawl4AI is a web scraping and search service for extracting structured data from websites.
It helps teams collect clean web data, monitor pages, and verify leads at scale.

## Categories

- developer tools & devops

## Toolkit Details

- Tools: 51

## Images

- Logo: https://logos.composio.dev/api/crawl4ai

## Authentication

- **Api Key**
  - Type: `api_key`
  - Description: Api Key authentication for Crawl4AI.
  - Setup:
    - Configure Api Key credentials for Crawl4AI.
    - Use the credentials when creating an auth config in Composio.

## Suggested Prompts

- Search web for competitor pricing pages
- Extract product data from retailer pages
- Verify email addresses from lead lists

## Supported Tools

| Tool slug | Name | Description |
|---|---|---|
| `CRAWL4AI_ANSWER_QUESTION` | Answer Question | Answer a question from web sources and return the answer with citations when available. This Crawl4AI endpoint is experimental. |
| `CRAWL4AI_CANCEL_RECIPE_JOB` | Cancel Recipe Job | Stop a running recipe job. The provider retains it in paused state and may report a pause reason and partial usage. |
| `CRAWL4AI_CANCEL_SITE_CRAWL` | Cancel Site Crawl | Stop a running site crawl job; the job ends in the paused state. |
| `CRAWL4AI_CONSUME_MONITOR_CHANGES` | Read Monitor Changes | Read the oldest new changes across monitors and mark the returned feed items as read. Without since, later calls start after consumed items; pass a prior cursor to replay. Requires confirm_mark_read=true; use agent to limit which agent's feed is consumed. |
| `CRAWL4AI_CREATE_MONITOR` | Create Monitor | Create a page-change monitor, or return the existing equivalent monitor, that performs recurring credit-consuming checks until paused or deleted and can send configured notifications. |
| `CRAWL4AI_CREATE_PLAN_CHECKOUT` | Create Plan Checkout | Create an account-specific Stripe checkout flow for a Crawl4AI paid plan. This only returns a checkout URL: it does not directly charge a card or start the plan. Open and pay the Stripe checkout URL to complete the subscription. Use GET_PRICES_AND_PLANS first for current plan names and prices. |
| `CRAWL4AI_DELETE_MONITOR` | Delete Monitor | Permanently delete one page-change monitor and its history. Use this for intentional removal and test cleanup. |
| `CRAWL4AI_ESTIMATE_COST` | Estimate Cost | Estimate the minimum and maximum Crawl4AI credits for a proposed API request without running that request. |
| `CRAWL4AI_EXTRACT_PRICES` | Extract Prices | Read pricing pages for up to 20 site-and-country combinations and return documented plans, billing periods, add-ons, discounts, trials, and source quotes. |
| `CRAWL4AI_EXTRACT_STRUCTURED_DATA` | Extract Structured Data | Extract structured records from exactly one URL or supplied content using an instruction, a JSON Schema, or both. |
| `CRAWL4AI_FORGET_EMAIL_VERIFICATION` | Forget Email Verification | Request permanent removal of Crawl4AI's stored verification answer for one email. This destructive privacy operation is documented as returning 204, but the provider may currently reject it with HTTP 405. |
| `CRAWL4AI_GET_BILLING_BALANCE` | Get Billing Balance | Return the connected account's current free-credit balance, allowance, plan, tier, card status, spending, and top-up balance. Ledger values can trail the exact response-header balance by about one minute. |
| `CRAWL4AI_GET_COMPANY` | Get Lead Company | Get one private-preview Lead company by domain, LinkedIn page, or an unambiguous name optionally narrowed by country. |
| `CRAWL4AI_GET_EMAIL_VERIFICATION_JOB` | Get Email Verification Job | Get later results for a batch email verification job, including background retries. |
| `CRAWL4AI_GET_MONITOR` | Get Monitor | Return one page-change monitor's configuration, health, baseline, cost, and notification settings. |
| `CRAWL4AI_GET_MONITOR_VERSION` | Get Monitor Version | Download the page exactly as Crawl4AI captured it for monitor evidence, returned as a Markdown text file. |
| `CRAWL4AI_GET_PERSON` | Get Lead Person | Get one private-preview Lead person and held contact data by one documented lookup. |
| `CRAWL4AI_GET_PERSON_EMAIL` | Get Lead Person Email | Get private-preview personal contact data: one person's checked email by one documented lookup. Lead may construct and verify a likely address from the company pattern before answering. |
| `CRAWL4AI_GET_PERSON_PHONE` | Get Lead Person Phone | Get private-preview personal contact data: a person's direct and company phones by one documented lookup. |
| `CRAWL4AI_GET_PRICE_HISTORY` | Get Price History | Return up to 1,000 observed price changes for one domain and optional country, newest first. |
| `CRAWL4AI_GET_PRICES_AND_PLANS` | Get Prices and Plans | Return Crawl4AI's current credit prices, effort levels, plans, allowances, and request limits from the authoritative live catalog. |
| `CRAWL4AI_GET_RECIPE_HEALTH` | Get Recipe Health | Get the latest regional health check and release stage for every recipe, keyed by its live catalog name. |
| `CRAWL4AI_GET_RECIPE_JOB` | Get Recipe Job | Get a recipe job's status, progress counts, pause reason, and usage. |
| `CRAWL4AI_GET_RECIPE_JOB_RESULTS` | Get Recipe Job Results | Download a recipe job's result rows as an NDJSON file. Each line is one dynamic recipe output row with a sequence number. |
| `CRAWL4AI_GET_SCRAPE_JOB` | Get Scrape Job | Return a scrape job's current status, URL counts, and credit usage, with optional per-URL detail. |
| `CRAWL4AI_GET_SCRAPE_JOB_RESULTS` | Get Scrape Job Results | Download one page of up to 500 scrape job results as NDJSON; pass the next result offset explicitly to retrieve another page. |
| `CRAWL4AI_GET_SERP` | Get SERP | Return structured Google rankings through Crawl4AI's GET transport, including source selection and the fixed device, engine, and result-type controls. Use SEARCH_SERP for the streamlined POST request. |
| `CRAWL4AI_GET_SITE_CRAWL_JOB` | Get Site Crawl Job | Read a site crawl job's state, progress counts, completion deadline, usage, and pause reason when applicable. |
| `CRAWL4AI_GET_SITE_CRAWL_RESULTS` | Get Site Crawl Results | Download one page of up to 500 site crawl result rows as NDJSON, optionally continuing after a previously returned sequence number. |
| `CRAWL4AI_LIST_BILLING_TOPUPS` | List Billing Topups | List account payments and credit additions newest first, including receipt URLs, one cursor-controlled page at a time. |
| `CRAWL4AI_LIST_LEAD_INDUSTRIES` | List Lead Industries | Return documented LinkedIn industry names or names closest to query words. |
| `CRAWL4AI_LIST_MONITOR_EVENTS` | List Monitor Events | List one monitor's change and health timeline oldest first, including facts and links to captured evidence versions. This does not mark the feed read. |
| `CRAWL4AI_LIST_MONITORS` | List Monitors | List the connected account's page-change monitors, optionally filtered by agent label. |
| `CRAWL4AI_LIST_RECIPES` | List Recipes | List the live ready-made scraper catalog, including each recipe's dynamic input schema, output fields, cost, stage, and health. |
| `CRAWL4AI_PURCHASE_EXTRA_CREDITS` | Purchase Extra Credits | Immediately charge the saved card for the connected account's paid Crawl4AI plan and add non-expiring extra credits. This is a real, irreversible card charge, not a checkout flow. Call only after the user explicitly approves the exact US-dollar amount and set confirm_charge=true. |
| `CRAWL4AI_RETRY_SCRAPE_JOB` | Retry Scrape Job | Re-run only the failed URLs in an existing scrape job. |
| `CRAWL4AI_RUN_RECIPE` | Run Recipe | Run one synchronous catalog recipe with inputs that match its current schema. Job-only recipes and catalog inputs marked secret are rejected. |
| `CRAWL4AI_SCAN_SITE_URLS` | Scan Site URLs | Discover a site's known public page URLs from sitemaps, its home page, Common Crawl, and optionally the Wayback Machine without crawling each URL. |
| `CRAWL4AI_SCRAPE_PAGE` | Scrape Page | Fetch one public web page and return its Markdown, HTML, and optional parsed links, media, metadata, or tables. |
| `CRAWL4AI_SCRAPE_PAGES` | Scrape Pages | Scrape up to 50 public URLs with shared options and return a downloadable NDJSON file of completion rows followed by a final summary row. |
| `CRAWL4AI_SEARCH_COMPANIES` | Search Lead Companies | Search private-preview Lead companies by documented company filters and optional description-ranking words, one provider-cursor page at a time. |
| `CRAWL4AI_SEARCH_PEOPLE` | Search Lead People | Search private-preview Lead people by documented company and person filters. Results can contain personal contact data; people who exercised removal are excluded permanently. Returns one provider-cursor page at a time. |
| `CRAWL4AI_SEARCH_SERP` | Search SERP | Return structured Google result ranks and SERP blocks for a localized query and explicit result page. |
| `CRAWL4AI_SEARCH_WEB` | Search Web | Search the web for ranked, cleaned results, optionally including related questions, queries, news, videos, and discussions. |
| `CRAWL4AI_SUBMIT_RECIPE_JOB` | Submit Recipe Job | Submit an asynchronous run for a job-capable recipe using its dynamic live catalog inputs. For execution-log safety, catalog inputs marked secret are intentionally unsupported and rejected before submission. |
| `CRAWL4AI_SUBMIT_SCRAPE_JOB` | Submit Scrape Job | Submit up to 10,000 public URLs for asynchronous scraping and return a job identifier for status polling and paged result retrieval. |
| `CRAWL4AI_SUBMIT_SITE_CRAWL` | Submit Site Crawl | Start a background crawl of public pages within one site and return the job identifier used to inspect its status and results. |
| `CRAWL4AI_UPDATE_MONITOR` | Update Monitor | Change a monitor's watch rule, schedule, notifications, schema, name, or paused state. At least one change must be supplied. |
| `CRAWL4AI_VERIFY_EMAIL` | Verify Email | Check whether one email can receive mail. Crawl4AI performs an SMTP conversation but sends no mail; stored answers may be reused for 30 days. |
| `CRAWL4AI_VERIFY_EMAIL_DOMAIN` | Verify Email Domain | Check one domain's mail acceptance, provider, gateway, catch-all, SPF, and DMARC results. |
| `CRAWL4AI_VERIFY_EMAILS` | Verify Emails | Check one to 1,000 email addresses without sending mail. Greylisted addresses return unknown initially and are retried in the background. |

## Supported Triggers

None listed.

## Installation and MCP Setup

### Path 1: SDK Installation

#### Path 1, Step 1: Install Composio

Install the Composio SDK, the OpenAI provider, and the OpenAI SDK
```python
pip install composio composio_openai openai
```

```typescript
npm install @composio/core @composio/openai openai
```

#### Path 1, Step 2: Create a Composio session

Initialize Composio with the OpenAI Responses provider and create a session scoped to Crawl4AI
```python
import json
from openai import OpenAI
from composio import Composio
from composio_openai import OpenAIResponsesProvider

composio = Composio(provider=OpenAIResponsesProvider())
client = OpenAI()

session = composio.create(user_id="your-user-id", toolkits=["crawl4ai"])
tools = session.tools()
```

```typescript
import OpenAI from "openai";
import { Composio } from "@composio/core";
import { OpenAIResponsesProvider } from "@composio/openai";

const composio = new Composio({ provider: new OpenAIResponsesProvider() });
const client = new OpenAI();

const session = await composio.create("your-user-id", { toolkits: ["crawl4ai"] });
const tools = await session.tools();
```

#### Path 1, Step 3: Run Crawl4AI tools with your agent

Send a request, execute the Crawl4AI tool calls through the session, and print the final answer
```python
response = client.responses.create(
    model="gpt-5.6-sol",
    tools=tools,
    input=[{"role": "user", "content": "Search the web for the latest pricing pages from three competing project management tools and summarize their plans."}],
)

while True:
    tool_calls = [o for o in response.output if o.type == "function_call"]
    if not tool_calls:
        break
    results = composio.provider.handle_tool_calls(response=response, session=session)
    response = client.responses.create(
        model="gpt-5.6-sol",
        tools=tools,
        previous_response_id=response.id,
        input=[
            {"type": "function_call_output", "call_id": call.call_id, "output": json.dumps(results[i])}
            for i, call in enumerate(tool_calls)
        ],
    )

print(response.output_text)
```

```typescript
let response = await client.responses.create({
  model: "gpt-5.6-sol",
  tools,
  input: [{ role: "user", content: "Search the web for the latest pricing pages from three competing project management tools and summarize their plans." }],
});

while (response.output.some((o) => o.type === "function_call")) {
  const outputs = await composio.provider.handleToolCalls(session, response.output);
  response = await client.responses.create({
    model: "gpt-5.6-sol",
    tools,
    previous_response_id: response.id,
    input: outputs,
  });
}

console.log(response.output_text);
```

### Path 2: MCP Server Setup

#### Path 2, Step 1: Install Composio

Install the Composio SDK and your agent framework
```python
pip install composio claude-agent-sdk
```

```typescript
npm install @composio/core ai @ai-sdk/mcp @ai-sdk/openai
```

#### Path 2, Step 2: Create a session with MCP enabled

Create a session scoped to Crawl4AI and read its MCP URL and headers
```python
from composio import Composio

composio = Composio()
session = composio.create(user_id="your-user-id", toolkits=["crawl4ai"], mcp=True)
```

```typescript
import { Composio } from "@composio/core";

const composio = new Composio();
const { mcp } = await composio.create("your-user-id", {
  toolkits: ["crawl4ai"],
  mcp: true,
});
```

#### Path 2, Step 3: Connect your agent to the MCP server

Pass the session's MCP URL and headers to your agent and run a Crawl4AI request
```python
import asyncio
from claude_agent_sdk import ClaudeSDKClient, ClaudeAgentOptions, AssistantMessage, TextBlock

options = ClaudeAgentOptions(
    mcp_servers={
        "composio": {
            "type": "http",
            "url": session.mcp.url,
            "headers": session.mcp.headers,
        }
    },
    system_prompt="You are a helpful assistant with access to Crawl4AI tools.",
    tools=[],
    allowed_tools=["mcp__composio__*"],
)

async def main():
    async with ClaudeSDKClient(options=options) as client:
        await client.query("Search the web for the latest pricing pages from three competing project management tools and summarize their plans.")
        async for message in client.receive_response():
            if isinstance(message, AssistantMessage):
                for block in message.content:
                    if isinstance(block, TextBlock):
                        print(block.text)

asyncio.run(main())
```

```typescript
import { createMCPClient } from "@ai-sdk/mcp";
import { openai } from "@ai-sdk/openai";
import { generateText, stepCountIs } from "ai";

const client = await createMCPClient({
  transport: { type: "http", url: mcp.url, headers: mcp.headers },
});

const { text } = await generateText({
  model: openai("gpt-5.6-sol"),
  tools: await client.tools(),
  prompt: "Search the web for the latest pricing pages from three competing project management tools and summarize their plans.",
  stopWhen: stepCountIs(10),
});

console.log(text);
await client.close();
```

## Why Use Composio?

### 1. AI Native Crawl4AI Integration

- Supports both Crawl4AI MCP and direct API based integrations
- Structured, LLM-friendly schemas for reliable web scraping and extraction
- Rich coverage for search, crawling, monitoring, lead intelligence, and email verification

### 2. Managed Auth

- Securely stores and uses your Crawl4AI API key without exposing it to agents
- Central place to manage, scope, and revoke Crawl4AI access
- Per user and per environment credentials instead of hard-coded keys

### 3. Agent Optimized Design

- Tools are tuned using real error and success rates to improve reliability over time
- Structured outputs help agents work with extracted page content and search results
- Comprehensive execution logs show what ran, when, and on whose behalf

### 4. Enterprise Grade Security

- Fine-grained RBAC so you control which agents and users can access Crawl4AI
- Scoped, least privilege access to Crawl4AI resources
- Full audit trail of agent actions to support review and compliance

## Use Crawl4AI with any AI Agent Framework

Choose a framework you want to connect Crawl4AI with:

- [ChatGPT](https://composio.dev/toolkits/crawl4ai/framework/chatgpt)
- [Claude Cowork](https://composio.dev/toolkits/crawl4ai/framework/claude-cowork)
- [Hermes](https://composio.dev/toolkits/crawl4ai/framework/hermes-agent)
- [Atomic Agent](https://composio.dev/toolkits/crawl4ai/framework/atomic-agent)

## Related Toolkits

- [Supabase](https://composio.dev/toolkits/supabase) - Supabase is an open-source backend platform offering scalable Postgres databases, authentication, storage, and real-time APIs. It lets developers build modern apps without managing infrastructure.
- [Codeinterpreter](https://composio.dev/toolkits/codeinterpreter) - Codeinterpreter is a Python-based coding environment with built-in data analysis and visualization. It lets you instantly run scripts, plot results, and prototype solutions inside supported platforms.
- [GitHub](https://composio.dev/toolkits/github) - GitHub is a code hosting platform for version control and collaborative software development. It streamlines project management, code review, and team workflows in one place.
- [1password](https://composio.dev/toolkits/_1password) - 1Password is a password manager and digital vault for storing logins, secrets, notes, and secure documents. It helps individuals and teams protect credentials, share access safely, and reduce password risk.
- [Ably](https://composio.dev/toolkits/ably) - Ably is a real-time messaging platform for live chat and data sync in modern apps. It offers global scale and rock-solid reliability for seamless, instant experiences.
- [Abuselpdb](https://composio.dev/toolkits/abuselpdb) - Abuselpdb is a central database for reporting and checking IPs linked to malicious online activity. Use it to quickly identify and report suspicious or abusive IP addresses.
- [Alchemy](https://composio.dev/toolkits/alchemy) - Alchemy is a blockchain development platform offering APIs and tools for Ethereum apps. It simplifies building and scaling Web3 projects with robust infrastructure.
- [Algolia](https://composio.dev/toolkits/algolia) - Algolia is a hosted search API that powers lightning-fast, relevant search experiences for web and mobile apps. It helps developers deliver instant, typo-tolerant, and scalable search without complex infrastructure.
- [Anakin](https://composio.dev/toolkits/anakin_io) - Anakin is a web automation platform offering scraping, crawling, search, and browser automation. Use it to extract structured data, automate site tasks, and monitor web changes reliably.
- [Anchor browser](https://composio.dev/toolkits/anchor_browser) - Anchor browser is a developer platform for AI-powered web automation. It transforms complex browser actions into easy API endpoints for streamlined web interaction.
- [AnyAPI](https://composio.dev/toolkits/anyapi) - AnyAPI is a marketplace of utility APIs for validation, currency conversion, holidays, documents, and web data. It brings common application workflows together through one practical API platform.
- [Apiflash](https://composio.dev/toolkits/apiflash) - Apiflash is a website screenshot API for programmatically capturing web pages. It delivers high-quality screenshots on demand for automation, monitoring, or reporting.
- [Apiverve](https://composio.dev/toolkits/apiverve) - Apiverve delivers a suite of powerful APIs that simplify integration for developers. It's designed for reliability and scalability so you can build faster, smarter applications without the integration headache.
- [Appcircle](https://composio.dev/toolkits/appcircle) - Appcircle is an enterprise-grade mobile CI/CD platform for building, testing, and publishing mobile apps. It streamlines mobile DevOps so teams ship faster and with more confidence.
- [Appdrag](https://composio.dev/toolkits/appdrag) - Appdrag is a cloud platform for building websites, APIs, and databases with drag-and-drop tools and code editing. It accelerates development and iteration by combining hosting, database management, and low-code features in one place.
- [Appveyor](https://composio.dev/toolkits/appveyor) - AppVeyor is a cloud-based continuous integration service for building, testing, and deploying applications. It helps developers automate and streamline their software delivery pipelines.
- [Authyo](https://composio.dev/toolkits/authyo) - Authyo is a REST API service for passwordless authentication, OTP verification, session management, and transactional notifications. Use it to add secure login flows and user verification without building auth infrastructure from scratch.
- [AWS Marketplace MCP](https://composio.dev/toolkits/aws_marketplace_mcp) - AWS Marketplace MCP provides MCP access to AWS Marketplace's cloud software, data, and services catalog. Use it to discover, compare, and evaluate 30K+ AWS Marketplace listings faster.
- [Azure Monitor Activity Log](https://composio.dev/toolkits/azure_monitor_activity_log) - Azure Monitor Activity Log is Azure's service that records management and control-plane events for a subscription. Use it to audit changes, troubleshoot issues, and track resource operations across your Azure resources.
- [Backendless](https://composio.dev/toolkits/backendless) - Backendless is a backend-as-a-service platform for mobile and web apps, offering database, file storage, user authentication, and APIs. It helps developers ship scalable applications faster without managing server infrastructure.

## Frequently Asked Questions

### Do I need my own developer credentials to use Crawl4AI with Composio?

Yes, Crawl4AI requires you to configure your own API key. Once set up, Composio handles secure credential storage and API request handling for you.

### Can I use multiple toolkits together?

Yes! Composio's Tool Router enables agents to use multiple toolkits. [Learn more](https://docs.composio.dev/tool-router/overview).

### Is Composio secure?

Composio is SOC 2 and ISO 27001 compliant with all data encrypted in transit and at rest. [Learn more](https://trust.composio.dev).

### What if the API changes?

Composio maintains and updates all toolkit integrations automatically, so your agents always work with the latest API versions.

---
[See all toolkits](https://composio.dev/toolkits) · [Composio docs](https://docs.composio.dev/llms.txt)
