Crawlbase MCP for AI Agents

Securely connect your AI agents and chatbots (Claude, ChatGPT, Cursor, etc) with Crawlbase MCP or direct API to crawl web pages, manage crawler queues, retrieve stored crawl results, and monitor account usage through natural language.

Crawlbase logoCrawlbase
Api Key

Crawlbase is a web crawling API platform for fetching pages, managing crawler queues, storing crawl results, and tracking usage. It helps developers collect web data at scale without building and maintaining crawler infrastructure.

8 Tools

Try Crawlbase now

Type what you want done — sign in and watch it run live in the Tool Router playground.

TOOL ROUTER PLAYGROUND
Crawlbase
Try asking
TOOLS

Supported Tools

Every Crawlbase action and event your agent gets out of the box.

Crawl URL

Fetch a public HTTP or HTTPS URL through Crawlbase using the connected Normal token and return its content with separate target and Crawlbase statuses.

Delete Stored Pages

Irreversibly delete explicitly selected pages from Crawlbase Cloud Storage by request ID and return each page's deletion outcome.

Generate User Agents

Generate one to ten realistic randomized User-Agent strings for a selected device class.

Get Crawling Usage

Return current Crawling API usage and optionally previous-month statistics for the connected Crawlbase account.

Get Storage Count

Return the number of pages currently held in Crawlbase Cloud Storage.

Get Stored Page

Retrieve the latest Crawlbase Cloud Storage page by exactly one request ID or original URL.

Get Stored Pages

Retrieve up to 100 Crawlbase Cloud Storage pages by request ID without deleting them.

List Stored Page IDs

Return one page of Crawlbase Cloud Storage request IDs and a short-lived continuation cursor.

SETUP GUIDE

Connect Crawlbase MCP Tool with your Agent

1

Install Composio

typescript
npm install @composio/core ai @ai-sdk/openai @ai-sdk/mcp
Install the Composio SDK for Python or TypeScript
2

Initialize Client and Create Tool Router Session

typescript
import { Composio } from '@composio/core';

const composio = new Composio({ apiKey: 'your-api-key' });
const session = await composio.create('your-user-id');
console.log(`Tool Router session created: ${session.mcp.url}`);
Import and initialize the Composio client, then create a Tool Router session for Crawlbase
3

Connect to AI Agent

typescript
import { openai } from '@ai-sdk/openai';
import { experimental_createMCPClient as createMCPClient } from '@ai-sdk/mcp';
import { generateText } from 'ai';

const client = await createMCPClient({
  transport: {
    type: 'http',
    url: session.mcp.url,
    headers: {
      'x-api-key': 'your-composio-api-key',
    },
  },
});

const tools = await client.tools();
const { text } = await generateText({
  model: openai('gpt-4o'),
  tools,
  messages: [{
    role: 'user',
    content: 'Crawl https://example.com and return the page title and main text'
  }],
  maxSteps: 5,
});

console.log(`Agent: ${text}`);
Use the MCP server with your AI agent (Anthropic Claude or Mastra)
SETUP GUIDE

Connect Crawlbase API Tool with your Agent

1

Install Composio

typescript
npm install @composio/openai
Install the Composio SDK
2

Initialize Composio and Create Tool Router Session

typescript
import OpenAI from 'openai';
import { Composio } from '@composio/core';
import { OpenAIResponsesProvider } from '@composio/openai';

const composio = new Composio({
  provider: new OpenAIResponsesProvider(),
});
const openai = new OpenAI({});
const session = await composio.create('your-user-id');
Import and initialize Composio client, then create a Tool Router session
3

Execute Crawlbase Tools via Tool Router with Your Agent

typescript
const tools = session.tools;
const response = await openai.responses.create({
  model: 'gpt-4.1',
  tools: tools,
  input: [{
    role: 'user',
    content: 'Crawl https://example.com and return the page title and main text'
  }],
});
const result = await composio.provider.handleToolCalls(
  'your-user-id',
  response.output
);
console.log(result);
Get tools from Tool Router session and execute Crawlbase actions with your Agent

Why Use Composio?

AI Native Crawlbase Integration

  • Supports both Crawlbase MCP and direct API based integrations
  • Structured, LLM-friendly schemas for reliable crawl requests and queue actions
  • Rich coverage for crawling pages, checking queues, reading stored results, and monitoring usage

Managed Auth

  • Securely store Crawlbase API keys without hard-coding them in your agent code
  • Central place to manage, scope, and revoke Crawlbase access
  • Per user and per environment credentials, so staging and production stay cleanly separated

Agent Optimized Design

  • Crawlbase tools are shaped so language models can pick the right crawl, queue, or result action
  • Tools are tuned using real error and success rates to improve reliability over time
  • Execution logs show what ran, when it ran, and which user triggered the crawl

Enterprise Grade Security

  • Fine-grained RBAC so you control which agents and users can access Crawlbase
  • Scoped, least privilege access for crawl requests, stored results, and usage data
  • Full audit trail of agent actions to support review and compliance
FRAMEWORKS

Use Crawlbase with any AI Agent Framework

Choose a Framework you want to connect Crawlbase with

FAQ

Frequently asked questions

Yes, Crawlbase requires you to configure your own API key credentials. Once set up, Composio handles secure credential storage and API request handling for you.

Yes! Composio's Tool Router enables agents to use multiple toolkits. Learn more.

Composio is SOC 2 and ISO 27001 compliant with all data encrypted in transit and at rest. Learn more.

Composio maintains and updates all toolkit integrations automatically, so your agents always work with the latest API versions.

Start with Crawlbase.It takes 30 seconds.

Managed auth, hosted MCP servers, and every Crawlbase tool your agent needs.Free to start.

Start building