# Webscraper io

```json
{
  "name": "Webscraper io",
  "slug": "webscraper_io",
  "url": "https://composio.dev/toolkits/webscraper_io",
  "markdown_url": "https://composio.dev/toolkits/webscraper_io.md",
  "logo_url": "https://logos.composio.dev/api/webscraper_io",
  "categories": [
    "productivity & project management"
  ],
  "is_composio_managed": false,
  "updated_at": "2026-03-29T06:55:06.134Z"
}
```

![Webscraper io logo](https://logos.composio.dev/api/webscraper_io)

## Description

Securely connect your AI agents and chatbots (Claude, ChatGPT, Cursor, etc) with Webscraper io MCP or direct API to run scraping jobs, retrieve extracted data, automate data collection, and monitor web sources through natural language.

## Summary

Webscraper io is a cloud-based web scraping platform for easy data extraction. Get structured web data quickly without complex coding or infrastructure.

## Categories

- productivity & project management

## Toolkit Details

- Tools: 10

## Images

- Logo: https://logos.composio.dev/api/webscraper_io

## Authentication

- **Api Key**
  - Type: `api_key`
  - Description: Api Key authentication for Webscraper io.
  - Setup:
    - Configure Api Key credentials for Webscraper io.
    - Use the credentials when creating an auth config in Composio.

## Suggested Prompts

- Run scraping job for latest news headlines
- Fetch extracted product prices from example.com
- Get all completed jobs and their results

## Supported Tools

| Tool slug | Name | Description |
|---|---|---|
| `WEBSCRAPER_IO_CREATE_SITEMAP` | Create Sitemap | Tool to create a new sitemap configuration for web scraping. Use when you need to define a new scraping structure with start URLs and selector rules for data extraction from a website. |
| `WEBSCRAPER_IO_DELETE_SITEMAP` | Delete Sitemap | Tool to permanently delete a sitemap configuration from Web Scraper Cloud account. Use when you need to remove a sitemap that is no longer needed. |
| `WEBSCRAPER_IO_DISABLE_SITEMAP_SCHEDULER` | Disable Sitemap Scheduler | Tool to disable automatic scheduling for a sitemap. Use when you need to stop automated scraping jobs from running on a schedule. |
| `WEBSCRAPER_IO_ENABLE_SITEMAP_SCHEDULER` | Enable Sitemap Scheduler | Tool to enable and configure automatic scheduling for sitemap scraping jobs. Use when you need to automate scraping jobs to run at specific times using cron expressions with customizable request intervals, page load delays, driver types, and proxy settings. |
| `WEBSCRAPER_IO_GET_ACCOUNT_INFO` | Get Account Info | Tool to retrieve account information including email and page credits. Use when you need to check account details or available credits. |
| `WEBSCRAPER_IO_GET_SCRAPING_JOBS` | Get Scraping Jobs | Tool to retrieve all scraping jobs for the account with optional filtering and pagination. Use when you need to list scraping jobs, check job status, or filter jobs by sitemap or tag. |
| `WEBSCRAPER_IO_GET_SITEMAP` | Get Sitemap | Tool to retrieve a specific sitemap configuration by ID. Use when you need to inspect or reference an existing sitemap's configuration. |
| `WEBSCRAPER_IO_GET_SITEMAPS` | Get Sitemaps | Tool to retrieve all sitemaps for the authenticated account with pagination support. Use when you need to list available sitemaps or filter them by tag. Supports optional pagination via page parameter and filtering by tag name. |
| `WEBSCRAPER_IO_GET_SITEMAP_SCHEDULER` | Get Sitemap Scheduler | Tool to retrieve scheduler configuration for a sitemap. Use when you need to check scheduling settings including cron configuration and proxy settings. |
| `WEBSCRAPER_IO_UPDATE_SITEMAP` | Update Sitemap | Tool to update an existing sitemap configuration including structure, URLs, and selectors. Use when you need to modify sitemap settings. |

## Supported Triggers

None listed.

## Installation and MCP Setup

### Path 1: SDK Installation

#### Path 1, Step 1: Install Composio

Install the Composio SDK
```python
pip install composio_openai
```

```typescript
npm install @composio/openai
```

#### Path 1, Step 2: Initialize Composio and Create Tool Router Session

Import and initialize Composio client, then create a Tool Router session
```python
from openai import OpenAI
from composio import Composio
from composio_openai import OpenAIResponsesProvider

composio = Composio(provider=OpenAIResponsesProvider())
openai = OpenAI()
session = composio.create(user_id='your-user-id')
```

```typescript
import OpenAI from 'openai';
import { Composio } from '@composio/core';
import { OpenAIResponsesProvider } from '@composio/openai';

const composio = new Composio({
  provider: new OpenAIResponsesProvider(),
});
const openai = new OpenAI({});
const session = await composio.create('your-user-id');
```

#### Path 1, Step 3: Execute Webscraper io Tools via Tool Router with Your Agent

Get tools from Tool Router session and execute Webscraper io actions with your Agent
```python
tools = session.tools
response = openai.responses.create(
  model='gpt-4.1',
  tools=tools,
  input=[{
    'role': 'user',
    'content': 'Start a new scraping job for example.com/products and return the results as a table.'
  }]
)
result = composio.provider.handle_tool_calls(
  response=response,
  user_id='your-user-id'
)
print(result)
```

```typescript
const tools = session.tools;
const response = await openai.responses.create({
  model: 'gpt-4.1',
  tools: tools,
  input: [{
    role: 'user',
    content: 'Start a new scraping job for example.com/products and return the results as a table.'
  }],
});
const result = await composio.provider.handleToolCalls(
  'your-user-id',
  response.output
);
console.log(result);
```

### Path 2: MCP Server Setup

#### Path 2, Step 1: Install Composio

Install the Composio SDK for Python or TypeScript
```python
pip install composio claude-agent-sdk
```

```typescript
npm install @composio/core ai @ai-sdk/openai @ai-sdk/mcp
```

#### Path 2, Step 2: Initialize Client and Create Tool Router Session

Import and initialize the Composio client, then create a Tool Router session for Webscraper io
```python
from composio import Composio
from claude_agent_sdk import ClaudeSDKClient, ClaudeAgentOptions

composio = Composio(api_key='your-composio-api-key')
session = composio.create(user_id='your-user-id')
url = session.mcp.url
```

```typescript
import { Composio } from '@composio/core';

const composio = new Composio({ apiKey: 'your-api-key' });
const session = await composio.create('your-user-id');
console.log(`Tool Router session created: ${session.mcp.url}`);
```

#### Path 2, Step 3: Connect to AI Agent

Use the MCP server with your AI agent (Anthropic Claude or Mastra)
```python
import asyncio

options = ClaudeAgentOptions(
    permission_mode='bypassPermissions',
    mcp_servers={
        'tool_router': {
            'type': 'http',
            'url': url,
            'headers': {
                'x-api-key': 'your-composio-api-key'
            }
        }
    },
    system_prompt='You are a helpful assistant with access to Webscraper io tools.',
    max_turns=10
)

async def main():
    async with ClaudeSDKClient(options=options) as client:
        await client.query('Start a new scraping job for example.com/products and return the results as a table.')
        async for message in client.receive_response():
            if hasattr(message, 'content'):
                for block in message.content:
                    if hasattr(block, 'text'):
                        print(block.text)

asyncio.run(main())
```

```typescript
import { openai } from '@ai-sdk/openai';
import { experimental_createMCPClient as createMCPClient } from '@ai-sdk/mcp';
import { generateText } from 'ai';

const client = await createMCPClient({
  transport: {
    type: 'http',
    url: session.mcp.url,
    headers: {
      'x-api-key': 'your-composio-api-key',
    },
  },
});

const tools = await client.tools();
const { text } = await generateText({
  model: openai('gpt-4o'),
  tools,
  messages: [{
    role: 'user',
    content: 'Start a new scraping job for example.com/products and return the results as a table.'
  }],
  maxSteps: 5,
});

console.log(`Agent: ${text}`);
```

## Why Use Composio?

### 1. AI Native Webscraper io Integration

- Supports both Webscraper io MCP and direct API based integrations
- Structured, LLM-friendly schemas for reliable tool execution
- Rich coverage for reading, writing, and querying your Webscraper io data

### 2. Managed Auth

- Built-in OAuth handling with automatic token refresh and rotation
- Central place to manage, scope, and revoke Webscraper io access
- Per user and per environment credentials instead of hard-coded keys

### 3. Agent Optimized Design

- Tools are tuned using real error and success rates to improve reliability over time
- Comprehensive execution logs so you always know what ran, when, and on whose behalf

### 4. Enterprise Grade Security

- Fine-grained RBAC so you control which agents and users can access Webscraper io
- Scoped, least privilege access to Webscraper io resources
- Full audit trail of agent actions to support review and compliance

## Use Webscraper io with any AI Agent Framework

Choose a framework you want to connect Webscraper io with:

- [OpenAI Agents SDK](https://composio.dev/toolkits/webscraper_io/framework/open-ai-agents-sdk)
- [Claude Agent SDK](https://composio.dev/toolkits/webscraper_io/framework/claude-agents-sdk)
- [Claude Code](https://composio.dev/toolkits/webscraper_io/framework/claude-code)
- [Claude Cowork](https://composio.dev/toolkits/webscraper_io/framework/claude-cowork)
- [Codex](https://composio.dev/toolkits/webscraper_io/framework/codex)
- [OpenClaw](https://composio.dev/toolkits/webscraper_io/framework/openclaw)
- [Hermes](https://composio.dev/toolkits/webscraper_io/framework/hermes-agent)
- [Google ADK](https://composio.dev/toolkits/webscraper_io/framework/google-adk)
- [LangChain](https://composio.dev/toolkits/webscraper_io/framework/langchain)
- [Vercel AI SDK](https://composio.dev/toolkits/webscraper_io/framework/ai-sdk)
- [Mastra AI](https://composio.dev/toolkits/webscraper_io/framework/mastra-ai)
- [LlamaIndex](https://composio.dev/toolkits/webscraper_io/framework/llama-index)
- [CrewAI](https://composio.dev/toolkits/webscraper_io/framework/crew-ai)
- [Pydantic AI](https://composio.dev/toolkits/webscraper_io/framework/pydantic-ai)
- [AutoGen](https://composio.dev/toolkits/webscraper_io/framework/autogen)

## Related Toolkits

- [Google Sheets](https://composio.dev/toolkits/googlesheets) - Google Sheets is a cloud-based spreadsheet tool for real-time collaboration and data analysis. It lets teams work together from anywhere, updating information instantly.
- [Notion](https://composio.dev/toolkits/notion) - Notion is a collaborative workspace for notes, docs, wikis, and tasks. It streamlines team knowledge, project tracking, and workflow customization in one place.
- [Airtable](https://composio.dev/toolkits/airtable) - Airtable combines the flexibility of spreadsheets with the power of a database for easy project and data management. Teams use Airtable to organize, track, and collaborate with custom views and automations.
- [Asana](https://composio.dev/toolkits/asana) - Asana is a collaborative work management platform for teams to organize and track projects. It streamlines teamwork, boosts productivity, and keeps everyone aligned on goals.
- [Google Tasks](https://composio.dev/toolkits/googletasks) - Google Tasks is a to-do list and task management tool integrated into Gmail and Google Calendar. It helps you organize, track, and complete tasks across your Google ecosystem.
- [Linear](https://composio.dev/toolkits/linear) - Linear is a modern issue tracking and project planning tool for fast-moving teams. It helps streamline workflows, organize projects, and boost productivity.
- [Jira](https://composio.dev/toolkits/jira) - Jira is Atlassian’s platform for bug tracking, issue tracking, and agile project management. It helps teams organize work, prioritize tasks, and deliver projects efficiently.
- [Clickup](https://composio.dev/toolkits/clickup) - ClickUp is an all-in-one productivity platform for managing tasks, docs, goals, and team collaboration. It streamlines project workflows so teams can work smarter and stay organized in one place.
- [Monday](https://composio.dev/toolkits/monday) - Monday.com is a customizable work management platform for project planning and collaboration. It helps teams organize tasks, automate workflows, and track progress in real time.
- [Addressfinder](https://composio.dev/toolkits/addressfinder) - Addressfinder is a data quality platform for verifying addresses, emails, and phone numbers. It helps you ensure accurate customer and contact data every time.
- [Agiled](https://composio.dev/toolkits/agiled) - Agiled is an all-in-one business management platform for CRM, projects, and finance. It helps you streamline workflows, consolidate client data, and manage business processes in one place.
- [Ascora](https://composio.dev/toolkits/ascora) - Ascora is a cloud-based field service management platform for service businesses. It streamlines scheduling, invoicing, and customer operations in one place.
- [Basecamp](https://composio.dev/toolkits/basecamp) - Basecamp is a project management and team collaboration tool by 37signals. It helps teams organize tasks, share files, and communicate efficiently in one place.
- [Beeminder](https://composio.dev/toolkits/beeminder) - Beeminder is an online goal-tracking platform that uses monetary pledges to keep you motivated. Stay accountable and hit your targets with real financial incentives.
- [Boxhero](https://composio.dev/toolkits/boxhero) - Boxhero is a cloud-based inventory management platform for SMBs, offering real-time updates, barcode scanning, and team collaboration. It helps businesses streamline stock tracking and analytics for smarter inventory decisions.
- [Breathe HR](https://composio.dev/toolkits/breathehr) - Breathe HR is cloud-based HR software for SMEs to manage employee data, absences, and performance. It simplifies HR admin, making it easy to keep employee records accurate and up to date.
- [Breeze](https://composio.dev/toolkits/breeze) - Breeze is a project management platform designed to help teams plan, track, and collaborate on projects. It streamlines workflows and keeps everyone on the same page.
- [Bugherd](https://composio.dev/toolkits/bugherd) - Bugherd is a visual feedback and bug tracking tool for websites. It helps teams and clients report website issues directly on live sites for faster fixes.
- [Canny](https://composio.dev/toolkits/canny) - Canny is a platform for managing customer feedback and feature requests. It helps teams prioritize product decisions based on real user insights.
- [Chmeetings](https://composio.dev/toolkits/chmeetings) - Chmeetings is a church management platform for events, members, donations, and volunteers. It streamlines church operations and improves community engagement.

## Frequently Asked Questions

### Do I need my own developer credentials to use Webscraper io with Composio?

Yes, Webscraper io requires you to configure your own API key. Once set up, Composio handles secure credential storage and API request handling for you.

### Can I use multiple toolkits together?

Yes! Composio's Tool Router enables agents to use multiple toolkits. [Learn more](https://docs.composio.dev/tool-router/overview).

### Is Composio secure?

Composio is SOC 2 and ISO 27001 compliant with all data encrypted in transit and at rest. [Learn more](https://trust.composio.dev).

### What if the API changes?

Composio maintains and updates all toolkit integrations automatically, so your agents always work with the latest API versions.

---
[See all toolkits](https://composio.dev/toolkits) · [Composio docs](https://docs.composio.dev/llms.txt)