# Scrapegraph ai

```json
{
  "name": "Scrapegraph ai",
  "slug": "scrapegraph_ai",
  "url": "https://composio.dev/toolkits/scrapegraph_ai",
  "markdown_url": "https://composio.dev/toolkits/scrapegraph_ai.md",
  "logo_url": "https://logos.composio.dev/api/scrapegraph_ai",
  "categories": [
    "data & analytics"
  ],
  "is_composio_managed": false,
  "updated_at": "2026-05-12T10:24:47.118Z"
}
```

![Scrapegraph ai logo](https://logos.composio.dev/api/scrapegraph_ai)

## Description

Securely connect your AI agents and chatbots (Claude, ChatGPT, Cursor, etc) with Scrapegraph ai MCP or direct API to scrape websites, extract structured tables, summarize online content, and automate data collection through natural language.

## Summary

Scrapegraph ai is an AI-powered web scraping API that lets you extract structured data from any website using natural language prompts. Easily turn web pages into ready-to-use data for your apps and agents.

## Categories

- data & analytics

## Toolkit Details

- Tools: 27

## Images

- Logo: https://logos.composio.dev/api/scrapegraph_ai

## Authentication

- **Api Key**
  - Type: `api_key`
  - Description: Api Key authentication for Scrapegraph Ai.
  - Setup:
    - Configure Api Key credentials for Scrapegraph Ai.
    - Use the credentials when creating an auth config in Composio.

## Suggested Prompts

- Extract product prices from Amazon search results
- Summarize latest news headlines from BBC homepage
- Convert Wikipedia article to markdown format
- Crawl top 10 blog posts from TechCrunch

## Supported Tools

| Tool slug | Name | Description |
|---|---|---|
| `SCRAPEGRAPH_AI_CONVERT_WEBPAGE_TO_MARKDOWN_V2` | Convert Webpage to Markdown (V2) | Tool to convert any webpage into clean, well-formatted Markdown with full parameter control. Use when you need advanced options like stealth mode, custom headers, or webhook notifications. Supports all Markdownify API parameters. |
| `SCRAPEGRAPH_AI_GENERATE_SCHEMA` | Generate Schema | Generate or modify a JSON schema based on a search query for structured data extraction. Use when you need a schema template for scraping specific data fields. |
| `SCRAPEGRAPH_AI_GET_AGENTIC_SCRAPER_HISTORY` | Get Agentic Scraper History | Retrieve paginated history of agentic scraper jobs. Use to view past scraping requests, their status, and results. |
| `SCRAPEGRAPH_AI_GET_CRAWLER_HISTORY` | Get Crawler History | Retrieve the history of crawler jobs for your account. Returns paginated list of past crawler requests with their status, results, and metadata. |
| `SCRAPEGRAPH_AI_GET_CREDITS` | Get Credits | Retrieve remaining and used credits for your ScrapeGraphAI account. Useful for checking credit availability before bulk scraping operations to avoid mid-run failures. |
| `SCRAPEGRAPH_AI_GET_ENDPOINT_SUGGESTIONS` | Get Endpoint Suggestions | Tool to get AI-powered suggestions for creating scraping endpoints. Use when you need to identify what data can be extracted from a website and how to structure the scraping logic. |
| `SCRAPEGRAPH_AI_GET_LIVE_SESSION_URL` | Get Live Session URL | Tool to get a URL for a live browser session. Use when you need to interact with a webpage in real-time through a controlled browser environment. |
| `SCRAPEGRAPH_AI_GET_MARKDOWNIFY_HISTORY` | Get Markdownify History | Tool to retrieve the history of markdownify webpage-to-Markdown conversion jobs. Use when you need to view past markdownify requests and their statuses. |
| `SCRAPEGRAPH_AI_GET_SCRAPE_HISTORY` | Get Scrape History | Retrieve the history of scrape jobs from your ScrapeGraphAI account. Use this to check the status of past scrapes, view results, and track credit usage. |
| `SCRAPEGRAPH_AI_GET_SEARCHSCRAPER_HISTORY` | Get Searchscraper History | Get the history of searchscraper jobs with pagination support. Use this to retrieve past searchscraper requests, their status, and results. |
| `SCRAPEGRAPH_AI_GET_SITEMAP_HISTORY` | Get Sitemap History | Tool to retrieve the history of sitemap extraction jobs. Use when you need to view past sitemap extraction requests, their status, and results. |
| `SCRAPEGRAPH_AI_GET_SMARTSCRAPER_HISTORY` | Get Smartscraper History | Tool to retrieve the history of smartscraper jobs. Use when you need to view past scraping requests and their results. |
| `SCRAPEGRAPH_AI_GET_USAGE_TIMELINE` | Get Usage Timeline | Tool to retrieve usage timeline statistics for your ScrapeGraphAI account. Use when you need to visualize or analyze service usage patterns over time. |
| `SCRAPEGRAPH_AI_GET_WEBHOOK_LOGS` | Get Webhook Logs | Tool to retrieve webhook delivery logs for a crawler job. Use when you need to check the status and history of webhook notifications sent for a specific crawler execution. |
| `SCRAPEGRAPH_AI_LIST_SCHEDULED_JOBS` | List Scheduled Jobs | Retrieve a paginated list of all scheduled scraping jobs for your account. Use this action to view and manage your scheduled jobs, including their configuration, cron schedules, and active status. Supports filtering by service type and active status. |
| `SCRAPEGRAPH_AI_MARKDOWNIFY_STATUS` | Markdownify Status | Check the status and retrieve results of a Markdownify webpage-to-Markdown conversion job. Use this action to poll for the status of an async Markdownify request started via SCRAPEGRAPH_AI_MARKDOWNIFY. Note: The ScrapeGraph AI API typically returns completed results synchronously, so this status endpoint is primarily useful for long-running conversions of large or complex webpages. |
| `SCRAPEGRAPH_AI_SAVE_ENDPOINT` | Save Endpoint Configuration | Tool to save custom scraping endpoint configurations to ScrapeGraphAI. Use when you need to create reusable scraping endpoints with specific parameters and extraction logic. |
| `SCRAPEGRAPH_AI_SEARCH_SCRAPER` | Search Scraper | Perform AI-powered web searches with structured, parsed results. Some sites block scrapers and return empty bodies; treat these as unrecoverable for that URL. JS-rendered pages may yield incomplete content. |
| `SCRAPEGRAPH_AI_SEARCH_SCRAPER_STATUS` | Check SearchScraper Status | Check the status and results of an asynchronous SearchScraper job. |
| `SCRAPEGRAPH_AI_SMART_CRAWLER_STATUS` | SmartCrawler Status | Check the status and retrieve results of a SmartCrawler web crawling job. Use this action to poll for completion and get the extracted content from a previously started SmartCrawler job. Returns the job status, crawled URLs, page content in markdown/HTML format, and LLM extraction results (if enabled). Implement a polling timeout (e.g., max retries or elapsed time cap) to avoid indefinite loops when waiting for long-running jobs. |
| `SCRAPEGRAPH_AI_SMART_SCRAPER_START` | Start Smart Scraper | Start AI-powered web scraping with natural language extraction prompts. When `wait` is false (default), returns a `request_id`; poll for results using SCRAPEGRAPH_AI_SMART_SCRAPER_STATUS. Check `error` and `job_status` fields in the response before using extracted data. |
| `SCRAPEGRAPH_AI_SMART_SCRAPER_STATUS` | SmartScraper Status | Check the status and retrieve results of a SmartScraper web scraping job. Use this action to poll for completion after starting a SmartScraper job with wait=false. The request_id is returned by the Start SmartScraper action. Typical workflow: 1. Start a scraping job with SCRAPEGRAPH_AI_SMART_SCRAPER_START (wait=false) 2. Use the returned request_id to check status with this action 3. Poll until status is 'completed' or 'failed' 4. When completed, the 'result' field contains the extracted data. When completed, also check the 'error' field before consuming 'result', as 'failed' status populates 'error' instead of 'result'. |
| `SCRAPEGRAPH_AI_START_SMART_CRAWLER` | Start Smart Crawler (Async) | Tool to start a multi-page web crawl using SmartCrawler for AI-powered data extraction. Use when you need to extract structured data from multiple pages of a website. Returns immediately with a task_id - use the status check action to monitor progress and retrieve results. |
| `SCRAPEGRAPH_AI_SUBMIT_FEEDBACK` | Submit Feedback | Submit feedback and ratings for completed ScrapeGraphAI requests. |
| `SCRAPEGRAPH_AI_SUBMIT_PRODUCT_FEEDBACK` | Submit Product Feedback | Submit product feedback for ScrapeGraphAI. Use to provide ratings, comments, suggestions, and other feedback about the product itself. |
| `SCRAPEGRAPH_AI_TOONIFY` | Convert JSON to TOON Format | Tool to convert JSON data to TOON (Token-Oriented Object Notation) format. Use when you need to reduce token usage for LLM processing while maintaining data structure. |
| `SCRAPEGRAPH_AI_VALIDATE_API_KEY` | Validate API Key | Validate your ScrapeGraphAI API key to ensure it is active and authorized. Use this action to check API key validity before making other API calls. |

## Supported Triggers

None listed.

## Installation and MCP Setup

### Path 1: SDK Installation

#### Path 1, Step 1: Install Composio

Install the Composio SDK
```python
pip install composio_openai
```

```typescript
npm install @composio/openai
```

#### Path 1, Step 2: Initialize Composio and Create Tool Router Session

Import and initialize Composio client, then create a Tool Router session
```python
from openai import OpenAI
from composio import Composio
from composio_openai import OpenAIResponsesProvider

composio = Composio(provider=OpenAIResponsesProvider())
openai = OpenAI()
session = composio.create(user_id='your-user-id')
```

```typescript
import OpenAI from 'openai';
import { Composio } from '@composio/core';
import { OpenAIResponsesProvider } from '@composio/openai';

const composio = new Composio({
  provider: new OpenAIResponsesProvider(),
});
const openai = new OpenAI({});
const session = await composio.create('your-user-id');
```

#### Path 1, Step 3: Execute Scrapegraph ai Tools via Tool Router with Your Agent

Get tools from Tool Router session and execute Scrapegraph ai actions with your Agent
```python
tools = session.tools
response = openai.responses.create(
  model='gpt-4.1',
  tools=tools,
  input=[{
    'role': 'user',
    'content': 'Extract product prices from the first page of https://www.example.com/shop'
  }]
)
result = composio.provider.handle_tool_calls(
  response=response,
  user_id='your-user-id'
)
print(result)
```

```typescript
const tools = session.tools;
const response = await openai.responses.create({
  model: 'gpt-4.1',
  tools: tools,
  input: [{
    role: 'user',
    content: 'Extract product prices from the first page of https://www.example.com/shop'
  }],
});
const result = await composio.provider.handleToolCalls(
  'your-user-id',
  response.output
);
console.log(result);
```

### Path 2: MCP Server Setup

#### Path 2, Step 1: Install Composio

Install the Composio SDK and Claude Agent SDK
```python
pip install composio claude-agent-sdk
```

```typescript
npm install @composio/core ai @ai-sdk/openai @ai-sdk/mcp
```

#### Path 2, Step 2: Create Tool Router Session

Initialize the Composio client and create a Tool Router session
```python
from composio import Composio
from claude_agent_sdk import ClaudeSDKClient, ClaudeAgentOptions

composio = Composio(api_key='your-composio-api-key')
session = composio.create(user_id='your-user-id')
url = session.mcp.url
```

```typescript
import { Composio } from '@composio/core';

const composio = new Composio({ apiKey: 'your-api-key' });

console.log("Creating Tool Router session...");
const { mcp } = await composio.create('your-user-id');
console.log(`Tool Router session created: ${mcp.url}`);
```

#### Path 2, Step 3: Connect to AI Agent

Use the MCP server with your AI agent
```python
import asyncio

options = ClaudeAgentOptions(
    permission_mode='bypassPermissions',
    mcp_servers={
        'tool_router': {
            'type': 'http',
            'url': url,
            'headers': {
                'x-api-key': 'your-composio-api-key'
            }
        }
    },
    system_prompt='You are a helpful assistant with access to Scrapegraph ai tools.',
    max_turns=10
)

async def main():
    async with ClaudeSDKClient(options=options) as client:
        await client.query('Extract all product titles from https://example.com/shop')
        async for message in client.receive_response():
            if hasattr(message, 'content'):
                for block in message.content:
                    if hasattr(block, 'text'):
                        print(block.text)

asyncio.run(main())
```

```typescript
import { openai } from '@ai-sdk/openai';
import { experimental_createMCPClient as createMCPClient } from '@ai-sdk/mcp';
import { generateText, stepCountIs } from 'ai';

const client = await createMCPClient({
  transport: {
    type: 'http',
    url: mcp.url,
    headers: { 'x-api-key': 'your-composio-api-key' }
  }
});

const tools = await client.tools();

const { text } = await generateText({
  model: openai('gpt-4o'),
  tools,
  messages: [{ role: 'user', content: 'Extract all product titles from https://example.com/shop' }],
  stopWhen: stepCountIs( 5 )
});

console.log(`Agent: ${text}`);
```

## Why Use Composio?

### 1. AI Native Scrapegraph ai Integration

- Supports both Scrapegraph ai MCP and direct API based integrations
- Structured, LLM-friendly schemas for reliable tool execution
- Rich coverage for reading, writing, and querying your Scrapegraph ai data

### 2. Managed Auth

- Built-in OAuth handling with automatic token refresh and rotation
- Central place to manage, scope, and revoke Scrapegraph ai access
- Per user and per environment credentials instead of hard-coded keys

### 3. Agent Optimized Design

- Tools are tuned using real error and success rates to improve reliability over time
- Comprehensive execution logs so you always know what ran, when, and on whose behalf

### 4. Enterprise Grade Security

- Fine-grained RBAC so you control which agents and users can access Scrapegraph ai
- Scoped, least privilege access to Scrapegraph ai resources
- Full audit trail of agent actions to support review and compliance

## Use Scrapegraph ai with any AI Agent Framework

Choose a framework you want to connect Scrapegraph ai with:

- [OpenAI Agents SDK](https://composio.dev/toolkits/scrapegraph_ai/framework/open-ai-agents-sdk)
- [Claude Agent SDK](https://composio.dev/toolkits/scrapegraph_ai/framework/claude-agents-sdk)
- [Claude Code](https://composio.dev/toolkits/scrapegraph_ai/framework/claude-code)
- [Claude Cowork](https://composio.dev/toolkits/scrapegraph_ai/framework/claude-cowork)
- [Codex](https://composio.dev/toolkits/scrapegraph_ai/framework/codex)
- [OpenClaw](https://composio.dev/toolkits/scrapegraph_ai/framework/openclaw)
- [Hermes](https://composio.dev/toolkits/scrapegraph_ai/framework/hermes-agent)
- [Google ADK](https://composio.dev/toolkits/scrapegraph_ai/framework/google-adk)
- [LangChain](https://composio.dev/toolkits/scrapegraph_ai/framework/langchain)
- [Vercel AI SDK](https://composio.dev/toolkits/scrapegraph_ai/framework/ai-sdk)
- [Mastra AI](https://composio.dev/toolkits/scrapegraph_ai/framework/mastra-ai)
- [LlamaIndex](https://composio.dev/toolkits/scrapegraph_ai/framework/llama-index)
- [CrewAI](https://composio.dev/toolkits/scrapegraph_ai/framework/crew-ai)
- [Pydantic AI](https://composio.dev/toolkits/scrapegraph_ai/framework/pydantic-ai)
- [AutoGen](https://composio.dev/toolkits/scrapegraph_ai/framework/autogen)

## Related Toolkits

- [Excel](https://composio.dev/toolkits/excel) - Microsoft Excel is a robust spreadsheet application for organizing, analyzing, and visualizing data. It's the go-to tool for calculations, reporting, and flexible data management.
- [21risk](https://composio.dev/toolkits/_21risk) - 21RISK is a web app built for easy checklist, audit, and compliance management. It streamlines risk processes so teams can focus on what matters.
- [Abstract](https://composio.dev/toolkits/abstract) - Abstract provides a suite of APIs for automating data validation and enrichment tasks. It helps developers streamline workflows and ensure data quality with minimal effort.
- [Addressfinder](https://composio.dev/toolkits/addressfinder) - Addressfinder is a data quality platform for verifying addresses, emails, and phone numbers. It helps you ensure accurate customer and contact data every time.
- [Agenty](https://composio.dev/toolkits/agenty) - Agenty is a web scraping and automation platform for extracting data and automating browser tasks—no coding needed. It streamlines data collection, monitoring, and repetitive online actions.
- [Ambee](https://composio.dev/toolkits/ambee) - Ambee is an environmental data platform providing real-time, hyperlocal APIs for air quality, weather, and pollen. Get precise environmental insights to power smarter decisions in your apps and workflows.
- [Ambient weather](https://composio.dev/toolkits/ambient_weather) - Ambient Weather is a platform for personal weather stations with a robust API for accessing local, real-time, and historical weather data. Get detailed environmental insights directly from your own sensors for smarter apps and automations.
- [Anonyflow](https://composio.dev/toolkits/anonyflow) - Anonyflow is a service for encryption-based data anonymization and secure data sharing. It helps organizations meet GDPR, CCPA, and HIPAA data privacy compliance requirements.
- [Api ninjas](https://composio.dev/toolkits/api_ninjas) - Api ninjas offers 120+ public APIs spanning categories like weather, finance, sports, and more. Developers use it to supercharge apps with real-time data and actionable endpoints.
- [Api sports](https://composio.dev/toolkits/api_sports) - Api sports is a comprehensive sports data platform covering 2,000+ competitions with live scores and 15+ years of stats. Instantly access up-to-date sports information for analysis, apps, or chatbots.
- [Apify](https://composio.dev/toolkits/apify) - Apify is a cloud platform for building, deploying, and managing web scraping and automation tools called Actors. It lets you automate data extraction and workflow tasks at scale—no infrastructure headaches.
- [Autom](https://composio.dev/toolkits/autom) - Autom is a lightning-fast search engine results data platform for Google, Bing, and Brave. Developers use it to access fresh, low-latency SERP data on demand.
- [Beaconchain](https://composio.dev/toolkits/beaconchain) - Beaconchain is a real-time analytics platform for Ethereum 2.0's Beacon Chain. It provides detailed insights into validators, blocks, and overall network performance.
- [Big data cloud](https://composio.dev/toolkits/big_data_cloud) - BigDataCloud provides APIs for geolocation, reverse geocoding, and address validation. Instantly access reliable location intelligence to enhance your applications and workflows.
- [Bigpicture io](https://composio.dev/toolkits/bigpicture_io) - BigPicture.io offers APIs for accessing detailed company and profile data. Instantly enrich your applications with up-to-date insights on 20M+ businesses.
- [Bitquery](https://composio.dev/toolkits/bitquery) - Bitquery is a blockchain data platform offering indexed, real-time, and historical data from 40+ blockchains via GraphQL APIs. Get unified, reliable access to complex on-chain data for analytics, trading, and research.
- [Brightdata](https://composio.dev/toolkits/brightdata) - Brightdata is a leading web data platform offering advanced scraping, SERP APIs, and anti-bot tools. It lets you collect public web data at scale, bypassing blocks and friction.
- [Builtwith](https://composio.dev/toolkits/builtwith) - BuiltWith is a web technology profiler that uncovers the technologies powering any website. Gain actionable insights into analytics, hosting, and content management stacks for smarter research and lead generation.
- [Byteforms](https://composio.dev/toolkits/byteforms) - Byteforms is an all-in-one platform for creating forms, managing submissions, and integrating data. It streamlines workflows by centralizing form data collection and automation.
- [Cabinpanda](https://composio.dev/toolkits/cabinpanda) - Cabinpanda is a data collection platform for building and managing online forms. It helps streamline how you gather, organize, and analyze responses.

## Frequently Asked Questions

### Do I need my own developer credentials to use Scrapegraph ai with Composio?

Yes, Scrapegraph ai requires you to configure your own API key credentials. Once set up, Composio handles secure credential storage and API request handling for you.

### Can I use multiple toolkits together?

Yes! Composio's Tool Router enables agents to use multiple toolkits. [Learn more](https://docs.composio.dev/tool-router/overview).

### Is Composio secure?

Composio is SOC 2 and ISO 27001 compliant with all data encrypted in transit and at rest. [Learn more](https://trust.composio.dev).

### What if the API changes?

Composio maintains and updates all toolkit integrations automatically, so your agents always work with the latest API versions.

---
[See all toolkits](https://composio.dev/toolkits) · [Composio docs](https://docs.composio.dev/llms.txt)