Diffbot Integration for AI Agents

Securely connect your AI agents and chatbots (Claude, ChatGPT, Cursor, etc) with Diffbot MCP or direct API to extract article content, analyze product listings, enrich structured web data, and automate web research through natural language.
BOOK A CALLGET STARTED FOR FREE
Diffbot Logo
Gradient Top
Gradient Middle
Gradient Bottom
divider

Supported Tools

Tools
Diffbot SearchTool to search data extracted by crawl or bulk jobs using dql queries.
Get Diffbot Account DetailsTool to retrieve account details, including plan information and usage statistics.
Diffbot AnalyzeTool to automatically determine a page's content type and route it to the appropriate extraction api.
Get Article DataTool to extract information from articles, including authors, publication dates, and images.
Get Discussion ThreadTool to extract threads of content from forums, comment sections, and review pages.
Diffbot Get EventTool to extract event details from web pages.
Diffbot Get ImageTool to extract detailed information about images, including dimensions and recognition data.
Diffbot Get ProductTool to extract product information such as specifications, prices, availability, and reviews.
Get Video DataTool to extract information from videos, including titles, descriptions, and embedded html.
List Bulk JobsTool to list all bulk jobs associated with a specific token.
Resolve Lost IDTool to resolve lost ids in the knowledge graph.
Start Bulk JobTool to start a bulk extract job.
Start Crawl JobTool to spider a site for links and process them with the extract api into a single collection.
Stop Bulk JobTool to stop a running bulk job.

Why Use Composio?

AI Native Diffbot Integration

  • Supports both Diffbot MCP and direct API based integrations
  • Structured, LLM-friendly schemas for reliable tool execution
  • Rich coverage for extracting, analyzing, and enriching web data

Managed Auth

  • Built-in API key management with secure storage and rotation
  • Central place to manage, scope, and revoke Diffbot API keys
  • Per user and per environment credentials instead of hard-coded keys

Agent Optimized Design

  • Tools are tuned using real error and success rates to improve reliability over time
  • Comprehensive execution logs so you always know what ran, when, and on whose behalf

Enterprise Grade Security

  • Fine-grained RBAC so you control which agents and users can access Diffbot
  • Scoped, least privilege access to Diffbot resources
  • Full audit trail of agent actions to support review and compliance

Connect Diffbot MCP Tool with your Agent

Python
TypeScript

Install Composio

python
pip install composio claude-agent-sdk
Install the Composio SDK and Claude Agent SDK

Create Tool Router Session

python
from composio import Composio
from claude_agent_sdk import ClaudeSDKClient, ClaudeAgentOptions

composio = Composio(api_key='your-composio-api-key')
session = composio.create(user_id='your-user-id')
url = session.mcp.url
Initialize the Composio client and create a Tool Router session

Connect to AI Agent

python
import asyncio

options = ClaudeAgentOptions(
    permission_mode='bypassPermissions',
    mcp_servers={
        'tool_router': {
            'type': 'http',
            'url': url,
            'headers': {
                'x-api-key': 'your-composio-api-key'
            }
        }
    },
    system_prompt='You are a helpful assistant with access to Diffbot tools.',
    max_turns=10
)

async def main():
    async with ClaudeSDKClient(options=options) as client:
        await client.query('Extract product details from https://www.example.com/product/12345')
        async for message in client.receive_response():
            if hasattr(message, 'content'):
                for block in message.content:
                    if hasattr(block, 'text'):
                        print(block.text)

asyncio.run(main())
Use the MCP server with your AI agent

Connect Diffbot API Tool with your Agent

Python
TypeScript

Install Composio

python
pip install composio_openai
Install the Composio SDK

Initialize Composio and Create Tool Router Session

python
from openai import OpenAI
from composio import Composio
from composio_openai import OpenAIResponsesProvider

composio = Composio(provider=OpenAIResponsesProvider())
openai = OpenAI()
session = composio.create(user_id='your-user-id')
Import and initialize Composio client, then create a Tool Router session

Execute Diffbot Tools via Tool Router with Your Agent

python
tools = session.tools
response = openai.responses.create(
  model='gpt-4.1',
  tools=tools,
  input=[{
    'role': 'user',
    'content': 'Extract product info from this Amazon page'
  }]
)
result = composio.provider.handle_tool_calls(
  response=response,
  user_id='your-user-id'
)
print(result)
Get tools from Tool Router session and execute Diffbot actions with your Agent

Use Diffbot with any AI Agent Framework

Choose a Framework you want to connect Diffbot with

OpenAI Agents SDK

OpenAI Agents SDK

Use Diffbot MCP with OpenAI Agents SDK

Claude Agents SDK

Claude Agents SDK

Use Diffbot MCP with Claude Agents SDK

Google ADK

Google ADK

Use Diffbot MCP with Google ADK

Langchain

Langchain

Use Diffbot MCP with Langchain

AI SDK

AI SDK

Use Diffbot MCP with AI SDK

Mastra AI

Mastra AI

Use Diffbot MCP with Mastra AI

LlamaIndex

LlamaIndex

Use Diffbot MCP with LlamaIndex

CrewAI

CrewAI

Use Diffbot MCP with CrewAI

Pydantic AI

Pydantic AI

Use Diffbot MCP with Pydantic AI

Autogen

Autogen

Use Diffbot MCP with Autogen

Frequently Asked Questions

Do I need my own developer credentials to use Diffbot with Composio?

Yes, Diffbot requires you to configure your own API key credentials. Once set up, Composio handles secure credential storage and API request handling for you.

Can I use multiple toolkits together?

Yes! Composio's Tool Router enables agents to use multiple toolkits. Learn more.

Is Composio secure?

Composio is SOC 2 and ISO 27001 compliant with all data encrypted in transit and at rest. Learn more.

What if the API changes?

Composio maintains and updates all toolkit integrations automatically, so your agents always work with the latest API versions.

Used by agents from

Context
ASU
Letta
glean
HubSpot
Agent.ai
Altera
DataStax
Entelligence
Rolai
Context
ASU
Letta
glean
HubSpot
Agent.ai
Altera
DataStax
Entelligence
Rolai
Context
ASU
Letta
glean
HubSpot
Agent.ai
Altera
DataStax
Entelligence
Rolai

Never worry about agent reliability

We handle tool reliability, observability, and security so you never have to second-guess an agent action.

Book a call↗Get started for free