SIGN UPLOG IN
X (Twitter)LinkedInGitHubDiscordYouTube

How to integrate Diffbot MCP with Pydantic AI

Connect Pydantic AI to Diffbot MCP. Extract specs and reviews from a product page, summarize key details from a news article url, and more using natural language, with authentication handled for you.

Get started for freeGet a demo
Diffbot logoDiffbot
Api Key

Diffbot is an AI-powered platform for extracting and structuring data from any web page. It transforms unstructured web content into rich, linked, and queryable data for analytics, research, and automation.

35 Tools

Introduction

This guide walks you through connecting Diffbot to Pydantic AI using the Composio tool router. By the end, you'll have a working Diffbot agent that can extract data from articles and products, search the Knowledge Graph, enrich entities, run crawl jobs, and manage bulk jobs through natural language commands.

This guide will help you understand how to give your Pydantic AI agent real control over a Diffbot account through Composio's Diffbot MCP server.

Before we dive in, let's take a quick look at the key ideas and tools involved.

For exact tool slugs and parameter tables, see the tool catalog in the Composio docs.

Also integrate Diffbot with

Claude CodeClaude CoworkOpenAI Agents SDKClaude Agent SDKCodexOpenClawHermesAtomic AgentGoogle ADKLangChainVercel AI SDKMastra AILlamaIndexCrewAIAutoGen

TL;DR

Here's what you'll learn:
  • How to set up your Composio API key and User ID
  • How to create a Composio Tool Router session for Diffbot
  • How to attach an MCP Server to a Pydantic AI agent
  • How to stream responses and maintain chat history
  • How to build a simple REPL-style chat interface to test your Diffbot workflows

What is Pydantic AI?

Pydantic AI is a Python framework for building AI agents with strong typing and validation. It leverages Pydantic's data validation capabilities to create robust, type-safe AI applications.

Key features include:

  • Type Safety: Built on Pydantic for automatic data validation
  • MCP Support: Native support for Model Context Protocol servers
  • Streaming: Built-in support for streaming responses
  • Async First: Designed for async/await patterns

What is the Diffbot MCP server, and what's possible with it?

The Diffbot MCP server is an implementation of the Model Context Protocol that connects your AI agent and assistants like Claude, Cursor, etc directly to your Diffbot account. It provides structured and secure access to web data extraction and analysis, so your agent can extract structured data from web pages, analyze content types, retrieve product details, manage bulk jobs, and search extracted datasets on your behalf.

  • Automatic content analysis and extraction: Let your agent analyze any web page and automatically extract structured data such as articles, products, events, images, or videos using AI-powered tools.
  • Article and discussion thread extraction: Effortlessly pull detailed metadata, authors, publication dates, and full discussion threads from news sites, blogs, forums, and comment sections.
  • Product and event data gathering: Instantly extract comprehensive product specifications, pricing, reviews, and event information including venues, dates, and descriptions from e-commerce or event pages.
  • Bulk job management and search: Enable your agent to list, monitor, and search across large-scale crawl or extraction jobs, making it easy to work with massive web data collections.
  • Account and usage insights: Retrieve your Diffbot account details, plan information, and usage statistics to stay on top of quotas and manage your web data operations efficiently.

What is the Composio tool router, and how does it fit here?

What is Composio SDK?

Composio's Composio SDK helps agents find the right tools for a task at runtime. You can plug in multiple toolkits (like Gmail, HubSpot, and GitHub), and the agent will identify the relevant app and action to complete multi-step workflows. This can reduce token usage and improve the reliability of tool calls. Read more here: Getting started with Composio SDK

The tool router generates a secure MCP URL that your agents can access to perform actions.

How the Composio SDK works

The Composio SDK follows a three-phase workflow:

  1. Discovery: Searches for tools matching your task and returns relevant toolkits with their details.
  2. Authentication: Checks for active connections. If missing, creates an auth config and returns a connection URL via Auth Link.
  3. Execution: Executes the action using the authenticated connection.

Step-by-step Guide

Step by step09 STEPS
1

Prerequisites

Before starting, make sure you have:
  • Python 3.9 or higher
  • A Composio account with an active API key
  • Basic familiarity with Python and async programming
2

Getting API Keys for OpenAI and Composio

OpenAI API Key
  • Go to the OpenAI dashboard and create an API key. You'll need credits to use the models, or you can connect to another model provider.
  • Keep the API key safe.
Composio API Key
  • Log in to the Composio dashboard.
  • Navigate to your API settings and generate a new API key.
  • Store this key securely as you'll need it for authentication.
3

Install dependencies

bash
pip install composio pydantic-ai python-dotenv

Install the required libraries.

What's happening:

  • composio connects your agent to external SaaS tools like Diffbot
  • pydantic-ai lets you create structured AI agents with tool support
  • python-dotenv loads your environment variables securely from a .env file
4

Set up environment variables

bash
COMPOSIO_API_KEY=your_composio_api_key_here
USER_ID=your_user_id_here
OPENAI_API_KEY=your_openai_api_key

Create a .env file in your project root.

What's happening:

  • COMPOSIO_API_KEY authenticates your agent to Composio's API
  • USER_ID associates your session with your account for secure tool access
  • OPENAI_API_KEY to access OpenAI LLMs
5

Import dependencies

python
import asyncio
import os
from dotenv import load_dotenv
from composio import Composio
from pydantic_ai import Agent
from pydantic_ai.mcp import MCPServerStreamableHTTP

load_dotenv()
What's happening:
  • We load environment variables and import required modules
  • Composio manages connections to Diffbot
  • MCPServerStreamableHTTP connects to the Diffbot MCP server endpoint
  • Agent from Pydantic AI lets you define and run the AI assistant
6

Create a Tool Router Session

python
async def main():
    api_key = os.getenv("COMPOSIO_API_KEY")
    user_id = os.getenv("USER_ID")
    if not api_key or not user_id:
        raise RuntimeError("Set COMPOSIO_API_KEY and USER_ID in your environment")

    # Create a Composio Tool Router session for Diffbot
    composio = Composio(api_key=api_key)
    session = composio.create(
        user_id=user_id,
        toolkits=["diffbot"],
    )
    url = session.mcp.url
    if not url:
        raise ValueError("Composio session did not return an MCP URL")
What's happening:
  • We're creating a Tool Router session that gives your agent access to Diffbot tools
  • The create method takes the user ID and specifies which toolkits should be available
  • The returned session.mcp.url is the MCP server URL that your agent will use
7

Initialize the Pydantic AI Agent

python
# Attach the MCP server to a Pydantic AI Agent
diffbot_mcp = MCPServerStreamableHTTP(url, headers={"x-api-key": COMPOSIO_API_KEY})
agent = Agent(
    "openai:gpt-5",
    toolsets=[diffbot_mcp],
    instructions=(
        "You are a Diffbot assistant. Use Diffbot tools to help users "
        "with their requests. Ask clarifying questions when needed."
    ),
)
What's happening:
  • The MCP client connects to the Diffbot endpoint
  • The agent uses GPT-5 to interpret user commands and perform Diffbot operations
  • The instructions field defines the agent's role and behavior
8

Build the chat interface

python
# Simple REPL with message history
history = []
print("Chat started! Type 'exit' or 'quit' to end.\n")
print("Try asking the agent to help you with Diffbot.\n")

while True:
    user_input = input("You: ").strip()
    if user_input.lower() in {"exit", "quit", "bye"}:
        print("\nGoodbye!")
        break
    if not user_input:
        continue

    print("\nAgent is thinking...\n", flush=True)

    async with agent.run_stream(user_input, message_history=history) as stream_result:
        collected_text = ""
        async for chunk in stream_result.stream_output():
            text_piece = None
            if isinstance(chunk, str):
                text_piece = chunk
            elif hasattr(chunk, "delta") and isinstance(chunk.delta, str):
                text_piece = chunk.delta
            elif hasattr(chunk, "text"):
                text_piece = chunk.text
            if text_piece:
                collected_text += text_piece
        result = stream_result

    print(f"Agent: {collected_text}\n")
    history = result.all_messages()
What's happening:
  • The agent reads input from the terminal and streams its response
  • Diffbot API calls happen automatically under the hood
  • The model keeps conversation history to maintain context across turns
9

Run the application

python
if __name__ == "__main__":
    asyncio.run(main())
What's happening:
  • The asyncio loop launches the agent and keeps it running until you exit

Complete Code

Here's the complete code to get you started with Diffbot and Pydantic AI:

python
import asyncio
import os
from dotenv import load_dotenv
from composio import Composio
from pydantic_ai import Agent
from pydantic_ai.mcp import MCPServerStreamableHTTP

load_dotenv()

async def main():
    api_key = os.getenv("COMPOSIO_API_KEY")
    user_id = os.getenv("USER_ID")
    if not api_key or not user_id:
        raise RuntimeError("Set COMPOSIO_API_KEY and USER_ID in your environment")

    # Create a Composio Tool Router session for Diffbot
    composio = Composio(api_key=api_key)
    session = composio.create(
        user_id=user_id,
        toolkits=["diffbot"],
    )
    url = session.mcp.url
    if not url:
        raise ValueError("Composio session did not return an MCP URL")

    # Attach the MCP server to a Pydantic AI Agent
    diffbot_mcp = MCPServerStreamableHTTP(url, headers={"x-api-key": COMPOSIO_API_KEY})
    agent = Agent(
        "openai:gpt-5",
        toolsets=[diffbot_mcp],
        instructions=(
            "You are a Diffbot assistant. Use Diffbot tools to help users "
            "with their requests. Ask clarifying questions when needed."
        ),
    )

    # Simple REPL with message history
    history = []
    print("Chat started! Type 'exit' or 'quit' to end.\n")
    print("Try asking the agent to help you with Diffbot.\n")

    while True:
        user_input = input("You: ").strip()
        if user_input.lower() in {"exit", "quit", "bye"}:
            print("\nGoodbye!")
            break
        if not user_input:
            continue

        print("\nAgent is thinking...\n", flush=True)

        async with agent.run_stream(user_input, message_history=history) as stream_result:
            collected_text = ""
            async for chunk in stream_result.stream_output():
                text_piece = None
                if isinstance(chunk, str):
                    text_piece = chunk
                elif hasattr(chunk, "delta") and isinstance(chunk.delta, str):
                    text_piece = chunk.delta
                elif hasattr(chunk, "text"):
                    text_piece = chunk.text
                if text_piece:
                    collected_text += text_piece
            result = stream_result

        print(f"Agent: {collected_text}\n")
        history = result.all_messages()

if __name__ == "__main__":
    asyncio.run(main())

Conclusion

You've built a Pydantic AI agent that can interact with Diffbot through Composio's Tool Router. With this setup, your agent can perform real Diffbot actions through natural language. You can extend this further by:
  • Adding other toolkits like Gmail, HubSpot, or Salesforce
  • Building a web-based chat interface around this agent
  • Using multiple MCP endpoints to enable cross-app workflows (for example, Gmail + Diffbot for workflow automation)
This architecture makes your AI agent "agent-native", able to securely use APIs in a unified, composable way without custom integrations.
TOOLS

Supported Tools

Every Diffbot action and event your agent gets out of the box.

Combine Entity Profiles

Combine entity information into an enhanced view using the Diffbot Knowledge Graph, matching identifying attributes such as name, email, employer, or URL.

Create Bulk Extract Job

Tool to submit a bulk extract job to process multiple URLs with Extract APIs.

Create or Update Custom API

Tool to create or update the parameters and ruleset of a Custom API.

Create Bulk Enhance Job

Submit an asynchronous bulk enhance job for Person or Organization records using entity descriptions or Knowledge Graph IDs.

Delete Custom API

Delete a Custom API definition and its extraction rules for a specified base API and URL pattern.

Delete KG Enhance Bulkjob

Tool to delete an Enhance Bulkjob.

Download Bulk Job Results

Download bulk enhance job results via POST, with filtering and export options for JSON, JSONL, CSV, XLS and XLSX.

Enhance Entity with Knowledge Graph

Enrich a person or organization with comprehensive data from the Diffbot Knowledge Graph.

Diffbot Extract Job

Tool to extract structured job posting data from job listing pages.

Diffbot Extract List

Tool to extract structured data from list-style pages like news indexes, product listings, and directory pages.

Get Diffbot Account Details

Retrieve Diffbot account details, including subscription plan credits, account status, account holder information, and daily usage history.

Diffbot Analyze

Automatically analyze a web page to determine its type and extract structured data.

Get Article Data

Tool to extract information from articles, including authors, publication dates, and images.

Get Bulk Job Data

Tool to download extracted results from a completed bulk job.

Get Bulk Job Status

Tool to poll the status of a specific Diffbot Knowledge Graph Enhance bulk job.

Get Bulk Job Results

Tool to download the results of a completed Enhance Bulkjob.

Get Bulk Single Result

Tool to download the result of a single job within a Diffbot bulk enhance job.

Get Crawl Data

Download extracted results from a completed crawl job, optionally limiting the number of results.

Get Discussion Thread

Extract structured discussion threads from web pages, including forums, comment sections and product reviews.

Diffbot Get Event

Tool to extract event details from web pages.

Diffbot Get Image

Tool to extract detailed information about images, including dimensions and recognition data.

Get KG Coverage Report by ID

Download Knowledge Graph coverage report by report ID.

Diffbot Get Product

Tool to extract product information such as specifications, prices, availability, and reviews.

Get Video Data

Tool to extract information from videos, including titles, descriptions, and embedded HTML.

List Bulk Jobs

Tool to list all Bulk jobs associated with a specific token.

List Bulk Jobs Status For Token

Tool to get the status of all bulk enhance jobs for a token.

List Custom APIs

Tool to retrieve all Custom APIs and their extraction rules currently defined on your Diffbot token.

Manage Crawl Job

Manages Diffbot crawl jobs: pause, restart, delete, or view status.

Resolve Lost ID

Tool to resolve lost IDs in the Knowledge Graph.

Diffbot Knowledge Graph Search

Search the Diffbot Knowledge Graph using DQL (Diffbot Query Language).

Search Crawl Job Data

Tool to query crawl job collections using DQL (Diffbot Query Language).

Start Bulk Job

Start an asynchronous Bulk Extract job for a list of URLs using a specified Extract API.

Start Crawl Job

Initiates a Diffbot crawl job that spiders a website starting from seed URLs and processes discovered pages with a specified Extract API.

Stop Bulk Job

Tool to pause (stop) a running Bulk job.

Stop KG Bulk Job By ID

Tool to stop an active Knowledge Graph Enhance bulk job by its ID.

FRAMEWORKS

How to build Diffbot MCP Agent with another framework

Claude Code logo

Claude Code

Use Diffbot MCP with Claude Code

Claude Cowork logo

Claude Cowork

Use Diffbot MCP with Claude Cowork

OpenAI Agents SDK logo

OpenAI Agents SDK

Use Diffbot MCP with OpenAI Agents SDK

Claude Agent SDK logo

Claude Agent SDK

Use Diffbot MCP with Claude Agent SDK

Codex logo

Codex

Use Diffbot MCP with Codex

OpenClaw logo

OpenClaw

Use Diffbot MCP with OpenClaw

Hermes logo

Hermes

Use Diffbot MCP with Hermes

Atomic Agent logo

Atomic Agent

Use Diffbot MCP with Atomic Agent

Google ADK logo

Google ADK

Use Diffbot MCP with Google ADK

LangChain logo

LangChain

Use Diffbot MCP with LangChain

Vercel AI SDK logo

Vercel AI SDK

Use Diffbot MCP with Vercel AI SDK

Mastra AI logo

Mastra AI

Use Diffbot MCP with Mastra AI

LlamaIndex logo

LlamaIndex

Use Diffbot MCP with LlamaIndex

CrewAI logo

CrewAI

Use Diffbot MCP with CrewAI

AutoGen logo

AutoGen

Use Diffbot MCP with AutoGen

FAQ

Frequently asked questions

Start with Diffbot.It takes 30 seconds.

Managed auth, hosted MCP servers, and every Diffbot tool your agent needs.Free to start.

Start building