# How to integrate Zenrows MCP with Pydantic AI ```json { "title": "How to integrate Zenrows MCP with Pydantic AI", "toolkit": "Zenrows", "toolkit_slug": "zenrows", "framework": "Pydantic AI", "framework_slug": "pydantic-ai", "url": "https://composio.dev/toolkits/zenrows/framework/pydantic-ai", "markdown_url": "https://composio.dev/toolkits/zenrows/framework/pydantic-ai.md", "updated_at": "2026-05-06T08:34:34.376Z" } ``` ## Introduction This guide walks you through connecting Zenrows to Pydantic AI using the Composio tool router. By the end, you'll have a working Zenrows agent that can download a pdf of this news article, extract plain text from the given webpage, get latest property data from zillow through natural language commands. This guide will help you understand how to give your Pydantic AI agent real control over a Zenrows account through Composio's Zenrows MCP server. Before we dive in, let's take a quick look at the key ideas and tools involved. ## Also integrate Zenrows with - [OpenAI Agents SDK](https://composio.dev/toolkits/zenrows/framework/open-ai-agents-sdk) - [Claude Agent SDK](https://composio.dev/toolkits/zenrows/framework/claude-agents-sdk) - [Claude Code](https://composio.dev/toolkits/zenrows/framework/claude-code) - [Claude Cowork](https://composio.dev/toolkits/zenrows/framework/claude-cowork) - [Codex](https://composio.dev/toolkits/zenrows/framework/codex) - [OpenClaw](https://composio.dev/toolkits/zenrows/framework/openclaw) - [Hermes](https://composio.dev/toolkits/zenrows/framework/hermes-agent) - [CLI](https://composio.dev/toolkits/zenrows/framework/cli) - [Google ADK](https://composio.dev/toolkits/zenrows/framework/google-adk) - [LangChain](https://composio.dev/toolkits/zenrows/framework/langchain) - [Vercel AI SDK](https://composio.dev/toolkits/zenrows/framework/ai-sdk) - [Mastra AI](https://composio.dev/toolkits/zenrows/framework/mastra-ai) - [LlamaIndex](https://composio.dev/toolkits/zenrows/framework/llama-index) - [CrewAI](https://composio.dev/toolkits/zenrows/framework/crew-ai) ## TL;DR Here's what you'll learn: - How to set up your Composio API key and User ID - How to create a Composio Tool Router session for Zenrows - How to attach an MCP Server to a Pydantic AI agent - How to stream responses and maintain chat history - How to build a simple REPL-style chat interface to test your Zenrows workflows ## What is Pydantic AI? Pydantic AI is a Python framework for building AI agents with strong typing and validation. It leverages Pydantic's data validation capabilities to create robust, type-safe AI applications. Key features include: - Type Safety: Built on Pydantic for automatic data validation - MCP Support: Native support for Model Context Protocol servers - Streaming: Built-in support for streaming responses - Async First: Designed for async/await patterns ## What is the Zenrows MCP server, and what's possible with it? The Zenrows MCP server is an implementation of the Model Context Protocol that connects your AI agent and assistants like Claude, Cursor, etc directly to your Zenrows account. It provides structured and secure access to advanced web scraping capabilities, so your agent can extract structured data, bypass CAPTCHAs, convert pages to PDF, and monitor your API usage on your behalf. - Intelligent web data extraction: Direct your agent to scrape and extract plain text or structured data from dynamic websites, including specialized real estate property data from platforms like Zillow or Idealista. - PDF and content generation: Ask your agent to convert any web page into a PDF or retrieve clean, formatted plain text for archiving, documentation, or offline reading. - Seamless CAPTCHA and block bypassing: Enable your agent to gather data from sites protected by CAPTCHAs or anti-bot systems without manual intervention. - Real-time API usage monitoring: Have the agent check your account’s current API usage, concurrency status, and limits to help manage credits and avoid interruptions. - Session and compression management: Instruct your agent to maintain consistent scraping sessions, handle compression to optimize bandwidth, and retrieve detailed response headers for debugging and performance optimization. ## Supported Tools | Tool slug | Name | Description | |---|---|---| | `ZENROWS_GET_API_USAGE` | Get ZenRows API Usage Statistics | This tool retrieves the current api usage statistics and limits for your zenrows account. it is an independent action that requires no additional parameters besides authentication. it is useful for monitoring api usage and ensuring that the account has sufficient remaining credits. | | `ZENROWS_GET_CONCURRENCY_STATUS` | Get Concurrency Status | This tool retrieves the current concurrency status of your zenrows api usage. it reports the maximum number of concurrent requests allowed by your plan and the number of available concurrent request slots. it is useful for monitoring api usage, implementing rate limiting, debugging request issues, and capacity planning. | | `ZENROWS_GET_CONCURRENCY_STATUS_DETAILED` | Get Detailed Concurrency Status | This tool provides detailed information about the current concurrency status and limits of your zenrows account by making a request to the api and analyzing the response headers. it is essential for monitoring real-time api usage, managing concurrent requests, and ensuring optimal performance within plan limits. | | `ZENROWS_GET_ORIGINAL_STATUS` | Get Original Status Code | This tool retrieves the original http status code returned by the target website, which is useful for debugging purposes. it returns the original status code in the response headers under 'x-zenrows-original-status'. it can also provide the full response body and error messages, helping with debugging scraping issues, verifying website responses, monitoring availability, and understanding website behavior. | | `ZENROWS_GET_PDF_FROM_URL` | Get PDF from URL | This tool generates a pdf version of the scraped content from a given url. it requires javascript rendering to be enabled and sets the response type to pdf, making it ideal for archiving web pages, creating documentation, generating reports, or saving articles for offline reading. | | `ZENROWS_GET_PLAINTEXT` | Get Plaintext Response | This tool extracts plain text content from a webpage using the zenrows api. by using the response type=plaintext parameter, it strips html tags and formats the content into clean, plain text. it's useful for extracting readable content for nlp, summarization, or archiving purposes. | | `ZENROWS_GET_REAL_ESTATE_DATA` | Get Real Estate Property Data | A specialized tool for extracting structured data from real estate platforms like zillow and idealista. it leverages zenrows' real estate api to fetch comprehensive property information, including property details, location information, features, seller details, and more, in a structured format. | | `ZENROWS_GET_RESPONSE_COMPRESSION` | Get Response with Compression | A tool to fetch content from a url using the zenrows api with compression enabled to optimize bandwidth usage and improve performance. it supports gzip, deflate, and br compression encodings, handles decompression automatically, and provides compression statistics along with the decompressed content. | | `ZENROWS_GET_RESPONSE_HEADERS` | Get response headers | A tool to retrieve and parse response headers from zenrows api requests. it provides critical metadata such as concurrency limits, available request slots, request cost, unique request id, and final url after redirects, which is essential for monitoring usage, debugging, and optimizing requests. | | `ZENROWS_GET_SESSION_ID` | Get Session ID | This tool implements zenrows' session management functionality to maintain the same ip address across multiple requests for up to 10 minutes. it supports parameters like url, session id, and premium proxy, and is useful for maintaining consistent scraping sessions, simulating real user behavior, and avoiding detection by anti-bot systems. | | `ZENROWS_GET_WALMART_PRODUCT` | Get Walmart Product Details | This tool allows users to extract detailed product information from walmart using zenrows' specialized e-commerce scraping api. it provides structured data for walmart products including product details, pricing, availability, and more. | | `ZENROWS_SCRAPE_URL` | Scrape url | Scrape and extract data from a specified url. this action allows you to collect and process web data effortlessly using the zenrows api. | | `ZENROWS_SCRAPE_URL_AUTOPARSE` | Scrape url autoparse | The zenrows scrape url autoparse tool automatically parses and extracts structured data from any given url using intelligent parsing capabilities. it eliminates the need for manual css selectors by auto-identifying relevant content on web pages, returning data such as titles, main content, meta descriptions, images, links, prices, and contact information in a structured json format. | | `ZENROWS_SCRAPE_URL_HTML` | Scrape URL HTML | This tool extracts raw html data from a given url using zenrows' universal scraper api. it focuses on retrieving the pure html content of the webpage without automatic parsing or data extraction. it supports parameters such as js render for enabling javascript rendering, custom headers for custom http headers, premium proxy for using premium proxies, and session id for maintaining the same ip across multiple requests. | | `ZENROWS_SCRAPE_WITH_CSS_SELECTORS` | Scrape URL with CSS Selectors | This tool allows users to scrape specific elements from a webpage using css selectors. it is particularly useful for targeted data extraction rather than retrieving the entire page content. the endpoint takes a url and a json object containing css selectors for parsing elements such as titles, links, images, and prices, and includes optional parameters like using premium proxies, specifying response wait times, and custom headers among others. | | `ZENROWS_SCREENSHOT_URL` | Screenshot URL | A tool to capture screenshots of web pages using zenrows api. this tool allows you to take screenshots of entire web pages or specific elements, with customizable options for format and quality. | ## Supported Triggers None listed. ## Creating MCP Server - Stand-alone vs Composio SDK The Zenrows MCP server is an implementation of the Model Context Protocol that connects your AI agent to Zenrows. It provides structured and secure access so your agent can perform Zenrows operations on your behalf through a secure, permission-based interface. With Composio's managed implementation, you don't have to create your own developer app. For production, if you're building an end product, we recommend using your own credentials. The managed server helps you prototype fast and go from 0-1 faster. ## Step-by-step Guide ### 1. Prerequisites Before starting, make sure you have: - Python 3.9 or higher - A Composio account with an active API key - Basic familiarity with Python and async programming ### 1. Getting API Keys for OpenAI and Composio OpenAI API Key - Go to the [OpenAI dashboard](https://platform.openai.com/settings/organization/api-keys) and create an API key. You'll need credits to use the models, or you can connect to another model provider. - Keep the API key safe. Composio API Key - Log in to the [Composio dashboard](https://dashboard.composio.dev?utm_source=toolkits&utm_medium=framework_docs). - Navigate to your API settings and generate a new API key. - Store this key securely as you'll need it for authentication. ### 2. Install dependencies Install the required libraries. What's happening: - composio connects your agent to external SaaS tools like Zenrows - pydantic-ai lets you create structured AI agents with tool support - python-dotenv loads your environment variables securely from a .env file ```bash pip install composio pydantic-ai python-dotenv ``` ### 3. Set up environment variables Create a .env file in your project root. What's happening: - COMPOSIO_API_KEY authenticates your agent to Composio's API - USER_ID associates your session with your account for secure tool access - OPENAI_API_KEY to access OpenAI LLMs ```bash COMPOSIO_API_KEY=your_composio_api_key_here USER_ID=your_user_id_here OPENAI_API_KEY=your_openai_api_key ``` ### 4. Import dependencies What's happening: - We load environment variables and import required modules - Composio manages connections to Zenrows - MCPServerStreamableHTTP connects to the Zenrows MCP server endpoint - Agent from Pydantic AI lets you define and run the AI assistant ```python import asyncio import os from dotenv import load_dotenv from composio import Composio from pydantic_ai import Agent from pydantic_ai.mcp import MCPServerStreamableHTTP load_dotenv() ``` ### 5. Create a Tool Router Session What's happening: - We're creating a Tool Router session that gives your agent access to Zenrows tools - The create method takes the user ID and specifies which toolkits should be available - The returned session.mcp.url is the MCP server URL that your agent will use ```python async def main(): api_key = os.getenv("COMPOSIO_API_KEY") user_id = os.getenv("USER_ID") if not api_key or not user_id: raise RuntimeError("Set COMPOSIO_API_KEY and USER_ID in your environment") # Create a Composio Tool Router session for Zenrows composio = Composio(api_key=api_key) session = composio.create( user_id=user_id, toolkits=["zenrows"], ) url = session.mcp.url if not url: raise ValueError("Composio session did not return an MCP URL") ``` ### 6. Initialize the Pydantic AI Agent What's happening: - The MCP client connects to the Zenrows endpoint - The agent uses GPT-5 to interpret user commands and perform Zenrows operations - The instructions field defines the agent's role and behavior ```python # Attach the MCP server to a Pydantic AI Agent zenrows_mcp = MCPServerStreamableHTTP(url, headers={"x-api-key": COMPOSIO_API_KEY}) agent = Agent( "openai:gpt-5", toolsets=[zenrows_mcp], instructions=( "You are a Zenrows assistant. Use Zenrows tools to help users " "with their requests. Ask clarifying questions when needed." ), ) ``` ### 7. Build the chat interface What's happening: - The agent reads input from the terminal and streams its response - Zenrows API calls happen automatically under the hood - The model keeps conversation history to maintain context across turns ```python # Simple REPL with message history history = [] print("Chat started! Type 'exit' or 'quit' to end.\n") print("Try asking the agent to help you with Zenrows.\n") while True: user_input = input("You: ").strip() if user_input.lower() in {"exit", "quit", "bye"}: print("\nGoodbye!") break if not user_input: continue print("\nAgent is thinking...\n", flush=True) async with agent.run_stream(user_input, message_history=history) as stream_result: collected_text = "" async for chunk in stream_result.stream_output(): text_piece = None if isinstance(chunk, str): text_piece = chunk elif hasattr(chunk, "delta") and isinstance(chunk.delta, str): text_piece = chunk.delta elif hasattr(chunk, "text"): text_piece = chunk.text if text_piece: collected_text += text_piece result = stream_result print(f"Agent: {collected_text}\n") history = result.all_messages() ``` ### 8. Run the application What's happening: - The asyncio loop launches the agent and keeps it running until you exit ```python if __name__ == "__main__": asyncio.run(main()) ``` ## Complete Code ```python import asyncio import os from dotenv import load_dotenv from composio import Composio from pydantic_ai import Agent from pydantic_ai.mcp import MCPServerStreamableHTTP load_dotenv() async def main(): api_key = os.getenv("COMPOSIO_API_KEY") user_id = os.getenv("USER_ID") if not api_key or not user_id: raise RuntimeError("Set COMPOSIO_API_KEY and USER_ID in your environment") # Create a Composio Tool Router session for Zenrows composio = Composio(api_key=api_key) session = composio.create( user_id=user_id, toolkits=["zenrows"], ) url = session.mcp.url if not url: raise ValueError("Composio session did not return an MCP URL") # Attach the MCP server to a Pydantic AI Agent zenrows_mcp = MCPServerStreamableHTTP(url, headers={"x-api-key": COMPOSIO_API_KEY}) agent = Agent( "openai:gpt-5", toolsets=[zenrows_mcp], instructions=( "You are a Zenrows assistant. Use Zenrows tools to help users " "with their requests. Ask clarifying questions when needed." ), ) # Simple REPL with message history history = [] print("Chat started! Type 'exit' or 'quit' to end.\n") print("Try asking the agent to help you with Zenrows.\n") while True: user_input = input("You: ").strip() if user_input.lower() in {"exit", "quit", "bye"}: print("\nGoodbye!") break if not user_input: continue print("\nAgent is thinking...\n", flush=True) async with agent.run_stream(user_input, message_history=history) as stream_result: collected_text = "" async for chunk in stream_result.stream_output(): text_piece = None if isinstance(chunk, str): text_piece = chunk elif hasattr(chunk, "delta") and isinstance(chunk.delta, str): text_piece = chunk.delta elif hasattr(chunk, "text"): text_piece = chunk.text if text_piece: collected_text += text_piece result = stream_result print(f"Agent: {collected_text}\n") history = result.all_messages() if __name__ == "__main__": asyncio.run(main()) ``` ## Conclusion You've built a Pydantic AI agent that can interact with Zenrows through Composio's Tool Router. With this setup, your agent can perform real Zenrows actions through natural language. You can extend this further by: - Adding other toolkits like Gmail, HubSpot, or Salesforce - Building a web-based chat interface around this agent - Using multiple MCP endpoints to enable cross-app workflows (for example, Gmail + Zenrows for workflow automation) This architecture makes your AI agent "agent-native", able to securely use APIs in a unified, composable way without custom integrations. ## How to build Zenrows MCP Agent with another framework - [OpenAI Agents SDK](https://composio.dev/toolkits/zenrows/framework/open-ai-agents-sdk) - [Claude Agent SDK](https://composio.dev/toolkits/zenrows/framework/claude-agents-sdk) - [Claude Code](https://composio.dev/toolkits/zenrows/framework/claude-code) - [Claude Cowork](https://composio.dev/toolkits/zenrows/framework/claude-cowork) - [Codex](https://composio.dev/toolkits/zenrows/framework/codex) - [OpenClaw](https://composio.dev/toolkits/zenrows/framework/openclaw) - [Hermes](https://composio.dev/toolkits/zenrows/framework/hermes-agent) - [CLI](https://composio.dev/toolkits/zenrows/framework/cli) - [Google ADK](https://composio.dev/toolkits/zenrows/framework/google-adk) - [LangChain](https://composio.dev/toolkits/zenrows/framework/langchain) - [Vercel AI SDK](https://composio.dev/toolkits/zenrows/framework/ai-sdk) - [Mastra AI](https://composio.dev/toolkits/zenrows/framework/mastra-ai) - [LlamaIndex](https://composio.dev/toolkits/zenrows/framework/llama-index) - [CrewAI](https://composio.dev/toolkits/zenrows/framework/crew-ai) ## Related Toolkits - [Supabase](https://composio.dev/toolkits/supabase) - Supabase is an open-source backend platform offering scalable Postgres databases, authentication, storage, and real-time APIs. It lets developers build modern apps without managing infrastructure. - [Codeinterpreter](https://composio.dev/toolkits/codeinterpreter) - Codeinterpreter is a Python-based coding environment with built-in data analysis and visualization. It lets you instantly run scripts, plot results, and prototype solutions inside supported platforms. - [GitHub](https://composio.dev/toolkits/github) - GitHub is a code hosting platform for version control and collaborative software development. It streamlines project management, code review, and team workflows in one place. - [Ably](https://composio.dev/toolkits/ably) - Ably is a real-time messaging platform for live chat and data sync in modern apps. It offers global scale and rock-solid reliability for seamless, instant experiences. - [Abuselpdb](https://composio.dev/toolkits/abuselpdb) - Abuselpdb is a central database for reporting and checking IPs linked to malicious online activity. Use it to quickly identify and report suspicious or abusive IP addresses. - [Alchemy](https://composio.dev/toolkits/alchemy) - Alchemy is a blockchain development platform offering APIs and tools for Ethereum apps. It simplifies building and scaling Web3 projects with robust infrastructure. - [Algolia](https://composio.dev/toolkits/algolia) - Algolia is a hosted search API that powers lightning-fast, relevant search experiences for web and mobile apps. It helps developers deliver instant, typo-tolerant, and scalable search without complex infrastructure. - [Anchor browser](https://composio.dev/toolkits/anchor_browser) - Anchor browser is a developer platform for AI-powered web automation. It transforms complex browser actions into easy API endpoints for streamlined web interaction. - [Apiflash](https://composio.dev/toolkits/apiflash) - Apiflash is a website screenshot API for programmatically capturing web pages. It delivers high-quality screenshots on demand for automation, monitoring, or reporting. - [Apiverve](https://composio.dev/toolkits/apiverve) - Apiverve delivers a suite of powerful APIs that simplify integration for developers. It's designed for reliability and scalability so you can build faster, smarter applications without the integration headache. - [Appcircle](https://composio.dev/toolkits/appcircle) - Appcircle is an enterprise-grade mobile CI/CD platform for building, testing, and publishing mobile apps. It streamlines mobile DevOps so teams ship faster and with more confidence. - [Appdrag](https://composio.dev/toolkits/appdrag) - Appdrag is a cloud platform for building websites, APIs, and databases with drag-and-drop tools and code editing. It accelerates development and iteration by combining hosting, database management, and low-code features in one place. - [Appveyor](https://composio.dev/toolkits/appveyor) - AppVeyor is a cloud-based continuous integration service for building, testing, and deploying applications. It helps developers automate and streamline their software delivery pipelines. - [Backendless](https://composio.dev/toolkits/backendless) - Backendless is a backend-as-a-service platform for mobile and web apps, offering database, file storage, user authentication, and APIs. It helps developers ship scalable applications faster without managing server infrastructure. - [Baserow](https://composio.dev/toolkits/baserow) - Baserow is an open-source no-code database platform for building collaborative data apps. It makes it easy for teams to organize data and automate workflows without writing code. - [Bench](https://composio.dev/toolkits/bench) - Bench is a benchmarking tool for automated performance measurement and analysis. It helps you quickly evaluate, compare, and track your systems or workflows. - [Better stack](https://composio.dev/toolkits/better_stack) - Better Stack is a monitoring, logging, and incident management solution for apps and services. It helps teams ensure application reliability and performance with real-time insights. - [Bitbucket](https://composio.dev/toolkits/bitbucket) - Bitbucket is a Git-based code hosting and collaboration platform for teams. It enables secure repository management and streamlined code reviews. - [Blazemeter](https://composio.dev/toolkits/blazemeter) - Blazemeter is a continuous testing platform for web and mobile app performance. It empowers teams to automate and analyze large-scale tests with ease. - [Blocknative](https://composio.dev/toolkits/blocknative) - Blocknative delivers real-time mempool monitoring and transaction management for public blockchains. Instantly track pending transactions and optimize blockchain interactions with live data. ## Frequently Asked Questions ### What are the differences in Tool Router MCP and Zenrows MCP? With a standalone Zenrows MCP server, the agents and LLMs can only access a fixed set of Zenrows tools tied to that server. However, with the Composio Tool Router, agents can dynamically load tools from Zenrows and many other apps based on the task at hand, all through a single MCP endpoint. ### Can I use Tool Router MCP with Pydantic AI? Yes, you can. Pydantic AI fully supports MCP integration. You get structured tool calling, message history handling, and model orchestration while Tool Router takes care of discovering and serving the right Zenrows tools. ### Can I manage the permissions and scopes for Zenrows while using Tool Router? Yes, absolutely. You can configure which Zenrows scopes and actions are allowed when connecting your account to Composio. You can also bring your own OAuth credentials or API configuration so you keep full control over what the agent can do. ### How safe is my data with Composio Tool Router? All sensitive data such as tokens, keys, and configuration is fully encrypted at rest and in transit. Composio is SOC 2 Type 2 compliant and follows strict security practices so your Zenrows data and credentials are handled as safely as possible. --- [See all toolkits](https://composio.dev/toolkits) · [Composio docs](https://docs.composio.dev/llms.txt)