SIGN UPLOG IN
X (Twitter)LinkedInGitHubDiscordYouTube

How to integrate Webscraping ai MCP with Google ADK

Connect Google ADK to Webscraping ai MCP. Extract product prices from amazon listings, get latest headlines from bbc homepage, and more using natural language, with authentication handled for you.

Get started for freeGet a demo
Webscraping ai logoWebscraping ai
Api Key

Webscraping ai is a powerful API for extracting website data with JS rendering and proxy rotation. It lets you reliably scrape and parse dynamic web content at scale.

7 Tools

Introduction

This guide walks you through connecting Webscraping ai to Google ADK using the Composio tool router. By the end, you'll have a working Webscraping ai agent that can fetch rendered HTML and text, extract fields with AI, ask questions about web pages, and check your quota through natural language commands.

This guide will help you understand how to give your Google ADK agent real control over a Webscraping ai account through Composio's Webscraping ai MCP server.

Before we dive in, let's take a quick look at the key ideas and tools involved.

Also integrate Webscraping ai with

Claude CodeClaude CoworkOpenAI Agents SDKClaude Agent SDKCodexOpenClawHermesAtomic AgentLangChainVercel AI SDKMastra AILlamaIndexCrewAIPydantic AIAutoGen

TL;DR

Here's what you'll learn:
  • Get a Webscraping ai account set up and connected to Composio
  • Install the Google ADK and Composio packages
  • Create a Composio Tool Router session for Webscraping ai
  • Build an agent that connects to Webscraping ai through MCP
  • Interact with Webscraping ai using natural language

What is Google ADK?

Google ADK (Agents Development Kit) is Google's framework for building AI agents powered by Gemini models. It provides tools for creating agents that can use external services through the Model Context Protocol.

Key features include:

  • Gemini Integration: Native support for Google's Gemini models
  • MCP Toolset: Built-in support for Model Context Protocol tools
  • Streamable HTTP: Connect to external services through streamable HTTP
  • CLI and Web UI: Run agents via command line or web interface

What is the Webscraping ai MCP server, and what's possible with it?

The Webscraping ai MCP server is an implementation of the Model Context Protocol that connects your AI agent and assistants like Claude, Cursor, etc directly to your Webscraping ai account. It provides structured and secure access to web scraping capabilities, so your agent can retrieve raw HTML, extract rendered web content, parse text, and monitor your API usage on your behalf.

  • Retrieve raw HTML from any webpage: Let your agent fetch the unprocessed HTML content of any website for analysis or processing.
  • Extract fully rendered HTML with JavaScript support: Have your agent capture the final, JavaScript-rendered HTML—great for sites that rely on dynamic content.
  • Get plain text from web pages: Direct your agent to extract just the readable text from any URL, skipping all the markup and scripts.
  • Monitor API usage and quota: Your agent can check your account's API call limits and current usage, so you always know where you stand.

What is the Composio tool router, and how does it fit here?

What is Composio SDK?

Composio's Composio SDK helps agents find the right tools for a task at runtime. You can plug in multiple toolkits (like Gmail, HubSpot, and GitHub), and the agent will identify the relevant app and action to complete multi-step workflows. This can reduce token usage and improve the reliability of tool calls. Read more here: Getting started with Composio SDK

The tool router generates a secure MCP URL that your agents can access to perform actions.

How the Composio SDK works

The Composio SDK follows a three-phase workflow:

  1. Discovery: Searches for tools matching your task and returns relevant toolkits with their details.
  2. Authentication: Checks for active connections. If missing, creates an auth config and returns a connection URL via Auth Link.
  3. Execution: Executes the action using the authenticated connection.

Step-by-step Guide

Step by step09 STEPS
1

Prerequisites

Before starting, make sure you have:
  • A Google API key for Gemini models
  • A Composio account and API key
  • Python 3.9 or later installed
  • Basic familiarity with Python
2

Getting API Keys for Google and Composio

Google API Key
  • Go to Google AI Studio and create an API key.
  • Copy the key and keep it safe. You will put this in GOOGLE_API_KEY.
Composio API Key and User ID
  • Log in to the Composio dashboard.
  • Go to Settings → API Keys and copy your Composio API key. Use this for COMPOSIO_API_KEY.
  • Decide on a stable user identifier to scope sessions, often your email or a user ID. Use this for COMPOSIO_USER_ID.
3

Install dependencies

bash
pip install google-adk composio python-dotenv

Inside your virtual environment, install the required packages.

What's happening:

  • google-adk is Google's Agents Development Kit
  • composio connects your agent to Webscraping ai via MCP
  • python-dotenv loads environment variables
4

Set up ADK project

bash
adk create my_agent

Set up a new Google ADK project.

What's happening:

  • This creates an agent folder with a root agent file and .env file
5

Set environment variables

bash
GOOGLE_API_KEY=your-google-api-key
COMPOSIO_API_KEY=your-composio-api-key
COMPOSIO_USER_ID=your-user-id-or-email

Save all your credentials in the .env file.

What's happening:

  • GOOGLE_API_KEY authenticates with Google's Gemini models
  • COMPOSIO_API_KEY authenticates with Composio
  • COMPOSIO_USER_ID identifies the user for session management
6

Import modules and validate environment

python
import os
import warnings

from composio import Composio
from dotenv import load_dotenv
from google.adk.agents.llm_agent import Agent
from google.adk.tools.mcp_tool.mcp_session_manager import StreamableHTTPConnectionParams
from google.adk.tools.mcp_tool.mcp_toolset import McpToolset

load_dotenv()

warnings.filterwarnings("ignore", message=".*BaseAuthenticatedTool.*")

GOOGLE_API_KEY = os.getenv("GOOGLE_API_KEY")
COMPOSIO_API_KEY = os.getenv("COMPOSIO_API_KEY")
COMPOSIO_USER_ID = os.getenv("COMPOSIO_USER_ID")

if not GOOGLE_API_KEY:
    raise ValueError("GOOGLE_API_KEY is not set in the environment.")
if not COMPOSIO_API_KEY:
    raise ValueError("COMPOSIO_API_KEY is not set in the environment.")
if not COMPOSIO_USER_ID:
    raise ValueError("COMPOSIO_USER_ID is not set in the environment.")
What's happening:
  • os reads environment variables
  • Composio is the main Composio SDK client
  • GoogleProvider declares that you are using Google ADK as the agent runtime
  • Agent is the Google ADK LLM agent class
  • McpToolset lets the ADK agent call MCP tools over HTTP
7

Create Composio client and Tool Router session

python
composio_client = Composio(api_key=COMPOSIO_API_KEY)

composio_session = composio_client.create(
    user_id=COMPOSIO_USER_ID,
    toolkits=["webscraping_ai"],
)

COMPOSIO_MCP_URL = composio_session.mcp.url,
print(f"Composio MCP URL: {COMPOSIO_MCP_URL}")
What's happening:
  • Authenticates to Composio with your API key
  • Declares Google ADK as the provider
  • Spins up a short-lived MCP endpoint for your user and selected toolkit
  • Stores the MCP HTTP URL for the ADK MCP integration
8

Set up the McpToolset and create the Agent

python
composio_toolset = McpToolset(
    connection_params=StreamableHTTPConnectionParams(
        url=COMPOSIO_MCP_URL,
        headers={"x-api-key": COMPOSIO_API_KEY}
    )
)

root_agent = Agent(
    model="gemini-2.5-flash",
    name="composio_agent",
    description="An agent that uses Composio tools to perform actions.",
    instruction=(
        "You are a helpful assistant connected to Composio. "
        "You have the following tools available: "
        "COMPOSIO_SEARCH_TOOLS, COMPOSIO_MULTI_EXECUTE_TOOL, "
        "COMPOSIO_MANAGE_CONNECTIONS, COMPOSIO_REMOTE_BASH_TOOL, COMPOSIO_REMOTE_WORKBENCH. "
        "Use these tools to help users with Webscraping ai operations."
    ),
    tools=[composio_toolset],
)

print("\nAgent setup complete. You can now run this agent directly ;)")
What's happening:
  • Connects the ADK agent to the Composio MCP endpoint through McpToolset
  • Uses Gemini as the model powering the agent
  • Lists exact tool names in instruction to reduce misnamed tool calls
9

Run the agent

bash
# Run in CLI mode
adk run my_agent

# Or run in web UI mode
adk web

Execute the agent from the project root. The web command opens a web portal where you can chat with the agent.

What's happening:

  • adk run runs the agent in CLI mode
  • adk web . opens a web UI for interactive testing

Complete Code

Here's the complete code to get you started with Webscraping ai and Google ADK:

python
import os
import warnings

from composio import Composio
from composio_google import GoogleProvider
from dotenv import load_dotenv
from google.adk.agents.llm_agent import Agent
from google.adk.tools.mcp_tool.mcp_session_manager import StreamableHTTPConnectionParams
from google.adk.tools.mcp_tool.mcp_toolset import McpToolset

load_dotenv()
warnings.filterwarnings("ignore", message=".*BaseAuthenticatedTool.*")

GOOGLE_API_KEY = os.getenv("GOOGLE_API_KEY")
COMPOSIO_API_KEY = os.getenv("COMPOSIO_API_KEY")
COMPOSIO_USER_ID = os.getenv("COMPOSIO_USER_ID")

if not GOOGLE_API_KEY:
    raise ValueError("GOOGLE_API_KEY is not set in the environment.")
if not COMPOSIO_API_KEY:
    raise ValueError("COMPOSIO_API_KEY is not set in the environment.")
if not COMPOSIO_USER_ID:
    raise ValueError("COMPOSIO_USER_ID is not set in the environment.")

composio_client = Composio(api_key=COMPOSIO_API_KEY, provider=GoogleProvider())

composio_session = composio_client.create(
    user_id=COMPOSIO_USER_ID,
    toolkits=["webscraping_ai"],
)

COMPOSIO_MCP_URL = composio_session.mcp.url


composio_toolset = McpToolset(
    connection_params=StreamableHTTPConnectionParams(
        url=COMPOSIO_MCP_URL,
        headers={"x-api-key": COMPOSIO_API_KEY}
    )
)

root_agent = Agent(
    model="gemini-2.5-flash",
    name="composio_agent",
    description="An agent that uses Composio tools to perform actions.",
    instruction=(
        "You are a helpful assistant connected to Composio. "
        "You have the following tools available: "
        "COMPOSIO_SEARCH_TOOLS, COMPOSIO_MULTI_EXECUTE_TOOL, "
        "COMPOSIO_MANAGE_CONNECTIONS, COMPOSIO_REMOTE_BASH_TOOL, COMPOSIO_REMOTE_WORKBENCH. "
        "Use these tools to help users with Webscraping ai operations."
    ),  
    tools=[composio_toolset],
)

print("\nAgent setup complete. You can now run this agent directly ;)")

Conclusion

You've successfully integrated Webscraping ai with the Google ADK through Composio's MCP Tool Router. Your agent can now interact with Webscraping ai using natural language commands.

Key takeaways:

  • The Tool Router approach dynamically routes requests to the appropriate Webscraping ai tools
  • Environment variables keep your credentials secure and separate from code
  • Clear agent instructions reduce tool calling errors
  • The ADK web UI provides an interactive interface for testing and development

You can extend this setup by adding more toolkits to the toolkits array in your session configuration.

TOOLS

Supported Tools

Every Webscraping ai action and event your agent gets out of the box.

Get account usage and quota

Tool to retrieve account API call quota and usage.

Ask Question About Web Page

Get an LLM answer to a question about a web page for AI-powered analysis or extraction.

Extract Fields with AI

Tool to extract structured data fields from a web page using AI.

Get Rendered HTML

Tool to retrieve fully rendered HTML of a webpage.

Get Selected HTML

Tool to extract HTML from specific page elements using CSS selectors.

Get Selected Multiple Elements

Extract HTML from multiple areas of a web page using its URL and CSS selectors, without parsing the page yourself.

Get Text

Retrieve text content from a specified web page.

FRAMEWORKS

How to build Webscraping ai MCP Agent with another framework

Claude Code logo

Claude Code

Use Webscraping ai MCP with Claude Code

Claude Cowork logo

Claude Cowork

Use Webscraping ai MCP with Claude Cowork

OpenAI Agents SDK logo

OpenAI Agents SDK

Use Webscraping ai MCP with OpenAI Agents SDK

Claude Agent SDK logo

Claude Agent SDK

Use Webscraping ai MCP with Claude Agent SDK

Codex logo

Codex

Use Webscraping ai MCP with Codex

OpenClaw logo

OpenClaw

Use Webscraping ai MCP with OpenClaw

Hermes logo

Hermes

Use Webscraping ai MCP with Hermes

Atomic Agent logo

Atomic Agent

Use Webscraping ai MCP with Atomic Agent

LangChain logo

LangChain

Use Webscraping ai MCP with LangChain

Vercel AI SDK logo

Vercel AI SDK

Use Webscraping ai MCP with Vercel AI SDK

Mastra AI logo

Mastra AI

Use Webscraping ai MCP with Mastra AI

LlamaIndex logo

LlamaIndex

Use Webscraping ai MCP with LlamaIndex

CrewAI logo

CrewAI

Use Webscraping ai MCP with CrewAI

Pydantic AI logo

Pydantic AI

Use Webscraping ai MCP with Pydantic AI

AutoGen logo

AutoGen

Use Webscraping ai MCP with AutoGen

FAQ

Frequently asked questions

With a standalone Webscraping ai MCP server, the agents and LLMs can only access a fixed set of Webscraping ai tools tied to that server. However, with the Composio Tool Router, agents can dynamically load tools from Webscraping ai and many other apps based on the task at hand, all through a single MCP endpoint.

Yes, you can. Google ADK fully supports MCP integration. You get structured tool calling, message history handling, and model orchestration while Tool Router takes care of discovering and serving the right Webscraping ai tools.

Yes, absolutely. You can configure which Webscraping ai scopes and actions are allowed when connecting your account to Composio. You can also bring your own OAuth credentials or API configuration so you keep full control over what the agent can do.

All sensitive data such as tokens, keys, and configuration is fully encrypted at rest and in transit. Composio is SOC 2 Type 2 compliant and follows strict security practices so your Webscraping ai data and credentials are handled as safely as possible.

Start with Webscraping ai.It takes 30 seconds.

Managed auth, hosted MCP servers, and every Webscraping ai tool your agent needs.Free to start.

Start building