How to integrate Scrapingdog MCP with Hermes

Connect Hermes to Scrapingdog MCP. Scrape product prices from amazon listings, search google for competitor websites, and more using natural language, with authentication handled for you.

Scrapingdog logoScrapingdog
Api Key

Scrapingdog is a web scraping API for websites, search engines, marketplaces, and social platforms. It returns structured data with proxy rotation and JavaScript rendering built in.

8 Tools

Introduction

Hermes is a 24/7 autonomous agent that lives on your computer or server — it remembers what it learns and evolves as your usage grows.

This guide explains the easiest and most robust way to connect your Scrapingdog account to Hermes. You can do this through either Composio Connect CLI or Composio Connect MCP. For personal use we recommend the CLI, but you won't go wrong with MCP either.

For exact tool slugs and parameter tables, see the tool catalog in the Composio docs.

Also integrate Scrapingdog with

What is Composio Connect?

Composio Connect is a consumer offering that lets anyone plug 1,700+ applications directly into their agent harness — including Hermes. It can:

  • Search and load tools from relevant toolkits on-demand, reducing context usage.
  • Chain multiple tools to accomplish complex workflows via a remote workbench, without excessive back-and-forth with the LLM.
  • Manage app authentication end-to-end with zero manual overhead.

Integrating Scrapingdog with Hermes

Using Composio Connect CLI

1. Install the Composio CLI

Run the install script directly, or paste https://composio.dev/hermes into your Hermes chat box to have it installed for you.

bash
curl -fsSL https://composio.dev/install | bash
Hermes authenticating with Composio

2. Authenticate

Once the CLI is installed, ask Hermes to authenticate with Composio.

3. Connect to Scrapingdog

Ask your agent to connect to Scrapingdog, or simply request any Scrapingdog-related task. Hermes will prompt you to authenticate and authorize access.

4. Done. You're all set with a new Scrapingdog connection.


Using Composio Connect MCP

1. Get your MCP URL and API Key

Go to dashboard.composio.dev and copy your Connect MCP URL and API key.

Copy MCP URL and API key from Composio dashboard

2. Open the Hermes config file

bash
nano ~/.hermes/config.yaml

3. Add the Composio Connect MCP server

bash
mcp_servers:
  composio:
    url: "https://connect.composio.dev/mcp"
    headers:
      x-consumer-api-key: "YOUR_COMPOSIO_API_KEY"
    connect_timeout: 60
    timeout: 180

Save with Ctrl + O, Enter, then exit with Ctrl + X.

4. Restart your Hermes agent

Once restarted, ask your agent to connect to Scrapingdog or request any Scrapingdog-related task. It will prompt you to authenticate and authorize access.

5. Done!

What is the Scrapingdog MCP server, and what's possible with it?

The Scrapingdog MCP server is an implementation of the Model Context Protocol that connects your AI agent and assistants like Claude, Cursor, etc directly to your Scrapingdog account. It provides structured and secure access to web scraping and search tools, so your agent can scrape webpages, search Google and Google Maps, find news, capture screenshots, and check API usage on your behalf.

  • Webpage data extraction: Have your agent scrape a target webpage as HTML, Markdown, a summary, links, or images.
  • Search and local business research: Let the agent run Google searches or find structured business listings through Google Maps.
  • News monitoring: Direct your agent to search Google News by topic, publication, keyword, or date and gather headlines, sources, snippets, and timestamps.
  • Webpage screenshots: Instruct your agent to capture a webpage as a PNG, JPG, or WebP image.
  • Usage tracking and form submissions: Have your agent check account usage and concurrency metrics, or send URL-encoded form data to a target webpage through Scrapingdog.

Way Forward

With Scrapingdog connected, Hermes can now act on your behalf whenever it detects a relevant task or you ask it to.

From here, you can extend Hermes further:

  • Connect more apps: Calendar, Slack, Notion, Linear, and hundreds of others are available through the same Composio Connect setup. Each new integration compounds what Hermes can do for you.
  • Build workflows across tools: Once multiple apps are connected, Hermes can chain actions together — turn an email into a calendar invite, a Slack message into a Linear ticket, or a meeting note into a follow-up draft.
  • Let it learn your patterns: The more you use Hermes, the better it gets at anticipating how you'd handle recurring tasks. Give it feedback on drafts and decisions, and it will adapt.

If you run into trouble or want to share what you've built, join the community or check out the Docs for deeper configuration options.

TOOLS

Supported Tools

Every Scrapingdog action and event your agent gets out of the box.

Get Account Usage

Get free account usage and concurrency metrics without exposing the API key.

Google Maps Search

Search Google Maps and return structured business listings.

Google News Search

Browse Google News by topic or publication token, or run a simple keyword query with absolute dates.

Google News SERP Search

Search the Google News SERP for headlines, snippets, sources, and relative timestamps.

Google Search

Search Google and return Scrapingdog's structured result data.

Scrape URL

Scrape a URL as HTML, Markdown, a summary, links, or images.

Scrape URL with POST

Send URL-encoded form data to a target URL through Scrapingdog and return its response.

Take Screenshot

Capture a webpage screenshot as a downloadable PNG, JPG, or WebP image for 5 credits.

FAQ

Frequently asked questions

Start with Scrapingdog.It takes 30 seconds.

Managed auth, hosted MCP servers, and every Scrapingdog tool your agent needs.Free to start.

Start building