# How to integrate Big data cloud MCP with CrewAI ```json { "title": "How to integrate Big data cloud MCP with CrewAI", "toolkit": "Big data cloud", "toolkit_slug": "big_data_cloud", "framework": "CrewAI", "framework_slug": "crew-ai", "url": "https://composio.dev/toolkits/big_data_cloud/framework/crew-ai", "markdown_url": "https://composio.dev/toolkits/big_data_cloud/framework/crew-ai.md", "updated_at": "2026-05-12T10:03:00.370Z" } ``` ## Introduction This guide walks you through connecting Big data cloud to CrewAI using the Composio tool router. By the end, you'll have a working Big data cloud agent that can check if this ip address is currently roaming, verify if an email address is valid, get country and demographic info for a given ip through natural language commands. This guide will help you understand how to give your CrewAI agent real control over a Big data cloud account through Composio's Big data cloud MCP server. Before we dive in, let's take a quick look at the key ideas and tools involved. ## Also integrate Big data cloud with - [OpenAI Agents SDK](https://composio.dev/toolkits/big_data_cloud/framework/open-ai-agents-sdk) - [Claude Agent SDK](https://composio.dev/toolkits/big_data_cloud/framework/claude-agents-sdk) - [Claude Code](https://composio.dev/toolkits/big_data_cloud/framework/claude-code) - [Claude Cowork](https://composio.dev/toolkits/big_data_cloud/framework/claude-cowork) - [Codex](https://composio.dev/toolkits/big_data_cloud/framework/codex) - [OpenClaw](https://composio.dev/toolkits/big_data_cloud/framework/openclaw) - [Hermes](https://composio.dev/toolkits/big_data_cloud/framework/hermes-agent) - [CLI](https://composio.dev/toolkits/big_data_cloud/framework/cli) - [Google ADK](https://composio.dev/toolkits/big_data_cloud/framework/google-adk) - [LangChain](https://composio.dev/toolkits/big_data_cloud/framework/langchain) - [Vercel AI SDK](https://composio.dev/toolkits/big_data_cloud/framework/ai-sdk) - [Mastra AI](https://composio.dev/toolkits/big_data_cloud/framework/mastra-ai) - [LlamaIndex](https://composio.dev/toolkits/big_data_cloud/framework/llama-index) ## TL;DR Here's what you'll learn: - Get a Composio API key and configure your Big data cloud connection - Set up CrewAI with an MCP enabled agent - Create a Tool Router session or standalone MCP server for Big data cloud - Build a conversational loop where your agent can execute Big data cloud operations ## What is CrewAI? CrewAI is a powerful framework for building multi-agent AI systems. It provides primitives for defining agents with specific roles, creating tasks, and orchestrating workflows through crews. Key features include: - Agent Roles: Define specialized agents with specific goals and backstories - Task Management: Create tasks with clear descriptions and expected outputs - Crew Orchestration: Combine agents and tasks into collaborative workflows - MCP Integration: Connect to external tools through Model Context Protocol ## What is the Big data cloud MCP server, and what's possible with it? The Big data cloud MCP server is an implementation of the Model Context Protocol that connects your AI agent and assistants like Claude, Cursor, etc directly to your Big data cloud account. It provides structured and secure access to advanced geolocation, reverse geocoding, ASN analysis, and data validation APIs, so your agent can perform actions like looking up IP details, verifying emails, assessing network risk, and analyzing BGP routing on your behalf. - IP geolocation and country insights: Let your agent instantly geolocate any IP address, retrieve country-level demographics, and pull rich metadata about locations worldwide. - Reverse geocoding with timezone detection: Have your agent translate GPS coordinates into precise locality information along with accurate timezone data—all in one go. - Email address verification and data hygiene: Ensure your agent can validate email addresses for proper syntax, domain legitimacy, and disposability to help maintain clean and reliable datasets. - ASN and BGP analytics: Allow your agent to analyze internet routing by fetching ranked lists of autonomous systems, upstream and downstream provider details, and active BGP prefixes for a given ASN. - Cybersecurity hazard assessment: Empower your agent to fetch and interpret hazard reports for IP addresses, identifying threats like VPN/proxy usage, blacklist status, and hosting risks. ## Supported Tools | Tool slug | Name | Description | |---|---|---| | `BIG_DATA_CLOUD_AM_I_ROAMING_API` | Am I Roaming API | Tool to determine if the user is roaming based on their IP address and GPS coordinates. Use after obtaining device location to verify roaming status before mobile actions. | | `BIG_DATA_CLOUD_ASN_EXTENDED_RECEIVING_FROM_INFO_API` | ASN Extended Receiving From Info API | Tool to return upstream providers (receivingFrom) for a given ASN. Use when you need a paginated list of ASes feeding traffic for the specified ASN. | | `BIG_DATA_CLOUD_ASN_EXTENDED_TRANSIT_TO_INFO_API` | ASN Extended Transit To Info API | Tool to return downstream customers (transitTo) for a given ASN. Use when you need a paginated list of ASes receiving traffic from a specific ASN. | | `BIG_DATA_CLOUD_ASN_RANK_LIST_API` | ASN Rank List API | Retrieves a ranked list of Autonomous Systems (ASNs) sorted by IPv4 address announcement volumes. Use cases: - Find the largest ASNs by IP address count (DoD, Amazon, Microsoft, etc.) - Look up ASN rankings for network analysis - Paginate through the global ASN database (79,000+ entries) - Sort ASNs by various criteria (rank, name, organisation, country) Returns paginated results with total count for navigation. | | `BIG_DATA_CLOUD_BGP_ACTIVE_PREFIXES_API` | BGP Active Prefixes API | Tool to retrieve IPv4 or IPv6 prefixes currently announced on BGP. Use when inspecting BGP routing announcements for a given ASN. | | `BIG_DATA_CLOUD_COUNTRY_BY_IP_ADDRESS_API` | Country by IP Address API | Tool to geolocate an IP address and retrieve country details and demographics. Use when you need country-level data after obtaining the target IP address. | | `BIG_DATA_CLOUD_COUNTRY_INFO_API` | Country Info API | Tool to fetch detailed country information by ISO code. Use when you need localized names, currencies, regions, and other metadata for a country. | | `BIG_DATA_CLOUD_EMAIL_ADDRESS_VERIFICATION_API` | Email Address Verification API | Tool to verify email addresses for syntax, domain validity, and disposability. Use after obtaining the email input. | | `BIG_DATA_CLOUD_HAZARD_REPORT_API` | Hazard Report API | Tool to fetch a cybersecurity hazard report for a specified IP address. Use when assessing an IP's threat profile (VPN, proxy, blacklists, hosting risk). | | `BIG_DATA_CLOUD_NETWORK_BY_CIDR` | Networks by CIDR | Tool to retrieve BGP-announced networks within a specified CIDR range. Use when you need to analyze network announcements within a particular CIDR after confirming the range format. | | `BIG_DATA_CLOUD_NETWORK_BY_IP_ADDRESS_API` | Network by IP Address API | Tool to retrieve registry, ASN, and BGP details for a given IP address’s network. Use when you need detailed network information (e.g., ASNs, prefixes) after confirming the target IP. | | `BIG_DATA_CLOUD_PHONE_NUMBER_VALIDATION_BY_IP` | Phone Number Validation by IP | Tool to validate phone numbers by inferring country from client IP. Use when you want to validate a number without specifying country. | | `BIG_DATA_CLOUD_REVERSE_GEOCODING_TIMEZONE_API` | Reverse Geocoding With Timezone API | Tool to return reverse geocoding and time zone info for given coordinates. Use when you need both locality details and timezone data in one call. | | `BIG_DATA_CLOUD_TIME_ZONE_BY_IP_ADDRESS_API` | Time Zone by IP Address API | Tool to retrieve time zone information for a given IP address. Use when you need DST status, UTC offsets, and local/UTC time for a specific IP. | | `BIG_DATA_CLOUD_TOR_EXIT_NODES_GEOLOCATED_API` | Tor Exit Nodes Geolocated API | Retrieve a paginated list of active TOR exit node IP addresses with geolocation and carrier (ASN) details. Use this tool to: - Get a list of known TOR exit node IPs to detect/block anonymous traffic - Analyze geographic distribution of TOR exit nodes by country - Look up carrier/ASN information for TOR nodes - Build IP blocklists or allowlists for TOR traffic Returns nodes with IP address, country info (when available), and detailed carrier/ASN data including BGP prefix counts and global ranking. | | `BIG_DATA_CLOUD_USER_AGENT_PARSER_API` | User Agent Parser API | Tool to parse a User-Agent string into device, OS, browser, and bot details. Use when you have a raw User-Agent header and need structured client info. | | `BIG_DATA_CLOUD_USER_RISK_API` | User Risk API | Tool to return a risk assessment for a user based on IP signals for fraud prevention. Use after initial IP checks to decide whether to bypass or require captcha challenges. | ## Supported Triggers None listed. ## Creating MCP Server - Stand-alone vs Composio SDK The Big data cloud MCP server is an implementation of the Model Context Protocol that connects your AI agent to Big data cloud. It provides structured and secure access so your agent can perform Big data cloud operations on your behalf through a secure, permission-based interface. With Composio's managed implementation, you don't have to create your own developer app. For production, if you're building an end product, we recommend using your own credentials. The managed server helps you prototype fast and go from 0-1 faster. ## Step-by-step Guide ### 1. Prerequisites Before starting, make sure you have: - Python 3.9 or higher - A Composio account and API key - A Big data cloud connection authorized in Composio - An OpenAI API key for the CrewAI LLM - Basic familiarity with Python ### 1. Getting API Keys for OpenAI and Composio OpenAI API Key - Go to the [OpenAI dashboard](https://platform.openai.com/settings/organization/api-keys) and create an API key. You'll need credits to use the models, or you can connect to another model provider. - Keep the API key safe. Composio API Key - Log in to the [Composio dashboard](https://dashboard.composio.dev?utm_source=toolkits&utm_medium=framework_docs). - Navigate to your API settings and generate a new API key. - Store this key securely as you'll need it for authentication. ### 2. Install dependencies **What's happening:** - composio connects your agent to Big data cloud via MCP - crewai provides Agent, Task, Crew, and LLM primitives - crewai-tools[mcp] includes MCP helpers - python-dotenv loads environment variables from .env ```bash pip install composio crewai crewai-tools[mcp] python-dotenv ``` ### 3. Set up environment variables Create a .env file in your project root. What's happening: - COMPOSIO_API_KEY authenticates with Composio - USER_ID scopes the session to your account - OPENAI_API_KEY lets CrewAI use your chosen OpenAI model ```bash COMPOSIO_API_KEY=your_composio_api_key_here USER_ID=your_user_id_here OPENAI_API_KEY=your_openai_api_key_here ``` ### 4. Import dependencies **What's happening:** - CrewAI classes define agents and tasks, and run the workflow - MCPServerHTTP connects the agent to an MCP endpoint - Composio will give you a short lived Big data cloud MCP URL ```python import os from composio import Composio from crewai import Agent, Task, Crew from crewai_tools import MCPServerAdapter import dotenv dotenv.load_dotenv() COMPOSIO_API_KEY = os.getenv("COMPOSIO_API_KEY") COMPOSIO_USER_ID = os.getenv("COMPOSIO_USER_ID") if not COMPOSIO_API_KEY: raise ValueError("COMPOSIO_API_KEY is not set") if not COMPOSIO_USER_ID: raise ValueError("COMPOSIO_USER_ID is not set") ``` ### 5. Create a Composio Tool Router session for Big data cloud **What's happening:** - You create a Big data cloud only session through Composio - Composio returns an MCP HTTP URL that exposes Big data cloud tools ```python composio_client = Composio(api_key=COMPOSIO_API_KEY) session = composio_client.create(user_id=COMPOSIO_USER_ID, toolkits=["big_data_cloud"]) url = session.mcp.url ``` ### 6. Initialize the MCP Server **What's Happening:** - Server Configuration: The code sets up connection parameters including the MCP server URL, streamable HTTP transport, and Composio API key authentication. - MCP Adapter Bridge: MCPServerAdapter acts as a context manager that converts Composio MCP tools into a CrewAI-compatible format. - Agent Setup: Creates a CrewAI Agent with a defined role (Search Assistant), goal (help with internet searches), and access to the MCP tools. - Configuration Options: The agent includes settings like verbose=False for clean output and max_iter=10 to prevent infinite loops. - Dynamic Tool Usage: Once created, the agent automatically accesses all Composio Search tools and decides when to use them based on user queries. ```python server_params = { "url": url, "transport": "streamable-http", "headers": {"x-api-key": COMPOSIO_API_KEY}, } with MCPServerAdapter(server_params) as tools: agent = Agent( role="Search Assistant", goal="Help users search the internet effectively", backstory="You are a helpful assistant with access to search tools.", tools=tools, verbose=False, max_iter=10, ) ``` ### 7. Create a CLI Chatloop and define the Crew **What's Happening:** - Interactive CLI Setup: The code creates an infinite loop that continuously prompts for user input and maintains the entire conversation history in a string variable. - Input Validation: Empty inputs are ignored to prevent processing blank messages and keep the conversation clean. - Context Building: Each user message is appended to the conversation context, which preserves the full dialogue history for better agent responses. - Dynamic Task Creation: For every user input, a new Task is created that includes both the full conversation history and the current request as context. - Crew Execution: A Crew is instantiated with the agent and task, then kicked off to process the request and generate a response. - Response Management: The agent's response is converted to a string, added to the conversation context, and displayed to the user, maintaining conversational continuity. ```python print("Chat started! Type 'exit' or 'quit' to end.\n") conversation_context = "" while True: user_input = input("You: ").strip() if user_input.lower() in ["exit", "quit", "bye"]: print("\nGoodbye!") break if not user_input: continue conversation_context += f"\nUser: {user_input}\n" print("\nAgent is thinking...\n") task = Task( description=( f"Conversation history:\n{conversation_context}\n\n" f"Current request: {user_input}" ), expected_output="A helpful response addressing the user's request", agent=agent, ) crew = Crew(agents=[agent], tasks=[task], verbose=False) result = crew.kickoff() response = str(result) conversation_context += f"Agent: {response}\n" print(f"Agent: {response}\n") ``` ## Complete Code ```python from crewai import Agent, Task, Crew, LLM from crewai_tools import MCPServerAdapter from composio import Composio from dotenv import load_dotenv import os load_dotenv() GOOGLE_API_KEY = os.getenv("GOOGLE_API_KEY") COMPOSIO_API_KEY = os.getenv("COMPOSIO_API_KEY") COMPOSIO_USER_ID = os.getenv("COMPOSIO_USER_ID") if not GOOGLE_API_KEY: raise ValueError("GOOGLE_API_KEY is not set in the environment.") if not COMPOSIO_API_KEY: raise ValueError("COMPOSIO_API_KEY is not set in the environment.") if not COMPOSIO_USER_ID: raise ValueError("COMPOSIO_USER_ID is not set in the environment.") # Initialize Composio and create a session composio = Composio(api_key=COMPOSIO_API_KEY) session = composio.create( user_id=COMPOSIO_USER_ID, toolkits=["big_data_cloud"], ) url = session.mcp.url # Configure LLM llm = LLM( model="gpt-5", api_key=os.getenv("OPENAI_API_KEY"), ) server_params = { "url": url, "transport": "streamable-http", "headers": {"x-api-key": COMPOSIO_API_KEY}, } with MCPServerAdapter(server_params) as tools: agent = Agent( role="Search Assistant", goal="Help users with internet searches", backstory="You are an expert assistant with access to Composio Search tools.", tools=tools, llm=llm, verbose=False, max_iter=10, ) print("Chat started! Type 'exit' or 'quit' to end.\n") conversation_context = "" while True: user_input = input("You: ").strip() if user_input.lower() in ["exit", "quit", "bye"]: print("\nGoodbye!") break if not user_input: continue conversation_context += f"\nUser: {user_input}\n" print("\nAgent is thinking...\n") task = Task( description=( f"Conversation history:\n{conversation_context}\n\n" f"Current request: {user_input}" ), expected_output="A helpful response addressing the user's request", agent=agent, ) crew = Crew(agents=[agent], tasks=[task], verbose=False) result = crew.kickoff() response = str(result) conversation_context += f"Agent: {response}\n" print(f"Agent: {response}\n") ``` ## Conclusion You now have a CrewAI agent connected to Big data cloud through Composio's Tool Router. The agent can perform Big data cloud operations through natural language commands. Next steps: - Add role-specific instructions to customize agent behavior - Plug in more toolkits for multi-app workflows - Chain tasks for complex multi-step operations ## How to build Big data cloud MCP Agent with another framework - [OpenAI Agents SDK](https://composio.dev/toolkits/big_data_cloud/framework/open-ai-agents-sdk) - [Claude Agent SDK](https://composio.dev/toolkits/big_data_cloud/framework/claude-agents-sdk) - [Claude Code](https://composio.dev/toolkits/big_data_cloud/framework/claude-code) - [Claude Cowork](https://composio.dev/toolkits/big_data_cloud/framework/claude-cowork) - [Codex](https://composio.dev/toolkits/big_data_cloud/framework/codex) - [OpenClaw](https://composio.dev/toolkits/big_data_cloud/framework/openclaw) - [Hermes](https://composio.dev/toolkits/big_data_cloud/framework/hermes-agent) - [CLI](https://composio.dev/toolkits/big_data_cloud/framework/cli) - [Google ADK](https://composio.dev/toolkits/big_data_cloud/framework/google-adk) - [LangChain](https://composio.dev/toolkits/big_data_cloud/framework/langchain) - [Vercel AI SDK](https://composio.dev/toolkits/big_data_cloud/framework/ai-sdk) - [Mastra AI](https://composio.dev/toolkits/big_data_cloud/framework/mastra-ai) - [LlamaIndex](https://composio.dev/toolkits/big_data_cloud/framework/llama-index) ## Related Toolkits - [Excel](https://composio.dev/toolkits/excel) - Microsoft Excel is a robust spreadsheet application for organizing, analyzing, and visualizing data. It's the go-to tool for calculations, reporting, and flexible data management. - [21risk](https://composio.dev/toolkits/_21risk) - 21RISK is a web app built for easy checklist, audit, and compliance management. It streamlines risk processes so teams can focus on what matters. - [Abstract](https://composio.dev/toolkits/abstract) - Abstract provides a suite of APIs for automating data validation and enrichment tasks. It helps developers streamline workflows and ensure data quality with minimal effort. - [Addressfinder](https://composio.dev/toolkits/addressfinder) - Addressfinder is a data quality platform for verifying addresses, emails, and phone numbers. It helps you ensure accurate customer and contact data every time. - [Agentql](https://composio.dev/toolkits/agentql) - Agentql is a toolkit that connects AI agents to the web using a specialized query language. It enables structured web interaction and data extraction for smarter automations. - [Agenty](https://composio.dev/toolkits/agenty) - Agenty is a web scraping and automation platform for extracting data and automating browser tasks—no coding needed. It streamlines data collection, monitoring, and repetitive online actions. - [Ambee](https://composio.dev/toolkits/ambee) - Ambee is an environmental data platform providing real-time, hyperlocal APIs for air quality, weather, and pollen. Get precise environmental insights to power smarter decisions in your apps and workflows. - [Ambient weather](https://composio.dev/toolkits/ambient_weather) - Ambient Weather is a platform for personal weather stations with a robust API for accessing local, real-time, and historical weather data. Get detailed environmental insights directly from your own sensors for smarter apps and automations. - [Anonyflow](https://composio.dev/toolkits/anonyflow) - Anonyflow is a service for encryption-based data anonymization and secure data sharing. It helps organizations meet GDPR, CCPA, and HIPAA data privacy compliance requirements. - [Api ninjas](https://composio.dev/toolkits/api_ninjas) - Api ninjas offers 120+ public APIs spanning categories like weather, finance, sports, and more. Developers use it to supercharge apps with real-time data and actionable endpoints. - [Api sports](https://composio.dev/toolkits/api_sports) - Api sports is a comprehensive sports data platform covering 2,000+ competitions with live scores and 15+ years of stats. Instantly access up-to-date sports information for analysis, apps, or chatbots. - [Apify](https://composio.dev/toolkits/apify) - Apify is a cloud platform for building, deploying, and managing web scraping and automation tools called Actors. It lets you automate data extraction and workflow tasks at scale—no infrastructure headaches. - [Autom](https://composio.dev/toolkits/autom) - Autom is a lightning-fast search engine results data platform for Google, Bing, and Brave. Developers use it to access fresh, low-latency SERP data on demand. - [Beaconchain](https://composio.dev/toolkits/beaconchain) - Beaconchain is a real-time analytics platform for Ethereum 2.0's Beacon Chain. It provides detailed insights into validators, blocks, and overall network performance. - [Bigpicture io](https://composio.dev/toolkits/bigpicture_io) - BigPicture.io offers APIs for accessing detailed company and profile data. Instantly enrich your applications with up-to-date insights on 20M+ businesses. - [Bitquery](https://composio.dev/toolkits/bitquery) - Bitquery is a blockchain data platform offering indexed, real-time, and historical data from 40+ blockchains via GraphQL APIs. Get unified, reliable access to complex on-chain data for analytics, trading, and research. - [Brightdata](https://composio.dev/toolkits/brightdata) - Brightdata is a leading web data platform offering advanced scraping, SERP APIs, and anti-bot tools. It lets you collect public web data at scale, bypassing blocks and friction. - [Builtwith](https://composio.dev/toolkits/builtwith) - BuiltWith is a web technology profiler that uncovers the technologies powering any website. Gain actionable insights into analytics, hosting, and content management stacks for smarter research and lead generation. - [Byteforms](https://composio.dev/toolkits/byteforms) - Byteforms is an all-in-one platform for creating forms, managing submissions, and integrating data. It streamlines workflows by centralizing form data collection and automation. - [Cabinpanda](https://composio.dev/toolkits/cabinpanda) - Cabinpanda is a data collection platform for building and managing online forms. It helps streamline how you gather, organize, and analyze responses. ## Frequently Asked Questions ### What are the differences in Tool Router MCP and Big data cloud MCP? With a standalone Big data cloud MCP server, the agents and LLMs can only access a fixed set of Big data cloud tools tied to that server. However, with the Composio Tool Router, agents can dynamically load tools from Big data cloud and many other apps based on the task at hand, all through a single MCP endpoint. ### Can I use Tool Router MCP with CrewAI? Yes, you can. CrewAI fully supports MCP integration. You get structured tool calling, message history handling, and model orchestration while Tool Router takes care of discovering and serving the right Big data cloud tools. ### Can I manage the permissions and scopes for Big data cloud while using Tool Router? Yes, absolutely. You can configure which Big data cloud scopes and actions are allowed when connecting your account to Composio. You can also bring your own OAuth credentials or API configuration so you keep full control over what the agent can do. ### How safe is my data with Composio Tool Router? All sensitive data such as tokens, keys, and configuration is fully encrypted at rest and in transit. Composio is SOC 2 Type 2 compliant and follows strict security practices so your Big data cloud data and credentials are handled as safely as possible. --- [See all toolkits](https://composio.dev/toolkits) · [Composio docs](https://docs.composio.dev/llms.txt)