Spider is a web scraping and data extraction tool that allows users to crawl websites, extract data, and export the results to various formats.
๐Ÿ”— Connect and Use Spider
1. ๐Ÿ”‘ Connect your Spider account
2. โœ… Select an action
3. ๐Ÿš€ Go live with the agent
What do you want to do?

API actions for Spider for AI assitants/agents

Language
JS
PYTHON
Framework

Web Crawl

Initiates a web crawl starting from a specified URL, following links to discover and index web pages.
from composio_crewai import ComposioToolSet, Action tool_set = ComposioToolSet() tools = tool_set.get_tools(actions=[Action.SPIDER_WEB_CRAWL])

Extract Data

Extracts specific data from web pages based on predefined patterns or selectors.
from composio_crewai import ComposioToolSet, Action tool_set = ComposioToolSet() tools = tool_set.get_tools(actions=[Action.SPIDER_EXTRACT_DATA])

Generate Sitemap

Creates a sitemap of a website by crawling and mapping its structure.
from composio_crewai import ComposioToolSet, Action tool_set = ComposioToolSet() tools = tool_set.get_tools(actions=[Action.SPIDER_GENERATE_SITEMAP])

Capture Screenshots

Takes screenshots of specified web pages during the crawling process.
from composio_crewai import ComposioToolSet, Action tool_set = ComposioToolSet() tools = tool_set.get_tools(actions=[Action.SPIDER_CAPTURE_SCREENSHOTS])

Download Files

Downloads files of specified types (e.g., PDFs, images) encountered during web crawling.
from composio_crewai import ComposioToolSet, Action tool_set = ComposioToolSet() tools = tool_set.get_tools(actions=[Action.SPIDER_DOWNLOAD_FILES])

Analyze Page Speed

Measures and reports the loading speed of crawled web pages.
from composio_crewai import ComposioToolSet, Action tool_set = ComposioToolSet() tools = tool_set.get_tools(actions=[Action.SPIDER_ANALYZE_PAGE_SPEED])

Generate Word Cloud

Creates a word cloud visualization based on the most frequent terms found during crawling.
from composio_crewai import ComposioToolSet, Action tool_set = ComposioToolSet() tools = tool_set.get_tools(actions=[Action.SPIDER_GENERATE_WORD_CLOUD])

Detect Content Changes

Compares current crawl results with previous ones to identify content changes on websites.
from composio_crewai import ComposioToolSet, Action tool_set = ComposioToolSet() tools = tool_set.get_tools(actions=[Action.SPIDER_DETECT_CONTENT_CHANGES])

Export Data To CSV

Exports extracted data or crawl results to a CSV file for further analysis.
from composio_crewai import ComposioToolSet, Action tool_set = ComposioToolSet() tools = tool_set.get_tools(actions=[Action.SPIDER_EXPORT_CSV])

Generate SEO Report

Creates a comprehensive SEO report based on the crawled data, including metadata analysis and keyword density.
from composio_crewai import ComposioToolSet, Action tool_set = ComposioToolSet() tools = tool_set.get_tools(actions=[Action.SPIDER_GENERATE_SEO_REPORT])

Detect Language

Automatically detects and reports the language of crawled web pages.
from composio_crewai import ComposioToolSet, Action tool_set = ComposioToolSet() tools = tool_set.get_tools(actions=[Action.SPIDER_DETECT_LANGUAGE])

Create Content Archive

Archives the content of crawled web pages for future reference or analysis.
from composio_crewai import ComposioToolSet, Action tool_set = ComposioToolSet() tools = tool_set.get_tools(actions=[Action.SPIDER_CREATE_CONTENT_ARCHIVE])

Analyze Internal Linking

Examines and reports on the internal linking structure of a website.
from composio_crewai import ComposioToolSet, Action tool_set = ComposioToolSet() tools = tool_set.get_tools(actions=[Action.SPIDER_ANALYZE_INTERNAL_LINKING])

Generate API Documentation

Automatically generates documentation for APIs discovered during web crawling.
from composio_crewai import ComposioToolSet, Action tool_set = ComposioToolSet() tools = tool_set.get_tools(actions=[Action.SPIDER_GENERATE_API_DOCS])

Perform Security Scan

Conducts a basic security scan of websites during crawling, identifying common vulnerabilities.
from composio_crewai import ComposioToolSet, Action tool_set = ComposioToolSet() tools = tool_set.get_tools(actions=[Action.SPIDER_SECURITY_SCAN])

Mobile Friendly Test

Performs a mobile-friendly test on crawled web pages and generates a report.
from composio_crewai import ComposioToolSet, Action tool_set = ComposioToolSet() tools = tool_set.get_tools(actions=[Action.SPIDER_MOBILE_FRIENDLY_TEST])

Generate Visual Sitemap

Creates a visual representation of the website structure based on crawl data.
from composio_crewai import ComposioToolSet, Action tool_set = ComposioToolSet() tools = tool_set.get_tools(actions=[Action.SPIDER_GENERATE_VISUAL_SITEMAP])

Crawl Depth Reached

Triggered when the spider reaches a specified crawl depth limit.
from composio_crewai import ComposioToolSet, Action tool_set = ComposioToolSet() tools = tool_set.get_tools(actions=[Action.SPIDER_CRAWL_DEPTH_REACHED])

New Domain Discovered

Fires when the spider encounters a link to a previously undiscovered domain.
from composio_crewai import ComposioToolSet, Action tool_set = ComposioToolSet() tools = tool_set.get_tools(actions=[Action.SPIDER_NEW_DOMAIN_DISCOVERED])

Crawl Completed

Triggered when a full web crawl operation is completed.
from composio_crewai import ComposioToolSet, Action tool_set = ComposioToolSet() tools = tool_set.get_tools(actions=[Action.SPIDER_CRAWL_COMPLETED])

Error Encountered

Fires when the spider encounters an error during crawling or data extraction.
from composio_crewai import ComposioToolSet, Action tool_set = ComposioToolSet() tools = tool_set.get_tools(actions=[Action.SPIDER_ERROR_ENCOUNTERED])

Data Threshold Reached

Triggered when the amount of extracted data reaches a specified threshold.
from composio_crewai import ComposioToolSet, Action tool_set = ComposioToolSet() tools = tool_set.get_tools(actions=[Action.SPIDER_DATA_THRESHOLD_REACHED])

New File Type Found

Fires when the spider discovers a new file type during crawling.
from composio_crewai import ComposioToolSet, Action tool_set = ComposioToolSet() tools = tool_set.get_tools(actions=[Action.SPIDER_NEW_FILE_TYPE_FOUND])

Duplicate Content Detected

Triggered when the spider identifies duplicate content across different pages.
from composio_crewai import ComposioToolSet, Action tool_set = ComposioToolSet() tools = tool_set.get_tools(actions=[Action.SPIDER_DUPLICATE_CONTENT_DETECTED])

Crawl Rate Limit Reached

Fires when the spider reaches the configured crawl rate limit to avoid overloading servers.
from composio_crewai import ComposioToolSet, Action tool_set = ComposioToolSet() tools = tool_set.get_tools(actions=[Action.SPIDER_RATE_LIMIT_REACHED])

New Technology Detected

Triggered when the spider detects a new technology or framework used on a website.
from composio_crewai import ComposioToolSet, Action tool_set = ComposioToolSet() tools = tool_set.get_tools(actions=[Action.SPIDER_NEW_TECHNOLOGY_DETECTED])

Content Update Detected

Fires when changes in content are detected on previously crawled pages.
from composio_crewai import ComposioToolSet, Action tool_set = ComposioToolSet() tools = tool_set.get_tools(actions=[Action.SPIDER_CONTENT_UPDATE_DETECTED])

Crawl Budget Exceeded

Triggered when the spider exceeds the allocated crawl budget (e.g., number of pages or data volume).
from composio_crewai import ComposioToolSet, Action tool_set = ComposioToolSet() tools = tool_set.get_tools(actions=[Action.SPIDER_CRAWL_BUDGET_EXCEEDED])

New Structured Data Found

Fires when the spider discovers new structured data (e.g., Schema.org markup) on a web page.
from composio_crewai import ComposioToolSet, Action tool_set = ComposioToolSet() tools = tool_set.get_tools(actions=[Action.SPIDER_NEW_STRUCTURED_DATA_FOUND])

Frequently asked questions

What is Composio.dev?

Composio.dev is a cutting-edge framework for building AI applications, designed to make the process of developing AI solutions super easy and fun! It's a collection of powerful tools and libraries that simplify the process of creating AI applications, allowing you to focus on the creative aspects of your project without getting bogged down by the technical details.

How does Composio.dev support Spider?

Composio.dev seamlessly integrates with Spider, making it a breeze to leverage its capabilities within the Composio.dev platform. You can use Spider to call functions on various platforms like Google, GitHub, and others, allowing you to incorporate different services into your AI applications with ease. It also supports user login via OAuth2 and can work with other popular frameworks such as LangChain and CrewAI, giving you the flexibility to build truly innovative AI solutions.

What models can I use with Spider and crewAI_python?

When using Spider and crewAI_python, you have access to a wide range of state-of-the-art language models, including GPT-4o (OpenAI), GPT-3.5 (OpenAI), GPT-4 (OpenAI), Claude (Anthropic), PaLM (Google), LLaMA and LLaMA 2 (Meta), Gemini, and many others. This flexibility allows you to choose the model that best suits your specific use case, whether you're building a chatbot, a content creation tool, or any other AI-powered application. You can experiment with different models and find the one that delivers the best performance for your project.

How can I integrate Spider with crewAI_python?

Integrating Spider with crewAI_python is super easy with Composio.dev! You can use the Composio.dev API to call functions from both Spider and crewAI_python, allowing you to tap into their capabilities with just a few lines of code. The SDK is available in Python, JavaScript, and TypeScript, so you can work with the language you're most comfortable with and integrate these powerful tools into your projects seamlessly.

What is the pricing for Spider and crewAI_python?

Both Spider and crewAI_python are completely free to use, with a generous free tier that allows up to 1000 requests per month. This makes them accessible for developers and organizations of all sizes, whether you're a student working on a personal project or a startup building the next big thing. You can get started with these powerful tools without worrying about breaking the bank.

What kind of authentication is supported for Spider and crewAI_python?

Spider and crewAI_python support OAuth2 authentication, ensuring secure and authorized access to their functionalities. You can use the Composio.dev API to handle authentication and call functions from both Spider and crewAI_python seamlessly. The SDK is available in Python, JavaScript, and TypeScript for your convenience, making it easy to integrate authentication into your projects and keep your users' data safe and secure.

Can I add Spider to my project?

Absolutely! You can easily incorporate Spider into your project by utilizing the Composio.dev API. This API allows you to call functions from both Spider and crewAI_python, enabling you to leverage their capabilities within your application. The SDK is available in Python, JavaScript, and TypeScript to facilitate integration, so you can work with the language you're most comfortable with and add these powerful tools to your project with ease.

What is the accuracy of Spider and crewAI_python?

Spider and crewAI_python are designed to provide highly accurate and reliable results, ensuring that your AI applications perform at their best. The integration with Composio.dev ensures precise function calls, enabling you to build robust and powerful AI applications with confidence. The comprehensive framework and the ability to leverage state-of-the-art models ensure reliable and accurate outcomes for your AI development needs, whether you're working on a chatbot, a content creation tool, or any other AI-powered project.

What are some common use cases for Spider and crewAI_python?

Spider and crewAI_python can be used for a wide range of AI applications, making them versatile tools for developers and creators alike. Some common use cases include natural language processing, text generation, question answering, sentiment analysis, and more. They're particularly useful for building chatbots, virtual assistants, content creation tools, and other AI-powered applications that can help you automate tasks, engage with users, and create compelling content. Whether you're working on a personal project or building a product for your startup, these tools can help you bring your ideas to life.

How does Spider handle data privacy and security?

Data privacy and security are crucial considerations when working with AI systems, and Spider takes these issues seriously. It follows industry best practices and adheres to strict data protection regulations, ensuring that your data is kept safe and secure. Spider provides robust security measures, such as encryption and access controls, to ensure the confidentiality and integrity of your data. You can rest assured that your sensitive information is protected when using Spider for your AI development needs.

Can I customize Spider and crewAI_python for my specific needs?

Absolutely! Spider and crewAI_python are highly customizable and extensible, allowing you to tailor their functionality, models, and configurations to meet your specific requirements. Whether you're building a chatbot, a content creation tool, or any other AI-powered application, you can customize these tools to fit your unique needs. Additionally, Composio.dev provides a flexible platform for integrating and orchestrating various AI tools and services, enabling you to create custom AI solutions that are tailored to your project.

What kind of support and documentation is available for Spider and crewAI_python?

Spider and crewAI_python have comprehensive documentation and a supportive community, making it easy for you to get started and find answers to your questions. Composio.dev also provides extensive resources, including tutorials, guides, and a dedicated support team to assist you throughout your AI development journey. Whether you're a beginner or an experienced developer, you'll have access to the resources you need to make the most of these powerful tools.
+ Integrate seamlessly with your agentic frameworks
Composio Works with All Shapes and SizesComposio Works with All Shapes and SizesComposio Works with All Shapes and SizesComposio Works with All Shapes and SizesComposio Works with All Shapes and Sizes
Building for AI across continents๐Ÿงช