LlamaIndex + Composio: Building Reliable & Secure AI Agents

July 10, 2024
AI Agents
16 min read

AI agents are all the rage. The ever-increasing quality of the Large Language Models (LLMs) and frameworks for building them has opened up the possibilities for creating intelligent and complex autonomous systems. However, it is still hard to build reliable and useful agents that work in the real world. Building reliable agents necessitates high-quality tools and integrations, superior language models, and an effective orchestration framework.

While the quality of LLMs and features of frameworks get much attention, the significance of AI Agent tools is usually understated. The tools allow the language model to interact with external environments. For example, an LLM can use a web scraping tool to gather the latest news articles or a scheduling tool to manage calendar events. Designing the right tools optimized for LLM tool calls can drastically improve the reliability of AI agents.

Additionally, implementing tool-level authentication and authorization is crucial for deploying AI agents in the real world. These measures are important to provide secure and controlled access to external applications like GitHub, Slack, etc.

Enter Composio home docs GitHub, an open-source tooling platform designed to solve these challenges. It offers over 150 tools and integrations to empower language models to accomplish complex agentic workflows reliably. With Composio’s managed user authentication and authorization, developers need not jump over hoops to manually set these things up. This complements the Llama Index’s orchestration framework. With Composio’s rich tool integrations and Llama Index Agent framework, developers can quickly go from prototypes to production builds.

This article explores the essentials and benefits of Composio; and explains how to build AI agents that can schedule events using Composio tools and Llama Index.

Why Use Composio?

Composio has been designed from the ground up keeping real-world readiness of AI agents in mind. Here are a few pros of Composio:

  • ● Managed Authentication and Authorization: Many applications like Slack, GitHub, and Google services require complex authentication protocols to ensure secure and controlled access. Composio addresses this by providing built-in support for authentication mechanisms like Oauth, ApiKey, JWT, and Basic authentication. It handles the authentication and authorization of your users, enabling the agents to integrate tools to perform actions on behalf of your users.

  • ● User Management: Composio offers comprehensive authentication management for agents during runtime. The embedded authentication management system can manage user credentials and access permissions across various applications. Composio provides a convenient way to handle scopes, authorization levels, and user roles for individuals, teams, or entire organizations.

  • ● Range of Tools: Composio has over 150 tools and integrations across industry verticals like CRM, Dev tools, Ticketing, etc, to make it easier for developers to build Effective AI agents capable of accomplishing complex real-world tasks.

  • ● Customizations: Composio offers a straightforward method for integrating applications not listed in the official tools catalog. To add an application to Composio, you will only need the OpenAPI specification of the application and a configuration file.

  • ● Monitoring: Composio provides a dedicated dashboard for monitoring all the tools and integrations. It lets you observe logs in real-time and explore actions and triggers.

Getting Started with Composio

Before building the Slack agent, let’s get acquainted with the key concepts of Composio.

  • ● Entities: In Composio, an “entity” is like a container for all the accounts and tools associated with a user or an entire organization. This allows you to manage all the accounts from a single dashboard.

  • ● Integrations: Integrations connect your account with external apps. This involves setting up authentication mechanisms, such as OAuth, and defining access permissions that dictate what actions the apps are allowed to perform. Once an integration is established, all the users can access the same integration through their accounts. For instance, consider a Slack integration. As a developer, you will only need to add it once in Composio, and all your users can access the integration with their Slack accounts, with the permissions you have defined. Each user will authenticate their Slack account, allowing them to interact with the AI agent through the shared integration. This means while the integration setup is centralized, the app access and authentication are individual to the user.

  • ● Actions: These are tasks that the integrated tools can perform. For example, sending a Slack message, scheduling a calendar event, etc.

  • ● Triggers: These are predefined conditions when met initiate triggers to your agents. Composio offers a built-in webhook to capture trigger events, The webhooks receive a payload from integrations in real time letting you perform actions on event data. For example, if a slack_receive_message trigger is configured for your Slack integration, the Slack app will send the event data like texts, time, and channel ID to the webhook at the backend.

Agents in LlamaIndex

LlamaIndex is one of the leading open-source frameworks for building LLM applications. The Llama Index simplifies the use of data from various sources and formats, facilitating data ingestion, indexing, and querying to build LLM applications.

Llama Index supports seamless integration with data stores, vector stores, graph stores, and SQL DB providers. Llama Hub provides the necessary tools and integrations for developing advanced RAG applications and AI agents.

In this article, we use LlamaIndex’s FunctionCallingAgentWorker to build the agent.

Now that you have grasped the basics of Composio, let’s build our Slack agent that will schedule meetings in Google Calendar using Llama Index’s Agent framework.

Workflow Overview

The workflow includes integrating both Slack and Google Calendar. The Slack integration is used to receive and send messages to Slack channels. The Calendar integration enables the Llama Index agent to schedule meetings and events based on the messages received from Slack.

Here is a high-level workflow of the system.

  1. 1. Slack Bot Setup: A Slack bot configured through Composio forwards the messages to a webhook.

  1. 2. Data formatting: The user IDs in the messages are replaced with the emails of the mentioned users.

  1. 3. Scheduling with Agent: The calendar agent receives messages with user emails and other instructions and schedules the event accordingly.

  1. 4. Sending Response to Slack: Once the event is scheduled, information about it is forwarded to the same Slack channel.

Integrating Slack bot and Calendar tools in Composio

To get started with building the agent, the first thing you need to do is create a Composio account if you haven’t already. Once done, go to the tools catalogue tab, search Google Calendar, and click on it. Now, click on the “Setup Google Calendar Integration”.

This will show a pop-up. Name the integration and save it. This will display another pop-up to link your Google account. Once you’ve completed the account linking, the integration will be displayed on the integrations page. For more advanced use cases, where you want to use your Google Calendar OAuth credentials, click on advanced and add your OAuth client ID and secret. Save and finish the flow to add Google Calendar with custom OAuth credentials.

Repeat the process for Slack, but this time, instead of using the default scopes, add two different scopes. Like with the default scopes, you won’t be able to access the user’s profile information, such as email addresses. However, Composio provides the option to add scopes to integrations. In this case, you need the user’s email address to schedule a meeting. For this, the Slack bot must have permission to access user profile information.

To access a user’s profile information on Slack, use the users.profile:read scope. Add it at the end of the User Scopes field and save it.

(Scopes: Scopes in the Slack Bot API specify the permissions and access levels granted to a bot within a Slack workspace. These scopes specify what a bot can and cannot do, such as reading messages, sending messages, managing channels, or accessing user data. )

To add custom scopes, click on Add Slack bot integration, then click on Choose scope & setup dev app (advanced), and toggle the Use composio app for authentication. Now add the scope at the end of the Bot Scopes and save it.

Now, you have both the Slack and Google Calendar integrations. You can monitor the connections in the integrations tab.

Prerequisites for Slack Agent

To complete the tutorial, you will need a Composio account. So, create one here. Once you log into the dashboard, click on the Settings tab to view your API keys. You will also need a function-calling model. You can use OpenAI, Anthropic models, or the Mixtral 8x7B model from Groq. This tutorial will use Mixtral as it is most accessible. So, get the Groq API key.

Setting Up Development Environment

Now that all the integrations are ready. we can start building our agentic workflow. Fork this Replit space, or clone this GitHub repository, and follow along.

Step 1: Installing Dependencies

If you run it locally, move to the root directory and execute the following CLI commands to set up the development environment.

chmod +x setup.sh
./setup.sh

This will install Poetry if it is not available. Then, proceed to create a virtual environment, install dependencies, and log in to your Composio account. Now, create a Poetry shell.

poetry shell

If you are using Replit, simply install dependencies using the following command.

petry install

And log in to your Composio account.

composio login

Step 2: Enable Slack Trigger

To begin, execute the following commands in the terminal to log into your Composio account, update apps, and enable the Slack message reception trigger.

composio login
composio apps update
composio triggers enable slackbot_receive_message

This activates the trigger within your account. You can also enable triggers from the dashboard.

Step 3: Setting up Environment Variables

Retrieve your Composio API key by running composio whoami or getting it from the dashboard and set it along with the Groq API key as environment variables:

Also, configure the Slack bot ID as an environment variable. You can find the Slack bot ID by selecting your bot in Slack.

For non-Replit users, set up these keys in a .env file and use dotenv for loading:

Bonus

You can also add new integrations from Composio’s Command Line Interface. This will add integrations with default scopes. For example,

composio add googlecalendar
composio add slack

Building Llama Index Agent with Composio Integrations

Now, let’s build our LLM agent. Create a agents.py file and import the required modules.

import os
from datetime import datetime

from composio.client.enums import Action, App, Tag

# Import from composio_llamaindex
from composio_llamaindex import ComposioToolSet
from llama_index.core.agent import FunctionCallingAgentWorker
from llama_index.core.llms import ChatMessage
from llama_index.llms.groq import Groq

Now, define the LLM and retrieve Composio’s Google Calendar tool.

# llm = OpenAI(model="gpt-4o")
llm = Groq(model="mixtral-8x7b-32768",
             api_key=os.environ["GROQ_API_KEY"])

# Get the Google Calendar integration
composio_toolset = ComposioToolSet()
tools = composio_toolset.get_tools(apps=[App.GOOGLECALENDAR])

Define a prefix system message to guide the LLMs response, emphasizing the agent’s role and operational details.

# Retreive the current date and time
date = datetime.today().strftime("%Y-%m-%d")
timezone = datetime.now().astimezone().tzinfo

prefix_messages = [
  ChatMessage(
      role="system",
      content=(
          f"You are now a calendar integration agent. \\
          You will try to execute the given task utilizing Google Calendar tools.\\
           Today's date is {date} (it's in YYYY-MM-DD format). \\
           Write the start and end time in the appropriate format for\\
            example 12pm as 'YYYY-MM-DDT12:00:00'. \\
            Infer the date of the event from the message. \\
            Fill in the date part accordingly. 
            Please remember the current date to be {date}. \\
            Always add a Google Meet event.\\
            Always send the mail to attendees.\\
            Always write the correct time mentioned in the request. \\
            Always return the HTML link of the meeting after execution. \\
            Remember the time zone is Indian Standard Time. \\
            Output meeting URL in the proper format,"
      ),
  )
]

Finally, define LlamaIndex’s function calling agent with the tools, LLM, and prefix_messages parameters.

agent = FunctionCallingAgentWorker(
  tools=tools,
  llm=llm,
  prefix_messages=prefix_messages,
  max_function_calls=10,
  allow_parallel_tool_calls=False,
  verbose=True,
).as_agent()

The max_function_calls parameter indicates the number of times the agent will attempt to complete the task in case of any errors before giving up. The verbose parameter will log execution details on the terminal.

Setting up the Agentic Workflow

In the previous section, we defined the tool and the LLM agent. This section explores setting up the event listener for Slack messages, processing the payloads, and running the agent.

Create a main.py file and set up the imports.

import os
import re

from agents import agent
from composio.client import App, Action
from composio import Composio

from composio.client.collections import TriggerEventData
from composio.tools import ComposioToolSet

BOT_ID = os.environ["BOT_ID"]

The next thing is to define the webhook for listening to Slack trigger events. Composio provides a decorator @listener.callbackthat registers a function as a callback for specific event types. The decorator allows the function to process incoming data when some conditions are met. This ensures the application works in real-time whenever the specified trigger is activated.

Here is how you can define Slack event listener.

toolset = ComposioToolSet()
listner = toolset.create_trigger_listener()

@listner.callback(filters={"trigger_name": "slackbot_receive_message"})
def event_handler(event: TriggerEventData) -> None:
    message = event.payload['event']['text']
    channel = event.payload['event']['channel']
    if BOT_ID not in message:
        return "Ignored", 204
    return run_agent(message, channel)

In the above code snippet, the callback function event_handler is called when the specified event matching the trigger ID occurs in Slack. If the message does not contain the BOT_ID, a 204 response is returned; otherwise, the function proceeds to execute the run_agents function.

The run_agent function coordinates message processing, running the agent, and returning the response to the Slack channel.


composio_client = Composio(api_key=os.getenv("COMPOSIO_API_KEY"))
entity = composio_client.get_entity("default")

def extract_user_ids(message, exclude_id=os.environ["BOT_ID"]):
    "Extracts Slack user IDs, optionally excluding the bot's ID."
    pattern = r"<@(\\w+)>"
    user_ids = re.findall(pattern, message)
    return [user_id for user_id in user_ids if user_id != exclude_id]
    
def fetch_user_details(user_ids):
    "Fetches user details from Slack and returns a mapping of user ID to email."
    uid_to_email = {}
    for user_id in user_ids:
        response = entity.execute(
            action=Action.SLACK_USERS_INFO,
            params={"user": user_id},
        )
        if response['response_data']['ok']:
            user = response['response_data']['user']
            profile = user['profile']
            uid_to_email[user_id] = profile['email']
            
    return uid_to_email
  
def replace_user_ids_with_emails(text, uid_to_email):
    "Replaces user IDs in the text with their corresponding emails."
    pattern = re.compile(r"<@(\\w+)>")
    return pattern.sub(lambda match: uid_to_email.get(match.group(1), match.group(0)), text) 

def replace_emails_with_uids(text, uids_to_emails):
    "Replace all the email addresses in the given text with their corresponding user IDs."
    # Invert the dictionary to map emails to user IDs
    emails_to_uids = {email: uid for uid, email in uids_to_emails.items()}
    email_pattern = re.compile(r'\\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\\.[A-Z|a-z]{2,}\\b')
    return email_pattern.sub(lambda match: '<@' + emails_to_uids.get(match.group(0), match.group(0)) + '>', text)
    
def run_agent(text, channel):
    user_ids = extract_user_ids(text)
    uid_to_email = fetch_user_details(user_ids)
    text_with_emails = replace_user_ids_with_emails(text, uid_to_email)

    response = agent.chat(text_with_emails)
    reply_with_usernames = replace_emails_with_usernames(response.response, email_to_uname)
     
    # Post the text with usernames to the Slack channel
    entity.execute(action=Action.SLACK_CHAT_POST_MESSAGE,
                   params={"text": reply_with_usernames, "channel": channel})
    return "Agent run completed", 200

The retrieved Slack message from the event listener contains User IDs instead of emails. But to schedule Calendar events, you will need emails. Hence, we retrieve emails from Slack user IDs using the Action SLACK_USERS_INFO. To use Actions directly, we defined the Composio client and retrieved the entity.

In the above code snippet,

  1. 1. First, we extract the mentioned user IDs from the message except the BOT_ID.

  2. 2. Then, we fetch emails from user IDs. We create two mappings: user IDs to emails and Emails to user names.

  3. 3. Replace the user IDs in the original message with emails for the Calendar agent.

  4. 4. Run the Calendar agent with formatted inputs.

  5. 5. Then, replace the emails with user IDs.

  6. 6. Finally, send the formatted message back to the Slack channel.

To start the webhook, execute the main.py file. This will activate the webhook, which monitors the associated Slack workspace. The webhook will trigger the workflow whenever a message includes a tag for the bot.

With the Calendar agent, you can schedule events, remove participants, delete schedules, and perform many other tasks.

Next Steps

Composio now officially supports the Llama Index. The article demonstrated how to integrate Slack and Google Calendar to automate scheduling tasks efficiently using Composio and Llama Index. You can enhance the agent’s versatility by adding more AI Agent Tools such as Exa, Tavily, and Notion. The agent can search and scrape the internet to prepare a report on a specific topic.

Conclusion

By combining Composio’s collection of tools and integrations with Llama Index, developers can build efficient AI automation workflows. As the technologies evolve, the potential for creating more efficient, sophisticated, and nuanced AI automation increases. Developers can leverage this to solve more challenging and complex problems. Check out all the tools and integrations in Composio to build reliable and secure AI agents.

Log In
Sign In