Agent inventory: Tracking which agents touch which systems

TL;DR: An agent inventory is a governed system of record that captures which AI agents exist, which systems they touch, who owns them, and what they are permitted to do. Building one requires discovery, cataloging, and refresh cadences tied to risk tier and staff movement. Policy-as-code enforcement turns the inventory from documentation into an enforced control that survives audits and team turnover.

Your developers deployed AI agents this quarter. Your audit is next quarter. You need an inventory that survives audit scrutiny, not a spreadsheet that falls apart when someone leaves.

This article provides a repeatable method to build and maintain that inventory: discovery, cataloging, ownership attribution, and refresh cadences that survive team turnover. It also shows how infrastructure-layer enforcement makes the inventory defensible rather than aspirational.

Why AI agent sprawl creates governance risk

An agentic loop is the cycle where an AI agent receives a prompt, reasons about which tools to call, executes those calls against external systems, and returns results. The execution space is the set of all actions an agent can take within that loop, defined by the tools it has access to and the permissions those tools carry.

The risk is unauthorized tool invocation. An agent with access to Slack, Salesforce, and GitHub can read messages, update records, and push code. If the agent's permissions exceed its intended function, or if a prompt injection convinces the model to call a destructive action, the execution space becomes a liability.

What makes agent inventory different

Traditional asset inventory tracks installations, not actions. Server and SaaS inventories assume static assets with clear installation locations. AI agents break this model: a developer can spin up an agent in a notebook, connect it to Gmail and Slack via OAuth, and have it reading and writing data within an hour. The agent does not appear in your asset inventory because it is not installed in the traditional sense.

Agents also act on systems, not just access them. A user with read access to Salesforce can view records. An agent with the same access can read every record, summarize them, and post the summary to Slack. Your inventory must capture which actions an agent can perform, not just which systems it touches. Read access to GitHub is different from delete access, so track permission scope at the action level.

Eliminate shadow IT through agent visibility

Shadow AI refers to unsanctioned LLM usage where business units call api.openai.com, generativelanguage.googleapis.com, api.anthropic.com, or hosted-LLM endpoints from spreadsheets, scripts, browser plug-ins, and self-built apps without the SOC's knowledge. Standard EDR and SIEM rules do not find it because the traffic looks like ordinary HTTPS to the wrong list of destinations.

Visibility is the prerequisite for governance: you cannot apply policy-as-code controls to agents that never entered your inventory. You cannot enforce least privilege on agents you do not know exist, review access permissions for agents deployed without IT review, or produce audit evidence for agents that never entered your inventory.

Automating evidence for AI audits

When auditors review your AI agent deployments, they ask for ownership, access scope, and change history. A live inventory provides this evidence on demand: when an auditor asks for a list of all agents with write access to production systems, you should produce it in minutes, not weeks. SOC 2 Type II access control criteria expect organizations to restrict system access to authorized identities, and an agent inventory supports this by documenting which non-human identities, such as agents, service accounts, and integrations, have access. ISO 27001 Annex A 5.15 requires access control rules to be established for all entities doing the accessing, including services, devices and machines, not only human users (High Table, ISO 27001 Annex A 5.15). An agent inventory supports this by documenting each agent's permission scope.

Uncovering hidden AI agent deployments

Discovery starts with network monitoring. Capture and label every connection to a known LLM API domain at the network boundary, and route traffic through a sanctioned AI gateway so previously unmonitored usage is logged and attributable. Review OAuth scope grants and correlate SIEM logs with known agent activity. Look for sudden API spikes, irregular request patterns, or access from unknown IPs or dormant accounts.

Integrating agent authentication and tool-call telemetry with SIEM and cloud monitoring tools improves detection by correlating this behavior across the environment.

Identifying hidden integration risks

Over-provisioned OAuth scopes create hidden access paths. A developer requests Gmail access for an agent that summarizes email threads: the OAuth grant includes read, send, and delete permissions. The agent only needs read access, but now it has send and delete access that no one reviewed. Service accounts and webhooks create similar ungoverned paths: a service account with broad permissions can be used by multiple agents without clear ownership, and a webhook triggering on Salesforce record changes can connect to an agent taking actions you did not approve.

Key attributes to track for every AI agent

Your inventory must capture enough metadata to answer auditor questions and support access reviews.

The table above is the recommended schema for this article, not a NIST AI RMF requirement.

Defining agent accountability and roles

Ownership attribution answers the question: who is accountable when this agent misbehaves? The owner is the human who approved the agent's deployment and is responsible for its behavior. Role-based access for inventory management determines who can view, edit, and approve changes. IT typically owns the inventory system, but business unit leaders should be able to view and attest to the agents their teams own.

Mapping AI agents to system integrations

Document each system touch point with the application name and the specific actions the agent can perform. An agent that reads Salesforce records and posts summaries to Slack has two touch points: Salesforce (read) and Slack (send). If the OAuth grant includes write access to Salesforce but the agent only reads records, flag this as scope creep. Build your inventory to surface these misalignments between intended function and actual permissions automatically, by comparing granted scopes against the actions recorded in the audit log.

Controlling AI agent access levels

Permission scope includes OAuth scopes, API keys, and service account permissions. Each should be reviewed against the agent's intended function. Composio's policy-as-code enforcement makes access levels enforceable: an admin sets which actions are permitted per user or role through a dashboard, and those restrictions are enforced before model interaction in the request path, not through prompt instructions the model can reason around.

Mapping agents to business owners

The technical owner is the developer or team that built and maintains the agent. The business owner is the leader accountable for the business process the agent supports. When a new agent is registered, the registrant must identify the business owner, and the business owner must attest that they are accountable for the agent's behavior. If no one claims ownership, flag the agent for review and potential decommissioning.

Auditing your current agent inventory and AI agent footprint

Before building the catalog, audit what already exists. This discovery exercise surfaces agents deployed outside IT review.

1. Enumerate credentials: List all active OAuth grants, API keys, and service accounts in your identity provider, secrets management system, and CI/CD pipelines.

2. Correlate activity: Match outbound connections to known LLM API domains with OAuth grants, API keys, and webhook activity in your SIEM and network logs.

3. Interview stakeholders: Ask business unit leaders which agents their teams use, which systems those agents access, and who owns them.

Auditing OAuth scopes and API keys

OAuth grants and API credentials issued during initial integration frequently cover more than the integration needs. Review each grant against the agent's documented function and compare granted scopes to the actions the agent actually performs. Once a grant is narrowed to the documented function, enforce that scope with the policy-as-code controls described under "Controlling AI agent access levels."

Verifying service account permissions

Service accounts are hidden agent identities. Microsoft's non-human identity guidance recommends tracking all non-human identities through a centralized IAM system so security teams understand what they can access. List all service accounts, identify which systems they can access, review their permissions, and assign a human owner to each one.

Auditing team-led AI deployments

Developer and product teams deploy agents outside IT review because they need to move fast, connecting SaaS tools via personal or team-level OAuth grants that never went through security review. Surface these deployments by monitoring OAuth grants and network traffic. When you discover a new agent, ask the team to register it in the inventory and frame this as enabling their work, not blocking it.

Auditing integration and webhook activity

Webhook and integration activity provides evidence of agent footprint. A webhook that triggers on Salesforce record changes and calls an external API is likely connected to an agent. Correlate activity logs with known agents: if you see webhook activity that matches no registered agent, investigate.

Building the initial agent inventory catalog

Once the audit is complete, build the initial catalog: the system of record that captures every agent, its attributes, and its ownership.

Security questionnaire readiness requirements:

• Agent inventory exists and is up to date

• Every agent has a named owner

• Every agent has documented system touch points

• Every agent has documented permission scope

• Permission scope is reviewed against intended function

• Audit logs capture all agent tool calls including denied actions

• Refresh cadence is defined and followed

• Offboarding procedure includes agent permission revocation

Choose your governance tracking model

Centralized model: A single system of record owned by IT, with all teams registering agents in the same system. This works well for small to mid-size enterprises and high-risk environments.

Federated model: Each business unit maintains its own inventory, and IT aggregates them. This works well for large distributed enterprises where business units have autonomy, but they require aggregation across multiple systems, adding complexity for SOC 2 and ISO reviews.

Define core metadata for AI agents

The minimum viable schema includes agent name, owner, business unit, systems touched, permission scope, creation date, last activity, and risk tier. You can extend it with approval records, change history, incident history, and cost center. Capture enough metadata to make the inventory useful without creating so much overhead that teams avoid registering their agents.

Audit trail for every agent action

A complete audit trail includes user, team, tool, action, outcome, and denied calls. This is what makes it usable as compliance evidence rather than activity summaries. Composio's centralized audit logging records every tool call with user, team, tool, action, and outcome, including denied calls:

ISO 27001:2022 Annex A 8.15 requires that logs be produced and analyzed, and its guidance recommends including user ID, system activity, date and time, device/system location, and network addresses in event logs. The audit log format above captures all of these fields.

Updating agent permissions for staff offboarding

Offboarding is the critical test of inventory quality. When an employee leaves, you must revoke their access and transfer or decommission the agents they owned. If your inventory is accurate and current, this is straightforward: if it is stale, you will miss agents and leave orphaned permissions.

Set refresh cadences by risk tier

A common refresh cadence model, organized by risk tier:

Trigger events such as owner departure or permission changes should always trigger an immediate review regardless of the scheduled cadence.

Scale visibility with automated scanning

Automated discovery cuts the manual effort your team spends maintaining the inventory. Composio's centralized credential vault enables enumeration because it holds all credentials in one governed location rather than leaving them scattered across developer environments and CI/CD secrets.

Verify agent permissions after staff moves

Permission verification after role changes and departures is the moment inventories fail. When an employee moves to a new role, review and potentially revoke their old permissions. When they depart, revoke permissions immediately. Most organizations fail here because the inventory is stale, so they cannot identify which agents the employee owned, and the offboarding process omits agent permission revocation.

Revoke agent permissions during offboarding

Immediate revocation procedure:

1. Block sign-in and revoke all active sessions in your identity provider within the first hour.

2. Revoke every API key, personal access token, and OAuth token associated with the employee within 24 hours.

3. Review service accounts and agents owned by the employee and either transfer ownership or decommission them.

4. Monitor logs for a defined period (typically 48-72 hours) and watch for failures in dependent systems.

5. Delete the service accounts and agent identities once the monitoring period shows no failures.

Aligning AI agent permissions with security policy

The inventory is not just a list. It is a system of record that feeds policy enforcement and access reviews. Policy-as-code turns inventory data into enforced controls through the request-path enforcement described under "Controlling AI agent access levels." The agent cannot reason its way past a restriction, and a prompt injection cannot widen its own scope.

Automating AI agent access reviews

Schedule access reviews based on risk tier: quarterly for high-risk agents, semi-annually for medium-risk, annually for low-risk. During each review, verify that the agent's permissions match its documented function and that the owner is still accountable. The per-user and per-role permission controls described under "Controlling AI agent access levels" reduce the need for manual permission checks.

Enforcing least privilege policies

Least privilege is an enforced state, not a documented aspiration. Because access restrictions are evaluated at the infrastructure layer, as described under "Controlling AI agent access levels," an admin's decision to disable delete actions holds regardless of what the prompt contains.

Mapping agent failures to recovery plans

Include incident history for each agent in your inventory with a documented recovery plan. When an agent fails or behaves unexpectedly, first revoke its permissions to contain the exposure, then investigate root cause in the audit logs, then either remediate the agent and restore permissions or decommission it and remove it from the inventory.

Documenting AI agent access logs

Auditors expect access logs to include denied calls, not just successful ones. A complete chain of custody shows not only what the agent did, but what it tried to do and was prevented from doing. Composio's centralized audit logging with denied calls provides complete chain of custody: every tool call is logged with the user, team, tool, action and outcome, denied calls included.

Book a call to walk through your security requirements before the next enterprise review. Contact the Composio team to discuss how Composio Enterprise helps you build a defensible agent inventory with policy-as-code enforcement and centralized audit logging.

FAQs

How often should we refresh the agent inventory?

Refresh cadence depends on risk tier: quarterly for high-risk agents, semi-annually for medium-risk, annually for low-risk. Trigger events such as owner departure or permission changes should always trigger an immediate review.

What if developers deploy agents without telling IT?

Monitor OAuth grants and network traffic to discover shadow deployments. When you find a new agent, ask the team to register it in the inventory and frame this as enabling their work, not blocking it.

Can we automate agent discovery completely?

Automated discovery can surface most agents through network monitoring, OAuth scope review, and SIEM correlation. However, you should also interview team leads to identify agents that may not appear in technical logs, such as agents that use personal API keys or run locally.

Who should own the inventory system?

IT typically owns the inventory system, but business unit leaders should be able to view and attest to the agents their teams own. The inventory is a governance tool, not just a technical tool.

How do we handle agents that span multiple business units?

Assign a primary owner in one business unit and secondary owners in the others. The primary owner is accountable for overall behavior while secondary owners are accountable for their unit's usage.

Key terms glossary

AI agent: An AI agent is a software program that uses a large language model to reason about which tools to call and executes those calls against external systems. Agents differ from traditional automation because they can adapt their behavior based on context.

Agent inventory: An agent inventory is a governed system of record that captures which AI agents exist, which systems they touch, who owns them, and what they are permitted to do. It is a living document that requires regular refresh to remain accurate.

System touch point: A system touch point is a connection between an agent and an external system that defines which application the agent can access and which actions it can perform within that application.

Ownership attribution: Ownership attribution is the process of assigning a human owner to each agent. The owner is accountable for the agent's behavior and is responsible for reviewing and approving changes to its permissions.

Refresh cadence: Refresh cadence is how often you review and update the inventory. It is typically organized by risk tier, with high-risk agents requiring more frequent reviews than low-risk agents.