AI agent observability for the tool layer

TL;DR: Your LLM tracing tool captures prompt spans, token costs, and reasoning chains, but it does not record what the agent actually executed against your systems of record. AI agent observability at the tool layer is a distinct problem: you need logs tied to specific users that capture every API action, every denied call, and every credential resolution, written at the execution boundary before the model sees a response. Without this layer, you cannot debug production failures, answer enterprise security questionnaires, or prove your access controls functioned during the audit period. This guide defines what that layer needs to contain and how to implement it.

Your LLM trace shows that the agent planned a Salesforce update, made the tool call, and got a confirmation back. But that’s about where the useful detail ends. You still can’t see which connected account was used, what data was actually sent to Salesforce, whether the agent was allowed to make that change, or who was responsible for it.

That becomes a real problem when a customer reports bad data or a security team asks, “Who changed this, with which credentials, and what exactly did they send?” This article looks at what’s missing from typical LLM traces, and what you need to capture to properly audit agent actions.

Beyond model traces: tracking agent tool calls

Model tracing and tool-layer logging answer different questions, and conflating them creates a dangerous blind spot in production.

Model tracing tools like LangSmith and Arize Phoenix capture the LLM's internal reasoning: prompt inputs, chain of thought steps, token counts, and response quality scores. That makes them valuable for prompt engineering and latency optimization. Composio complements that layer by logging what happens at the execution boundary, the point where an HTTP request crosses your application layer and mutates a record in Salesforce, pushes a commit to GitHub, or sends a message in Slack, with credential isolation and policy enforcement that model tracing cannot provide by design.

Once an agent can call external tools, its behavior is no longer limited to generating text: a tool call may retrieve private data, modify external states, execute code, or trigger irreversible downstream actions.

Mapping tool calls to agent intent

The gap between model intent and tool execution is not theoretical. Consider these concrete divergence scenarios:

• Token expiration: The model constructs a valid tool call. Composio's proxy layer receives a 401 Unauthorized from the upstream API because the access token expired under Slack's token rotation policy, which applies when an app has rotation explicitly enabled. The model's trace shows a successful intent. The tool log shows an authentication failure.

• Rate limiting: A per-IP rate limit returns 429 Too Many Requests. The agent enters a retry loop. Your model trace shows repeated identical tool calls with no explanation for why none completed.

• Schema drift: An upstream API updates its endpoint schema. The agent sends a payload that matched last month's spec. The API returns a 400 Bad Request that LLM reasoning cannot detect as an error, because the model receives only what the agent framework surfaces.

Essential data for agent debugging

Debugging a production agent failure without tool-layer logs forces your team to reconstruct events from model outputs that may not accurately reflect what actually executed. The minimum telemetry required to diagnose a failing agent includes:

1. Exact HTTP status code returned by the third-party API

2. Timestamp of the execution attempt

3. User identity that authorized the call

4. Connected account ID used to resolve credentials

5. Outcome classification (success, denied, error, timeout)

Mapping actions to specific identities

Multi-agent systems introduce a delegation problem: when Agent B calls a tool on behalf of a task initiated by a human user through Agent A, your log must capture the full delegation chain, not just the service account that made the API call.

Composio's SCIM 2.0 integration (System for Cross-domain Identity Management) makes this identity chain operational at the directory level. SCIM maps directory groups from Okta, Entra ID, or Google Workspace to Composio teams, so every tool call log entry carries an identity that traces directly back to your identity provider, not a generic service account name. When a team member is offboarded, SCIM revocation propagates at the directory level and the audit record preserves their historical actions for the configured retention window.

Verifying tool usage and side effects

Every tool call log must capture the action type, HTTP status code, credential reference, user identity, and outcome for every execution attempt. When a customer reports that their Salesforce record was overwritten with incorrect data, you need the metadata record that confirms which connected account executed which action and when, not a model trace showing that the agent intended to make an update.

The agent never touches the raw credential, and audit records contain metadata only (action, outcome, HTTP status, and credential reference), with no payload data stored at rest.

Identifying security gaps in blocked tool requests

Successful tool calls prove what your agent did. Denied tool calls prove that your controls worked. This distinction matters enormously in a security review.

SOC 2 Type II reports require demonstrating the operational effectiveness of security mechanisms across the full observation window, typically six to twelve months. A log that only records successful actions cannot prove that unauthorized actions were prevented. Denied calls are the concrete evidence that policy enforcement is active, not just configured.

How to verify tool invocation logs

Shadow AI use, where team members or external inputs attempt to invoke tools outside their authorized scope, surfaces only in a log that captures denied requests. Without it, you have no visibility into whether agents attempted to exceed their permissions or whether prompt injection attempts reached the execution layer.

OWASP's Top 10 for LLM Applications identifies LLM01: Prompt Injection as the highest-priority vulnerability for agentic systems, noting that neither RAG nor fine-tuning fully mitigates this class of attack. The recommended defense combines least-privilege tooling, input and output filtering, human approval for high-risk actions, and infrastructure-layer enforcement. Monitoring the execution path for unauthorized invocation attempts is the only way to detect injection attempts that successfully reached the tool layer before being blocked.

Audit logs for denied tool calls

Composio's centralized audit log records every denied call when an agent attempts to execute an action that violates an administrative policy. The security and compliance overview covers how Composio's audit system handles both permitted and refused execution attempts as first-class events.

Denied call entries follow the same structured format as successful ones, with status and metadata fields that indicate the call was blocked and why.

A denied call entry looks structurally identical to a successful one, with "outcome": "denied" and "policy_result": "blocked" replacing the success fields. This means your SIEM receives a complete picture of execution attempts, not just the subset that completed, and auditors can verify that your controls actively blocked unauthorized actions during the observation period.

Distinguishing agent tool logging from model traces

The two logging layers are architecturally complementary, not redundant. Running model tracing alongside tool-layer logging gives your team complete observability: what the agent reasoned and what it executed. The goal is to run both, correlated on session ID, for full-stack incident investigation.

Audit logs for agent tool execution

Composio decouples tool execution logs from the LLM context for two reasons. First, if credentials appear anywhere in the LLM context window, they can be exfiltrated through prompt injection. Second, logging at the execution layer creates an independent audit trail outside the model's reasoning chain.

Composio's proxy execution layer resolves this structurally. When a tool call is invoked, Composio decrypts the credential inside an isolated runtime, injects it into the outbound HTTP request, and returns only the API response to the agent. The credential never enters the LLM context or application memory, and the log is captured at the execution layer independently of the model.

Auditing tool calls and API responses

When an upstream API changes its authentication model or an OAuth token expires, the failure is invisible to model tracing. The agent produces output that looks plausible while the tool call silently failed or returned stale data.

Slack access tokens carry a short expiry window under Slack's token rotation policy, when an app has rotation explicitly enabled. Without a tool-layer log capturing the 401 Unauthorized response, your debugging workflow starts with the model output and works backward through code, reaching the auth layer after hours of investigation. With a tool-layer log, you query by HTTP status 401, filter by toolkit, and identify the failed refresh in under a minute. The troubleshooting workflow is: identify the incident timestamp from the user report, query the tool execution log for that user and time window, inspect the HTTP status and execution metadata for the affected toolkit, and resolve at the credential or policy layer rather than the model layer.

Defining data retention policies for agent logs

Log retention is a compliance requirement, not just an operational convenience. SOC 2 does not specify an exact retention period, but auditors expect logs to cover the full observation window plus a buffer.

At the infrastructure layer, Composio processes more than 300 million tool calls monthly across 1,000+ apps, which means your retention policy and log architecture need to be designed for operational scale from the start, not retrofitted after your agent fleet grows.

Different agent projects carry different compliance requirements, and Composio supports configurable metadata retention per project, allowing your team to set longer retention windows for regulated workflows and shorter windows for low-sensitivity internal tools. Composio supports configurable retention periods ranging from 7 days to 1 year.

What a retention policy needs to answer

Use this pre-filled snippet to answer the three questions enterprise security questionnaires consistently ask about agent log handling:

Credential storage: API tokens are encrypted with AES-256 encryption and stored in Composio's managed credential vault. Credentials are isolated from the LLM context, application code, and agent memory at the infrastructure layer. No recoverable credential copy is accessible outside the proxy execution runtime.

Audit log retention: Composio's centralized audit log records every tool call including denied calls with user, team, tool, action, and outcome. Audit records hold metadata only, tool-call payloads are never stored. The retention window is configurable per project. Logs are accessible for investigation and can be exported for external SIEM ingestion.

Composio's trust center provides the underlying SOC 2 Type II and ISO/IEC 27001:2022 documentation to attach as evidence alongside this response.

If you are preparing for an enterprise security review, book a technical architecture review to walk through your specific credential handling and audit requirements before the questionnaire arrives.

FAQs

How does tool-layer logging differ from traditional LLM tracing?

LLM tracing captures the model's internal reasoning steps, prompt spans, and token costs, while tool-layer logging records the actual API execution, HTTP status codes, exact payloads, and credential resolutions at the system-of-record boundary. The two layers are complementary: model tracing shows what the agent thought, and tool-layer logging proves what it executed.

What is the standard data retention period for agent tool call logs?

Composio Enterprise supports configurable data TTL per project ranging from 7 days to 1 year, so you align log retention with your compliance requirements.

Can we self-host Composio's logging and credential layer?

Yes. Composio supports self-hosting under the Enterprise tier for teams with strict data residency requirements, allowing you to deploy the credential vault and audit logging layer within your own infrastructure boundary.

What happens to denied tool calls in the audit log?

Denied calls are logged as first-class events with the same field set as successful calls, including the attempted action, the policy rule that blocked it, the user identity, and the timestamp. These entries are the concrete evidence that your policy-as-code controls functioned during the audit period.

Does Composio replace my existing model tracing tool?

No. Composio's tool-layer logging runs alongside LLM tracing tools without replacing them. Model tracing captures the reasoning chain inside your agent framework, while Composio logs what actually executed at the API boundary. Both are needed for complete production observability.

Key terms glossary

Credential isolation: An architectural security pattern where API tokens are stored in an encrypted vault and resolved inside an isolated runtime, preventing them from entering the LLM context or application memory.

Policy-as-code: Security rules evaluated directly in the request path as executable code, ensuring access controls are enforced structurally rather than relying on soft prompt instructions.

Identity anchor: A metadata tag that binds an agent's tool execution back to the specific human user or system process that authorized the task, typically populated via SCIM 2.0 from your identity provider.

Proxy Execute: Composio's server-side execution layer that routes tool calls through a secure proxy, keeping raw credentials entirely out of the agentic application layer and writing the audit log entry before returning the response to the agent.

Non-repudiation: The property of an audit log that makes it impossible for a user or system to credibly deny having performed an action, achieved by binding every log entry to a verified identity and storing it outside the application layer.

Token lifecycle: The full sequence of OAuth operations covering initial authorization, access token issuance, refresh token management, scope validation, and revocation, each of which can fail in ways that are invisible to model tracing but visible to tool-layer logs.