Consider a customer-support agent handling a refund. It reads the customer's message, retrieves an order, checks account notes, and drafts a response. The production database holding that information may be encrypted, tightly permissioned, and audited.

The trace of the interaction can tell a different story. Depending on the tracing library and its configuration, it may contain the original message, the agent's private instructions, the arguments sent to each tool, the full customer record returned by the database, and the final response. That information can then be exported into a separate observability platform used by developers to diagnose failures.

The trace started as a debugging aid. Once it contains interaction content, it has become a second store of customer and application data.

This article calls that store a shadow database. The name is an explanatory label, not an established industry term. It describes a system that accumulates sensitive, structured information as a side effect of operating another system. Teams may govern the production database carefully while giving the trace backend different access rules, longer retention, or additional downstream exports.

AI applications make this problem unusually easy to create. A conventional application trace might record that a database query took 80 milliseconds. An AI trace can record the conversation that caused the query, the data returned by it, and the model's next action.

The practical conclusion is straightforward: production AI tracing should be metadata-only by default. Raw prompts, responses, and tool data should be captured only when there is a defined need, transformed before they leave the trusted environment, and removed on a short schedule.

A trace is a record of how one request moved through a system. It is divided into spans, which represent individual operations such as calling a model, searching a database, or executing a tool. The spans form a timeline that lets a developer see where an operation slowed down or failed.

For an AI agent, execution includes both conversation and action. That makes its spans far more content-rich than ordinary performance telemetry.

The OpenTelemetry Generative AI conventions standardize fields for details such as the model used, token counts, response status, prompts, outputs, and tool activity. OpenTelemetry's May 2026 walkthrough says prompt and tool content are disabled by default in its example configuration. When content capture is enabled, the resulting attributes can include full prompt messages, system instructions, tool schemas, tool arguments, and tool results.

That detail is valuable when an agent behaves strangely. A developer can follow the exact sequence of model calls and tool invocations instead of guessing. The same detail can also reproduce a private interaction inside a system with a completely different audience.

Tool results are particularly important. A final answer may reveal only two fields from a customer record while the trace preserves the entire object returned to the agent. An application-level privacy control can therefore succeed while the observability layer still keeps the unused data.

No single tracing default applies across the AI ecosystem.

The OpenAI Agents SDK documents that generation spans store model inputs and outputs, while function spans store the inputs and outputs of tool calls. Its trace_include_sensitive_data setting is enabled by default. The documentation also says audio spans include base64-encoded audio data by default unless sensitive audio capture is disabled.

Arize's OpenInference configuration guidance exposes separate controls for inputs, outputs, messages, images, prompts, embedding vectors, and model invocation parameters such as temperature. In the documented configuration table, those hiding controls default to false.

Other systems provide their own mechanisms. LangSmith can hide all trace inputs and outputs, transform metadata, redact recognizable secrets, or disable tracing for requests with zero-retention requirements.

These differences do not mean one tracing project is universally safe and another is universally unsafe. They show that changing an instrumentation library can also change the application's data-collection policy. A team that reviews only its model provider's privacy settings may never examine what the tracing SDK exports.

This becomes more complicated when a trace passes through several components. An application may send spans to a local collector, which forwards them to a commercial backend, which retains them in an index and archive. The team may later export selected traces into an evaluation dataset. Every step can create another copy, another access path, and another deletion obligation.

Useful observability does not require every trace to contain the full interaction. Most routine operational questions can be answered with metadata:

The raw prompt may be necessary for investigating a model-quality failure, but it is unnecessary for measuring latency or identifying a failing service. Combining those use cases under one capture policy encourages teams to collect the maximum amount of data all the time.

A safer design separates traces into three content tiers:

The distinction matters because raw content is not simply a more detailed version of metadata. It is a different class of information. It may contain personal data, credentials, proprietary instructions, customer documents, source code, or records fetched from another system.

The most important place to control trace content is before it crosses into the observability backend.

OpenTelemetry's sensitive-data guidance recommends avoiding collection when possible. Its Collector can also remove attributes, filter entire spans, transform values, or apply a redaction processor that deletes fields outside an allowlist. The Collector security guidance specifically describes scrubbing data before it is exported.

An allowlist is often safer than trying to enumerate every possible secret. A blocklist might recognize a credit-card number or known API-key format, but it will not recognize an unreleased product name, a medical diagnosis, or a confidential sentence written in ordinary language. An allowlist exports only the fields that the team has deliberately approved.

Hashing identifiers also requires care. OpenTelemetry warns that hashing may not provide meaningful anonymity when the original values come from a small or predictable set, such as sequential user IDs. A hash can help correlate events without displaying an identifier, but it should not automatically be treated as anonymous data.

Backend controls still matter. Raw-content traces should have narrower read permissions than ordinary performance dashboards. Access should be recorded, retention should be explicit, and tenant boundaries should continue into the trace store. Encryption protects stored and transmitted traces, but it does not reduce the amount of information available to an authorized account or a compromised account with the same privileges.

The long-standing logging principle still applies. NIST's Guide to Computer Security Log Management recommends avoiding unneeded sensitive data and limiting access to log files. OWASP's Logging Cheat Sheet similarly says access tokens, passwords, encryption keys, sensitive personal data, commercially sensitive information, and data above the logging system's permitted classification should generally be removed, masked, hashed, sanitized, or encrypted rather than recorded directly.

AI traces do not replace those principles. They increase the number of places where teams must apply them.

Trace configuration deserves its own security tests.

A practical test can submit synthetic customer data and a fake credential through a staging interaction, then inspect every destination that receives the trace. The test should verify that prohibited fields never appear in exported spans, backend search results, archives, debugging downloads, or evaluation datasets. Because the values are synthetic, their appearance can be checked without exposing a real secret.

Teams should repeat this test when they change SDKs, enable automatic instrumentation, add an exporter, or modify trace schemas. The failure being tested is data propagation, so reviewing the application's final response is insufficient. The response may be clean while an intermediate span still contains the prohibited value.

Raw-content capture also needs a shutoff mechanism. If a developer enables it to diagnose one incident, the configuration should expire automatically or be tied to a specific request, tenant, or sampling rule. A temporary debugging change that becomes permanent is one of the simplest ways to create a shadow database.

Metadata-only tracing cannot explain every failure. Some model behaviors depend on exact wording, retrieved context, or tool output. Redaction can also remove the detail that a developer needs to reproduce a bug.

That tradeoff does not support unrestricted collection. It supports deliberate escalation. A team can begin with operational metadata, enable redacted context for selected cases, and authorize raw capture when the diagnostic value justifies the exposure. Highly sensitive interactions may need local inspection without external export.

Redaction itself is imperfect. Pattern matching misses context-sensitive information, transformations can break as schemas change, and metadata can still reveal relationships between users, models, and internal tools. A secured collector also cannot compensate for an observability account with excessive privileges or a retention policy that is never enforced.

The goal is therefore not to make tracing risk-free. It is to make the amount of captured content proportional to the operational question being answered.

AI systems need traces. Agents combine probabilistic model behavior with ordinary software and external tools, which makes their failures difficult to reconstruct from error codes alone.

That need for visibility should not turn every private interaction into permanent debugging data. Once a trace contains prompts, tool results, and media, it deserves the same design questions as any other data system: what enters it, where it is transformed, who can read it, where copies go, and when they are deleted.

If a team cannot answer those questions, its trace backend is already functioning as an undefined data pipeline. Calling it observability does not make the data less sensitive.