Architecture
End-to-end tracing for an AI agent behind two API gateways
A reference architecture that keeps one W3C trace from the browser to the model, enforces baggage trust boundaries, and keeps prompts out of telemetry.
AI-generated, human-reviewed: the first draft was generated by an AI agent; I fact-checked, tested, and edited it. AI use policy
Architecture#
Dataflow#
- The browser SPA starts a W3C trace and calls
POST /api/chatwithtraceparent. - The front gateway applies CORS and per-IP rate limits, deletes caller
baggage, and forwards the trace headers unchanged. - AgentApi creates
conversationIdandturnId, sets one approved baggage value, and runs the agent. - The agent calls the model through the AI gateway on the same trace; the AI gateway deletes baggage again and authenticates with a managed identity.
- When the model asks for a tool, the agent runs it and calls the model a second time, still on the same trace.
- AgentApi replies first, then queues an evaluation record. Service Bus carries the trace to a Function that writes the record to Cosmos DB.
- Every hop reports privacy-safe markers to Application Insights, where one query replays the journey.
Components#
| Component | Role in this architecture |
|---|---|
| Application Insights JS SDK | Starts the W3C trace in the browser; no headers or cookies collected |
| API Management (front gateway) | CORS, rate limits, baggage trust boundary, trace pass-through |
| ASP.NET Core 10 (AgentApi) | Hosts the chat endpoint and the agent; OpenTelemetry instrumentation |
| Microsoft Agent Framework | Runs the agent, its tools, and GenAI semantic-convention spans |
| API Management (AI gateway) | Token limits, managed identity to the model, second baggage boundary |
| Microsoft Foundry model | Answers the prompt or requests a tool call |
| Service Bus and Azure Functions | Carry evaluation records after the reply, with trace context |
| Cosmos DB | Stores prompts, replies, feedback, and evaluation results, keyed by turnId |
| Application Insights | Stores IDs, timings, outcomes, and hop markers, never content |
Considerations#
Reliability#
The content lane uses a bounded in-memory queue so the reply never waits for storage. Records can be dropped on a crash or when the queue is full; the POC counts drops, and production should alert on them or use an outbox. Conversation memory is in-process and does not scale across instances.
Security#
Caller baggage is deleted at both gateways. AgentApi rejects calls that skip
the front gateway, and the AI gateway reaches the model with a managed
identity. The reference deployment uses public endpoints with IP restrictions;
production should consider private endpoints and virtual network integration.
Cost optimization#
Two API Management Developer instances dominate the bill. Cosmos DB serverless, Service Bus Basic, a B1 App Service plan, and a Flex Consumption Function keep the rest small. Delete the deployment when you finish testing.
Operational excellence#
Hop markers, KQL queries, and a workbook make one trace readable by humans. The POC samples at 100 percent; production needs aligned sampling across the browser, gateways, API, and Function so traces stay whole.
Performance efficiency#
The reply-first content lane keeps evaluation writes off the user’s critical path. Returning the trace ID with each answer makes slow hops quick to find.
When to use this architecture#
- You run an AI agent behind API Management and need to explain where time goes, end to end.
- You must evaluate answer quality without putting prompts and replies in operational telemetry.
When not to use it#
- A single internal service with no gateway and no browser client; standard ASP.NET Core instrumentation is enough.
- You cannot run two gateways; fold the AI gateway policies into one gateway, and keep the baggage boundary.
Build it#
The post Tracing an AI agent from click to model walks through it, and these complete snippets are the working parts: