Tempo MCP Server

• mcp-servers/tempo/internal/tempo/client.go
• mcp-servers/tempo/internal/tempo/format.go
• mcp-servers/tempo/internal/tools/tools.go
• SearchInput
• GetTraceInput
• TraceSummary
• SpanSet
• SearchSpan
• ServiceStats
• Attribute

mcp-tempo is Korvus's distributed-tracing tool server: a standalone process that exposes two read-only MCP tools — a TraceQL search and a fetch of one trace by ID — over the Grafana Tempo HTTP API. It exists because an incident investigation usually needs to move from "the checkout service got slow at 02:14" to "these five requests failed, and here is the span where the time went", and that chain runs through Tempo's query API rather than through logs or metrics. Logs tell you what was said, metrics tell you how much, traces tell you which call.

The server is deliberately a separate deployment unit. Korvus never spawns it; the container runtime does, and Korvus reaches it over an HTTP MCP endpoint listed in mcp.servers (run, configuration example). It imports nothing from the root module, which keeps its dependency budget down to one MCP SDK. It shares its shape with the other bundled servers — see the Bundled MCP Servers Overview — and it reads the same TimeQL/TraceQL-adjacent conventions as the Loki MCP Server and the Prometheus MCP Server.

How it works

The process starts in main, which delegates to run so that every deferred cleanup runs before the process exits (main). run reads two flags — -addr (default :9102, overridable with MCP_TEMPO_ADDR) and -tempo-url (default http://localhost:3200, overridable with TEMPO_URL) — builds a tempo.Client with a 30-second per-request timeout, creates one sdkmcp.Server, and calls tools.Register on it (run). The same server instance is handed to every MCP session because the tools are stateless and the upstream HTTP client is safe for concurrent use. The mux exposes /mcp for the streamable HTTP handler and /healthz for liveness probes.

The tool surface

Register adds exactly two tools, both annotated read-only (Register).

tempo_search takes a TraceQL query, a start and end time, and an optional limit, modelled by SearchInput (SearchInput). The tool validates before it calls: a blank query is rejected with an example expression, the two timestamps are parsed as RFC 3339, and end must be strictly after start. The limit is clamped rather than rejected — a non-positive value becomes the default of 20 traces and anything above 100 is cut to 100, because spending a whole tool call teaching the model a constant is worse than answering the question it meant to ask (clampLimit). The default and the ceiling live with the client and the formatter, so DefaultSearchLimit and MaxSearchLimit are shared by the tool layer and the rendering layer (the default trace budget).

tempo_get_trace takes a trace ID and an optional span ID, modelled by GetTraceInput (GetTraceInput). Without span_id it returns a whole-trace summary — services, error spans, span tree. With span_id it returns that one span in full. The trace ID is normalized to 32 lower-case hex characters before use, so the ID the model reads out of a search result is byte-identical to the one it would grep a log for, even though Tempo's search output strips leading zeros (the trace ID normalizer). A span ID must be exactly 16 hex characters, and a malformed one is rejected with the shape the model should pass.

Timestamps are the one argument form the model can get wrong in many ways, so the accepted format is stated identically in every time argument, in one shared sentence that the other bundled servers reuse (the shared timestamp rule). The parser accepts exactly RFC 3339 and, on failure, names the offending field and shows the expected shape — the model's next action is to retry, and it can only fix what the message tells it (parseTime).

The client

tempo.Client is a thin HTTP client for one Tempo API root, holding the base URL and an http.Client whose timeout bounds each request end to end (Client, NewClient). It deliberately does not import Grafana's own Tempo packages: those pull in the whole Tempo server tree for two GETs.

A search builds a query string with q, start, end, limit and spss (matched spans per spanset, fixed at 3) and issues GET /api/search; the end timestamp is rounded up to the next whole second so a span starting in the final fraction of a second stays inside the window the model asked about (the search request). A trace fetch issues GET /api/v2/traces/{id} with the ID escaped as a path segment, so an unescaped value could never reach a different endpoint (the trace fetch).

The response decoding is where most of the real work lives. Search results become a SearchResult holding TraceSummary values; each summary carries the trace ID, root service and operation, start, duration, its SpanSet groups and per-service ServiceStats (SearchResult, TraceSummary). A SpanSet is one TraceQL spanset selector's group of matched spans: Matched counts every match, while Spans holds only the first few Tempo returned (SpanSet, SearchSpan). ServiceStats counts one service's spans and errors within the matched trace (ServiceStats), and every key/value pair is flattened into an Attribute whose value is already rendered as text (Attribute). Tempo's deprecated singular spanSet field is read only when the plural spanSets is absent, never in addition to it.

Trace fetches return OTLP/JSON, which the client flattens: resourceSpans / scopeSpans nesting is unwrapped into a flat list of Span values, each carrying its parent ID, service name, attributes, resource attributes and events (Span). Two encoding quirks are handled here. Span and trace IDs arrive either as hex or as base64 (the protobuf JSON mapping of a bytes field), and the two are told apart by length — a base64 span ID is 12 characters with padding, a hex one 16 — so both are decoded to lower-case hex. OTLP enums arrive either as a name (SPAN_KIND_SERVER) or as the integer the spec prescribes, and are normalized to short strings such as server and to UNSET / OK / ERROR status codes (the OTLP decoding).

A Span's duration is clamped at zero: a span whose end precedes its start, which happens under clock skew between hosts, reports zero rather than a negative duration (Span::Duration). Error detection is a status comparison, not a heuristic (Span::IsError).

Rendering

format.go is where a raw Tempo payload becomes something a model can read in one tool call. The guiding constraint is Korvus's own agent loop: any tool result above 32 KiB is truncated head-and-tail, which for a trace is the worst possible cut, because it keeps the root and the last leaves and drops the middle of the tree where the failing call usually is. Every renderer therefore works inside a 28 KiB budget, with headroom for the notices that explain a trim (the result byte budget).

FormatSearch renders a search result oldest trace first — Tempo returns traces in the order its search jobs complete, which differs between two runs of the same query, so sorting makes the output deterministic (FormatSearch). Each hit gets an ID line, per-service span and error counts, and the first spans that matched. When the whole first trace does not fit, its header alone is still emitted, because a reply with no trace ID gives the model nothing to do but repeat the same query. Two different trims are reported separately: traces omitted for budget reasons, and the case where Tempo stopped at the requested limit — the latter is the fact that makes a model mistake a sample for the whole picture.

FormatTrace renders a whole trace as a summary, never as raw spans (FormatTrace). It leads with the services involved and the error spans — each with its status message and exception summary — and then prints a span tree in which runs of same-named siblings are collapsed into a single line, since a fan-out of hundreds of identical cache calls is one fact, not hundreds. The error section is capped at half the budget so a trace in which everything failed still shows the tree that explains how the failures relate. Spans whose parent is missing from the trace become extra roots and are reported as a partial trace, because hiding the fragments of an incomplete trace would hide exactly the spans that did arrive. A parent-ID cycle, which has no root, is broken by promoting the unreached spans.

FormatSpan is the drill-in: one span's attributes, resource attributes, events, parent and children, with the attribute list sorted and each value bounded so a single huge db.statement cannot spend the whole budget (FormatSpan).

Errors

Tempo reports a failed request as an HTTP status plus a bare text/plain body, so the client classifies the fault itself. QueryError carries a Kind and Tempo's own message verbatim, and the kind set is this package's own vocabulary — Tempo supplies none (QueryError, the error kinds). A 400 is bad_query, a 404 is not_found, a 429 is too_many_requests, a 5xx is server_error; a client deadline is timeout, a request that never produced a response is unreachable, and a response that was not the API's shape is bad_response (status classification).

One case is deliberately not an error: Tempo answers an unknown trace ID with HTTP 200 and an empty trace, so the client converts that into ErrTraceNotFound (ErrTraceNotFound). The tool layer treats it as an answer, not a failure — it returns a text result explaining that the trace may be older than the retention window or sampled out, and telling the model not to retry the same ID but to search the incident's time range instead.

Every other failure is rewritten by explain into a sentence the model can act on: a bare "parse error at line 1, col 12" says the call failed but not that the fix is the model's own expression, so the bad-query message shows the brace syntax and the span. / resource. scoping rule, the timeout message asks for a narrower range, and the rate-limit message asks for a narrower next query rather than a repeat (explain). The response body is read through a 64 MiB limit, since a trace with tens of thousands of spans legitimately runs to tens of megabytes and the rendered result is a few KiB regardless (the response size limit).

Relationships

graph LR
  main["main / run"] --> `Register`
  `Register` --> `SearchInput`
  `Register` --> `GetTraceInput`
  `Register` --> `Client`
  `Client` --> `SearchResult`
  `SearchResult` --> `TraceSummary`
  `TraceSummary` --> `SpanSet`
  `SpanSet` --> `SearchSpan`
  `TraceSummary` --> `ServiceStats`
  `Client` --> `Trace`
  `Trace` --> `Span`
  `Span` --> `Attribute`
  `Client` --> `QueryError`
  `Client` --> `ErrTraceNotFound`
  `Register` --> `FormatSearch`
  `Register` --> `FormatTrace`
  `Register` --> `FormatSpan`
  `Register` --> `explain`

Configuration

In the dev stack, Tempo itself runs as grafana/tempo:3.1.0 with its query API on port 3200 and OTLP on 4317/4318, and the checkout demo service exports spans into it (docker-compose.yml). The trace data this server serves therefore comes from real OTLP exporters — see OpenTelemetry Setup and GenAI Semantic Attributes and Checkout Demo Service and Telemetry.

Reference

Generated by astrolabe from commit 0c2dca73 on branch main.