OpenTelemetry Setup and GenAI Semantic Attributes

• internal/telemetry/telemetry.go
• internal/telemetry/genai.go
• Setup
• buildResource
• Version
• GenAIOperationChat
• GenAIProviderName
• GenAIRequestModel
• GenAIUsageInputTokens
• GenAIUsageOutputTokens
• GenAIResponseFinishReasons
• ErrorType

Korvus is an incident-RCA agent: it investigates alerts, calls language models, and produces a report. That work is invisible to an operator unless it is traced. internal/telemetry is the small package that makes the rest of the codebase observable — it stands up an OpenTelemetry SDK, attaches the service identity to every span, and provides the handful of GenAI attribute constructors the agent uses so the emitted attribute names are spec-conformant rather than bespoke.

The package's whole reason to exist is the ordering problem it solves. internal/agent takes its tracer from the global provider at construction time (the tracer is resolved in agent.New), and the Connect layer does the same through an interceptor (the Connect interceptor is built during mux construction). If the SDK lands after those constructors run, every early span is silently dropped — no error, no trace, just a gap where the evidence should be. So the SDK has to be installed before anything instrumented starts, and that is why the CLI installs telemetry third, ahead of the database, the MCP registry, and the agent (see the startup-order contract).

Two design commitments run through the package. The first is that tracing is best-effort: a dropped span must never affect incident correctness, which is why the resource and the exporter are configured to stay off the incident path and why the audit counters persisted in the report keep a session self-contained regardless (the package doc says so explicitly). The second is that attribute names come from the OpenTelemetry semantic conventions, so a conformant backend can group on them without configuration (see the contract the tests pin).

How it works

Setup: one provider, installed globally

Setup is the entry point. It takes the config.Telemetry block and returns a shutdown function the caller defers (Setup).

The first thing it checks is whether an OTLP endpoint was configured. An empty OTLPEndpoint is not an error and not a half-configured state: it installs the OTel no-op provider, so every span created anywhere in the process becomes a cheap no-op, and the returned shutdown is a no-op too (the disabled branch). This is the correct default for a local run, and the config struct documents the same intent (Telemetry).

When tracing is enabled, the flow is: build the resource, create a gRPC OTLP exporter pointed at the endpoint, pick a sampler, construct a TracerProvider, and install it as the global provider. Two details matter to operators. The exporter uses insecure credentials only when cfg.Insecure is set (the exporter options), and the provider is configured with a batcher, never a synchronous processor — the comment in the code is explicit that export must not sit on the incident path (the provider construction). The sampler is parent-based over a ratio, so a sampled trace stays sampled end to end (the sampler).

Finally, Setup installs the W3C TraceContext propagator globally (the propagator). This is what lets a span started inside Korvus continue into the MCP servers it calls: the MCP client's tracing transport injects the context into outbound headers so the server's own spans land in the same trace rather than starting a parallel one (the MCP transport) — the reason the MCP client page treats tracing as a transport concern and not just a tool-call wrapper.

The returned shutdown calls provider.Shutdown and logs a warning if the flush fails, rather than failing the process (the shutdown closure). The CLI does the flushing last in its own shutdown sequence, on a fresh context with a bounded deadline — the original context is already cancelled by then, and an export on a cancelled context would drop exactly the spans the flush exists to save (the CLI's flush, bounded by FlushDeadline).

buildResource: the service identity

buildResource merges resource.Default() with a small set of explicit attributes: service.name from config, service.version from the package-level Version variable, and the deployment environment (buildResource).

It is a separate function rather than a few lines inside Setup for a specific reason: tests can exercise the exact attribute set that Setup installs without needing a live OTLP endpoint, because the test suite builds a real SDK provider over an in-memory span recorder using this same function (the recorder helper).

Version defaults to "dev" and is overridden at build time with -ldflags "-X .../telemetry.Version=v1.2.3" (Version). The CLI logs it at startup, so the version a run reports matches the binary that produced it (the startup log line).

The deployment environment is a constant, "development", with no config knob yet (the environment constant). It exists so Tempo queries can already filter on it before the deployment story settles.

The GenAI attributes: pinned, and deliberately so

internal/telemetry/genai.go holds the attribute constructors. Every one of them returns an attribute.KeyValue built from the generated gen_ai.* semantic-convention package, so the wire names are exactly the ones a conformant backend expects (the GenAI helpers).

The interesting engineering decision in this file is a version split that looks like a bug and is not. The GenAI attributes are generated from semconv v1.41.0, while buildResource uses v1.43.0 (the two imports and the GenAI import). The generated Go package dropped every gen_ai.* declaration in v1.42.0 — the conventions were removed "due to lack of applicability" — but the attributes remain current in the specification itself. Pinning the last version that still generates them is what keeps the emitted names spec-conformant instead of hand-rolled; the resource keeps the newer schema URL because nothing it carries moved (the rationale, in full).

Drift is caught rather than tolerated: genai_test.go asserts the literal wire strings — gen_ai.operation.name, gen_ai.provider.name, gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, gen_ai.response.finish_reasons, and error.type — because comparing against the semconv package's own constants would be a tautology that moves with every version bump (the literal-name test). A semconv bump that renames or re-scopes one of these names fails the tests instead of silently emitting an attribute no backend recognises.

What each attribute carries

GenAIOperationChat marks a span as a chat completion — the only GenAI operation Korvus instruments today. The embeddings path is not yet traced and gets its own operation value later (GenAIOperationChat).

GenAIProviderName maps a Korvus provider type onto the spec's gen_ai.provider.name enum. anthropic maps to anthropic; openai_compatible maps to openai, because the spec's discriminator is the telemetry flavour rather than the vendor — anything speaking the OpenAI REST API reports openai even when it proxies another vendor's model (the mapping). An unrecognised type is passed through rather than dropped: a wrong-but-present discriminator is more useful to an operator than a missing one, and the model attributes still identify the endpoint. The test pins all three cases (the enum test).

GenAIRequestModel carries the model id asked for (GenAIRequestModel), and the token counters GenAIUsageInputTokens and GenAIUsageOutputTokens carry the per-call counts (the token helpers). Those counters are the attributes cost dashboards aggregate on, which is the whole reason standard names are used here.

GenAIResponseFinishReasons carries Korvus's normalized stop reason. The spec types it as a string slice because a provider may return one reason per generated choice; Korvus requests a single completion, so it always emits exactly one element — and the cardinality is part of the contract, not an implementation detail, so the test asserts the slice rather than a bare string (the helper, the cardinality test).

ErrorType is the standard error.type discriminator. Korvus sets it to its own error-kind string so retry behaviour is queryable: the kind is what decides retry-versus-failover, so it is the field an operator filters on when asking why a run stalled (ErrorType, and the error-kind taxonomy in the model providers page).

How the agent uses these attributes

The agent's model-call path is the main consumer. A logical model call opens one parent span named gen_ai.chat, carrying the operation attribute and the incident id (the parent span). The provider identity — provider name and requested model — is set on that parent each time a provider is entered, deliberately not only on the success path, so a call that never succeeds still names what it was talking to; re-setting the same keys is last-write-wins in OTel, so the success path needs no copy of it (the provider identity block).

On success, the parent gains the token counters, the finish reason, and the attempt and failover counts (the usage block). On failure, recordCallFailure sets error.type from the error kind alongside the same counters, so the failure case — when an operator most wants to know how much the loop spent before giving up — carries the same shape as the success case (the failure path).

Each individual attempt gets its own child span, gen_ai.chat.attempt, with the provider and model attributes and the attempt number, so a retried call shows each attempt with its own duration and error kind rather than collapsing into one (the per-attempt span). Between attempts the parent records a gen_ai.retry event carrying the attempt number, the error type, and the backoff delay (the retry event); a provider failover is likewise an event rather than an attribute, because a session may fail over more than once and an attribute would keep only the last (the failover event).

The rest of the tree is instrumented in the same style but with different attribute vocabularies: the database layer emits db.query and db.batch spans through a pgx tracer so incident writes appear in the same trace as the loop that caused them (see the persistence page), and the Connect API is traced by the interceptor mentioned above (see the Connect API page). The MCP servers Korvus queries are separate deployment units with their own spans — the Tempo server is the one that reads them back (see the Tempo MCP server page) — and the deployment stack is where the collector, Tempo, and the OTLP endpoint are configured (see the deployment page).

Relationships

graph TD
    CLI["cmd/korvus/main.go"] --> Setup
    Setup --> buildResource
    buildResource --> Version
    Setup --> noop["noop provider (tracing disabled)"]
    Setup --> exporter["otlptracegrpc exporter"]
    Setup --> propagator["TraceContext propagator"]

    AGENT["internal/agent/caller.go"] --> GenAIOperationChat
    AGENT --> GenAIProviderName
    AGENT --> GenAIRequestModel
    AGENT --> GenAIUsageInputTokens
    AGENT --> GenAIUsageOutputTokens
    AGENT --> GenAIResponseFinishReasons
    AGENT --> ErrorType

    MCP["internal/mcp/tracing.go"] --> propagator
    STORE["internal/store/tracer.go"] --> Setup
    TRANSPORT["internal/transport/transport.go"] --> propagator

    CLI --> CONFIG["internal/config/config.go"]

The dashed edge in words: internal/store/tracer.go resolves its tracer from the global provider, so it only starts emitting once Setup has installed the SDK — and if the no-op provider was installed instead, delegation is once-only and that tracer stays dead for the life of the process.

Configuration

Everything Setup reads comes from the telemetry block of the YAML file, described in the configuration reference:

Reference

Generated by astrolabe from commit 0c2dca73 on branch main.