Checkout Demo Service and Telemetry

• demo/checkout/cmd/demo-checkout/main.go
• demo/checkout/internal/shop/shop.go
• demo/checkout/internal/shop/http.go
• demo/checkout/internal/shop/load.go
• demo/checkout/internal/telemetry/loki.go
• demo/checkout/internal/telemetry/metrics.go
• demo/checkout/internal/telemetry/otlp.go
• Service
• Release
• LokiPusher
• SpanExporter
• GenerateLoad

The demo/checkout module is the Korvus demo's toy backend: a checkout endpoint that runs under steady synthetic load and emits real Prometheus metrics, Loki logs and Tempo traces. Its purpose is to give the rest of the stack — the MCP servers, the dashboards, the runbook — something honest to talk to. The demo imports nothing from the root Korvus module, and Korvus never talks to it directly except through the bundled MCP servers (Bundled MCP Servers Overview).

The interesting part is the fault it ships with. A "bad release" lowers the simulated database pool from 16 connections to 2, so under load requests queue for a connection, time out and fail with 503 demo/checkout/internal/shop/shop.go. The regression is then visible in four independent signals: the deploy log line, the pool-saturation gauge, the error logs, and the db.pool.acquire spans — which is exactly the correlation an investigator is meant to practise.

How it works

The binary is demo-checkout, a main package that wires everything together demo/checkout/cmd/demo-checkout/main.go. It starts with the healthy release, registers the metric families, and starts two background flushers — one for logs, one for traces — each ticking every two seconds flushInterval. If LOKI_URL is empty the log sink is left unset, and if OTLP_ENDPOINT is empty the span sink is left unset; in both cases the service substitutes an internal no-op so it still runs without a telemetry backend main.go and nopLogs.

The request path is a small state machine in the shop package. Service holds a mutable Release and a pool — a channel-based semaphore whose capacity is the configured db.pool.max_conns Service and newPool.

Checkout is the heart of it Service::Checkout. It mints a trace ID and a root span ID, then tries to take a connection from the pool with acquire, which returns ErrPoolExhausted if no slot frees up within the acquire timeout pool::acquire. The wait is timed and recorded as its own db.pool.acquire span carrying db.pool.max_conns and db.system attributes, and it is exported whether it succeeded or failed — so the queueing delay is visible in the trace, not just inferred from the error. Only after a connection is held does the service run the two downstream Steps (a database query and a payment authorisation), each wrapped in its own client-kind span Service::downstream.

A successful request increments checkout_requests_total{code="200"} and observes checkout_request_duration_seconds; a failure increments the code="503" series and stamps the root span and the log line with the error text Service::Checkout. The error returned to the HTTP layer becomes a 503 there http.go.

Fault injection is a deploy, not a special code path. Deploy swaps in a fresh pool sized to the new release and leaves the old one alone, so in-flight requests keep the connection they hold and return it to the pool they took it from Service::Deploy. It also records the config delta as db.pool.max_conns 16 -> 2 in the deploy log line, which is the single cheapest clue in the whole demo.

HTTP surface

Handler mounts five routes on a ServeMux Handler:

Load generation

GenerateLoad fires Checkout at a fixed arrival rate, each call in its own goroutine, so a slow request never throttles the arrival rate — a stalled pool must show up as queueing rather than as fewer requests GenerateLoad. It stops new arrivals on context cancellation but lets in-flight checkouts finish, deliberately: a shutdown that logged context-cancelled database errors would be read by the runbook as a slow-database signal.

Metrics

The telemetry package implements a small Prometheus registry from scratch — stdlib only, on the grounds that five metric families do not justify pulling in client_golang metrics.go. Registry holds metric families and renders them in the Prometheus text exposition format, with series sorted by label string so scrapes are byte-stable Registry::WriteText.

Histograms count an observation into the first bucket whose upper bound it fits under, keeping cumulative bucket counts plus a running sum and count Histogram::Observe. Counters and gauges are per-label-set series; GaugeFunc registers an unlabelled gauge whose value is read at scrape time, which is how the live pool occupancy is exposed Registry::GaugeFunc.

The service registers five families New:

The service also zero-initialises the 200 and 503 request series at startup, so a pre-fault baseline reads as 0 instead of "no data" and an error-rate query does not divide by nothing New.

Logs

LokiPusher buffers log lines grouped by their label set and ships them to Loki's push API at baseURL + /loki/api/v1/push NewLokiPusher. Push never blocks on the network; it appends the line to the stream matching its labels, creating the stream on first use LokiPusher::Push. Flush sends everything buffered so far in one request and drops a failed batch rather than retrying it, because a retry queue would reorder lines behind newer ones LokiPusher::Flush.

The service writes one logfmt line per request through Service::log, with the stream labels service=checkout, env=demo and level=… attached to the line Service::log. The line body carries order_id, status, duration_ms, version and trace_id, plus err on failure — the trace_id is what lets a log line be pivoted into the trace and vice versa. Non-request events such as deploys are additionally mirrored to slog so they show up in docker compose logs, whereas per-request lines are not, because at 25 rps they would drown stdout Service::log.

Traces

SpanExporter buffers finished spans and ships them as OTLP/HTTP JSON to endpoint + /v1/traces, tagging the resource with service.name NewSpanExporter. Flush converts each span into an otlpSpan — hex trace and span IDs, nanosecond start and end, sorted attributes, and a status code of 1 for OK or 2 with a message for errors — and posts them wrapped in resourceSpans / scopeSpans SpanExporter::Flush. A failed batch is dropped, mirroring the Loki sink's policy SpanExporter::Flush.

Both sinks share a post helper that treats any non-2xx response as an error and surfaces the first 512 bytes of the body post.

Failure handling and back-pressure

Neither sink may block the request path, so both are bounded: a per-sink cap of 20 000 buffered entries, after which new entries are counted as dropped and the demo keeps serving maxPending and LokiPusher::Push. The drop count is logged on the next flush as "buffer full, lines dropped".

RunFlusher drives a sink on a ticker and, when the context is cancelled, performs one final flush on a fresh five-second context so the last batch before shutdown is not lost RunFlusher. Flush failures are logged as warnings rather than propagated.

On the service side, the only expected error is ErrPoolExhausted, which surfaces as a 503 with the pool size and the wait time folded into the message Service::Checkout.

Relationships

graph TD
  main["demo-checkout main"] --> shop["shop.Service"]
  main --> load["GenerateLoad"]
  main --> reg["telemetry.Registry"]
  main --> loki["telemetry.LokiPusher"]
  main --> exp["telemetry.SpanExporter"]
  load --> shop
  shop --> loki
  shop --> exp
  shop --> reg
  handler["shop.Handler"] --> shop
  handler --> reg
  loki --> flush["RunFlusher"]
  exp --> flush

The demo's signals are consumed by the bundled MCP servers that Korvus itself uses: the Prometheus MCP Server scrapes the /metrics exposition, the Loki MCP Server queries the log streams the pusher writes, and the Tempo MCP Server queries the traces the exporter writes. How those are wired into a running stack is covered in Deployment Stack and Example Configuration; for how the surrounding repository fits together, start at Korvus Overview.

Reference

Generated by astrolabe from commit 0c2dca73 on branch main.