Alert Ingestion, Fingerprinting and Debounce

• internal/incident/ingest.go
• internal/incident/alert.go
• Ingestor
• IngestorConfig
• AlertGroup
• IncomingAlert
• AlertStore
• SessionStarter
• groupFingerprint
• alertPrompt

The alert path is how Korvus turns raw monitoring traffic into incidents. Something fires — an Alertmanager webhook, or a synthetic trigger from the API — and the path has to decide three things: is this the same problem we already know about, should we open an incident right now, and should we spend an agent session on it. Doing that naively is expensive: a single flapping alert can produce a dozen deliveries in a minute, and each one would otherwise open its own incident and start its own investigation session.

The whole design lives in internal/incident, split across two files. internal/incident/alert.go holds the wire types — what a delivery is — and internal/incident/ingest.go holds the engine that decides what a delivery means. The engine is a single type, Ingestor, and it is deliberately boring: one mutex, one map of pending opens, one timer per fingerprint.

The delivery

A delivery is an AlertGroup: common labels, common annotations, and a list of IncomingAlert values AlertGroup. The Alertmanager webhook builds one from its JSON payload alertGroup, and the synthetic alert path builds a group of one whose common labels are its own labels AlertGroup. IncomingAlert is the alert as received: a status, its own labels and annotations, a generator URL, and start/end times. It carries no ID, no incident reference and no suppression state — ingestion is what decides those IncomingAlert.

One detail in that type is worth spelling out because it is a real trap: EndsAt is nil while an alert is firing, and Alertmanager sends 0001-01-01T00:00:00Z rather than omitting the field, so a receiver must map the zero time to nil instead of storing it IncomingAlert.

The transport layer consumes the ingestor through a narrow interface, AlertIngestor, whose only method is Ingest(ctx, incident.AlertGroup) AlertIngestor. The webhook handler calls it detached from the request context, so that a sender hanging up mid-write cannot leave alerts recorded but never attached Ingest.

Fingerprinting: what counts as "the same problem"

The dedup key is a SHA-256 over the group's common labels minus the configured ignored labels groupFingerprint. The fingerprint is a property of the group, not of the individual alerts — two deliveries that share their common labels are the same problem, even if their individual alert sets differ.

The hashing detail matters more than it looks. The labels are encoded as JSON before hashing, which sorts the map keys, so the key is independent of map iteration order and unambiguous for any label value: a value containing = or a newline cannot collide with a different label set the way a naive k=v\n join could groupFingerprint. The incident.ignored_labels list exists for exactly this reason — labels like pod, instance or uid change on every restart of the target, and including them would make every delivery look like a new problem.

Ingest: the decision tree

Ingest validates, fingerprints, and then takes one of four paths Ingest. Every decision runs under a single mutex held across the store calls, which is what makes "a burst opens exactly one incident" true within this process: the dedup read and the pending-open bookkeeping cannot interleave with a concurrent delivery or with the timer's own open Ingestor.

The first check is the group itself. A delivery with no alerts, or with an alert whose status is neither firing nor resolved, is rejected with ErrInvalidAlertGroup and nothing is written for it validateGroup.

Then the group is split by whether every alert in it resolved, mirroring Alertmanager's own group status allResolved.

For a firing group the order is:

1. An active incident already owns this fingerprint. The store is asked for an open incident with this fingerprint created within the dedup window; the newest match wins GetActiveIncidentByFingerprint. If there is one, the group's alerts are attached to it and marked suppressed — recorded, but not the reason anything happened. Nothing is started.
2. A debounce window is already pending for this fingerprint. The alerts are stored unattached and appended to the pending open. The window's "resolved" flag is cleared, because a fresh firing means the problem is back.
3. Nothing pending, no active incident. This is a new fingerprint. The alerts are stored unattached and, if incident.debounce_delay is zero, the incident opens immediately and the alerts are attached right away Ingest. Otherwise a debounce timer starts.

The store.ErrNotFound returned by the dedup lookup is the "no active incident" signal; any other error aborts the delivery rather than risk opening a duplicate incident Ingest.

Debounce: one incident per burst

A new fingerprint does not open its incident at once. Its alerts are stored unattached and a timer of incident.debounce_delay starts; every further group with that fingerprint joins the pending open, marked suppressed. When the timer fires, one incident opens, every pending alert is attached, and one session starts Ingestor.

The timer callback is fire, which takes the same mutex, checks that its pending entry is still the one it was registered for (a timer that fired just as shutdown cleared the set finds its entry gone and opens nothing), and then calls open under a 30-second timeout — the timer runs detached from any request, so nothing else would bound it fire debounceOpenTimeout.

open is where the incident actually comes into existence. It creates the incident, attaches the pending alert IDs to it, and logs the result open. The incident's title prefers the group's summary annotation, then its alertname label, and falls back to a truncated fingerprint, because an incident list must never show a blank title alertTitle. Its severity comes from the severity label, defaulting when absent, and its started_at is the earliest startsAt among the group's alerts, so the incident reflects the problem's real onset rather than the moment the webhook arrived alertIncident.

Two failure modes here are deliberate. If the group resolved during the debounce window — the flap the delay exists to absorb — the incident is kept as the record of it but closed without a session, because spending an agent run on a problem that has already gone away is waste open. And if the session fails to start, the incident is still opened and the failure is only logged: reporting an error would make Alertmanager redeliver, and the redelivery would dedup into this very incident, so it could never start the session either. An operator can start the session by hand open.

The session itself is started through the SessionStarter interface SessionStarter. The command binary satisfies it with a small adapter that fills the same deployment defaults CreateSession would use — primary model, session.default_template, and the workspace revision currently being served StartSession. The prompt handed to the session is built by alertPrompt: the group's common labels and annotations, sorted so the same group always yields the same prompt, followed by the first delivery's alerts (capped at 20, with a count of the remainder — a group can carry hundreds, but every alert is kept in the alerts table regardless) alertPrompt.

Resolution: closing what the alert opened

A group whose every alert resolved takes a different path, ingestResolved ingestResolved. It closes every open incident of the fingerprint whatever its age — the dedup window bounds which incident a new firing joins, not how long an incident may wait for its resolution, and a re-fire past that window can legitimately leave two incidents open that one resolution must close ListOpenIncidentsByFingerprint.

The resolution's own alerts attach to the newest open incident. If a debounce window is still pending for the fingerprint, the pending open is marked resolved so that its incident opens already-closed and starts no session. A resolution with nothing open and nothing pending is recorded and otherwise ignored — it must never open an incident ingestResolved.

Closing is driven by incident.close_on_resolved, and the close reason names the reason and the config key closeResolved. An incident an operator closed in the meantime is already where this would put it, so store.ErrAlreadyClosed is swallowed rather than treated as a failure closeResolved.

Persistence and lifecycle

The store side of the contract is AlertStore, the alert half of the incident store AlertStore. It is declared in internal/incident rather than in internal/store because an interface belongs to the package that consumes it, and the PostgreSQL implementation lives in internal/store and satisfies it LogAlert AttachAlerts.

Every alert of a group is stored individually with a UUIDv7 ID, the group fingerprint, the received timestamp, and the suppression flag; IncidentID stays nil until the incident exists, which is precisely why the column is nullable and why AttachAlerts exists as a retroactive attribution record Alert.

NewIngestor refuses a config the alert path cannot run with, rather than deferring the failure to the first alert: store and sessions are required, the dedup window must be positive, and the debounce delay must not be negative NewIngestor.

Pending opens live in memory. Stop abandons every pending debounce and waits for a timer already firing to finish; deliveries after Stop are refused with ErrIngestorStopped so the sender retries against the next process — Alertmanager does so on a 5xx Stop. A restart during a debounce window leaves that window's alerts unattached, and the next delivery for the fingerprint starts a fresh window Ingestor.

Configuration

Everything the ingestor does is driven by the incident.* block Incident, which is loaded and validated like the rest of the surface described in Configuration Loading and Validation.

The defaults are populated by Default, so an omitted key means "use the default" rather than "use the zero value" Default. The ingestor is wired from the command binary in newAlertIngestor, which is where the config block becomes an IngestorConfig newAlertIngestor.

Error handling

The webhook maps the first two to HTTP 400 and 503 respectively, and any other failure to a 500, so Alertmanager's redelivery semantics line up with the error kinds Ingest.

Relationships

flowchart LR
  subgraph transport["internal/transport"]
    WH["webhook handler"]
    AG["alertGroup"]
    AI["AlertIngestor"]
    SY["SystemService.TriggerAlert"]
  end

  subgraph incidentpkg["internal/incident"]
    INC["Ingestor"]
    CFG["IngestorConfig"]
    GRP["AlertGroup"]
    INCAL["IncomingAlert"]
    GF["groupFingerprint"]
    AP["alertPrompt"]
    SS["SessionStarter"]
  end

  subgraph storepkg["internal/store"]
    AS["AlertStore"]
    IS["IncidentStore"]
    PS["PostgresStore"]
    SI["store.Incident"]
    SA["store.Alert"]
  end

  WH --> AG
  AG --> GRP
  SY --> GRP
  WH --> AI
  AI --> INC
  CFG --> INC
  GRP --> INC
  INCAL --> GRP
  INC --> GF
  INC --> AP
  AP --> GRP
  INC --> SS
  INC --> AS
  AS --> IS
  IS --> PS
  PS --> SI
  PS --> SA

The ingestor sits between the transport layer and the store: it receives groups over Watch Streams, Read APIs, Webhooks and Proto Conversion and the Connect API Server and Service Handlers, and it writes the incident and alert rows described in Storage Domain Model and Store Interfaces and PostgreSQL Persistence, Migrations and Vector Search. The incidents it opens are the same incidents the Incident Manager: Open, List, Update, Close works with, and the session it starts runs under the Session Manager and Lifecycle. Process-level wiring lives in korvus CLI and Process Wiring; a worked example of the configuration is in Deployment Stack and Example Configuration.

Reference

Generated by astrolabe from commit 0c2dca73 on branch main.