Agent Run Loop, Tool Execution and Provider Calls

• internal/agent/agent.go
• internal/agent/caller.go
• internal/agent/toolexec.go
• internal/agent/guard.go
• internal/agent/layered.go
• Agent
• RunRequest
• ToolExecutor
• TurnLogger
• InheritedArtifact
• WorkspaceRef
• callModel
• completeOnce
• executeToolCall
• repeatGuard
• layeredTools
• truncateResult
• ErrProviderExhausted

The internal/agent package is the heart of Korvus: the bounded ReAct loop that investigates one incident and always lands on a report. Everything else in the repository is plumbing around it — the MCP Client, Registry and Server Connection supplies the tools, the Model Providers, Embeddings and Error Kinds supply the completions, and the Session Manager and Lifecycle owns the rows that the loop fills in. This page walks the loop itself: how one model call is made and retried, how one tool call is dispatched and turned back into a transcript entry, and which rails stop a run from spinning.

The package states its own philosophy at the top of agent.go: the loop is bounded, the system prompt is static, sessions are stateless from the agent's point of view, and all I/O goes through MCP agent.go. That last rule is why the loop has no HTTP client, no database handle and no file access of its own — it only knows the ToolExecutor interface ToolExecutor, which *mcp.Registry satisfies.

How it works

Entering a run

Agent.Run is the front door. It validates the request, builds the session, opens the first burst and then runs the investigation to a landing Agent::Run. Validation happens first and is deliberately paranoid about the incident boundary: a request whose session belongs to a different incident, or that carries an inherited artifact from another incident, is rejected before any of it reaches a prompt RunRequest::validate. The database enforces the same invariant with composite foreign keys, but the agent re-checks it because by the time a mismatched artifact has been described to the model the leak has already happened.

The request itself is a plain struct — session, incident, operator prompt, inherited artifacts, workspace reference, per-session tools, template path and skill catalog RunRequest. Two of those fields exist purely to keep the agent ignorant. Workspace is a WorkspaceRef that locates the pinned checkout, but the agent never touches the directory: it reaches the files only through the workspace tools that the caller injects as SessionTools. And TemplatePath is a path, never the template's content, because the model reads the house format out of the workspace repository at the pinned commit.

The Agent itself is small and safe for concurrent use — all mutable state lives in the per-incident Session Agent. It holds the ordered provider list, the shared tool executor, the turn logger, an optional broadcaster and a tracer.

The burst loop

The unit of work is a burst: one autonomous stretch of the ReAct loop. Run opens burst one; Continue opens the next on a session that is still open Agent::Continue. A burst always begins by persisting the message that prompted it as a user turn, before the first model call, so a replay of the session shows both halves of the conversation rather than an assistant talking to nobody Agent::openBurst.

runBurst is the loop body itself Agent::runBurst. Every iteration begins with the same checks, in this order: the context is alive, the session wall clock has room, the providers are not exhausted, and both iteration ceilings have room. Only then does it increment the two turn counters and call the model. The rule the code enforces is simple — no path reaches the model without incrementing both counters.

The two ceilings are deliberate constants rather than configuration safety rails: 20 model calls per burst and 60 cumulative per session. Both are pathology rails, not budget caps; no session is ever stopped for spend, because killing a run partway has already cost the money and thrown away the benefit. The wall clock is checked here rather than carried on the context, because no single context spans the gap between bursts — a follow-up arrives minutes later on a request context of its own. It counts active time, so reopening a session hours later does not trip it immediately.

What ends a burst is always an ExitReason, never an error. The loop's only error return is a turn-persistence failure; analysis outcomes are recorded as exit reasons and land on the session ExitReason. Every break in runBurst sets a reason, and an invariant guard at the bottom catches a future break that forgets to.

One model call

callModel is the only place model-call errors are retried and the only code that knows more than one provider exists Agent::callModel. It opens one gen_ai.chat span for the logical call, then walks the provider list in order. The failover is one-way and sticky for the session: sess.providerIdx only moves forward, so once a session has fallen over to the fallback it stays there.

Within a provider, up to maxCallAttempts attempts are made, each one a full Provider.Complete round-trip with its own timeout and its own gen_ai.chat.attempt span Agent::completeOnce. Classification decides what happens next. Overflow and bad-request errors are deterministic — retrying is spam — so the call fails immediately caller.go. Auth errors fail over immediately, because a wrong key stays wrong. Everything else (rate limit, overloaded, server, timeout) is transient and earns a jittered exponential backoff, unless the provider supplied a Retry-After, which is honoured but capped at 60 seconds so a sleeping session does not eat its budget caller.go. The jitter is not optional: Korvus's normal operating condition is an alert storm where several sessions hit the same limit at once, and unjittered retries synchronize newBackoff.

When every provider has run out of attempts, the call returns an error wrapping ErrProviderExhausted ErrProviderExhausted. That condition is sticky and is checked at the top of every burst, so a follow-up on a dead session exits as provider_failed before making any model call at all Agent::providersExhausted.

One overflow deserves special mention. When the provider reports a context overflow, the loop makes exactly one mechanical prune of old tool results and one retry — never a loop of prunes, never an LLM summarization sub-call agent.go. If nothing can be pruned the run exits as context_overflow rather than re-sending a byte-identical request.

One tool call

The model answers with content and possibly a list of tool calls. If the stop reason is length, the calls are dropped outright — the arguments may be cut mid-string, and executing them would break the transcript pairing invariant agent.go. The assistant turn is persisted before any tool runs, so a crash mid-execution does not lose the transcript.

executeToolCall dispatches one call and never returns an error Agent::executeToolCall. Every failure — cancelled session, unknown tool, malformed arguments, MCP error, timeout — comes back as an error-flagged tool result, because the transcript pairing invariant has to hold by construction and the model needs to see what went wrong in order to self-correct. The error text is deliberate prompt engineering: specific messages ("arguments for tool X were not valid JSON… re-emit the call with corrected arguments") produce self-correction, vague ones produce doom loops. Malformed argument JSON is never repaired silently, because that would execute a query the model never wrote and the audit trail would lie.

Before the call goes out, a tool_start event is published Agent::executeToolCall. It is announced before dispatch rather than after, because the call can block for up to a minute and a watcher that only saw completed turns would show nothing during exactly the window someone is watching. The event is deliberately unpersisted — the result turn that follows is the durable record, and the broadcasting layer is described in Agent Session State, Resume and Event Broadcasting.

The result that comes back is capped by truncateResult truncateResult. It keeps the head and the tail of the payload and drops the middle, because the last log lines before a crash are the valuable ones for an RCA; the notice spliced into the middle tells the model how to narrow the query. Cuts land on line boundaries and never split a UTF-8 rune cutAtBoundary.

Rails and pathology detection

Three separate mechanisms stop the loop from spinning, and each has its own exit reason.

The repeat guard detects no-progress loops by hashing both the call (name plus canonicalized arguments) and the result repeatGuard. Matching on both matters: a repeated call whose result changes can be legitimate polling, while identical results prove nothing is happening. At two repeats a warning is injected into the result the model sees; at four the burst halts with repeat_call_halt Agent::runBurst.

Bad tool use is counted as strikes: an unknown tool name or unparseable arguments increments a counter, and three consecutive strikes halt the burst with bad_tool_use. Strikes must be consecutive — three malformed calls scattered across twenty productive turns are not the pathology the rail exists for — so a turn that produced no strike resets the counter agent.go. The cap on tool calls per turn is separate: past eight calls in one turn, the surplus calls get an error result explaining they were not executed, because leaving a call without a result would earn a 400 on the next request agent.go.

Empty or truncated responses are handled by handleNoToolCalls Agent::handleNoToolCalls. A filtered response ends the burst immediately, because content filters do not un-trigger. A length-truncated response gets a nudge asking the model to re-issue the calls, and an empty response gets a "continue the investigation" nudge; after two empty retries the burst ends as empty_exhausted.

Tool layering

Tools reach the agent through the ToolExecutor interface, but a session may need its own tools — the workspace tools rooted in that session's scratch directory, since the agent is shared and every session's directory differs. layeredTools puts the session's tools in front of the agent's shared executor layeredTools. A name present in both resolves to the session's tool, and the shadowed shared tool is left out of the advertised list so the tools the model is offered and the set Call dispatches agree layeredTools. Refusing the collision instead, the way the MCP registry does, would break every session of a deployment whose remote server happens to export a workspace_* name. The merged list is sorted by name, matching the registry's ordering, because the tool list is part of the prompt and a stable order keeps the provider's prompt cache warm.

The workspace tools themselves are covered in Workspace Sandbox Client and Built-in Tools and Workspace Sync, Checkout and Materialization.

Landing

Whatever the burst ended on, investigateAndLand converges it: the burst, then the author phase, then finalization Agent::investigateAndLand. The author phase runs before finalize, never after — the JSON report is the guaranteed landing and authoring is the richer artifact layered on top of it, so authoring must not be able to prevent it. Finalization, prompt construction and the report schema are described in Agent Prompts, Report Authoring and Finalization and Incident Report Schema.

Error handling

The agent distinguishes sharply between infrastructure failures and analysis outcomes. Run returns an error only for turn persistence and a malformed request; a session that hit the budget, timed out, lost its provider or was halted by a pathology still returns a session with an exit reason and a report Agent::Run. The one exception is a malformed request, which returns a nil session because validation runs before any session exists.

Provider errors are classified once, at the adapter boundary, into a model.ErrorKind KindOf, and the loop switches on that classification rather than on error strings. The Model Providers, Embeddings and Error Kinds page covers the classification in detail.

Turn persistence uses a context with its own 10-second timeout, detached from the session's cancellation, because the database is the source of truth and a cancelled incident must not lose its transcript Agent::logAndBroadcast. The write happens before the broadcast for the same reason: a client that drains then subscribes can never miss a turn.

Observability

The loop emits three nested spans. agent.run wraps the whole investigation; gen_ai.chat wraps one logical model call; gen_ai.chat.attempt wraps one attempt Agent::completeOnce. The middle span is what makes "how many attempts did this one call cost" answerable, and it carries the attempt and failover counts even on the failure path, which is when an operator most wants them Agent::recordCallFailure. A provider failover is recorded as a span event rather than an attribute, because a session can fail over more than once and an attribute would keep only the last one caller.go. Tool calls emit an mcp.tool_call span carrying the tool name, duration, result size and whether the result was truncated Agent::executeToolCall. The attribute conventions are described in OpenTelemetry Setup and GenAI Semantic Attributes.

Relationships

flowchart TD
    SM["Session Manager"] -->|"Run / Continue"| AG["Agent"]
    AG --> RB["runBurst"]
    RB --> CM["callModel"]
    CM --> CO["completeOnce"]
    CO --> PR["model.Provider"]
    PR --> MP["Model Providers"]
    RB --> EX["executeToolCall"]
    EX --> TE["ToolExecutor"]
    TE --> RG["mcp.Registry"]
    TE --> LT["layeredTools"]
    LT --> WT["Session workspace tools"]
    RB --> GU["repeatGuard"]
    RB --> TR["truncateResult"]
    CM --> PE["ErrProviderExhausted"]
    AG --> TL["TurnLogger"]
    TL --> ST["store.Store"]
    AG --> BC["TurnBroadcaster"]

Reference

Generated by astrolabe from commit 0c2dca73 on branch main.