Documentation Chunking, Indexing and Search

• internal/docs/chunk.go
• internal/docs/index.go
• internal/docs/search.go
• Chunk
• Indexer
• SplitMarkdown
• sections
• EmbeddingInput

The internal/docs package turns the markdown under a workspace checkout into an embedded corpus and serves it back to the agent as a single tool, docs_search. The package exists because the agent's questions about a service ("who owns Checkout?", "what does the runbook say about the payment timeout?") are answerable from prose in the repository, and grep cannot find prose — only keywords. Korvus therefore embeds the documentation and lets the model retrieve it by meaning, as a tool call it makes when it realises it needs the documentation, rather than as text pasted into the prompt. The package doc states the two constraints that shape everything below: indexing happens on workspace sync and never per session, and retrieval is a tool rather than pre-injected prompt text, so the system prompt stays static (chunk.go).

The work splits into three files: chunk.go parses markdown into retrievable slices, index.go decides when those slices get embedded and stored, and search.go exposes the result to the agent as an MCP tool.

How it works

Chunking a markdown file

SplitMarkdown is the entry point: given a path and a file's text, it returns the chunks that will be indexed (chunk.go). It first normalises CRLF line endings, then walks the file's heading sections, skipping any section with no body, and finally splits each remaining section into pieces.

The section splitter, sections, tracks a heading breadcrumb while scanning line by line (chunk.go). A line that parses as an ATX heading ends the current section and starts a new one, and the breadcrumb is rebuilt from the levels above it, so a chunk knows it sits under "Services > Checkout". The scanner is fence-aware: while inside a fenced code block, a line that looks like a heading is still content, because "a shell comment in a runbook's code sample is not a heading" (chunk.go).

Not every section deserves a chunk. A section carries a hasBody flag that reports whether it holds a non-blank line other than the heading itself, because "a heading immediately followed by a sub-heading is structure, not content, and indexing it would put an empty chunk in every top-k" (chunk.go). SplitMarkdown drops those sections before splitting (chunk.go).

Long sections are cut by splitSection, which packs the section's lines into pieces of at most maxChunkBytes and cuts only between lines (chunk.go). The budget is 2000 bytes, chosen so a chunk stays inside any embeddings model's input limit while still holding a whole runbook step or ownership table (chunk.go). Each later piece repeats the trailing whole lines of the previous one that fit within overlapBytes (300), so a sentence cut across the boundary is not half in each piece and fully in neither (chunk.go). The overlap is applied only inside a section: a heading is already a topic boundary, and carrying the previous section's tail across it would blend two topics into one embedding (chunk.go). A single line longer than the budget is split at rune boundaries by hardWrap; only pathological input reaches it, and without that split one such line would be a chunk of unbounded size (chunk.go).

The result is a Chunk: the file path relative to the checkout root, the chunk's 0-based position within the file, the heading breadcrumb and the content (chunk.go). What actually goes to the embeddings endpoint is EmbeddingInput, which prefixes the path and heading to the body, because "the path and heading often carry the very words a question uses ("services.md", "Checkout") when the body does not" (chunk.go).

Indexing a workspace revision

Indexer is the component that owns the timing and the bookkeeping (index.go). It holds a Store — the slice of the persistence layer it needs, declared locally because "an interface belongs to the package that consumes it" (index.go) — and a model.Embedder, the narrow embedding interface that lets a deployment use one provider for completion and another for retrieval (embedder.go).

Refresh is the workspace-sync hook. It bounds itself with RefreshTimeout, a two-minute ceiling, so an embeddings endpoint that hangs cannot hold the sync — and with it startup and the SyncWorkspace RPC — for longer than that (index.go, index.go). On failure it logs and returns, because "a broken embeddings endpoint must degrade retrieval, not fail the sync that keeps skills and runbooks current" (index.go). The deployment wires this hook in docsAfterSync, which passes the revision and checkout path straight to Refresh (main.go); the workspace syncer guarantees the tree stays at that revision for the whole call (sync.go).

Index does the actual work (index.go). It serialises itself on indexMu so two overlapping runs cannot commit in the opposite order to the syncs that started them, returns early if the ref was already settled in this process, and otherwise asks the store whether the ref already has chunks — one EXISTS-style query and no embedding requests for an unchanged ref, including after a restart. Embedding happens only for a ref with no chunks yet, and ReplaceDocChunks is all-or-nothing, so "a ref that has any chunks has all of them" (index.go).

The corpus comes from readDocs, which walks docs/ in the checkout through os.Root and skips symlinks rather than following them, since the checkout is remote content and a link to /etc would otherwise be embedded and handed to the model as documentation (index.go). Files larger than maxDocBytes (1 MiB) are skipped as not plausibly hand-written documentation, since a generated API dump would dominate every top-k (index.go, index.go). Files that are not valid UTF-8 text are skipped with a warning, because one of them would fail the whole all-or-nothing replace (index.go).

Embedding itself happens in embed, which batches chunks through the embedder in groups of embedBatchSize (64) so the request stays under the input cap of OpenAI-compatible endpoints (index.go, index.go). Vectors are paired to chunks by position, and a count mismatch is an error rather than a silent truncation, since a short response would index past the end and panic the sync goroutine (index.go). Each row becomes a store.DocChunk carrying the workspace ref, path, chunk number, heading, content and the vector (store.go).

After the rows are written, promote makes the ref current and prunes the refs no running session is pinned to (index.go). The keep-list is the current ref plus every pinned ref (index.go). A prune failure is returned but the ref still becomes current: "the stale rows cost disk, not correctness, since every search filters by ref" (index.go). While a pinned ref other than the new one survives, the indexer is not "settled", so the next sync of the same ref prunes again and collects the rows of sessions that have ended since, instead of returning early and keeping them until the repository moves (index.go).

Pin keeps a ref's chunks out of every prune until its release function is called, and the release is idempotent (index.go). A pin taken while a prune is already in flight may lose the ref's rows to that prune; the session's docs_search then falls back to the current revision and says so in its answer (index.go).

Answering a search

Client is an in-process MCP client shaped like the workspace tools, so the agent reaches the documentation only through the registry (search.go). It advertises one tool, docs_search, whose top_k argument is bounded between 1 and 20 with a default of 5 (search.go, search.go). The server name it is registered under is docs (search.go).

CallTool reads the query, embeds it, and searches the store for the nearest chunks of the ref it should answer from (search.go). The ref is the indexer's current ref, unless the client is pinned to one — a session pinned to a commit keeps answering from that commit's documentation after a later sync moves current on. If the pinned ref has nothing indexed, the client answers from current instead and prefixes a note naming both commits, so the model does not take another commit's documentation for the one it is investigating (search.go).

The result is rendered as plain text: a header with the number of excerpts, the query and the commit, then each excerpt with its path and section heading (search.go). The underlying search is a pgvector cosine query over one workspace ref, ORDER BY embedding <=> $2 with a LIMIT, using the operator that matches the HNSW index the migration builds (postgres_report.go).

Retrieval is optional end to end. Without an embeddings provider there is no indexer and therefore no docs_search tool at all (main.go). The per-session wiring follows the same pattern: a session registers its own pinned docs client under the same server name, shadowing the deployment-wide one that reads whatever revision is current (scratch.go). The agent's system prompt mentions docs_search only when the deployment registered it (prompt.go).

Error handling

CallTool separates failures the model can act on from failures the tool could not attempt at all (search.go). An empty or missing query, an out-of-range or fractional top_k, an embedding failure, a store failure and an empty index all come back as an error-flagged result whose text tells the model what to do next (search.go, search.go). A top_k that is not a whole number or falls outside 1..20 is refused rather than clamped, so the model learns the bound instead of silently receiving a different count than it asked for (search.go). A call naming an unknown tool returns a Go error (search.go).

On the indexing side, every failure is wrapped with the ref it happened for, and nothing is written: the previous ref keeps serving (index.go). The vector width is fixed by the docs_chunks.embedding column, and startup validates the configured embedder's width against the connected database rather than trusting the config (embedder.go, embedding_dims.go).

Relationships

graph TD
  subgraph docs["internal/docs"]
    chunkfile["internal/docs/chunk.go"]
    indexfile["internal/docs/index.go"]
    searchfile["internal/docs/search.go"]
    Chunk["Chunk"]
    SplitMarkdown["SplitMarkdown"]
    sections["sections"]
    EmbeddingInput["EmbeddingInput"]
    Indexer["Indexer"]
  end

  SplitMarkdown --> Chunk
  SplitMarkdown --> sections
  Chunk --> EmbeddingInput
  indexfile --> SplitMarkdown
  indexfile --> EmbeddingInput
  indexfile --> Indexer
  searchfile --> Indexer

Reference

Related pages: Workspace Sync, Checkout and Materialization, PostgreSQL Persistence, Migrations and Vector Search, Model Providers, Embeddings and Error Kinds, MCP Client, Registry and Server Connection, Session Manager and Lifecycle, Agent Prompts, Report Authoring and Finalization, Korvus Overview.

Generated by astrolabe from commit 0c2dca73 on branch main.