tatara-operator¶

The central component of the tatara platform. A controller-runtime Kubernetes operator that reconciles the platform CRDs, drives every Task through its 15-stage lifecycle, receives SCM and Grafana webhooks, provisions per-project memory stacks, gates agent-pod concurrency, walks the post-review merge sequence, and enforces the security model.

Repository: github.com/szymonrychu/tatara-operator

What it does¶

• Reconciles six CRDs, all tatara.dev/v1alpha1, namespaced: Project, Repository, Task, QueuedEvent, Issue (short name iss), MergeRequest (short name mr). Subtask is deleted. WorkItem was never a CRD - it was an embedded Go slice on Task.Status - and it is gone entirely along with the field.
• Receives HMAC-verified GitHub and GitLab webhooks and bearer-verified Grafana alert webhooks on a shared HTTP listener.
• Provisions per-project memory stacks (CNPG Postgres + Neo4j + LightRAG + tatara-memory service).
• Schedules repo-ingest jobs (tatara-memory-repo-ingester) on push and on cron.
• Admits queued work against per-project agent-pod concurrency (maxConcurrentAgents), then spawns tatara-claude-code-wrapper pods for agent turns.
• Drives the Task state machine end to end: new classifies the origin and mints Issue CRs, refined runs the approval gate (an implement pod, for most origins) and, once granted, code, under-implementation -> awaiting-review -> merged -> deployed -> done, with the nightly documentation batch and each dependency-upgrade cron tick minted straight into under-implementation.
• Writes results back to the SCM through a mirror-first REST layer: opens MRs, posts comments, posts review verdicts as COMMENT-type reviews (never a forge-native approve), and merges directly once a review approves and CI is green.
• Walks the sequential per-repo merge order (spec.mergeOrder) after review, re-verifying the live head SHA and CI status immediately before each merge.
• Reaps orphaned agent pods and GCs terminal Tasks and stale-labelled Issues/MergeRequests per a fixed retention table.
• Exposes an OIDC-gated REST API (used by tatara-cli and agent pods).

Listener ports¶

The manager binds four separate addresses. Only the public HTTP listener is routed through the ingress.

Layout¶

cmd/manager/                       # controller-runtime entrypoint + wiring
api/v1alpha1/                      # CRD types: Project/Repository/Task/QueuedEvent/Issue/MergeRequest
internal/controller/               # the sweep, admission dispatcher, stage machine, merge walker, reaper
internal/agent/                    # agent Pod/Service builder + turn session/callback
internal/ingest/                   # repo-ingest Job builder
internal/memory/                   # per-project memory stack builders
internal/scm/                      # GitHub/GitLab clients + provider registry
internal/restapi/                  # OIDC-gated CRUD REST API
internal/webhook/                  # HMAC-verified SCM + bearer-verified Grafana webhook server
internal/auth/                     # OIDC verifier + client-credentials token source
internal/config/                   # env-scalar config
internal/obs/                      # JSON slog + Prometheus metrics
charts/tatara-operator/            # cluster-agnostic Helm chart + CRDs

The Task state machine¶

Every Task carries status.state, one of 8 values: new, refined, under-implementation, awaiting-review, merged, deployed, done, rejected. Whether it is stalled is a separate, orthogonal field, status.parkReason (28 closed values, empty when not parked) - the pre-#521 machine folded parking into the stage enum itself, which is how parked ended up simultaneously a stage, a terminal, and a pod-less marker. Only the operator writes status.state / status.parkReason - an agent never does. A transition outside the fixed table is rejected and counted (operator_illegal_stage_transition_total - the metric kept its pre-redesign name).

Task.spec.kind is the origin, immutable, one of brainstorm, incident, implement, refine, review, documentation, takeover, upgrade. Task.status.agentKind is the currently running agent, one of seven: brainstorm, incident, implement, refine, review, documentation, upgrade. implement is both an agent kind and, since

521, an origin kind - it is SweepIssueKind, the value stamped on any Task minted¶

from a new issue (webhook or backlog sweep), the same role clarify used to play before it was deleted platform-wide. The three live states (refined, under-implementation, awaiting-review) map to an agent kind and spawn a pod named after that kind and the Task's target (see Pod naming); new, merged, and deployed run no pod at all - they are pure operator logic.

For full details on the transition table, the three-clock deadline model (admission / readiness / work), and the per-state retention windows, see the Task state machine reference and Approval gates.

Admission is a separate concern from the stage machine: a producer stashes a QueuedEvent (class normal or alert), and the dispatcher admits it against the project's agent-pod pool before any pod is spawned.

stateDiagram-v2
    [*] --> Queued: QueuedEvent created
    Queued --> Admitted: pod slot free
    Admitted --> [*]: pod spawned for the current stage

Queue admission and concurrency¶

Agent work is not spawned directly from a webhook. Producers stash a QueuedEvent and an in-operator dispatcher admits events against per-project pod-slot capacity, so a burst of issues cannot fan out into unbounded concurrent agent pods.

Over-capacity events wait in Queued and are admitted when a pod slot frees; the alert pool has reserved slots so an incident is never starved by a backlog of normal work.

Two further caps bound the mint side, independent of the concurrency gate:

agentPodTTLSeconds (default 3600, minimum 300) bounds one pod's life, not the Task: a Task persists across as many pods as it takes, each one picking up continuity from Task.status.notes. maxBundleBytes (default 400000, roughly 100k tokens) is the hard byte budget for a rendered context bundle; the oldest comments elide first, behind an explicit marker.

Stall detection and the residency cap¶

Token-metered spend gates (maxTaskTokens, the tokenBudget admission mode) are gone. A second generation of backstops - turn- and pod-recreation-count budgets on Project.spec.agent (maxTurnsPerTask, maxReviewRounds, maxPodRecreations) - is also gone as of tatara-operator#582 (O3): a turn, review-round, or respawn count measures how much an agent has done, not whether it is stuck, and each of those ceilings had killed healthy long-running work. maxTurnsPerPod survives as a field with zero effect - see AgentSpec. The human-review round bound is still enforced, but it is the MaxHumanReviewRounds constant and never was a field: writing agent.maxHumanReviewRounds into a Project is pruned silently by the apiserver.

What replaced them is two-part:

1. Stall detection (probe, interrupt, stop). turnTimeoutSeconds of inactivity no longer kills a turn - it triggers a probe (POST /v1/probe on the wrapper). Unanswered past stallProbeGraceSeconds, retried up to stallProbeMaxAttempts, the operator interrupts the session (POST /v1/interrupt - ESC on the PTY, ~40ms, session and context survive) and runs the ordinary stop-and-handoff sequence. See Agent Execution and the stall-probe runbook.
2. The residency cap. stage.ResidencyCapAll = 24h, a hardcoded constant (not a Project field), applied uniformly to all three live states since O3 - a dead-man switch for the population stall detection cannot reach at all: a wedged pod, a boot-crash loop, an operator bug. See the residency section.

The accepted cost: repeated pod recreation (a boot-crash loop, say) is no longer capped short of 24h residency - at a 5-minute respawn cycle, on the order of 288 pods. The compensating control is the operator_pod_recreations_total alert, which pages rather than silently parking, and whose reason label (tatara-operator#587) now distinguishes a vanished pod, an OOMKilled one, a non-zero container exit, phase Failed, a boot timeout, and a clean exit with no agent handoff - the operator previously only detected the vanished case, so a Failed-phase or OOMKilled pod sat undetected for up to an hour on the turn-inactivity clock alone. An OOMKilled pod also gets an OOM-specific synthetic handoff note, warning the next agent that the workspace was ephemeral and to verify branches/commits against the remote rather than trust a note that may describe a commit that never reached origin.

The review post and the merge¶

There is exactly one bot identity on the platform, and GitHub/GitLab both reject an APPROVE or REQUEST_CHANGES review from the same identity that opened the PR (422). So the operator never attempts either: it posts a COMMENT-type review carrying the verdict text, and the actual approval of record is the merge itself. Auto-merge is never armed on any tatara-opened PR.

verdict=approve          -> operator posts a COMMENT review, then MERGES directly
verdict=request_changes  -> operator posts a COMMENT review with inline findings,
                             Task returns to implementing

A review-kind Task - a human's own PR under review - can never reach merging by any path: it parks at awaiting-human instead, bounded by maxHumanReviewRounds (5), and only a human's next comment un-parks it. Merging is exclusively an operator action; no MCP tool exposes it to an agent.

Once a Task's review approves, the operator walks spec.mergeOrder sequentially: for each repo it re-reads the live head SHA (never the mirror), confirms CI is green, and merges. If the live head moved since the reviewed SHA, the Task returns to reviewing rather than merging a stale review. See Merge and deploy for the full walker.

CI truth and the readiness gate¶

Before tatara-operator#592, MergeRequest.status.ciStatus was only ever set at MR-mint time; an MR the agent opened itself stayed "" forever, so agents had no visibility into their own pipeline and the operator advanced Tasks to awaiting-review without regard to a red build. Two changes close that:

1. Ingest. The webhook server decodes GitHub check_suite/check_run/status and GitLab Pipeline Hook events (previously received and discarded) into a normalized CI status, joined onto the owning MR by head SHA. A single check-run or commit-status context can prove red on its own but never green - only an aggregate (a check-suite conclusion, a GitLab pipeline status, or a live re-read) writes green. Backstopped by a 5-minute poll for MRs owned by a live Task. ciStatus/ciUpdatedAt are rendered into the agent's own context bundle - see MergeRequest.
2. The readiness gate. Since tatara-operator#594, an implement or review-approve outcome is refused - a structured 409, nothing written - if any open MR the Task owns has red CI, a real base conflict, or an unanswered request_changes. The agent is instructed to wait for its own PR's pipeline (bounded ~20 minutes) before submitting; separately, Task.status.ciWaitSince holds an un-parked Task up to 30 minutes for a still-pending pipeline before advancing anyway. See CI readiness gate.

Reaper and GC¶

A background sweep keeps state bounded. Every terminal ages out on a fixed clock: a park (except backlog-sweep, which is exempt - it owns an Issue at zero agent cost and is reaped only when that Issue closes) after 7 days, rejected after 24 hours, done after 48 hours (and only once the nightly documentation batch has covered it, or it provably has nothing to document). There is no separate failed terminal any more - every pre-#521 failed(...) reason is now either a park or lands directly on rejected. The reaper also GCs orphaned agent pods (pods whose owning Task is gone or terminal, after a grace period). Each path is metered so leaks are visible - see Metrics.

Leader election and metrics¶

The operator runs multi-replica with leader election. Metrics that can only be observed on the leader (reconcile state, queue depth) are exported with sum by() / max by() aggregates so Prometheus correctly handles the non-leader replicas reporting zero.

The turn-complete callback server itself is not leader-gated - it runs on every replica, so a naive write from inside its handler lands on a non-leader roughly two-thirds of the time in a 3-replica prod deployment and is silently discarded (the tokenBudget gate's in-process store is per-process memory). AccountUsageFeedReconciler is the pattern for anything downstream of that callback that needs a single fleet-wide value: the callback handler only parks the wrapper's newest snapshot on Task.status.accountUsage, and this leader-only reconciler is what folds the newest snapshot across every Task into the store the claudeSubscription gate actually reads (tatara-operator#633). See Tuning for the gate itself.

Helm chart¶

The chart at charts/tatara-operator/ is cluster-agnostic. Cluster-specific configuration (ingress host, storage class, imagePullSecrets, OIDC URLs) comes from the tatara-helmfile values files.

The chart packages both the operator itself and charts/tatara-project/ as a sibling chart. The tatara-project chart templates Project and Repository CRs declaratively from helmfile values (replacing raw YAML presync manifests).

CRD ownership¶

CRDs are templated in the chart and applied via helm upgrade. On initial install or first upgrade, pre-existing CRDs need a one-time ownership annotation (helm.sh/resource-policy: keep + managed-by Helm annotations) before the chart can adopt them.

Key configuration¶

Operator configuration is env scalars. The webhook signing secrets are not operator env: they are read per-Project from Kubernetes Secrets referenced by the Project CR (see the note below).

Metrics¶

The operator exposes Prometheus series on :9090/metrics. A representative subset - see Observability for the alerting-relevant set in full:

See Observability for the complete metric and alert catalogue.