tatara-operator¶

The central component of the tatara platform. A controller-runtime Kubernetes operator that reconciles the platform CRDs, drives the agentic development lifecycle, receives SCM and Grafana webhooks, provisions per-project memory stacks, gates spend, supervises deploys, and enforces the security model.

Repository: github.com/szymonrychu/tatara-operator

What it does¶

• Reconciles four CRDs with a controller-runtime manager: Project, Repository, Task, and QueuedEvent. A fifth CRD, Subtask, is a data-only object written and read over the REST API (the agent self-planning ledger); it has no reconciler. WorkItem is not a CRD - it is a plain Go struct that backs the Task.Status.WorkItems project-level work-item ledger.
• Receives HMAC-verified GitHub and GitLab webhooks and bearer-verified Grafana alert webhooks on a shared HTTP listener
• Provisions per-project memory stacks (CNPG Postgres + Neo4j + LightRAG + tatara-memory service)
• Schedules repo-ingest jobs (tatara-memory-repo-ingester) on push and on cron
• Admits queued work against per-project concurrency and token-budget gates, then spawns tatara-claude-code-wrapper pods for agent turns
• Drives the turn loop: submits prompts, receives callbacks, transitions task states
• Writes results back to the SCM: opens PRs, posts comments, applies labels, merges on approval
• Supervises the post-merge push-CD cascade to a tatara-helmfile apply
• Reaps orphaned pods and GCs terminal Tasks and stale conversation transcripts
• Exposes an OIDC-gated REST API (used by tatara-cli and agent pods)

Listener ports¶

The manager binds four separate addresses. Only the public HTTP listener is routed through the ingress.

Layout¶

cmd/manager/                       # controller-runtime entrypoint + wiring
api/v1alpha1/                      # CRD types: Project/Repository/Task/QueuedEvent/Subtask (+ WorkItem struct)
internal/controller/               # reconcilers + turn loop + writeback + queue dispatcher + reaper
internal/agent/                    # agent Pod/Service builder + turn session/callback
internal/ingest/                   # repo-ingest Job builder
internal/memory/                   # per-project memory stack builders
internal/budget/                   # token-budget admission gate
internal/scm/                      # GitHub/GitLab clients + provider registry
internal/restapi/                  # OIDC-gated CRUD REST API
internal/webhook/                  # HMAC-verified SCM + bearer-verified Grafana webhook server
internal/auth/                     # OIDC verifier + client-credentials token source
internal/config/                   # env-scalar config
internal/obs/                      # JSON slog + Prometheus metrics
charts/tatara-operator/            # cluster-agnostic Helm chart + CRDs

Reconciliation model¶

Each controller reconciles its resource type independently. The most complex is the Task reconciler, which drives a state machine across the full lifecycle of an agent session. Two orthogonal state fields are in play and it is important not to conflate them (see Dual Phase / LifecycleState below).

Admission happens first, on the QueuedEvent:

stateDiagram-v2
    [*] --> Queued: QueuedEvent created
    Queued --> Admitted: pool slot free + budget gates pass
    Admitted --> [*]: Task created

A single-shot task kind (implement, review, brainstorm, incident, triageIssue, selfImprove, refine) then runs a pod-backed Status.Phase machine:

stateDiagram-v2
    [*] --> Planning
    Planning --> Running
    Running --> Succeeded
    Running --> Failed
    Running --> Deploying: implement PR merged
    Deploying --> Succeeded: helmfile applied
    Deploying --> Failed: deploy-timeout (parks recoverable)

issueLifecycle tasks are the exception: one durable long-lived Task spans the whole issue-to-deploy arc (Triage, Conversation, Implement, MRCI, Merge, Deploy). These Tasks leave Status.Phase empty for their whole life and signal progress and completion through Status.LifecycleState.

Dual Phase / LifecycleState¶

Task.Status.Phase and Task.Status.LifecycleState are two distinct fields with different owners:

• Status.Phase (bare strings Planning / Running / Succeeded / Failed / Deploying) is the pod-backed execution phase used by single-shot kinds. Deploying is the pod-less post-merge deploy-supervision phase and is deliberately not terminal.
• Status.LifecycleState (Triage / Conversation / Implement / MRCI / Merge / Deploying / Done / Stopped / Parked) is the issue-level state carried by issueLifecycle Tasks, which keep Status.Phase empty by design.

Terminality is therefore a function of both fields: a Task is terminal when Phase is Succeeded/Failed or LifecycleState is Done/Stopped/Parked. Auditing issueLifecycle Tasks by Phase alone reports every one of them as non-terminal churn, which is wrong. The TaskTerminal helper is the source of truth; controller code and any external audit must use it rather than testing Phase directly.

Queue admission and concurrency¶

Agent work is not spawned directly from a webhook. Producers stash a QueuedEvent (class normal or alert) and an in-operator dispatcher admits events against per-project pool capacity, so a burst of issues cannot fan out into unbounded concurrent agent pods.

Over-capacity events wait in Queued and are admitted when a slot frees; the alert pool has reserved slots so an incident is never starved by a backlog of normal work.

Token conservation and budget gates¶

The operator has three independent spend controls, from coarsest to finest:

1. Concurrency cap - spec.maxConcurrentTasks (above). Bounds how many agent pods can run at once.
2. Per-Task token ceiling - spec.agent.maxTaskTokens. A cumulative output-token ceiling per Task; when a Task crosses it the operator terminates the pod and marks the Task failed with reason TokenBudgetExceeded. This is the runaway backstop, independent of the admission gate.
3. Per-Project token-budget admission gate - spec.tokenBudget plus operator env. When enabled, the dispatcher pauses admission before spawning new work once the Project crosses a usage threshold within a window. Two thresholds: proactivePercent pauses the normal pool (brainstorm/implement/review/...), emergencyPercent pauses the alert pool (incidents) so incidents keep running longer than routine work.

The gate has two modes, selected by the TOKEN_BUDGET_MODE operator env (default customWindow):

Deploy supervision (push-CD)¶

An implement Task does not go terminal when its PR merges. It enters Deploying and the operator drives the post-merge push-CD cascade: it tracks the artifact through CascadeStage (tagged -> parent-pr-open -> parent-merged -> helmfile-applied) toward a terminal tatara-helmfile apply, then resolves the Task Succeeded (or Done for issueLifecycle) and closes the originating issue. A wall-clock deadline of now + spec.deployBudgetSeconds (with a spec.deploySingleHopBudgetSeconds per-artifact override) bounds the wait; on exceed the Task parks recoverable with reason deploy-timeout. This is how a merged change is followed all the way to running in the cluster rather than being assumed deployed at merge.

Reaper and GC¶

A background sweep keeps state bounded: it reaps orphaned agent pods (pods whose owning Task is gone or terminal, after a grace period), GCs terminal Tasks, and GCs stale conversation transcripts. Each path is metered (operator_orphan_reaped_total, operator_tasks_gc_total, operator_conversation_gc_total) so leaks are visible.

Leader election and metrics¶

The operator runs multi-replica with leader election. Metrics that can only be observed on the leader (reconcile state, queue depth) are exported with sum by() / max by() aggregates so Prometheus correctly handles the non-leader replicas reporting zero.

Helm chart¶

The chart at charts/tatara-operator/ is cluster-agnostic. Cluster-specific configuration (ingress host, storage class, imagePullSecrets, OIDC URLs) comes from the tatara-helmfile values files.

The chart packages both the operator itself and charts/tatara-project/ as a sibling chart. The tatara-project chart templates Project and Repository CRs declaratively from helmfile values (replacing raw YAML presync manifests).

CRD ownership¶

CRDs are templated in the chart and applied via helm upgrade. On initial install or first upgrade, pre-existing CRDs need a one-time ownership annotation (helm.sh/resource-policy: keep + managed-by Helm annotations) before the chart can adopt them.

Key configuration¶

Operator configuration is env scalars. The webhook signing secrets are not operator env: they are read per-Project from Kubernetes Secrets referenced by the Project CR (see the note below).

Metrics¶

The operator exposes roughly sixty Prometheus series on :9090/metrics. A representative subset: