Observability¶

The tatara operator ships observability first-class: a ServiceMonitor for Prometheus scraping and structured log/slog output that doubles as the platform's audit trail (K.3 below) - both cluster-agnostic, both enabled by default. A companion repository, tatara-observability, holds the full Grafana alert rule set for the stage machine, managed as code and applied by Terraform CI.

Signal flow¶

graph LR
    A["operator :9090\n/metrics"] -->|"ServiceMonitor\n30 s interval"| P[Prometheus]
    OA["tatara-observability\nalerts/*.yaml"] -->|"PR = plan\nmerge = apply"| GA["Grafana\nTatara folder rules"]
    P --> GA
    GA -->|"system=tatara fires"| WH["operator\n/webhooks/tatara/grafana"]
    WH --> IT["Task: kind=incident\nstate=refined, agentKind=incident"]
    IT -->|"submit_outcome(file_issue)"| CL["state=refined, agentKind=implement"]
    O2["operator stdout\nJSON logs"] -->|"Loki"| L[Loki]

The operator is the only Loki-scraped component; agent pods are not (K.3). Alert verdicts route back in as incident-kind Tasks, which run at state=refined with agentKind=incident and, if the alert corresponds to real work, file a tracker Issue and hand off to implement's conversation phase (same state, agentKind now implement) - see the state reference for the full transition table.

1. Metrics catalog¶

The operator exposes /metrics on the metricsAddr port (default :9090, exposed as the metrics port on the operator Service). This port is not ingress-routed. It is only reachable in-cluster via the Service.

Task state metrics¶

These replace every phase / lifecycleState / cascadeStage series the previous design used. operator_task_state is the single source of truth for "what is a Task doing right now" - see the state machine reference for the full 8-member enum and transition table.

Agent pod and contract metrics¶

Merge, deploy, and sweep metrics¶

Bundle and object-size guard metrics¶

SCM metrics¶

2. Dashboard: Tatara Loop¶

The chart ships the "Tatara Loop" dashboard as a ConfigMap labelled grafana_dashboard: "1" for automatic sidecar discovery. When the Grafana sidecar (kiwigrid/k8s-sidecar) is deployed with the matching label selector it loads the dashboard without manual import.

# values.yaml knobs
dashboard:
  enabled: true          # default: true
  folder: "tatara"       # Grafana folder for placement
  additionalLabels: {}   # match non-default sidecar label selectors

The dashboard hardcodes no datasource UID. A $datasource template variable lets you select any Prometheus instance in the cluster. Two additional template variables, $project and $repo, filter every panel to a specific project or repository.

Panels that read Task.status.phase / lifecycleState are re-keyed onto operator_task_state{state,kind} (renamed from operator_task_stage by the #521 redesign) - a per-state breakdown (one series per state, faceted by kind) replaces the old two-value phase timeseries with something that actually shows where Tasks are queued up. dashboards/chat.json is deleted along with the rest of the tatara-chat footprint.

3. Alerts¶

Tatara defines alert rules in four places, but only one delivers on this cluster: the per-component rule set in tatara-observability, applied via Terraform CI and routed through the Tatara contact point into the operator's incident webhook. The other three - tatara-operator's chart PrometheusRule, tatara-memory's chart PrometheusRule, and tatara-operator's own Go-defined memoryAlertRules (internal/memory/monitoring.go) - are non-delivering specifications: this cluster's Alertmanager never loads them (prometheusRule.enabled: false in tatara-helmfile, issue #440), and check_alert_plane_parity.py in tatara-observability reconciles them against the delivering plane rather than deploying them. The stage-machine rewrite below concerns the delivering plane: any rule in tatara-observability keyed on phase, lifecycleState, cascadeStage, implementGiveUps, or linksSyncFailures no longer has a metric to read.

Rewritten (8 rules keyed on deleted fields)¶

Deleted¶

tatara-chat is archived and fully decommissioned (its helm release is removed from the cluster - see Deployment). Delete alerts/tatara-chat.yaml (10 rules), the chat log-burst rule in the Loki-based log-alert file, and dashboards/chat.json.

New (minimum set)¶

Do not key the incident-starvation alert on operator_task_state_age_seconds{kind="incident",state="refined"} even though it looks intuitive - refined carries only the idle-conversation budget (60m default) once a pod is up, and, more importantly, an unadmitted incident Task sits in new (a 5-minute, no-pod state) or in the admission queue, never reaching a running incident pod at all while starved. The queue-age metric is the correct signal because it measures time waiting for admission, not time spent working.

CI pipeline (tatara-observability)¶

PR opened     ->  terraform plan  ->  sticky comment on PR showing planned rule changes
Merge to main ->  terraform apply ->  rules live in Grafana within ~60 s

Required GitHub Actions secrets: AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY (S3 Terraform state), TF_VAR_GRAFANA_API_KEY (Grafana Editor SA token), TF_VAR_GRAFANA_URL.

To add or modify a rule, edit the relevant alerts/tatara-<component>.yaml and open a PR. The Terraform module handles the Grafana API interaction; no Terraform edits are required for rule changes.

4. Alert routing¶

Alerts route to an operator webhook that mints an incident-kind Task, entering state=refined with agentKind=incident (an incident agent pod, budgeted by the refined state's idle clock before parking). The routing is label-driven.

Required labels on any alert that should open an incident:

labels:
  homelab: "true"      # matches the global homelab notification policy
  system: "tatara"     # routes to the tatara-specific contact point
  severity: "warning"  # or "critical" - both trigger an incident
                        # omit system= on info-only rules (email only)

Boundary: Contact points and the system=tatara notification policy live in the global infra Terraform (infra/terraform/grafana), not in tatara-observability. The tatara-observability repo owns only the Grafana Tatara folder and the tatara-* rule groups. Routing works regardless of folder or rule ownership.

5. Structured logging: the operator is the audit trail¶

Agent pods are not Loki-scraped - they are ephemeral, per-turn processes with no stable identity to attach a log stream to. The operator is, so it carries audit-trail responsibility for anything that needs to outlive a Task's TTL (up to 48h for done, 24h for rejected, 7d for a terminal park). The operator logs the following actions at INFO on its own stdout:

stage_transition in particular gives you a full, Loki-queryable history of every Task's path through the state machine at the cost of one slog.Info call per transition - useful when a Task has already been reaped and operator_task_state_age_seconds no longer has a series for it. Approval-gate audit fields (approval_verified / approval_refused) are the log-side complement to the check itself - see Security: approval gates for what the operator verifies about the cited comment: that it exists, its author, and that the quoted text is really there.

6. Scrape gotchas¶

Leader-only metrics¶

The operator runs with leaderElection: true (default). Business metrics - task state, tokens, sweep heartbeats, queue ages - are emitted only by the leader replica. If you run replicaCount > 1 for high availability, always aggregate with sum() or max() rather than querying a single instance:

# Correct: aggregates across all replicas
sum(operator_task_state{state="under-implementation"})

# Wrong with HA: single-replica query may hit a non-leader
operator_task_state{instance="tatara-operator-abc:9090",state="under-implementation"}

Workload infrastructure metrics from kube-state-metrics (pod restart counts, readiness, container waiting reasons) are always available regardless of leader election.

Memory API pods are not scraped¶

The tatara-memory API pods (mem-*) do not have an active ServiceMonitor. Component- specific rules in alerts/tatara-memory.yaml that reference memory service metrics are dark until scraping is wired. The default_no_data_state: "OK" setting is correct here - the absence of these series is an intentional gap, not a heartbeat that should alert. The workload-generic pod-health rules (kube-state-metrics) fire correctly without scraping.

Viewing raw metrics in-cluster¶

# Port-forward the metrics port (not exposed via Ingress)
kubectl -n tatara port-forward svc/tatara-operator 9090:metrics

# Sample Task state
curl -s http://localhost:9090/metrics | grep -E '^operator_task_state'

# Check for the C.9 accepted-risk detector firing
curl -s http://localhost:9090/metrics | grep operator_unexpected_merge_total