tatara-observability¶

Alerts-as-code for the tatara platform. Grafana alert rules live in versioned YAML files under alerts/, rendered into Grafana by Terraform on PR merge. Both humans and enrolled agents change alerts through the same PR workflow - open a PR, review the terraform plan diff, merge to apply.

Repository: github.com/szymonrychu/tatara-observability

What it is¶

The repo contains one YAML file per component under alerts/tatara-<component>.yaml. Each file maps to exactly one Grafana rule group in the Tatara folder. The modules/grafana_alert Terraform module reads every alerts/*.yaml and renders the rules into Grafana via the Grafana provider.

Current rule groups:

tatara-cd covers the merge/deploy cascade (sequential-merge stalls, apply failures), tatara-quality the model-keyed review/CI quality-feedback signals, and tatara-usage-gate originally covered the turn- and pod-recreation budget gates (maxTurnsPerTask, maxPodRecreations, maxReviewRounds) that replaced the retired per-Task token-budget admission gate. Those three are themselves now deprecated with zero effect (tatara-operator#582) - what bounds a runaway Task today is the 24h residency cap plus the operator_pod_recreations_total alert.

alerts/tatara-chat.yaml (10 rules) and its log-burst rule are deleted along with the tatara-chat component itself. alerts/tatara-cd.yaml and alerts/tatara-operator.yaml also gain the new alert classes below - both merge-blocked/deploy-blocked (cascade) and the platform-wide operator health signals (contract mismatch, unexpected merge, sweep heartbeat) attach to their existing per-component groups; no new file is introduced for them.

All rules land in the Grafana Tatara folder, which is managed exclusively by this repo. The infra/terraform/grafana state never touches the Tatara folder, so the two states never collide.

Alert-as-code flow¶

sequenceDiagram
    participant E as Editor<br/>(human or agent)
    participant GH as GitHub
    participant CI as GitHub Actions
    participant TF as Terraform
    participant G as Grafana

    E->>GH: open PR editing alerts/*.yaml
    GH->>CI: pull_request event
    CI->>TF: terraform plan
    TF->>G: read current state (provider)
    TF-->>CI: plan output
    CI->>GH: sticky comment with plan diff
    E->>GH: review plan, merge PR
    GH->>CI: push event (main)
    CI->>TF: terraform apply
    TF->>G: create/update/delete rule group
    G-->>TF: confirmed
    CI-->>GH: apply success

Rule schema¶

Each alerts/tatara-<component>.yaml file is a rule group definition. The file is parsed by grafana.tf with yamldecode and passed to modules/grafana_alert.

Group-level fields¶

Rule-level fields¶

Query object fields¶

Query pipeline¶

The module builds a four-stage pipeline for every rule:

expression (PromQL/LogQL)
   └─► Reduce    (last value of each series)
       └─► Round (to decimal_points precision)
           └─► Threshold (math_operator compared against threshold)

expression provides the raw value. The comparison is expressed entirely through math_operator and threshold. For example, "alert when the error ratio exceeds 20%":

math_operator: ">"
threshold: 0.2

And "alert when fewer than one replica is up" (reverse comparison):

math_operator: "<"
threshold: 1

Annotation templates¶

Grafana annotation values are Go templates evaluated at alert time. The module exposes two template variables:

• {{ index $values "C" }} - the numeric result of the reduce/round stage (the value compared against the threshold)
• {{ index $labels "<key>" }} - a label value from the firing series (useful for pod, reason, component, etc.)

annotations:
  summary: >
    tatara-operator reconcile error ratio is {{ index $values "C" }} (>0.20) over 15m.
    Reconciles are failing en masse.

Example: Prometheus rule¶

interval_seconds: 60
default_no_data_state: "OK"
rules:
  - name: "Operator reconcile error ratio high"
    queries:
      - expression: |
          sum(increase(operator_reconcile_total{namespace="tatara",job="tatara-operator",result="error"}[15m]))
          / clamp_min(sum(increase(operator_reconcile_total{namespace="tatara",job="tatara-operator"}[15m])), 1)
    math_operator: ">"
    threshold: 0.2
    for: 15m
    decimal_points: 2
    annotations:
      summary: "Reconcile error ratio is {{ index $values \"C\" }} (>0.20) over 15m."
    labels:
      homelab: "true"
      system: "tatara"
      component: "operator"
      severity: "warning"

Example: Loki rule¶

For log-based alerts, override datasource_uid and set query_type: "loki" on the query object. The expression becomes a LogQL stream selector + pipeline:

rules:
  - name: "Tatara agent reported platform problem"
    queries:
      - expression: |
          sum by (description, category, severity) (
            count_over_time(
              {namespace="tatara", app="tatara-claude-code-wrapper"}
              | pattern `<_> <_> <_> <body>`
              | line_format `{{.body}}`
              | json
              | action="internal_issue_report"
              [5m]
            )
          )
        datasource_uid: "efihqbqlmroqod"
        query_type: "loki"
    math_operator: ">"
    threshold: 0
    for: 1m
    decimal_points: 0
    annotations:
      summary: "Agent reported a platform problem: {{ index $labels \"description\" }}"
    labels:
      homelab: "true"
      system: "tatara"
      component: "wrapper"
      severity: "warning"

Label requirements for routing¶

Every rule must carry four labels. The labels replace the module defaults - there is no merge. Omitting a label removes it from the rule.

Severity and incident routing¶

When an alert with homelab=true + system=tatara + severity=warning|critical fires, Grafana sends a POST to the operator's alert webhook (/operator/webhooks/tatara/grafana). The operator validates the bearer token, deduplicates by alert group, enqueues a QueuedEvent of class alert, and spawns an incident Task. The agent runs with access to a grafana-mcp sidecar for in-session Grafana queries.

Operator metrics this release depends on¶

The task-centric redesign deletes the operator's phase/lifecycleState/cascadeStage series wholesale and replaces them with a stage-keyed model. Any rule still querying a deleted metric does not error - it just returns no data, and under the file's default_no_data_state: "OK" that reports as a silently, permanently green alert. The rules keyed on phase, lifecycleState, cascadeStage, implementGiveUps, and linksSyncFailures - eight rules in total, including both CD-cascade alerts - must be rewritten against the metrics below, not left in place.

The metrics this release's rules key on (non-exhaustive; see tatara-operator for the full catalogue):

New alert classes for this release (minimum set; land in the existing per-component group that owns the metric - tatara-operator.yaml for platform-wide operator health, tatara-cd.yaml for the merge/deploy cascade - no new rule-group file is introduced):

- incident starvation (CRITICAL):
    operator_queue_age_seconds{class="alert",state="Queued"} > 300
- agent contract mismatch (CRITICAL):
    increase(operator_agent_contract_mismatch_total[5m]) > 0
- unexpected merge (CRITICAL):
    increase(operator_unexpected_merge_total[15m]) > 0
- object too large (CRITICAL):
    increase(operator_object_too_large_total[15m]) > 0
- illegal stage transition (WARNING):
    increase(operator_illegal_stage_transition_total[15m]) > 0
- sweep heartbeat stalled (CRITICAL, no_data_state: Alerting):
    time() - operator_sweep_last_success_timestamp_seconds > 7200
- SCM rate limited (WARNING):
    increase(operator_scm_ratelimited_total[10m]) > 0
- merge / deploy blocked (WARNING):
    increase(operator_task_parked_total{stageReason=~"merge-blocked|deploy-blocked"}[1h]) > 0
- docs never written (WARNING):
    increase(operator_doc_task_abandoned_total{reason="never_ran"}[25h]) > 0

CI¶

GitHub Actions in .github/workflows/apply.yml drives the full terraform lifecycle.

Triggers¶

The workflow runs on PRs and pushes to main for changes to:

• alerts/**
• **.tf
• modules/**
• .github/workflows/apply.yml

Steps¶

The plan comment is posted (and updated on each push) via marocchino/sticky-pull-request-comment with header tatara-observability-plan. The concurrency group tatara-observability-tf is set to cancel-in-progress: false so concurrent applies queue rather than cancel.

Required secrets¶

Ownership boundary¶

This repository has a deliberately narrow scope.

tatara-observability (this repo)
├── Grafana folder: "Tatara"
└── Rule groups: tatara-*

infra/terraform/grafana (separate repo + state)
├── Contact points (operator incident webhook, email)
├── Notification policies
│   ├── homelab root policy (homelab=true)
│   └── system=tatara child route -> operator webhook contact point
└── All other Grafana folders and rule groups

The separation is intentional: tatara agents are enrolled on GitHub repos only. infra/terraform/grafana lives on GitLab and is not enrolled on the tatara Project. Moving alert rules to this GitHub repo lets agents propose and review changes to their own observability without access to global homelab infrastructure.

How agents edit alerts¶

tatara agents are enrolled on this repository as a standard Repository CR. The standard PR-based workflow applies:

1. An agent (or a human) edits one or more files in alerts/.
2. The agent opens a PR via the SCM API.
3. GitHub Actions runs terraform plan and posts the diff as a sticky PR comment.
4. A human reviewer reads the plan diff - it shows exactly which Grafana rule groups will change, what threshold values shift, and what labels are added or removed.
5. On approval and merge, terraform apply pushes the changes to Grafana immediately.

No Terraform knowledge is required to author a rule. The YAML schema is the only interface. The plan comment makes the Grafana impact visible before any change reaches production.

Operations notes¶

Loki datasource UID is hardcoded. Rules in tatara-logs.yaml use datasource_uid: "efihqbqlmroqod". This is the UID of the Loki datasource in the target Grafana instance. If you install tatara-observability against a different Grafana, update this value to match your Loki datasource UID (find it in Grafana under Configuration -> Data sources).

Ingest alert mode selector. The rule "Tatara ingest job failing" in tatara-ingester.yaml uses mode="full" in its PromQL selector. This is intentional and must not be removed. Failed incremental ingest jobs self-heal via the full-ingest fallback and are benign; alerting on them produces chronic noise. Only terminal full-ingest failures (which mean the recall corpus is going stale) warrant a page.

No-data defaults to OK, except for heartbeats. All tatara rule groups set default_no_data_state: "OK" at the file level. This suppresses spurious alerts during scrape gaps (component restarts, transient probe failures) for a gauge that legitimately disappears when idle. A component that is genuinely down will fire via the up == 0 or pod-not-ready rules, not via no-data state.

This default is wrong, and dangerous, for a heartbeat metric. A heartbeat series (e.g. operator_sweep_last_success_timestamp_seconds, emitted every sweep pass) stopping altogether - the sweep loop itself wedging - means the query returns no series at all, and under the file default that reports as OK forever: the exact failure mode invisible behind a green dashboard. Every heartbeat/liveness rule must set no_data_state: "Alerting" as a per-rule override, overriding the file default:

rules:
  - name: "Operator sweep heartbeat stalled"
    queries:
      - expression: "time() - operator_sweep_last_success_timestamp_seconds"
    math_operator: ">"
    threshold: 7200
    for: 5m
    no_data_state: "Alerting"
    labels:
      homelab: "true"
      system: "tatara"
      component: "operator"
      severity: "critical"

This bit the platform once already (recorded in MEMORY) and the redesign re-arms it at scale: deleting the pre-redesign phase/lifecycle metrics silently green-lights every alert still keyed on them (see below) unless each is rewritten against a metric that still exists.