Observability¶

The tatara operator ships observability first-class: a ServiceMonitor for Prometheus scraping, a Grafana dashboard as a sidecar-discovered ConfigMap, and a PrometheusRule for loop-failure alerting - all enabled by default, all cluster-agnostic. A companion repository, tatara-observability, holds the full per-component Grafana alert rule set managed as code and applied by CI.

Signal flow¶

graph LR
    A["operator :9090\n/metrics"] -->|"ServiceMonitor\n30 s interval"| P[Prometheus]
    W["wrapper pods\nephemeral"] -->|"POST /internal/metrics/push"| PR["operator\npush receiver"]
    PR --> P
    P --> D["Grafana\nTatara Loop dashboard"]
    P --> R["PrometheusRule\ntatara-loop group"]
    OA["tatara-observability\nalerts/*.yaml"] -->|"PR = plan\nmerge = apply"| GA["Grafana\nTatara folder rules"]
    R -->|"system=tatara fires"| WH["operator\n/webhooks/tatara/grafana"]
    GA -->|"system=tatara fires"| WH
    WH --> IT["incident Task"]
    IT --> IB["brainstorm issue\n(emergency cycle)"]

1. Metrics catalog¶

Operator metrics¶

The operator exposes /metrics on the metricsAddr port (default :9090, exposed as the metrics port on the operator Service). This port is not ingress-routed. It is only reachable in-cluster via the Service; the optional Ingress covers only the main HTTP port (:8080).

Core counters¶

Gauges¶

Histograms¶

There is no operator_reconcile_duration_seconds histogram. operator_agent_http_total (labels method, outcome; outcome includes ok, unreachable, timeout, transport_error) is a counter, not a histogram - the wrapper HTTP call latency lives in operator_agent_http_duration_seconds above.

Push-receiver / wrapper series¶

Ephemeral agent pods (wrapper, ingest) cannot be scraped directly - they have no stable /metrics endpoint and a per-run lifetime measured in minutes. Instead they push their metrics to the operator's push receiver at the end of each run. The operator re-exposes them through its own /metrics.

The allowlist is controlled by the pushMetricsAllowedPrefixes Helm value. The chart default (kept in sync with the code fallback DefaultAllowedPrefixes) is:

wrapper_,agent_,ccw_,tatara_wrapper_,ingest_,analyzer_,semantic_,scip_,llm_

ccw_ and tatara_wrapper_ (the wrapper families) and ingest_,analyzer_,semantic_,scip_,llm_ (the repo-ingester families) are all admitted out of the box, so the ccw_* series below are present with no allowlist edit. Series whose name matches no prefix are dropped and counted in operator_push_series_dropped_total{reason="reserved_name"}.

2. Dashboard: Tatara Loop¶

The chart ships the "Tatara Loop" dashboard as a ConfigMap labelled grafana_dashboard: "1" for automatic sidecar discovery. When the Grafana sidecar (kiwigrid/k8s-sidecar) is deployed with the matching label selector it loads the dashboard without manual import.

# values.yaml knobs
dashboard:
  enabled: true          # default: true
  folder: "tatara"       # Grafana folder for placement
  additionalLabels: {}   # match non-default sidecar label selectors

The dashboard hardcodes no datasource UID. A $datasource template variable lets you select any Prometheus instance in the cluster. Two additional template variables, $project and $repo, filter every panel to a specific project or repository.

Panel layout¶

3. Alerts shipped¶

Tatara ships alert rules from two sources with different scopes and update paths.

Chart PrometheusRule (tatara-loop group)¶

A PrometheusRule CR in the tatara-loop group is rendered by the chart and enabled by default (prometheusRule.enabled: true). These rules cover only the operator's own continuously-present series and are intended as a minimal, always-correct baseline that installs automatically with the operator.

# values.yaml knobs
prometheusRule:
  enabled: true
  severityLabel: "warning"       # stamped on every alert
  tasksInflightThreshold: 8      # cap for the TataraTasksInflightPinned deadman
  additionalLabels: {}           # cluster-specific: match Prometheus ruleSelector

Class A: deadman / liveness¶

These alerts catch silent stalls - conditions that emit no error event but indicate the loop has stopped producing work.

Class B: active failures¶

Class C: latency / saturation¶

tatara-observability alert rules¶

The tatara-observability repository holds a richer, per-component rule set managed as code and applied via Terraform CI. These rules live in the Grafana Tatara folder and complement the chart PrometheusRule with additional coverage per component, workload-generic pod-health rules, Loki-based log alerts, and finer-grained thresholds.

Each alerts/tatara-<component>.yaml file defines one Grafana rule group:

interval_seconds: 60
default_no_data_state: "OK"    # absent series do not fire
rules:
  - name: "Operator reconcile loop wedged"
    queries:
      - expression: |
          sum(increase(operator_reconcile_total{namespace="tatara",job="tatara-operator"}[15m]))
    math_operator: "<"
    threshold: 1
    for: 15m
    annotations:
      summary: "No reconciles in 15m (increase={{ index $values \"C\" }})."
    labels:
      homelab: "true"
      system: "tatara"          # routes to the incident webhook (omit on info rules)
      component: "operator"
      severity: "critical"      # warning|critical trigger an incident; info -> email only

The expression produces the VALUE; the comparison is math_operator + threshold. The Terraform module builds the reduce/round/threshold chain automatically. For LogQL rules (Loki), add datasource_uid and query_type: "loki" under the query.

Coverage by component file:

4. Alert routing¶

Tatara alerts route to an operator webhook that creates an incident Task, kicking off an emergency brainstorm cycle. The routing is label-driven.

Required labels on any alert that should open an incident:

labels:
  homelab: "true"      # matches the global homelab notification policy
  system: "tatara"     # routes to the tatara-specific contact point
  severity: "warning"  # or "critical" - both trigger an incident
                        # omit system= on info-only rules (email only)

Boundary: Contact points and the system=tatara notification policy live in the global infra Terraform (infra/terraform/grafana), not in tatara-observability. The tatara-observability repo owns only the Grafana Tatara folder and the tatara-* rule groups. Routing works regardless of folder or rule ownership.

CI pipeline (tatara-observability):

PR opened     ->  terraform plan  ->  sticky comment on PR showing planned rule changes
Merge to main ->  terraform apply ->  rules live in Grafana within ~60 s

Required GitHub Actions secrets: AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY (S3 Terraform state), TF_VAR_GRAFANA_API_KEY (Grafana Editor SA token), TF_VAR_GRAFANA_URL.

To add or modify a rule, edit the relevant alerts/tatara-<component>.yaml and open a PR. The terraform module handles the Grafana API interaction; no terraform edits required for rule changes.

5. Scrape gotchas¶

Leader-only metrics¶

The operator runs with leaderElection: true (default). Business metrics - task tokens, terminal transitions, scan counts, memory stack gauges, lightrag documents - are emitted only by the leader replica. If you run replicaCount > 1 for high availability, always aggregate with sum() or max() rather than querying a single instance:

# Correct: aggregates across all replicas
sum(rate(operator_reconcile_total{result="error"}[5m]))

# Wrong with HA: single-replica query may hit a non-leader
rate(operator_reconcile_total{instance="tatara-operator-abc:9090",result="error"}[5m])

Workload infrastructure metrics from kube-state-metrics (pod restart counts, readiness, container waiting reasons) are always available regardless of leader election.

Push receiver allowlist¶

The push receiver enforces a prefix allowlist on metric names. The chart default already admits every family the wrapper and repo-ingester pods actually push:

pushMetricsAllowedPrefixes: "wrapper_,agent_,ccw_,tatara_wrapper_,ingest_,analyzer_,semantic_,scip_,llm_"

Series whose names do not match any prefix are dropped and counted in operator_push_series_dropped_total{reason="reserved_name"}. Because ccw_ and tatara_wrapper_ are in the default, the ccw_* wrapper push-series rules in tatara-observability are active out of the box - no allowlist edit is required. Only edit pushMetricsAllowedPrefixes if a component starts pushing a new metric family under a prefix not in the list above (TataraPushMetricsDropped fires and the receiver logs the dropped family name at WARN).

Memory API pods are not scraped¶

The tatara-memory API pods (mem-*) do not have an active ServiceMonitor. The component-specific rules in alerts/tatara-memory.yaml that reference memory service metrics (http_requests_total, lightrag_calls_total, lightrag_call_duration_seconds, ingest_jobs_total, ingest_items_total, code_graph_query_total, etc.) are dark until scraping is wired. The default_no_data_state: "OK" setting prevents these from false-firing. The workload-generic pod-health rules (kube-state-metrics) fire correctly without scraping.

Viewing raw metrics in-cluster¶

# Port-forward the metrics port (not exposed via Ingress)
kubectl -n tatara port-forward svc/tatara-operator 9090:metrics

# Sample operator series
curl -s http://localhost:9090/metrics | grep -E '^operator_task_tokens_total'

# Check push receiver drops
curl -s http://localhost:9090/metrics | grep operator_push_series_dropped_total