Installing the Operator¶

The tatara operator deploys exclusively through the tatara-helmfile GitOps repository. There is no supported path using helm install directly or kubectl apply from local files. Every deploy is a PR, reviewed via a rendered diff, and applied by an in-cluster Actions Runner Controller (ARC) pipeline.

Prerequisites¶

Before the first deploy, verify:

• An in-cluster ARC runner set labeled arc-runner-tatara-helmfile exists (the runner set, ServiceAccount tatara-helmfile-deployer, and its cluster-admin ClusterRoleBinding are provisioned separately in your cluster bootstrap helmfile, not in tatara-helmfile itself).
• A Harbor OCI registry (or compatible substitute) is accessible from the cluster and from the ARC runner for chart pulls.
• A SOPS PGP key is available; encrypted files in values/tatara-operator/ are decrypted at deploy time.
• An OIDC provider (Keycloak or compatible) is running with the required client registrations (see OIDC clients below).

1. Repo layout¶

Fork the tatara-helmfile repository into your organization and treat it as a private, team-restricted repo. The runner ServiceAccount is cluster-admin scoped - any code merged here can modify any cluster resource.

tatara-helmfile/
  helmfile.yaml.gotmpl          # single 'default' env; 4 releases
  .hook.sh                      # presync hook: applies raw/*.pre.yaml,
                                #   sops-decrypts *.pre.secrets.yaml
  values/
    common.yaml                 # bucket-wide: imagePullSecrets: regcred
    tatara-operator/
      common.yaml               # image.tag pin (the only env-specific override)
      default.yaml              # ingress, webhook, OIDC, memory images, S3, scheduling
      default.secrets.yaml      # sops-encrypted: operator OIDC secret + SCM PAT
      raw/
        *.pre.yaml              # plain manifests applied pre-sync (e.g. OBC)
        *.pre.secrets.yaml      # sops-encrypted Secrets applied pre-sync
    project-tatara/
      common.yaml               # Project + Repository CR values (tatara-project chart)
    project-infrastructure/
      common.yaml               # second Project's CR values
    tatara-chat/
      default.yaml              # ingress host + path for the chat UI
  .github/workflows/
    diff.yaml                   # PR trigger: helmfile diff -> sticky comment
    apply.yaml                  # push to main trigger: helmfile apply

The four Helm releases are:

The project-* releases declare needs: tatara/tatara-operator in helmfile.yaml.gotmpl, which forces the operator - and therefore its CRDs - to apply before any Project or Repository CR is rendered.

2. OIDC clients and SCM secret¶

The operator requires several credential groups before it can start reconciling. These are provisioned as Kubernetes Secrets and referenced by name in the chart values.

OIDC clients¶

Five Keycloak clients are needed for the full platform. See Identity & OIDC for the authoritative client inventory. This section covers the two credential groups the operator chart renders directly:

The chart renders the CLI OIDC credentials into a Secret named by cliOidcSecretName (keys client-id, client-secret) and the operator OIDC client secret into a separate Secret (key OPERATOR_OIDC_CLIENT_SECRET). Supply these values through default.secrets.yaml (SOPS-encrypted). The remaining three clients (tatara-memory, tatara-claude-code-wrapper, tatara-chat) are used by their respective component charts.

SCM secret¶

The SCM secret holds the bot identity token and the webhook HMAC secret. The chart renders it when scmToken, scmWebhookSecret, and scmSecretName are all set in the SOPS values file.

# values/tatara-operator/default.secrets.yaml (sops-encrypt before commit)
scmSecretName: "tatara-scm"
scmToken: "<github-fine-grained-pat>"
scmWebhookSecret: "<random-32-byte-hex>"

# values/tatara-operator/default.secrets.yaml (sops-encrypt before commit)
scmSecretName: "tatara-scm"
scmToken: "<gitlab-personal-access-token>"
scmWebhookSecret: "<random-32-byte-hex>"

The rendered Secret contains keys token and webhookSecret. The operator references it via the SCM_SECRET_NAME ConfigMap key.

Anthropic and OpenAI secrets¶

Two additional Secrets must exist before the operator starts agent pods:

# tatara-anthropic: oauth-token key
# Rendered by the chart when anthropicOauthToken + anthropicSecretName are set.
anthropicSecretName: "tatara-anthropic"
anthropicOauthToken: "<anthropic-oauth-token>"

# lightrag-openai: LLM_BINDING_API_KEY key
# Used by the per-Project lightrag (memory) instance.
openaiSecretName: "lightrag-openai"
openaiApiKey: "<openai-api-key>"

Optional: callback HMAC secret¶

If you want the operator to verify HMAC-SHA256 signatures on internal turn-complete callbacks from wrapper pods (recommended for defense-in-depth alongside NetworkPolicy):

callbackHmacSecretName: "tatara-callback-hmac"
callbackHmacSecret: "<random-32-byte-hex>"

The rendered Secret contains key callback-hmac-secret.

3. Operator release values¶

Edit values/tatara-operator/default.yaml. Every scalar maps 1:1 to a SCREAMING_SNAKE ConfigMap key consumed by the manager via envFrom. No inline Pod-spec env values are used.

Ingress and URLs¶

# Operator's own Ingress (public webhook + API endpoint).
ingress:
  enabled: true
  host: tatara.example.com
  path: /
  className: nginx

# externalWebhookBase is stamped into Project.status.webhookURL.
# Must match the public hostname and the operator's internal webhook route prefix.
externalWebhookBase: "https://tatara.example.com/operator/webhooks"

# callbackUrl is the in-cluster Service the wrapper pods POST turn results to.
# Use the internal Service DNS (tatara-operator-internal Service, port 8082).
callbackUrl: "http://tatara-operator-internal.tatara.svc:8082"

OIDC¶

# Use a dedicated realm for tatara's service clients. Do not use Keycloak's
# built-in "master" realm for application clients.
oidcIssuer: "https://auth.example.com/realms/tatara"
oidcAudience: "tatara-operator"
operatorOidcClientId: "tatara-operator"

# Secret references (match the values in default.secrets.yaml)
scmSecretName: "tatara-scm"
anthropicSecretName: "tatara-anthropic"
cliOidcSecretName: "tatara-cli-oidc"
openaiSecretName: "lightrag-openai"

Image pins¶

Tatara-built images are pinned by semver (vX.Y.Z); third-party images pin their own upstream tags or digests. The operator stamps these into the native objects it provisions per Project. Under semver push-CD (section 6) these pins are normally moved forward by the release pipeline, not hand-edited.

# Operator manager image tag (in values/tatara-operator/common.yaml)
image:
  tag: "v0.4.11"          # semver; the pipeline propagates this on release

# Per-Project memory stack images (in values/tatara-operator/default.yaml)
memoryImage:    "harbor.example.com/containers/tatara-memory:v0.4.2"
lightragImage:  "harbor.example.com/proxy-ghcr/hkuds/lightrag@sha256:<digest>"
neo4jImage:     "neo4j:2026.04.0"
grafanaMcpImage: "grafana/mcp-grafana:0.17.0"
ingesterImage:  "harbor.example.com/containers/tatara-memory-repo-ingester:v0.2.10"

# Pull secret for all operator-spawned workloads (neo4j, lightrag, memory, cnpg).
imagePullSecret: "regcred"

Agent scheduling and security¶

agentRunAsNonRoot: true
agentRunAsUser: 10001    # must be numeric; the chart fail-renders if missing

agentScheduling:
  nodeSelector:
    kubernetes.io/os: linux
    # Pin to specific nodes with the resources and network access agents need.
    nas: "true"

agentRunAsNonRoot: false
# agentRunAsUser: omit or leave empty

agentScheduling:
  nodeSelector:
    kubernetes.io/os: linux

Agent pod resource bounds:

agentCpuRequest: "250m"
agentCpuLimit: "2"
agentMemoryRequest: "512Mi"
agentMemoryLimit: "4Gi"

Per-Project memory ingress¶

The operator creates an Ingress for each Project's memory stack at reconcile time. Supply the cluster-specific IngressClass and rewrite annotation:

ingressClassName: "nginx"
ingressRewriteTarget: "/$2"

S3 conversation persistence (optional)¶

When s3Bucket is non-empty, the operator and wrapper store each issue's Claude conversation transcript in an S3-compatible bucket, allowing a fresh pod to resume a prior conversation. Leave s3Bucket empty to disable.

s3Endpoint: "http://rook-ceph-rgw-ceph-objectstore.rook-ceph.svc"
s3Bucket: "tatara-conversations"
s3Region: "us-east-1"       # Ceph RGW ignores region but the AWS SDK requires it
s3KeyPrefix: "conversations"
s3ForcePathStyle: true       # required for Ceph RGW and MinIO; false for AWS S3
s3SecretName: "tatara-conversations"  # Secret with AWS_ACCESS_KEY_ID + AWS_SECRET_ACCESS_KEY
s3ConversationRetentionHours: 72

4. Release ordering¶

Helmfile applies releases in the order they appear in helmfile.yaml.gotmpl, subject to needs: declarations:

tatara-chat          (no dependencies; can be deployed independently)
tatara-operator      (installs CRDs via templates/crds.yaml)
project-tatara       (needs: tatara/tatara-operator)
project-infrastructure (needs: tatara/tatara-operator)

The project-* releases render Project and Repository custom resources via the tatara-project chart. Because they declare needs: tatara/tatara-operator, Helmfile blocks their apply until the operator release (and therefore all tatara.dev CRDs) is confirmed healthy. Never apply project-* releases to a cluster where the operator CRDs are absent.

for crd in projects.tatara.dev repositories.tatara.dev tasks.tatara.dev \
            subtasks.tatara.dev queuedevents.tatara.dev workitems.tatara.dev; do
  kubectl label crd "$crd" app.kubernetes.io/managed-by=Helm
  kubectl annotate crd "$crd" \
    meta.helm.sh/release-name=tatara-operator \
    meta.helm.sh/release-namespace=tatara
done

5. The deploy flow¶

Every change to the cluster - image bumps, config changes, enrollment CR updates - follows the same PR-based flow:

sequenceDiagram
    participant Dev as Developer
    participant PR as Pull Request
    participant ARC as ARC Runner
    participant K8s as Kubernetes

    Dev->>PR: Open PR (bump image.tag and/or chart version)
    PR->>ARC: Trigger diff workflow
    ARC->>ARC: helmfile -e default diff --detailed-exitcode --suppress-secrets
    ARC->>PR: Post sticky diff comment (ANSI-stripped, truncated at 60 KB)
    alt Exit code 1 or other error
        ARC->>PR: Block merge (chart-not-found / render / sops decrypt failure)
    else Exit code 0 or 2
        ARC->>PR: Check passes (no change or reviewable diff)
    end
    Dev->>PR: Review rendered diff, approve, merge to main
    PR->>ARC: Trigger apply workflow (concurrency-guarded, cancel-in-progress: false)
    ARC->>K8s: helmfile -e default apply --suppress-secrets
    alt Apply succeeds
        K8s-->>ARC: All releases healthy
        ARC->>K8s: Apply pre-sync raw manifests (kubectl apply raw/*.pre.yaml: OBC + SCM/Grafana Secrets)
    else Apply fails
        K8s-->>ARC: --rollback-on-failure triggers helm rollback
    end

Diff workflow¶

The diff.yaml workflow runs on every PR targeting main. It:

1. Installs tooling via mise install (helm, helmfile, kubectl, sops plugins).
2. Imports the GPG private key from the GPG_PRIVATE_RSA_B64 Actions secret to decrypt SOPS files.
3. Logs in to Harbor OCI using the HARBOR_ROBOT_KUBERNETES_USERNAME/PASSWORD secrets.
4. Runs helmfile -e default diff --detailed-exitcode --suppress-secrets.
5. Posts (or updates) a sticky PR comment via marocchino/sticky-pull-request-comment@v2 - even when the diff errors, so the reviewer sees the failure reason.
6. Blocks the merge if the exit code is anything other than 0 (no change) or 2 (diff present). Exit code 1 or any other value indicates a chart-not-found, render, or sops decrypt failure.

Apply workflow¶

The apply.yaml workflow runs on every push to main. Key properties:

• Concurrency group: tatara-helmfile-apply, cancel-in-progress: false. Overlapping pushes queue; they never cancel a running apply.
• Timeout: 900 seconds per release (helmDefaults.timeout), generous enough for image pulls and ServiceMonitor/CRD settling.
• Rollback: --rollback-on-failure is set in helmDefaults.syncArgs. A failed apply triggers an automatic Helm rollback to the previous release revision.
• Server-side apply: Helm 4 uses server-side apply by default (--server-side=auto). --force-conflicts lets the GitOps deploy reclaim fields previously touched by emergency kubectl operations.
• Pre-sync raw manifests: After helmfile apply, the workflow re-applies the plain manifests in values/tatara-operator/raw/ using kubectl apply. These are the conversation-bucket ObjectBucketClaim and the SCM/Grafana Secrets (sops-decrypted), not Project or Repository CRs. Applying them explicitly makes them idempotent on every run, even when Helm decides the operator release is unchanged and skips the presync hook. The Project and Repository CRs themselves are rendered by the tatara-project chart via the project-tatara and project-infrastructure releases (sections 1 and 4), not from raw/.

6. Release versioning (semver push-CD)¶

Deploys are semver, pipeline-driven, and largely hands-off. You almost never hand-edit a pin.

How a release ships¶

Every merged PR declares its significance: a human sets a semver:<level> label (major / minor / patch) on the PR; bot-authored PRs carry a change_significance on their change summary that maps to the same label. On merge to the component's main, the release pipeline:

1. Cuts the tag. Computes the next vX.Y.Z from the merged PR's semver:* label.
2. Publishes artifacts. Builds and pushes the image at :vX.Y.Z (the required-checks pipeline already pushed a :<shortSHA> traceability tag for the same commit; Harbor's containers project has tag immutability). It packages both charts (tatara-operator and tatara-project) at the bare X.Y.Z, with appVersion carrying vX.Y.Z, and pulls them back to prove neither is missing (guards against a partial-publish wedge).
3. Propagates the pin. Opens a bot PR against tatara-helmfile that rewrites all four pins atomically in one commit: the three chart-version pins (tatara-operator plus both project-tatara / project-infrastructure releases) take the bare X.Y.Z; the operator image.tag takes vX.Y.Z. Pins are matched by release name, so tatara-chat's pin is never clobbered.
4. Applies and closes. That helmfile PR auto-merges on green checks and the apply workflow rolls it out (section 5). On a successful apply the operator closes the originating issue.

# What the pipeline writes into tatara-helmfile (do not hand-edit in normal flow):

# values/tatara-operator/common.yaml
image:
  tag: "v0.4.11"            # image at :vX.Y.Z

# helmfile.yaml.gotmpl
- name: tatara-operator
  version: 0.4.11           # chart at bare X.Y.Z
- name: project-tatara
  version: 0.4.11
- name: project-infrastructure
  version: 0.4.11

Break-glass: manual dual-pin bump¶

Only when the pipeline is unavailable (e.g. recovering from a stuck release), bump the pins by hand. The invariant is that the chart version and image tag move together:

Harbor's retention policy GCs old chart tags, so pinning backward to a GC'd X.Y.Z fails apply with FetchReference ... not found. Roll forward to a published version rather than back. Find published versions via helm search repo or the Harbor UI.

Local validation¶

mise install                                       # installs helm, helmfile, kubectl, sops
helm registry login <your-registry>               # OCI chart pull
helmfile -e default diff --suppress-secrets        # validate against current kube-context