tatara-memory-repo-ingester¶

Stateless Go batch tool that walks a git repository, runs language-aware static analysis, and pushes a deterministic code graph plus enriched semantic chunks to tatara-memory. It is the bulk knowledge loader that seeds the platform's persistent code graph before any agent works on a repository.

Repository: github.com/szymonrychu/tatara-memory-repo-ingester

What it does¶

The ingester is a short-lived binary (tatara-ingest) with no persistent state of its own. Each run:

1. Computes the set of files to process - either a git diff window (incremental) or the full tracked file list (git ls-files for first or forced full runs).
2. Routes each file to the first matching language analyzer in a fixed precedence order.
3. Runs all matched analyzers in sequence, each producing entities, directed edges, and semantic text chunks.
4. Pushes the aggregate code graph to tatara-memory via POST /code-graph:bulk.
5. Pushes semantic chunks to tatara-memory via POST /memories:bulk, with per-file reconciliation so stale chunks for deleted or renamed files are purged.
6. Optionally runs a best-effort LLM semantic extraction stage (Phase 2) that enriches the graph with concept and rationale nodes for files the server's content-SHA cache marks as unseen.

The ingester is fail-closed on hard analyzer errors: if any analyzer fails for its entire file batch (as opposed to a single-file soft parse error), the run aborts before pushing anything and exits non-zero so the operator retries the same commit window without permanently losing graph coverage.

flowchart LR
    subgraph Source
        R[(git repo)]
    end
    subgraph tatara-ingest
        W[walk / diff] --> G[group by analyzer]
        G --> A1[Go AST\ngo/packages]
        G --> A2[Python\ntree-sitter]
        G --> A3[JavaScript\ntree-sitter]
        G --> A4[Terraform\nHCL parser]
        G --> A5[Helm\ntemplate parser]
        G --> A6[Docs\nmarkdown chunker]
        A1 & A2 & A3 & A4 & A5 & A6 --> AGG[aggregate\nentities / edges / chunks]
        AGG --> SEM[semantic stage\nLLM - optional]
    end
    subgraph tatara-memory
        CG[(code graph\nLightRAG)]
        MEM[(semantic chunks\nvector store)]
    end
    R --> W
    AGG --> CG
    AGG --> MEM
    SEM --> CG

Language analyzers¶

The analyzer registry has a fixed precedence order. Each file is claimed by the first matching analyzer; files matching none are silently skipped. Helm is registered before Docs so chart YAML is not consumed by the markdown/text path.

Go analyzer - confidence scoring¶

The Go analyzer uses go/packages to perform full type resolution. Call edges carry a numeric confidence score and a tier:

TierForScore assigns EXTRACTED only for a score >= 1.0, INFERRED for (0.3, 1.0), and AMBIGUOUS for <= 0.3. The highest-confidence Go call edge (type_resolved, 0.98) still lands in the INFERRED band, so in practice no Go call edge reaches the EXTRACTED tier - that tier is reserved for score-1.0 sources such as a SCIP index. Filter on ?tier=INFERRED (not EXTRACTED) to pick up type-resolved Go calls.

Cross-repo symbol references (packages under a configurable crossRepoPrefix, defaulting to github.com/szymonrychu/) are emitted as SymbolRow records so tatara-memory can link entities across repository boundaries.

Two ingest modes¶

tatara-ingest \
  --repo-root /path/to/repo \
  --repo-name my-service \           # optional; defaults to basename of repo-root
  --since <base-commit>  \           # incremental: diff since this SHA
  --base-url http://tatara-memory:8080

tatara-ingest \
  --scip /path/to/index.scip \
  --scip-repo my-service \           # required
  --base-url http://tatara-memory:8080

Incremental re-ingest¶

The ingester does not maintain local state between runs. Incrementality is driven entirely by git:

1. The operator records status.lastIngestedCommit on the Repository CR after each successful ingest Job.
2. On the next ingest trigger the operator passes --since <lastIngestedCommit> to the Job.
3. The ingester runs git diff -M -z --name-status <since>..HEAD to produce the change set.
4. If <since> is no longer in history (force-push, rebase, GC) the diff command fails; the ingester logs a WARN and falls back to a full git ls-files pass automatically. No manual intervention is needed.

Reconcile-per-file (idempotency)¶

For each ingest run the ingester sends a reconcile_files list alongside the chunk push. The tatara-memory server purges prior chunks for every file in that list before inserting the new ones. This means:

• Deleted files: chunks are removed from the vector store.
• Renamed files: old-path chunks are purged; new-path chunks are inserted.
• Re-analyzed files: stale chunks are replaced, not duplicated.

Files where an analyzer produced a per-file soft parse error are excluded from reconcile_files so existing chunks for those files survive until the file is parseable again.

A full re-ingest (--full or first ingest) is insert-only - no reconcile list is sent - so it is safe to run over an already-populated store without a destructive purge pass.

Operator-managed Jobs¶

The tatara operator manages ingest as short-lived Kubernetes Job resources. Operators do not invoke tatara-ingest directly.

Job lifecycle¶

sequenceDiagram
    participant SCM as SCM (GitHub/GitLab)
    participant OP as tatara-operator
    participant JOB as ingest Job (clone + ingest containers)
    participant MEM as tatara-memory

    SCM->>OP: push webhook
    OP->>OP: read Repository.status.lastIngestedCommit
    OP->>JOB: create Job (--since <last-sha>)
    JOB->>JOB: init: git clone (SCM token in init container)
    JOB->>MEM: POST /code-graph:bulk
    JOB->>MEM: POST /memories:bulk
    JOB->>OP: kubectl patch ConfigMap <repo>-ingest-result sha=<HEAD>
    OP->>OP: update status.lastIngestedCommit, phase=Ingested

Job structure¶

Each Job has two containers sharing an emptyDir workspace volume:

• init container (clone): runs git clone using the project's SCM token (sourced via SecretKeyRef from spec.scmSecretRef). The token is never embedded in the Job spec stored in etcd.
• main container (ingest): runs tatara-ingest --repo-root ... --repo-name ... [--since ...], then patches the result ConfigMap with the cloned HEAD SHA.

# Abridged Job spec produced by the operator
apiVersion: batch/v1
kind: Job
metadata:
  name: my-service-ingest-x7k2m
  labels:
    tatara.dev/repository: my-service
    tatara.dev/ingest-mode: incremental   # or: full
spec:
  backoffLimit: 0          # incremental: 0 (fail fast, let operator escalate to full)
  ttlSecondsAfterFinished: 600
  template:
    spec:
      restartPolicy: Never
      serviceAccountName: tatara-ingest
      initContainers:
      - name: clone
        image: <ingesterImage>
        command: ["/bin/sh", "-c"]
        args:
        - git clone --branch "$GIT_BRANCH" "$GIT_CLONE_URL" "$GIT_REPO_DIR"
        env:
        - name: SCM_TOKEN
          valueFrom:
            secretKeyRef:
              name: <project.spec.scmSecretRef>
              key: token
      containers:
      - name: ingest
        image: <ingesterImage>
        env:
        - name: OIDC_CLIENT_SECRET
          valueFrom:
            secretKeyRef:
              name: <oidcSecretName>
              key: OPERATOR_OIDC_CLIENT_SECRET
        - name: HTTP_TIMEOUT
          value: "300s"
        - name: SEMANTIC_INGEST
          value: "true"     # mirrors Repository.spec.semanticIngest

Retry and backoff¶

The operator applies exponential backoff between failed Job recreations using status.ingestFailureCount and status.lastIngestFailureTime on the Repository CR.

Repository CR - ingest-relevant fields¶

apiVersion: tatara.dev/v1alpha1
kind: Repository
metadata:
  name: my-service
spec:
  projectRef: my-project
  url: https://github.com/myorg/my-service
  defaultBranch: main
  ingestEnabled: true           # default: true; set false to suspend all ingest
  semanticIngest: true          # default: true; set false for AST-only (no LLM cost)
  reingestSchedule: "0 6 * * *" # cron: daily catch-up ingest at 06:00 UTC
status:
  phase: Ingested               # Ingesting | Ingested | Failed
  lastIngestedCommit: "a1b2c3d"
  lastIngestTime: "2026-06-27T06:00:00Z"
  ingestFailureCount: 0
  jobName: "my-service-ingest-x7k2m"

reingestSchedule is required. It fires a periodic full-history catch-up in addition to push-webhook-triggered incremental ingests, guarding against missed events or webhook delivery failures.

Authentication¶

All HTTP calls to tatara-memory are authenticated with an OIDC client-credentials bearer token. The ingester fetches a short-lived token from the issuer's token endpoint before each run and attaches it to every request. Configuration is via environment variables:

When OIDC_CLIENT_ID is unset, the ingester makes unauthenticated requests. This is acceptable in local development but should not be used in production.

Observability¶

The ingester is a short-lived Job and cannot be scraped by Prometheus during its run. It pushes gathered metrics to the operator's /internal/metrics/push receiver at Job completion. The operator's tatara.dev/ingest-mode label on the Job distinguishes incremental from full runs for alert routing.

Key metrics emitted: