Incremental Re-ingest (Operator): Full-History Namespace Clone + Per-Repository Cron Schedule -- Implementation Plan¶
Header¶
- Date: 2026-06-09
- Repo:
tatara-operator(/Users/szymonri/Documents/tatara/tatara-operator) - Module:
github.com/szymonrychu/tatara-operator, Go1.26.3 - Specs (obey exactly):
docs/superpowers/specs/2026-06-09-incremental-reingest-design.md(Component 1 + Phase 0)docs/superpowers/specs/2026-06-09-phase0-contract-lock.md(namespacePath rule, locked)docs/superpowers/specs/2026-06-09-namespace-clone-layout-notes.md(clone layout)- Scope (operator only):
internal/ingest/job.go: drop--depth 1fromcloneCmd(full history so<since>resolves); set clone destination to/workspace/<namespacePath(url)>and point ingester--repo-rootat it; add a tinynamespacePathhelper.api/v1alpha1/repository_types.go: add requiredSpec.ReingestSchedule string(kubebuilder Required + cron pattern) andStatus.LastScheduledReingest *metav1.Time; regenerate CRD (make manifests) + deepcopy (make generate).internal/controller/repository_controller.go: after a repo is Ingested, parsespec.reingestSchedulewithgithub.com/robfig/cron/v3; compute next fire from base (lastScheduledReingest | lastIngestTime | creationTimestamp); if due andnow.After(lastIngestTime)stamp the existingtatara.dev/reingest-requestedannotation + setstatus.lastScheduledReingest; elseRequeueAfter = next-now(clamped). Bad cron: log ERROR, skip. Do not break the webhook path.deploy-samples/tatara-project.yaml: addreingestSchedule: "0 6 * * *"to every Repository.
Repo conventions (verified by reading the code)¶
- Tests: stdlib
testing, plaint.Run/top-levelTest*funcs. Theingestpackage uses pure unit tests (strings.Containsassertions on the built Job). Thecontrollerpackage uses envtest (internal/controller/suite_test.goboots one control plane; helpersmkProject,mkRepo,getRepo,reconcileRepo,listIngestJobs,waitRepoJob,setProjectMemoryReady,findCond).testifyis present ingo.modbut the operator's controller/ingest tests do not use it -- match that, use stdlibt.Errorf/t.Fatalf. - Annotation constant:
tataradevv1alpha1.ReingestRequestedAnnotation = "tatara.dev/reingest-requested"(inapi/v1alpha1/annotations.go), aliased in the controller asReingestAnnotation. The webhook stamps it withtime.Now().UTC().Format(time.RFC3339)(internal/webhook/server.go:133). Match that exact format. - Logging: controller uses
log.FromContext(ctx)(logr). For the bad-cron ERROR, usel.Error(err, "...", "key", val). - Make targets:
make generate(deepcopy via controller-gen),make manifests(CRD intocharts/tatara-operator/crds),make test(envtest:KUBEBUILDER_ASSETS=... go test ./... -race -count=1),make build. - CRD validation is live in envtest.
mkRepo(repository_controller_test.go) andmkTaskRepository(task_controller_test.go)CreateRepositories against the real API server; oncereingestScheduleis Required they MUST set it or every controller test fails at create. Thewebhookandrestapitests use a fake client (no CRD validation) and are unaffected.
Sequencing / dependencies¶
- Task 1 (namespacePath helper) and Task 2 (clone command) are in
internal/ingest, pure unit tests, no CRD dependency -- do them first. - Task 3 (CRD fields + regen) must precede Task 4 and Task 5 because the controller tests run against envtest with the regenerated CRD, and the new Required field forces the test-helper updates in Task 4.
- Task 5 (scheduler) depends on the new spec/status fields (Task 3) and the
robfig/cron/v3dependency (Task 5 step 0). - Task 6 (samples) is independent; do it last.
Run each task's tests with the envtest harness. The exact command for internal/controller tests is:
KUBEBUILDER_ASSETS="$(go run sigs.k8s.io/controller-runtime/tools/setup-envtest@release-0.21 use 1.33.0 -p path)" go test ./internal/controller/... -race -count=1 -run <RegExp>
For internal/ingest (no envtest needed):
Task 1: namespacePath helper in internal/ingest¶
Maps a git clone URL to its on-disk subpath owner[/subgroups]/repo, dropping scheme, host, userinfo, and a trailing .git. Per the contract lock (2026-06-09-phase0-contract-lock.md), implement it tiny and independently (no shared module).
Files¶
- Create
/Users/szymonri/Documents/tatara/tatara-operator/internal/ingest/namespace.go - Create
/Users/szymonri/Documents/tatara/tatara-operator/internal/ingest/namespace_test.go
Steps¶
Step 1.1 -- Write the failing test (full code).
Create /Users/szymonri/Documents/tatara/tatara-operator/internal/ingest/namespace_test.go:
package ingest
import "testing"
func TestNamespacePath(t *testing.T) {
tests := []struct {
name string
url string
want string
}{
{"https github with .git", "https://github.com/szymonrychu/tatara-cli.git", "szymonrychu/tatara-cli"},
{"https github no .git", "https://github.com/szymonrychu/tatara-cli", "szymonrychu/tatara-cli"},
{"https gitlab subgroups", "https://gitlab.com/szymonrychu/infra/helmfile", "szymonrychu/infra/helmfile"},
{"scp-like git@", "git@github.com:szymonrychu/tatara-cli.git", "szymonrychu/tatara-cli"},
{"ssh url with port", "ssh://git@host:22/group/sub/repo.git", "group/sub/repo"},
{"trailing slash", "https://github.com/acme/widgets/", "acme/widgets"},
{"https with userinfo", "https://x-access-token@github.com/acme/widgets.git", "acme/widgets"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
if got := namespacePath(tt.url); got != tt.want {
t.Errorf("namespacePath(%q) = %q, want %q", tt.url, got, tt.want)
}
})
}
}
Step 1.2 -- Run it, expect FAIL.
Expected FAIL: ./namespace_test.go:... undefined: namespacePath (build failure -- the function does not exist yet).
Step 1.3 -- Minimal implementation (full code).
Create /Users/szymonri/Documents/tatara/tatara-operator/internal/ingest/namespace.go:
package ingest
import "strings"
// namespacePath maps a git clone URL to the on-disk subpath:
// owner[/subgroups]/repo, dropping scheme, host, userinfo, and a trailing
// ".git". Keeps the owner.
//
// https://github.com/szymonrychu/tatara-cli.git -> szymonrychu/tatara-cli
// https://gitlab.com/szymonrychu/infra/helmfile -> szymonrychu/infra/helmfile
// git@github.com:szymonrychu/tatara-cli.git -> szymonrychu/tatara-cli
// ssh://git@host:22/group/sub/repo.git -> group/sub/repo
func namespacePath(cloneURL string) string {
s := strings.TrimSpace(cloneURL)
// Drop scheme (https://, ssh://, git://, ...).
if i := strings.Index(s, "://"); i >= 0 {
s = s[i+3:]
}
// scp-like "git@host:owner/repo" has no scheme; split host from path on
// the first ":" when there is no "/" before it.
if !strings.Contains(s[:firstSlash(s)], "/") {
if i := strings.Index(s, ":"); i >= 0 {
s = s[:i] + "/" + s[i+1:]
}
}
// Drop userinfo before the host ("git@host", "x-access-token@host").
if i := strings.Index(s, "@"); i >= 0 && i < firstSlash(s) {
s = s[i+1:]
}
// Drop the host segment (everything up to and including the first slash).
if i := strings.Index(s, "/"); i >= 0 {
s = s[i+1:]
}
s = strings.Trim(s, "/")
s = strings.TrimSuffix(s, ".git")
return s
}
// firstSlash returns the index of the first "/" in s, or len(s) when absent.
func firstSlash(s string) int {
if i := strings.Index(s, "/"); i >= 0 {
return i
}
return len(s)
}
Step 1.4 -- Run it, expect PASS.
Expected PASS: ok github.com/szymonrychu/tatara-operator/internal/ingest.
Step 1.5 -- Commit.
git add internal/ingest/namespace.go internal/ingest/namespace_test.go
git commit -m "feat(ingest): add namespacePath helper for namespace-mirrored clone dir"
Task 2: Full-history clone into namespace dir (internal/ingest/job.go)¶
Drop --depth 1, clone into /workspace/<namespacePath(url)>, and point --repo-root (and the git -C HEAD resolution) at that same dir. Apply both edits to the one cloneCmd together (per the layout notes).
Files¶
- Modify
/Users/szymonri/Documents/tatara/tatara-operator/internal/ingest/job.go - Modify
/Users/szymonri/Documents/tatara/tatara-operator/internal/ingest/job_test.go
Steps¶
Step 2.1 -- Write the failing test (full code).
Add these two test functions to /Users/szymonri/Documents/tatara/tatara-operator/internal/ingest/job_test.go (append after TestBuildJob_IncrementalIngest):
func TestBuildJob_FullHistoryClone(t *testing.T) {
job := BuildJob(testProject(), testRepository(), "", testBaseURL, testConfig())
clone := job.Spec.Template.Spec.InitContainers[0]
cloneCmd := strings.Join(clone.Command, " ") + " " + strings.Join(clone.Args, " ")
if strings.Contains(cloneCmd, "--depth") {
t.Errorf("clone must be full history (no --depth): %q", cloneCmd)
}
if !strings.Contains(cloneCmd, "--branch main") {
t.Errorf("clone cmd missing branch: %q", cloneCmd)
}
}
func TestBuildJob_NamespaceCloneDir(t *testing.T) {
job := BuildJob(testProject(), testRepository(), "", testBaseURL, testConfig())
// widgets repo URL is https://github.com/acme/widgets.git -> acme/widgets
const wantDir = "/workspace/acme/widgets"
clone := job.Spec.Template.Spec.InitContainers[0]
cloneCmd := strings.Join(clone.Command, " ") + " " + strings.Join(clone.Args, " ")
if !strings.Contains(cloneCmd, wantDir) {
t.Errorf("clone must target namespace dir %q: %q", wantDir, cloneCmd)
}
main := job.Spec.Template.Spec.Containers[0]
cmd := strings.Join(main.Command, " ") + " " + strings.Join(main.Args, " ")
if !strings.Contains(cmd, "--repo-root "+wantDir) {
t.Errorf("ingest cmd must use namespace repo-root %q: %q", wantDir, cmd)
}
if !strings.Contains(cmd, "git -C "+wantDir+" rev-parse HEAD") {
t.Errorf("HEAD resolution must run in namespace dir %q: %q", wantDir, cmd)
}
}
Also update the existing TestBuildJob_FullIngest assertion that hard-codes /workspace/repo so it does not regress. In TestBuildJob_FullIngest, change the line:
if !strings.Contains(cmd, "tatara-ingest --repo-root /workspace/repo --repo-name widgets --base-url http://mem-acme.tatara.svc:8080") {
to:
if !strings.Contains(cmd, "tatara-ingest --repo-root /workspace/acme/widgets --repo-name widgets --base-url http://mem-acme.tatara.svc:8080") {
Step 2.2 -- Run it, expect FAIL.
go test ./internal/ingest/... -race -count=1 -run 'TestBuildJob_FullHistoryClone|TestBuildJob_NamespaceCloneDir|TestBuildJob_FullIngest'
Expected FAIL: clone must be full history (no --depth) and clone must target namespace dir "/workspace/acme/widgets" and ingest cmd must use namespace repo-root "/workspace/acme/widgets" (the current code clones --depth 1 into the const /workspace/repo).
Step 2.3 -- Minimal implementation (full code).
Edit /Users/szymonri/Documents/tatara/tatara-operator/internal/ingest/job.go.
(a) Remove the now-unused repoDir const and keep workspaceVolume/workspaceMount. Change the const block from:
to:
(b) Inside BuildJob, derive the per-repo clone dir and drop --depth 1. Replace the cloneCmd/ingestArgs/mainScript block:
// Use git credential helper to inject SCM_TOKEN without embedding it in
// the URL string. The full URL appears literally in the command so tests
// can assert on it; the token is supplied via SecretKeyRef env var.
cloneCmd := fmt.Sprintf(
`set -e; git -c "credential.helper=!f() { echo username=x-access-token; echo password=${SCM_TOKEN}; }; f" `+
`clone --depth 1 --branch %s %s %s`,
repo.Spec.DefaultBranch, repo.Spec.URL, repoDir)
ingestArgs := fmt.Sprintf(
"tatara-ingest --repo-root %s --repo-name %s --base-url %s",
repoDir, repo.Name, baseURL)
if since != "" {
ingestArgs += " --since " + since
}
// After a successful ingest, resolve HEAD and patch the result ConfigMap
// via the in-cluster API (the Job ServiceAccount has patch on it).
resultCM := ResultConfigMapName(repo)
mainScript := fmt.Sprintf(
"set -e; %s; "+
"SHA=$(git -C %s rev-parse HEAD); "+
"kubectl -n %s patch configmap %s --type merge "+
"-p \"{\\\"data\\\":{\\\"sha\\\":\\\"${SHA}\\\"}}\"",
ingestArgs, repoDir, cfg.Namespace, resultCM)
with:
// Clone into a directory that mirrors the repo namespace (owner/.../repo),
// not a flat "/workspace/repo", so concurrent clones never collide.
repoDir := workspaceMount + "/" + namespacePath(repo.Spec.URL)
// Use git credential helper to inject SCM_TOKEN without embedding it in
// the URL string. The full URL appears literally in the command so tests
// can assert on it; the token is supplied via SecretKeyRef env var.
// Full-history clone (no --depth): the incremental diff needs <since> in
// history, and a shallow clone exits 128 when <since> is absent.
cloneCmd := fmt.Sprintf(
`set -e; git -c "credential.helper=!f() { echo username=x-access-token; echo password=${SCM_TOKEN}; }; f" `+
`clone --branch %s %s %s`,
repo.Spec.DefaultBranch, repo.Spec.URL, repoDir)
ingestArgs := fmt.Sprintf(
"tatara-ingest --repo-root %s --repo-name %s --base-url %s",
repoDir, repo.Name, baseURL)
if since != "" {
ingestArgs += " --since " + since
}
// After a successful ingest, resolve HEAD and patch the result ConfigMap
// via the in-cluster API (the Job ServiceAccount has patch on it).
resultCM := ResultConfigMapName(repo)
mainScript := fmt.Sprintf(
"set -e; %s; "+
"SHA=$(git -C %s rev-parse HEAD); "+
"kubectl -n %s patch configmap %s --type merge "+
"-p \"{\\\"data\\\":{\\\"sha\\\":\\\"${SHA}\\\"}}\"",
ingestArgs, repoDir, cfg.Namespace, resultCM)
Step 2.4 -- Run it, expect PASS.
Expected PASS: ok github.com/szymonrychu/tatara-operator/internal/ingest (all TestBuildJob_* pass, including the updated TestBuildJob_FullIngest).
Step 2.5 -- Commit.
git add internal/ingest/job.go internal/ingest/job_test.go
git commit -m "feat(ingest): full-history clone into namespace-mirrored workspace dir"
Task 3: CRD fields -- Spec.ReingestSchedule (required) + Status.LastScheduledReingest¶
Add the API fields, then regenerate deepcopy and CRD. The new spec field is Required with a cron-shaped pattern.
Files¶
- Modify
/Users/szymonri/Documents/tatara/tatara-operator/api/v1alpha1/repository_types.go - Modify (regenerated)
/Users/szymonri/Documents/tatara/tatara-operator/api/v1alpha1/zz_generated.deepcopy.go - Modify (regenerated)
/Users/szymonri/Documents/tatara/tatara-operator/charts/tatara-operator/crds/tatara.dev_repositories.yaml - Create
/Users/szymonri/Documents/tatara/tatara-operator/api/v1alpha1/repository_types_test.go
Steps¶
Step 3.1 -- Write the failing test (full code).
Create /Users/szymonri/Documents/tatara/tatara-operator/api/v1alpha1/repository_types_test.go. This is a pure-Go test asserting the new fields exist, deepcopy works, and the JSON tags are correct (no envtest needed):
package v1alpha1
import (
"testing"
"time"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)
func TestRepositorySpec_ReingestScheduleField(t *testing.T) {
r := &Repository{}
r.Spec.ReingestSchedule = "0 6 * * *"
if r.Spec.ReingestSchedule != "0 6 * * *" {
t.Fatalf("ReingestSchedule = %q, want %q", r.Spec.ReingestSchedule, "0 6 * * *")
}
}
func TestRepositoryStatus_LastScheduledReingestField(t *testing.T) {
r := &Repository{}
now := metav1.NewTime(time.Now())
r.Status.LastScheduledReingest = &now
if r.Status.LastScheduledReingest == nil {
t.Fatal("LastScheduledReingest should round-trip a *metav1.Time")
}
}
func TestRepository_DeepCopyCopiesLastScheduledReingest(t *testing.T) {
now := metav1.NewTime(time.Now())
r := &Repository{}
r.Spec.ReingestSchedule = "0 6 * * *"
r.Status.LastScheduledReingest = &now
cp := r.DeepCopy()
if cp.Spec.ReingestSchedule != "0 6 * * *" {
t.Errorf("deepcopy lost ReingestSchedule: %q", cp.Spec.ReingestSchedule)
}
if cp.Status.LastScheduledReingest == nil {
t.Fatal("deepcopy lost LastScheduledReingest")
}
// Must be a distinct pointer (deep, not shallow).
if cp.Status.LastScheduledReingest == r.Status.LastScheduledReingest {
t.Error("deepcopy must allocate a new LastScheduledReingest pointer")
}
}
Step 3.2 -- Run it, expect FAIL.
Expected FAIL: build error r.Spec.ReingestSchedule undefined (type RepositorySpec has no field or method ReingestSchedule) and r.Status.LastScheduledReingest undefined.
Step 3.3 -- Minimal implementation: add the fields (full code).
Edit /Users/szymonri/Documents/tatara/tatara-operator/api/v1alpha1/repository_types.go.
Replace the RepositorySpec struct:
// RepositorySpec defines the desired state of a Repository.
type RepositorySpec struct {
ProjectRef string `json:"projectRef"`
URL string `json:"url"`
// +kubebuilder:default="main"
// +optional
DefaultBranch string `json:"defaultBranch,omitempty"`
// +kubebuilder:default=true
// +optional
IngestEnabled bool `json:"ingestEnabled,omitempty"`
}
with:
// RepositorySpec defines the desired state of a Repository.
type RepositorySpec struct {
ProjectRef string `json:"projectRef"`
URL string `json:"url"`
// +kubebuilder:default="main"
// +optional
DefaultBranch string `json:"defaultBranch,omitempty"`
// +kubebuilder:default=true
// +optional
IngestEnabled bool `json:"ingestEnabled,omitempty"`
// ReingestSchedule is a standard 5-field cron expression (e.g. "0 6 * * *")
// that triggers a periodic catch-up re-ingest in addition to push webhooks.
// +kubebuilder:validation:Required
// +kubebuilder:validation:MinLength=9
// +kubebuilder:validation:Pattern=`^(\S+\s+){4}\S+$`
ReingestSchedule string `json:"reingestSchedule"`
}
Replace the RepositoryStatus struct by adding the new status field after LastIngestTime:
// RepositoryStatus defines the observed state of a Repository.
type RepositoryStatus struct {
// +kubebuilder:validation:Enum=Pending;Ingesting;Ingested;Failed
// +optional
Phase string `json:"phase,omitempty"`
// +optional
LastIngestedCommit string `json:"lastIngestedCommit,omitempty"`
// +optional
LastIngestTime *metav1.Time `json:"lastIngestTime,omitempty"`
// LastScheduledReingest is the last time the cron schedule stamped a
// reingest-requested annotation; used as the base for the next fire.
// +optional
LastScheduledReingest *metav1.Time `json:"lastScheduledReingest,omitempty"`
// +optional
JobName string `json:"jobName,omitempty"`
// +optional
// +listType=map
// +listMapKey=type
Conditions []metav1.Condition `json:"conditions,omitempty"`
}
Step 3.4 -- Regenerate deepcopy + CRD.
Expected: make generate updates api/v1alpha1/zz_generated.deepcopy.go so RepositoryStatus.DeepCopyInto now also deep-copies LastScheduledReingest (mirroring the existing LastIngestTime block). make manifests adds reingestSchedule (with minLength: 9, the pattern, and a required: [projectRef, reingestSchedule, url] list) to charts/tatara-operator/crds/tatara.dev_repositories.yaml, plus lastScheduledReingest (format: date-time) under status.
Verify the regen actually happened (read-only check):
git diff --stat api/v1alpha1/zz_generated.deepcopy.go charts/tatara-operator/crds/tatara.dev_repositories.yaml
Expected: both files show as modified.
Step 3.5 -- Run the API test, expect PASS.
Expected PASS: ok github.com/szymonrychu/tatara-operator/api/v1alpha1 (all three new tests pass; deepcopy now allocates a distinct LastScheduledReingest pointer).
Step 3.6 -- Commit.
git add api/v1alpha1/repository_types.go api/v1alpha1/repository_types_test.go api/v1alpha1/zz_generated.deepcopy.go charts/tatara-operator/crds/tatara.dev_repositories.yaml
git commit -m "feat(api): add required Repository.spec.reingestSchedule and status.lastScheduledReingest"
Task 4: Make controller envtest helpers set the now-Required reingestSchedule¶
The Required field added in Task 3 makes envtest reject every Repository create that omits it. mkRepo (repository_controller_test.go) and mkTaskRepository (task_controller_test.go) both Create against the real API server. Update them so the existing controller test suite keeps creating valid objects. This is a pure test-infra change; its "test" is the existing suite passing again.
Files¶
- Modify
/Users/szymonri/Documents/tatara/tatara-operator/internal/controller/repository_controller_test.go - Modify
/Users/szymonri/Documents/tatara/tatara-operator/internal/controller/task_controller_test.go
Steps¶
Step 4.1 -- Run the existing controller suite to observe the breakage (expect FAIL).
KUBEBUILDER_ASSETS="$(go run sigs.k8s.io/controller-runtime/tools/setup-envtest@release-0.21 use 1.33.0 -p path)" go test ./internal/controller/... -race -count=1 -run 'TestRepoReconcile|TestTask'
Expected FAIL: creates now fail with an admission error like create repo ...: Repository.tatara.dev "..." is invalid: spec.reingestSchedule: Required value (from mkRepo/mkTaskRepository).
Step 4.2 -- Fix mkRepo (full edited helper).
In /Users/szymonri/Documents/tatara/tatara-operator/internal/controller/repository_controller_test.go, replace the mkRepo helper:
func mkRepo(t *testing.T, name, projectRef string) *tataradevv1alpha1.Repository {
t.Helper()
r := &tataradevv1alpha1.Repository{}
r.Name = name
r.Namespace = testNS
r.Spec.ProjectRef = projectRef
r.Spec.URL = "https://github.com/acme/" + name + ".git"
r.Spec.DefaultBranch = "main"
r.Spec.IngestEnabled = true
if err := k8sClient.Create(context.Background(), r); err != nil {
t.Fatalf("create repo %s: %v", name, err)
}
return r
}
with (add the ReingestSchedule):
func mkRepo(t *testing.T, name, projectRef string) *tataradevv1alpha1.Repository {
t.Helper()
r := &tataradevv1alpha1.Repository{}
r.Name = name
r.Namespace = testNS
r.Spec.ProjectRef = projectRef
r.Spec.URL = "https://github.com/acme/" + name + ".git"
r.Spec.DefaultBranch = "main"
r.Spec.IngestEnabled = true
r.Spec.ReingestSchedule = "0 6 * * *"
if err := k8sClient.Create(context.Background(), r); err != nil {
t.Fatalf("create repo %s: %v", name, err)
}
return r
}
Step 4.3 -- Fix mkTaskRepository (full edited helper).
In /Users/szymonri/Documents/tatara/tatara-operator/internal/controller/task_controller_test.go, replace:
func mkTaskRepository(t *testing.T, name, projectRef string) {
t.Helper()
r := &tatarav1alpha1.Repository{}
r.Name = name
r.Namespace = testNS
r.Spec.ProjectRef = projectRef
r.Spec.URL = "https://git/acme/" + name
r.Spec.DefaultBranch = "main"
if err := k8sClient.Create(context.Background(), r); err != nil {
t.Fatalf("create repository: %v", err)
}
}
with:
func mkTaskRepository(t *testing.T, name, projectRef string) {
t.Helper()
r := &tatarav1alpha1.Repository{}
r.Name = name
r.Namespace = testNS
r.Spec.ProjectRef = projectRef
r.Spec.URL = "https://git/acme/" + name
r.Spec.DefaultBranch = "main"
r.Spec.ReingestSchedule = "0 6 * * *"
if err := k8sClient.Create(context.Background(), r); err != nil {
t.Fatalf("create repository: %v", err)
}
}
Step 4.4 -- Run the suite again, expect PASS.
KUBEBUILDER_ASSETS="$(go run sigs.k8s.io/controller-runtime/tools/setup-envtest@release-0.21 use 1.33.0 -p path)" go test ./internal/controller/... -race -count=1 -run 'TestRepoReconcile|TestTask'
Expected PASS: ok github.com/szymonrychu/tatara-operator/internal/controller (creates succeed again).
Step 4.5 -- Commit.
git add internal/controller/repository_controller_test.go internal/controller/task_controller_test.go
git commit -m "test(controller): set required reingestSchedule in envtest repo helpers"
Task 5: Per-Repository cron scheduler in the reconciler¶
Add the github.com/robfig/cron/v3 dependency, then a scheduleNextReingest step run at the end of Reconcile for already-Ingested repos. It parses spec.reingestSchedule, computes the next fire from base = lastScheduledReingest | lastIngestTime | creationTimestamp, and either stamps the existing annotation (when due and now.After(lastIngestTime)) or returns RequeueAfter = next-now clamped to 6h. Bad cron logs ERROR and skips (no requeue, no crash). The webhook path is untouched: the scheduler only stamps the same annotation ingestDecision already consumes.
Files¶
- Modify
/Users/szymonri/Documents/tatara/tatara-operator/go.mod(andgo.sum, viago get) - Modify
/Users/szymonri/Documents/tatara/tatara-operator/internal/controller/repository_controller.go - Modify
/Users/szymonri/Documents/tatara/tatara-operator/internal/controller/repository_controller_test.go
Steps¶
Step 5.0 -- Add the dependency (this changes state via go get; it is required and explicitly in scope).
Expected: go.mod gains github.com/robfig/cron/v3 v3.0.1 in the require block and go.sum gains the matching hashes. Verify:
Expected output: github.com/robfig/cron/v3 v3.0.1.
Step 5.1 -- Write the failing tests (full code).
Append to /Users/szymonri/Documents/tatara/tatara-operator/internal/controller/repository_controller_test.go. These use the existing envtest harness and helpers. They cover: due -> stamps annotation + sets lastScheduledReingest; not-yet-due -> RequeueAfter set, no stamp; bad cron -> no stamp, no requeue, no error; lastScheduledReingest prevents double-fire within one interval.
func setRepoIngested(t *testing.T, name, sha string, lastIngest time.Time) {
t.Helper()
r := getRepo(t, name)
r.Status.LastIngestedCommit = sha
lt := metav1.NewTime(lastIngest)
r.Status.LastIngestTime = <
r.Status.Phase = "Ingested"
if err := k8sClient.Status().Update(context.Background(), r); err != nil {
t.Fatalf("seed ingested status for %s: %v", name, err)
}
}
func TestRepoReconcile_ScheduleStampsAnnotationWhenDue(t *testing.T) {
mkProject(t, "rp-sch1", "rp-sch1-scm")
mkSecret(t, "rp-sch1-scm", map[string][]byte{"token": []byte("x"), "webhookSecret": []byte("y")})
mkRepo(t, "sch1", "rp-sch1")
setProjectMemoryReady(t, "rp-sch1", "http://mem-rp-sch1.tatara.svc:8080")
// Schedule fires every minute; last ingest was an hour ago, so it is due now.
r := getRepo(t, "sch1")
r.Spec.ReingestSchedule = "* * * * *"
if err := k8sClient.Update(context.Background(), r); err != nil {
t.Fatalf("set schedule: %v", err)
}
setRepoIngested(t, "sch1", "shaSch1", time.Now().Add(-1*time.Hour))
if _, err := reconcileRepo(t, "sch1"); err != nil {
t.Fatalf("reconcile: %v", err)
}
got := getRepo(t, "sch1")
if got.Annotations[ReingestAnnotation] == "" {
t.Fatal("due schedule must stamp the reingest-requested annotation")
}
if got.Status.LastScheduledReingest == nil {
t.Fatal("due schedule must set status.lastScheduledReingest")
}
// No Job yet: the annotation re-triggers reconcile via the watch; this
// reconcile pass only stamps.
if jobs := listIngestJobs(t, "sch1"); len(jobs) != 0 {
t.Fatalf("schedule stamp pass must not itself launch a job, got %d", len(jobs))
}
}
func TestRepoReconcile_ScheduleRequeuesWhenNotDue(t *testing.T) {
mkProject(t, "rp-sch2", "rp-sch2-scm")
mkSecret(t, "rp-sch2-scm", map[string][]byte{"token": []byte("x"), "webhookSecret": []byte("y")})
mkRepo(t, "sch2", "rp-sch2")
setProjectMemoryReady(t, "rp-sch2", "http://mem-rp-sch2.tatara.svc:8080")
// Far-future daily schedule + a fresh ingest => not due; expect a requeue.
r := getRepo(t, "sch2")
r.Spec.ReingestSchedule = "0 6 * * *"
if err := k8sClient.Update(context.Background(), r); err != nil {
t.Fatalf("set schedule: %v", err)
}
setRepoIngested(t, "sch2", "shaSch2", time.Now())
res, err := reconcileRepo(t, "sch2")
if err != nil {
t.Fatalf("reconcile: %v", err)
}
if res.RequeueAfter <= 0 {
t.Errorf("not-due schedule must set RequeueAfter, got %v", res.RequeueAfter)
}
if res.RequeueAfter > 6*time.Hour {
t.Errorf("RequeueAfter must be clamped to 6h, got %v", res.RequeueAfter)
}
if getRepo(t, "sch2").Annotations[ReingestAnnotation] != "" {
t.Error("not-due schedule must not stamp the annotation")
}
if getRepo(t, "sch2").Status.LastScheduledReingest != nil {
t.Error("not-due schedule must not set lastScheduledReingest")
}
}
func TestRepoReconcile_ScheduleBadCronSkips(t *testing.T) {
mkProject(t, "rp-sch3", "rp-sch3-scm")
mkSecret(t, "rp-sch3-scm", map[string][]byte{"token": []byte("x"), "webhookSecret": []byte("y")})
mkRepo(t, "sch3", "rp-sch3")
setProjectMemoryReady(t, "rp-sch3", "http://mem-rp-sch3.tatara.svc:8080")
// A syntactically-shaped but semantically invalid cron (bad minute field).
r := getRepo(t, "sch3")
r.Spec.ReingestSchedule = "99 6 * * *"
if err := k8sClient.Update(context.Background(), r); err != nil {
t.Fatalf("set schedule: %v", err)
}
setRepoIngested(t, "sch3", "shaSch3", time.Now().Add(-1*time.Hour))
res, err := reconcileRepo(t, "sch3")
if err != nil {
t.Fatalf("bad cron must not error the reconcile: %v", err)
}
if res.RequeueAfter != 0 {
t.Errorf("bad cron must skip scheduling (no requeue), got %v", res.RequeueAfter)
}
if getRepo(t, "sch3").Annotations[ReingestAnnotation] != "" {
t.Error("bad cron must not stamp the annotation")
}
}
func TestRepoReconcile_ScheduleNoDoubleFireWithinInterval(t *testing.T) {
mkProject(t, "rp-sch4", "rp-sch4-scm")
mkSecret(t, "rp-sch4-scm", map[string][]byte{"token": []byte("x"), "webhookSecret": []byte("y")})
mkRepo(t, "sch4", "rp-sch4")
setProjectMemoryReady(t, "rp-sch4", "http://mem-rp-sch4.tatara.svc:8080")
r := getRepo(t, "sch4")
r.Spec.ReingestSchedule = "0 6 * * *"
if err := k8sClient.Update(context.Background(), r); err != nil {
t.Fatalf("set schedule: %v", err)
}
setRepoIngested(t, "sch4", "shaSch4", time.Now().Add(-25*time.Hour))
// lastScheduledReingest is recent => next fire is in the future => not due,
// even though lastIngestTime is old. Guards against double-fire.
r = getRepo(t, "sch4")
just := metav1.NewTime(time.Now().Add(-1 * time.Minute))
r.Status.LastScheduledReingest = &just
if err := k8sClient.Status().Update(context.Background(), r); err != nil {
t.Fatalf("seed lastScheduledReingest: %v", err)
}
res, err := reconcileRepo(t, "sch4")
if err != nil {
t.Fatalf("reconcile: %v", err)
}
if getRepo(t, "sch4").Annotations[ReingestAnnotation] != "" {
t.Error("recent lastScheduledReingest must prevent a second stamp this interval")
}
if res.RequeueAfter <= 0 {
t.Errorf("expected a requeue to the next fire, got %v", res.RequeueAfter)
}
}
Step 5.2 -- Run them, expect FAIL.
KUBEBUILDER_ASSETS="$(go run sigs.k8s.io/controller-runtime/tools/setup-envtest@release-0.21 use 1.33.0 -p path)" go test ./internal/controller/... -race -count=1 -run 'TestRepoReconcile_Schedule'
Expected FAIL: TestRepoReconcile_ScheduleStampsAnnotationWhenDue fails -- the annotation is empty and LastScheduledReingest is nil because the reconciler has no scheduling step yet. (TestRepoReconcile_ScheduleRequeuesWhenNotDue may currently pass trivially since RequeueAfter is 0; that is acceptable -- the suite as a whole is red.)
Step 5.3 -- Minimal implementation (full code).
Edit /Users/szymonri/Documents/tatara/tatara-operator/internal/controller/repository_controller.go.
(a) Add the import. Change the import block to include cron:
import (
"context"
"fmt"
"time"
"github.com/robfig/cron/v3"
tataradevv1alpha1 "github.com/szymonrychu/tatara-operator/api/v1alpha1"
"github.com/szymonrychu/tatara-operator/internal/ingest"
"github.com/szymonrychu/tatara-operator/internal/obs"
batchv1 "k8s.io/api/batch/v1"
corev1 "k8s.io/api/core/v1"
apierrors "k8s.io/apimachinery/pkg/api/errors"
"k8s.io/apimachinery/pkg/api/meta"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/runtime"
"k8s.io/apimachinery/pkg/types"
ctrl "sigs.k8s.io/controller-runtime"
"sigs.k8s.io/controller-runtime/pkg/client"
"sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
"sigs.k8s.io/controller-runtime/pkg/log"
)
(b) Add a clamp const near the top (after ReingestAnnotation):
// maxScheduleRequeue bounds the cron requeue so clock skew or long sleeps still
// re-evaluate the schedule reasonably soon.
const maxScheduleRequeue = 6 * time.Hour
© Replace the post-ingestDecision "no ingest wanted" early return so it runs the scheduler instead of returning bare. Currently:
since, want := r.ingestDecision(&repo)
if !want {
r.Metrics.ReconcileResult("Repository", "success")
return ctrl.Result{}, nil
}
becomes:
since, want := r.ingestDecision(&repo)
if !want {
res, err := r.scheduleNextReingest(ctx, &repo)
if err != nil {
r.Metrics.ReconcileResult("Repository", "error")
return ctrl.Result{}, err
}
r.Metrics.ReconcileResult("Repository", "success")
return res, nil
}
(d) Add the scheduler method. Append after ingestDecision:
// scheduleNextReingest applies the per-Repository cron schedule for an
// already-ingested repo. It parses spec.reingestSchedule and computes the next
// fire from base = lastScheduledReingest | lastIngestTime | creationTimestamp.
// When the fire is due (and strictly after lastIngestTime, so an in-flight
// ingest from another trigger is not double-stamped), it stamps the existing
// reingest-requested annotation and records lastScheduledReingest; the
// annotation change re-triggers reconcile, which launches the Job via the
// existing path. Otherwise it requeues at the next fire (clamped). A bad cron
// expression is logged at ERROR and skipped (no requeue, no error).
func (r *RepositoryReconciler) scheduleNextReingest(ctx context.Context, repo *tataradevv1alpha1.Repository) (ctrl.Result, error) {
l := log.FromContext(ctx)
// Only schedule once a repo has been ingested at least once; a first full
// ingest is driven by ingestDecision, not the cron.
if repo.Status.LastIngestedCommit == "" || repo.Spec.ReingestSchedule == "" {
return ctrl.Result{}, nil
}
schedule, err := cron.ParseStandard(repo.Spec.ReingestSchedule)
if err != nil {
l.Error(err, "invalid reingestSchedule, skipping cron",
"action", "ingest_schedule_invalid", "resource_id", repo.Name,
"schedule", repo.Spec.ReingestSchedule)
return ctrl.Result{}, nil
}
var lastIngestTime time.Time
if repo.Status.LastIngestTime != nil {
lastIngestTime = repo.Status.LastIngestTime.Time
}
base := repo.CreationTimestamp.Time
if repo.Status.LastIngestTime != nil {
base = repo.Status.LastIngestTime.Time
}
if repo.Status.LastScheduledReingest != nil {
base = repo.Status.LastScheduledReingest.Time
}
now := time.Now()
next := schedule.Next(base)
if now.Before(next) {
requeue := next.Sub(now)
if requeue > maxScheduleRequeue {
requeue = maxScheduleRequeue
}
return ctrl.Result{RequeueAfter: requeue}, nil
}
// Due. Guard against firing while an ingest from another trigger is still
// in flight or just finished: only stamp when now is strictly after the
// last successful ingest.
if !now.After(lastIngestTime) {
return ctrl.Result{RequeueAfter: maxScheduleRequeue}, nil
}
if repo.Annotations == nil {
repo.Annotations = map[string]string{}
}
repo.Annotations[ReingestAnnotation] = now.UTC().Format(time.RFC3339)
if err := r.Update(ctx, repo); err != nil {
return ctrl.Result{}, fmt.Errorf("stamp scheduled reingest annotation: %w", err)
}
scheduled := metav1.NewTime(now)
repo.Status.LastScheduledReingest = &scheduled
if err := r.Status().Update(ctx, repo); err != nil {
return ctrl.Result{}, fmt.Errorf("update lastScheduledReingest: %w", err)
}
l.Info("scheduled re-ingest requested",
"action", "ingest_schedule_fire", "resource_id", repo.Name,
"schedule", repo.Spec.ReingestSchedule)
return ctrl.Result{}, nil
}
Notes for the implementer: stamp the annotation with r.Update (spec/metadata write) BEFORE the status write, matching the webhook's Update ordering and the existing ingestDecision annotation contract. The metadata Update bumps the object so the subsequent Status().Update operates on the returned object's resourceVersion in-place (controller-runtime mutates repo on Update). The annotation re-triggers reconcile via the existing For(&Repository{}) watch; that next pass goes through ingestDecision, sees requested.After(lastIngestTime), and launches the incremental Job -- the webhook path is wholly reused and untouched.
Step 5.4 -- Run them, expect PASS.
KUBEBUILDER_ASSETS="$(go run sigs.k8s.io/controller-runtime/tools/setup-envtest@release-0.21 use 1.33.0 -p path)" go test ./internal/controller/... -race -count=1 -run 'TestRepoReconcile'
Expected PASS: ok github.com/szymonrychu/tatara-operator/internal/controller (all four new TestRepoReconcile_Schedule* pass and the existing TestRepoReconcile_* still pass -- the not-due/stale-annotation cases now also exercise the requeue branch without launching jobs).
Step 5.5 -- Commit.
git add go.mod go.sum internal/controller/repository_controller.go internal/controller/repository_controller_test.go
git commit -m "feat(controller): per-Repository cron schedule stamps reingest annotation"
Task 6: Add reingestSchedule to every sample Repository¶
The new field is Required, so the samples must carry it. Add reingestSchedule: "0 6 * * *" to all six Repository entries.
Files¶
- Modify
/Users/szymonri/Documents/tatara/tatara-operator/deploy-samples/tatara-project.yaml
Steps¶
Step 6.1 -- Edit the samples (full intended content of each Repository block).
In /Users/szymonri/Documents/tatara/tatara-operator/deploy-samples/tatara-project.yaml, add reingestSchedule: "0 6 * * *" to each Repository's inline spec map. The six spec lines become:
spec: {projectRef: tatara, url: "https://github.com/szymonrychu/tatara-memory", defaultBranch: main, reingestSchedule: "0 6 * * *"}
spec: {projectRef: tatara, url: "https://github.com/szymonrychu/tatara-cli", defaultBranch: main, reingestSchedule: "0 6 * * *"}
spec: {projectRef: tatara, url: "https://github.com/szymonrychu/tatara-operator", defaultBranch: main, reingestSchedule: "0 6 * * *"}
spec: {projectRef: tatara, url: "https://github.com/szymonrychu/tatara-chat", defaultBranch: main, reingestSchedule: "0 6 * * *"}
spec: {projectRef: tatara, url: "https://github.com/szymonrychu/tatara-memory-repo-ingester", defaultBranch: main, reingestSchedule: "0 6 * * *"}
spec: {projectRef: tatara, url: "https://github.com/szymonrychu/tatara-claude-code-wrapper", defaultBranch: main, reingestSchedule: "0 6 * * *"}
(The Project block at the top is unchanged.)
Step 6.2 -- Validate the samples against the regenerated CRD (read-only dry-run; no cluster mutation).
If a cluster/kubeconfig is available:
Expected: repository.tatara.dev/tatara-memory configured (dry run) etc., with no spec.reingestSchedule: Required value error. If no cluster is reachable, instead grep-verify all six entries carry the field:
Expected: 6.
Step 6.3 -- Commit.
git add deploy-samples/tatara-project.yaml
git commit -m "chore(samples): set reingestSchedule on every sample Repository"
Final verification (whole-repo gate)¶
After all tasks, run the full quality gate the repo's make ci uses, to confirm nothing regressed:
make generate
make manifests
git diff --exit-code api/v1alpha1/zz_generated.deepcopy.go charts/tatara-operator/crds/tatara.dev_repositories.yaml
make fmt
make lint
make test
make build
Expected: - git diff --exit-code on the generated files is clean (regen is idempotent; if it is not, the committed CRD/deepcopy was stale -- re-commit). - make lint passes (or exits 5 = "no issues", which the Makefile treats as success). - make test -> ok for ./api/v1alpha1/..., ./internal/ingest/..., ./internal/controller/..., and all other packages (webhook/restapi unaffected -- they use the fake client and never set the Required field). - make build produces bin/tatara-operator.
If make test reaches the internal/webhook or internal/restapi packages and they fail on the new Required field, that means a fake-client test there started enforcing CRD schema (it does not today) -- recheck, but no change is expected.
Notes carried from the specs (do not lose)¶
- The memory
repoLABEL stays the logical identifier (Repository CR name); only the clone DIRECTORY mirrors the namespace. No re-keying. (2026-06-09-phase0-contract-lock.md,2026-06-09-namespace-clone-layout-notes.md.) - The clone keeps the existing inline credential helper that injects
SCM_TOKEN; no other Job-spec change. (2026-06-09-incremental-reingest-design.md1a.) - The scheduler adds no CronJob resource, no new RBAC, no new image -- it only reuses the existing webhook re-ingest mechanism with a time-based stamper. (
2026-06-09-incremental-reingest-design.md1b.) - Out of operator scope here (other repos / later MR): the live-CR patch of
reingestScheduleon the 6 running Repositories, the ingesterwalk/pushchanges, the memory migrations, and chart/image version bumps. This plan covers only the four operator code/CRD/sample edits.
Critical Files for Implementation¶
- /Users/szymonri/Documents/tatara/tatara-operator/internal/ingest/job.go
- /Users/szymonri/Documents/tatara/tatara-operator/internal/controller/repository_controller.go
- /Users/szymonri/Documents/tatara/tatara-operator/api/v1alpha1/repository_types.go
- /Users/szymonri/Documents/tatara/tatara-operator/internal/controller/repository_controller_test.go
- /Users/szymonri/Documents/tatara/tatara-operator/Makefile