Ownership¶

Every SCM artifact the platform touches is mirrored into the cluster as a CR: an Issue or a MergeRequest. Nothing in the operator walks those CRs and deletes them. Kubernetes' garbage collector does, and the operator's only job is to keep the owner references honest.

That is not a stylistic preference. It is the mechanism the whole redesign rests on. The refine fold, the reaper, the delivery cleanup and the invariant that no SCM artifact may exist without a Task responsible for it are all consequences of one three-line property of the GC - and there is not a single line of cascade code anywhere in the operator.

The GC invariant¶

The Kubernetes garbage collector deletes a dependent object only when every owner reference resolves to gone (attemptToDeleteItem deletes only once len(solid) == 0). And an object with zero owner references is never collected.

Both halves are load-bearing, and they pull in opposite directions:

• Zero owners means it leaks forever. So "no SCM artifact without a Task" is not a rule the operator enforces in code - it is structural. An Issue CR that no Task owns would sit in etcd until a human deleted it. The sweep's job is to make sure one always does own it.
• All-owners-gone means delete. So the refine fold and the delivery cleanup need no cascade code at all. The operator drops an owner reference, or deletes a Task, and the API server does the rest.

Who owns what¶

An Issue and a MergeRequest are the only objects with more than one owner, and that is the entire point: a refine umbrella and the member Task it folded in can both hold a reference to the same issue while exactly one of them is responsible for it.

flowchart LR
    U["Task: refine umbrella"]
    M["Task: brainstorm member"]
    I["Issue CR: iss-tatara-operator-412"]
    U == "ownerRef controller=true, blockOwnerDeletion=true" ==> I
    M -. "ownerRef controller=false, blockOwnerDeletion=true" .-> I

Five rules follow, and each one exists because something breaks without it.

1. Exactly one owner carries controller=true. It is the TASK print column on kubectl get iss, it is the anti-race lock, and - the part that is easy to miss - it is the authorization for every SCM write against that artifact. The issue_write and mr_write MCP tools refuse with 409 task does not own this issue when the calling Task is not the controller owner. Without that check, two Tasks that both own an issue could each spawn a pod and hold a conversation with each other on a human's thread.

2. Every owner reference sets blockOwnerDeletion: true. Explicitly, on every write, including the plain ones.

3. Non-controller owners are plain references. controller unset or false, blockOwnerDeletion still true. Their only job is to hold GC open.

4. A Task never owns a Task. A Task-owns-Task edge would cascade-delete exactly the artifacts a refine fold exists to preserve. The umbrella adopts the members' artifacts and then deletes the member Tasks; if it owned them, deleting the members would take their issues with them.

5. An Issue or MergeRequest must never have zero controller owners. Every path that removes one - the fold, the reap - hands the flag to a survivor first. An artifact with plain owners and no controller owner is worked by nobody and re-minted by nobody: the sweep's orphan predicate looks for an issue with no controller owner, but it also sees an issue that is owned, and the two conditions are not the same. A reconcile guard detects the state, logs at ERROR, increments operator_orphan_no_controller_total, and repairs it by promoting the oldest surviving owner.

The refine fold¶

A refine Task groups several existing Tasks under one umbrella. It does not adopt the Tasks - rule 4 forbids that - it adopts their artifacts, and then deletes the Tasks. The sequence is adopt, verify, then delete.

Before anything is written, /outcome checks that every named member exists, is in the same project, and is not live: a member with a running pod (status.podName != "") or a live post-approved stage is refused with 409 fold target has work in flight. That is the same liveness gate the closes[] directive gets.

1. status.foldInFlight = [M1..Mn]                       (one Status().Update)

2. for each artifact A owned by Mi:
     ONE Update on A that atomically:
       - APPENDS U's ownerRef   (controller=false, blockOwnerDeletion=true)
       - REWRITES Mi's ownerRef to controller=false
       - REWRITES U's ownerRef  to controller=true

3. RE-LIST every named artifact; VERIFY U is a solid owner with controller=true
   on each. On ANY mismatch: stage -> failed(fold-adoption-unverified),
   foldInFlight cleared, members NOT deleted. Nothing is lost; a human sees a
   failed umbrella.

4. only then: delete M1..Mn

5. foldInFlight = []

Step 2 must be one PUT. The API server rejects an object carrying two controller=true references, so the swap cannot be done as two writes: there is no ordering of "add the new controller" and "demote the old one" that does not either transit through an illegal object or through a window with no controller owner at all.

A crash between steps 2 and 4 is safe and idempotent - the artifacts have both owners, the umbrella is already the controller, and the next reconcile finishes the job. A crash after step 4 leaves foldInFlight set, and the reconciler clears it once the members are gone.

Reaping¶

Terminal Tasks age out. Before the reaper deletes any Task, it settles that Task's controller ownership, for every Issue and MergeRequest the Task controller-owns:

if the artifact has >= 1 SURVIVING plain owner:
    hand controller=true to the OLDEST surviving owner
    (the same atomic single-PUT swap as the fold), THEN delete the Task.
else:
    delete the Task; the artifact cascades with it.

This is a routine consequence of multi-owner GC - a folded umbrella reaped while a sibling still holds a plain reference - not an exotic race. Rule 5's repair guard is the belt; this is the braces.

Every terminal stage ages out¶

Two of those rows are the ones readers get wrong.

delivered is reaped at 48h, not 24h, and only once status.documentedBy is stamped or the Task provably had zero merged MRs. The nightly documentation batch runs once a day, so a Task delivered at 23:30 would have had thirty minutes of margin against a 24h TTL. The margin should not be decorative. A stuck documentation batch cannot pin its parents either: it is force-moved to delivered at docStageBudget and stamps documentedBy on every Task in its spec.documentsTasks on the way out - unless it never ran a pod at all, in which case it stamps nothing and the next night's batch picks the same Tasks up.

parked(backlog-sweep) is never reaped on age. It is exempt from parkRetention and is reaped only when every Issue it owns has state == closed. It is the anchor that holds the mirror open for a backlog issue nobody is working; ageing it out would churn mint and reap forever. See Task stages for the full set of park reasons and which of them have a re-entry path.

failed releases its Issues immediately¶

A failed Task that kept controller-ownership of its issues for seven days would hold them hostage: the orphan predicate would not re-mint them, no bot comment would explain why, and nothing would move. So the terminal path runs in a fixed order, and the ordering is the load-bearing part:

on entering failed / rejected, or being reaped from parked:

  1. post a bot comment on every owned OPEN Issue naming stageReason
     (BLOCKING: a 403 requeues the reap)

  2. stamp the tatara-parked label on the SCM issue
     (BLOCKING TOO - the sweep's mint-stage predicate READS this label)

  3. RELEASE controller-ownership FIRST:
       - hand it to the oldest SURVIVING plain owner, if any
       - else DROP the ownerRef entirely, so the next sweep re-mints the issue
         as parked(backlog-sweep) and ADOPTS the ownerless CR

  4. ONLY NOW, and ONLY for MRs WE CREATED: see below.

  5. the Task CR itself survives 7 days as a DEBUGGING ARTIFACT - owning
     nothing, blocking nothing.

Step 3 leaves an Issue CR with zero owners when there is no survivor, and per the GC invariant a zero-owner object is never collected. That is deliberate: the CR is still there, with its mirrored thread intact, waiting for the next sweep to adopt it.

The reaper skips any Task named in a live Task's status.foldInFlight.

Closing what the platform created¶

Step 4 above is the cleanup of open merge requests, and it runs after the ownership hand-off, never before. Getting the order backwards means closing a merge request and then handing the corpse to the survivor that is still working it.

On a Task entering failed or rejected, or being reaped from parked, the operator closes an owned open MergeRequest and deletes its head branch only when all four of these hold:

1. the merge request is bot-authored (mr.status.author equals Project.spec.scm.botLogin)
2. its head branch (mr.status.headBranch) is task/<this-task-name>
3. this Task is (or was) the controller owner
4. no surviving plain owner remains

Otherwise it is left completely alone. No close, no branch delete, no comment.

The branch delete is retained only under all four clauses: only for a branch this Task itself pushed and that nobody else now owns. Where the four clauses do not hold, the branch is preserved so a human can reopen the merge request and retry.

The sweep¶

The sweep is what keeps the structural invariant true. It runs hourly and nightly, over the same one predicate, in one function.

The orphan predicate¶

ISSUE ORPHAN (mint a Task) iff ALL of:
    a. issue.state == "open"                          (SCM truth)
    b. no Issue CR for (repo, number) has a controller=true owner
    c. IsAllowedReporter(project, repo, issue.author)

Clause b is answered by the issueKey field index (<repositoryRef>#<number>), never by a hashed name and never by a label selector. It is the same index, and the same question, that the QueuedEvent producer uses to decide whether an issue is already being worked. One source of truth.

Clause c is the reporter intake gate. It is closed-by-default when configured: an empty allowlist preserves the open default, and an empty login fails closed under an active gate. Its entire purpose is that an issue injected by an untrusted reporter never becomes a Task.

Two mint stages, and the cost decision¶

The orphan predicate says whether to mint. The mint stage says what it costs.

• stage = triaging (active: this spawns a pod) when the Task is webhook-originated, or the thread's last comment is human-authored. A human is waiting on us.
• stage = parked, stageReason = backlog-sweep (spawns zero pods) otherwise. That is every cron-discovered backlog issue, and every re-mint of an issue whose previous Task the operator parked and the reaper collected.

A parked(backlog-sweep) Task owns its Issue CRs and spends no agent time. It preserves "no SCM artifact without a Task" at zero pod cost, and it is what makes the ownership invariant affordable across a 150-issue backlog. Without it, the post-cutover sweep would mint 150 active Tasks queueing against three agent slots and spend somewhere between 17 and 100 pod-hours re-triaging a backlog that was already triaged.

It also breaks what would otherwise be an infinite re-mint loop: a re-minted backlog Task owns the issue, so on the next sweep the issue is no longer an orphan. Ownership breaks the loop, not a comment heuristic.

parked(backlog-sweep) is not free¶

Its Issue CRs are mirrored, and a mirror costs forge requests: 150 backlog issues plus their threads is several hundred requests per sweep. So the mirror cadence is keyed on the Task's stage:

The on-demand sync is not an optimisation. A non-bot comment un-parks a parked(identity-unverified) Task and spawns a fresh implement pod, and that pod needs the approving comment in the mirror, with its external id, both to render it into the turn-0 bundle the agent cites from and because the operator's restapi.verifyApprovalScope enforces single-use evidence against that same mirror on the next submit_outcome. A TaskEvent carries no external id. Without the sync, the fresh pod would see a bundle that does not contain the comment that triggered it, could not cite it, and the Task would park again, silently.

The mint adopts, it never blindly creates¶

Because a failed Task drops its owner reference, a zero-owner Issue CR is a normal, expected state - and per the GC invariant it is still there. So the mint is adopt-or-create:

mintTaskForIssue(repo, number):
    task := create the Task
    iss  := GET iss-<repo>-<number>
    if iss EXISTS:                      // the ownerless survivor of a failed Task
        APPEND task as its controller=true owner   (an UPDATE)
    else:
        CREATE it with task as controller owner

A blind Create would AlreadyExists on every re-mint of every previously failed Task, which is exactly the recovery path the immediate release exists to enable.

Creation budgets¶

per sweep pass:
    mint at most Project.spec.maxNewTasksPerSweep (default 5) Tasks
    AND never let ACTIVE Tasks exceed Project.spec.maxOpenTasks (default 6)
        ACTIVE = stage NOT IN (parked, delivered, rejected, failed)
        parked(backlog-sweep) Tasks are NOT active and do not count

Remaining orphans are minted on the next pass. The predicate is stateless, so nothing is lost. The operator records operator_tasks_minted_per_sweep{project,stage} on every pass and operator_sweep_mint_cap_hit_total{project,cap} with a WARN when either cap binds.

Backlog Tasks minted at parked(backlog-sweep) enqueue nothing, because they spawn no pod. Admission is per pod-spawn, not per Task - a QueuedEvent holds a pod slot, maxConcurrentAgents is its capacity, and maxConcurrentAgents: 0 is the whole-project kill switch. A backlog of parked Tasks never crowds that queue at all.

Pull requests: adopt, ignore, or review¶

An open pull request the sweep finds gets exactly one of four dispositions.

1. ADOPT into an existing Task's mrRefs iff ALL of:
       a. author       == Project.spec.scm.botLogin
       b. head branch  == "task/<owning-task-name>"
       c. head.repo    == base.repo               (NO forks, ever)

2. BOT-AUTHORED and NOT ADOPTABLE  ->  IGNORE. FULL STOP.
       No Task. No pod. No tokens. Never a review Task.

3. HUMAN-AUTHORED  ->  mint a review-kind Task iff the PR is in reaction scope:
       the author is a trusted insider, OR Scm.PRReactionScope is not
       "labeledOrMentioned", OR the PR carries the trigger label, OR it
       @-mentions the bot.

4. Everything else is IGNORED.

A fork pull request may name its head branch anything it likes, including task/<a-real-task>. Clause c is what stops an outside contributor injecting a merge request into a trusted Task's merge stream. It is the only clause of the three that an attacker fully controls.

Clause 2 is what keeps the review population honest, and it closes two real holes. An orphaned agent PR - one whose Task failed and was reaped, leaving the pull request open on the forge - would otherwise be minted as a review Task, approved by the documented-flaky review agent, and merged by the operator: push-CD would ship an abandoned change that no gate ever approved. And a CI pin-bump PR - the bot bump PR that tatara-helmfile opens on every release, and the daily one the wrapper opens to refresh its pinned Claude Code version - would have minted a review Task, eaten the maxOpenTasks budget, and raced the forge's own merge automation on a pull request the platform has no business touching.

Both are ignored, and the orphaned PR is closed by the four-clause rule above rather than adopted. The consequence is a simplification worth stating: every review-kind Task is now, by construction, non-bot-authored. That is what makes "never close what we did not create" enforceable, and it is why a review Task has no edge into implementing or merging at all. Merging a human's pull request is a human's job.

An unwritable Task pins its artifacts forever¶

Ownership is what makes one otherwise-unremarkable failure mode the worst in the design. A Task CR that grows past etcd's object ceiling cannot be written. A 413 is not retried on conflict, so every writer fails, the stage never advances, the Task never goes terminal, the reaper never runs - and the issues and merge requests it controller-owns stay pinned by that ownership forever.

So the operator runs a byte-exact guard before every write, budgeted at 800,000 bytes, half the ceiling. The headroom is for metadata.managedFields, which grows without bound under repeated server-side-apply status patches and counts against the same limit. Oldest comments and oldest notes are spilled to tatara-memory first, and dropped from the CR only on spill success - and the spill runs outside the retry closure, so a conflict cannot re-fire it.

When even that cannot win - an enormous goal, a wall of conditions, nothing left to evict - the Task fails loudly: stage = failed, stageReason = object-too-large, written as a minimal JSON merge patch that carries only the two status fields, so the escape hatch itself cannot 413. And because failed releases its issues immediately, the artifacts are freed rather than pinned.

A count cap would not do this job. Two hundred notes of 4 KB is 800 KB; two hundred notes of 40 KB is 8 MB. Only bytes are bytes.

MR Ownership¶

MR ownership is the push/merge agency of a merge request - distinct from the Kubernetes owner-reference "who owns what" structure above. Every MergeRequest carries an ownership field: tatara (the platform may push, review, and merge) or external (review rounds only; the operator never pushes, awaiting a human hand). The distinction is not author-ship - even a tatara-authored MR can be external - it is agency.

Ownership signals¶

The machine signal is the last bot-pushed head SHA, stored on the MergeRequest CR's status.lastBotHeadSHA. If the MR's current head resolves to the same commit, ownership is tatara. If a human pushed a different commit, ownership is external.

Ownership is not authoritative until the operator writes it to the MergeRequest CR's status.ownership field, together with a status.ownershipReason (one of: initial - MR was never delegated to tatara; takeover-requested-by:<user> - a maintainer requested takeover; external-push:<sha> - a human pushed to an MR a maintainer had handed over; adopted-external-push:<sha> - a human pushed to an adopted dependency-upgrade MR). The last two are deliberately different strings: the merge gate tests for the external-push: prefix exactly, so the adopted shape does not open it (see The merge gate). The field is read-only from the agent's perspective: only the operator sets it, on every reconcile loop, by comparing the head SHA to lastBotHeadSHA. The status.lastMirroredCommentID field tracks the sweep's comment-redelivery cursor, ensuring bot comments are not re-posted on every sync.

Flipping ownership: human push¶

A push of any commit other than lastBotHeadSHA to the MR's head branch flips ownership from tatara to external, with reason external-push:<sha> on an MR a maintainer handed over and adopted-external-push:<sha> on an adopted dependency-upgrade MR. The operator detects it on the next webhook or reconcile and stamps the flip; DrainOwnershipAnnouncement, running on the next MergeRequest reconcile, posts the stand-down announcement: "A commit outside tatara landed on this branch - standing down from pushing. I'll keep reviewing, and the operator will still merge on an approved review. Comment 'take over' to hand it back." The bot stops pushing commits, but the review continues, and the operator will still merge on an approved review - unless the flip carried the adopted reason, which the merge gate refuses.

Flipping ownership back: maintainer comment¶

A project maintainer (an account in the project or repository maintainerLogins set, closed by default) can hand an external MR back to tatara by posting a natural-language comment requesting a takeover, for example "please take over, resolve the conflicts, and merge once green". There is no fixed command syntax. The review agent reads the comment, judges whether it is a hand-over request, and if so, calls the MCP tool mr_takeover_request(repo, number, comment_external_id) to recommend takeover to the operator.

The operator receives the recommendation and re-checks the comment author server-side against the effective maintainer set (Project/Repository maintainerLogins). The agent's judgment alone can never change ownership. If the author is a maintainer, the operator stamps status.ownership = tatara with reason takeover-requested-by:<user>, bumps lastBotHeadSHA to the live head, and calls MintOrUnparkTakeoverTask.

There is exactly one deterministically named, full-lifecycle takeover Task per MR, reused across every take-over / stand-down / re-take round, so its history is never reset by a fresh mint. That one call mints it straight into under-implementation if none exists (not refined: a takeover's source is always a PR, so it owns zero Issue CRs and the approval gate there could never be passed), un-parks it back into under-implementation if it already exists and is parked(ownership-lost), and returns it unchanged in any other live state. It is bound to the MR's existing head branch, so the implement pod pushes there rather than to a derived tatara/* branch. From that point on, the agent may push and merge.

The endpoint posts no comment of its own: DrainOwnershipAnnouncement posts the take-over announcement on the next MergeRequest reconcile, once, keyed on the flip timestamp.

If the author is not a maintainer, the endpoint answers HTTP 403 with the body author not in maintainer set: only a project maintainer may hand this merge request over, and nothing changes. No comment is posted to the forge on that path either - refusing the requester conversationally is the review skill's job, not the operator's.

A take-over comment can only be acted on if the kind=review Task gets a live turn to read it, and both of its parked(awaiting-human) re-entry paths are normally bounded by maxHumanReviewRounds (5). An externally-owned MR is exempt from that cap - the round budget bounds ordinary review ping-pong, not a maintainer reclaiming a stood-down MR - so a take-over comment still spawns a pod even after the cap was already spent on unrelated review rounds.

Adopting a dependency engine's merge requests¶

A merge request a dependency engine (Renovate) opens on its own can reach tatara ownership without ever going through a takeover comment. Two separate predicates are at work here, and they answer different questions.

ownershipForAuthor classifies a never-seen MR and checks the author only - it never looks at the branch. An author this project owns (scm.botLogin, or a login listed in Project.spec.upgradePolicy.upgradeEngineLogins - see Project reference) is classified tatara; anything else - a human, an un-allowlisted bot, or an unknown/empty author - is external. There is no lastBotHeadSHA comparison and no ownershipReason: initial in between.

AdoptUpgradeMR is the separate predicate deciding which of those MRs the platform adopts into its own kind=upgrade Task (MintAdoptedUpgradeTask). The branch prefix is one of its clauses, not an ownership signal: it gates which Task gets minted, not how ownership is classified. Adoption requires all of:

• upgradePolicy.adoptBranchPrefix is configured. Empty is the default and disables adoption outright
• the MR's head branch starts with that prefix
• the author is adoptable - the same predicate ownershipForAuthor delegates to, so the identity that permits adoption and the identity the merge corridor checks are one fact, not two
• the head repo equals the base repo. No forks, ever; an unknown head repo fails closed
• no Task owns the branch and no live Task owns the mirror CR. This is what stops re-adoption
• the MR does not carry the project's triggerLabel. Labelling a prefixed branch is the human override - an explicit ask for a review rather than an adoption
• its mirror carries no durable adoption-refusal marker (AnnAdoptionRefused) for this head SHA. The marker is keyed on the refused SHA, so a declined tree stays declined while a genuinely new bump force-pushed onto the same branch gets a fresh decision
• its mirror has not stood down to an external push, which is exactly what the merge gate refuses

The adopted upgrade Task writes no ownership of its own, because the author predicate the adoption uses is the one ownershipForAuthor has already answered. From the review agent's next turn, approve merges the engine's own MR exactly like any other owned MR - see PR/MR Review: adopted merge requests.

This is a separate path from maintainer takeover above: takeover is a one-off, comment-triggered hand-over of a single MR; adoption is a standing policy that claims every future MR matching the prefix-and-author pair. Admission is not paced by maxOpenUpgrades - each matching MR is queued and admitted the moment the project's general concurrency pool has room, the same immediate-admit path any other queued event uses. See Upgrade workflow: adopting the engine's own merge requests for the queueing mechanics.

The merge gate¶

The merge gate merges when status.ownership == tatara or status.ownership == external with a reason prefix external-push:. The first case is a fully delegated MR; the second is an MR that tatara previously owned and a human paused, but may resume pushing and merging it. An MR with ownership external and reason initial (never delegated to tatara) will not be merged by the operator.

The prefix test is exact, and adopted-external-push: does not match it. A human pushing to an adopted dependency-upgrade MR therefore makes it human-merged-only: the operator keeps reviewing but will never merge it, and the same holds for the merge-after-stand-down path. The two reasons cannot share a string. Merge-after-stand-down exists because a maintainer explicitly asked the platform to take an MR over, so continuing across their later push is the behaviour they asked for; nobody asks for an adoption, and reusing external-push: there would hand a human's commit to a pod-less merge corridor on a decision no human ever made.

Where to go next¶

• Issue and MergeRequest are the owned artifacts: their fields, their mirror, and the write path.
• Task stages covers the stage machine, the park reasons, and which of them have a re-entry path.
• Task covers foldInFlight, documentedBy, and the rest of the status surface this page relies on.
• Handing an MR to tatara is the maintainer's how-to for requesting a takeover.