Incremental Re-ingest + Phase 0 Capture (tatara-memory-repo-ingester) Implementation Plan¶
Header¶
- Repo:
tatara-memory-repo-ingester(/Users/szymonri/Documents/tatara/tatara-memory-repo-ingester) - Module:
github.com/szymonrychu/tatara-memory-repo-ingester - Go:
1.25.0 - Spec sources (obey byte-for-byte):
docs/superpowers/specs/2026-06-09-incremental-reingest-design.md(Component 2 + Phase 0)docs/superpowers/specs/2026-06-09-phase0-contract-lock.md(LOCKED wire types)docs/superpowers/specs/2026-06-09-namespace-clone-layout-notes.md- Test style: stdlib
testing, table-drivent.Run,github.com/stretchr/testify/require,httptestfor HTTP, synthetic git repos built withexec.Command("git", ...)int.TempDir(). Loggers uselog/slog. YAML viasigs.k8s.io/yaml(direct dep, JSON tags). - Scope of this plan (this repo only):
internal/contract/contract.go— mirror the locked Phase 0 wire types (Edge confidence fields, Entity provenance fields,Hyperedge+GraphPush.Hyperedges, new entity-type + relation constants, confidence-tier helper). Updatecontract_shape_test.go.internal/walk/walk.go— replaceChangedwith a status-awareDiffreturningChanges{Files []Change, FullSet bool}; ls-files for full/empty-since;git diff --name-statusotherwise; WARN + ls-files fallback when diff errors;ContentSHA(sha256 of working-tree file) for A/M.internal/analyze/docs.go—DocsAnalyzeremits a doc entity per file (Type=doc_file, iddoc:file:<path>), routes doc files into the code-graph, captures YAML frontmatter (source_url/author/captured_at) onto the entity.internal/push+cmd/tatara-ingest/run.go— drive reconcile-per-file: A/M/renamed-new analyze+chunk; D/renamed-old go into code-graphFilesand memoriesreconcile_fileswith no entities/items;/memories:bulkcarriesreconcile_files;/code-graph:bulkFilesincludes deleted paths.- Out of scope (other repos / other plans): tatara-operator clone + cron, tatara-memory migrations +
DeleteMemoriesBySource+ bulk handler, charts. - Commit discipline: one commit per TDD task (test+impl together, conventional message). Branch off
mainin a worktree before starting (per CLAUDE.md rule 10).
Every task is RED → GREEN → COMMIT. Run commands from the repo root
/Users/szymonri/Documents/tatara/tatara-memory-repo-ingesterunless stated. Code blocks are complete; no placeholders.
Task 1: Contract — add Edge confidence fields + ConfidenceTier helper¶
Files - Modify: internal/contract/contract.go - Modify: internal/contract/contract_shape_test.go
Steps
- Write the failing test. Append to
internal/contract/contract_shape_test.go:
func TestEdgeConfidenceFields(t *testing.T) {
e := contract.Edge{
From: "a", To: "b", Relation: contract.RelCalls, SrcFile: "x.go",
ConfidenceScore: 0.98, ConfidenceTier: contract.TierInferred,
Properties: map[string]string{"resolution": contract.ResTypeResolved},
}
b, err := json.Marshal(e)
require.NoError(t, err)
var got map[string]any
require.NoError(t, json.Unmarshal(b, &got))
require.ElementsMatch(t,
[]string{"from", "to", "relation", "src_file", "confidence_score", "confidence_tier", "properties"},
keys(got))
require.InDelta(t, 0.98, got["confidence_score"], 1e-9)
require.Equal(t, "INFERRED", got["confidence_tier"])
}
func TestEdgeConfidenceOmitEmpty(t *testing.T) {
e := contract.Edge{From: "a", To: "b", Relation: contract.RelCalls, SrcFile: "x.go"}
b, err := json.Marshal(e)
require.NoError(t, err)
var got map[string]any
require.NoError(t, json.Unmarshal(b, &got))
_, hasScore := got["confidence_score"]
_, hasTier := got["confidence_tier"]
require.False(t, hasScore, "confidence_score must be absent when zero")
require.False(t, hasTier, "confidence_tier must be absent when empty")
}
func TestTierForScore(t *testing.T) {
cases := []struct {
name string
score float64
tier string
}{
{"extracted", 1.0, contract.TierExtracted},
{"inferred_high", 0.98, contract.TierInferred},
{"inferred_mid", 0.45, contract.TierInferred},
{"ambiguous_boundary", 0.3, contract.TierAmbiguous},
{"ambiguous_low", 0.0, contract.TierAmbiguous},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
require.Equal(t, tc.tier, contract.TierForScore(tc.score))
})
}
}
-
Run it (expect FAIL).
Expected FAIL:undefined: contract.TierInferred(compile error ininternal/contract/contract_shape_test.go). -
Minimal implementation. In
internal/contract/contract.go, replace theEdgestruct (lines 16-23) with:
// Edge is a directed, typed relationship between two entities.
type Edge struct {
From string `json:"from"`
To string `json:"to"`
Relation string `json:"relation"`
SrcFile string `json:"src_file"`
ConfidenceScore float64 `json:"confidence_score,omitempty"` // 1.0 EXTRACTED, <1 INFERRED
ConfidenceTier string `json:"confidence_tier,omitempty"` // EXTRACTED|INFERRED|AMBIGUOUS
Properties map[string]string `json:"properties,omitempty"`
}
Then add, immediately after the ConfidenceFor function (end of file):
// Confidence tier values (typed column on code_edges; promoted from the scalar score).
const (
TierExtracted = "EXTRACTED"
TierInferred = "INFERRED"
TierAmbiguous = "AMBIGUOUS"
)
// TierForScore maps a confidence score to a tier:
// 1.0 -> EXTRACTED; (0.3,1.0) -> INFERRED; <=0.3 -> AMBIGUOUS.
func TierForScore(score float64) string {
switch {
case score >= 1.0:
return TierExtracted
case score > 0.3:
return TierInferred
default:
return TierAmbiguous
}
}
-
Run to pass.
Expected PASS:ok github.com/szymonrychu/tatara-memory-repo-ingester/internal/contract. -
Commit.
Task 2: Contract — add Entity provenance fields + doc entity-type constants¶
Files - Modify: internal/contract/contract.go - Modify: internal/contract/contract_shape_test.go
Steps
- Write the failing test. Append to
internal/contract/contract_shape_test.go:
func TestEntityProvenanceFields(t *testing.T) {
e := contract.Entity{
ID: "doc:file:README.md", Name: "README.md", Type: contract.EntityDocFile,
FilePath: "README.md", LineStart: 1, LineEnd: 42,
SourceURL: "https://example.com/x", Author: "alice", CapturedAt: "2026-06-09T00:00:00Z",
}
b, err := json.Marshal(e)
require.NoError(t, err)
var got map[string]any
require.NoError(t, json.Unmarshal(b, &got))
require.ElementsMatch(t,
[]string{"id", "name", "type", "file_path", "line_start", "line_end", "source_url", "author", "captured_at"},
keys(got))
}
func TestEntityProvenanceOmitEmpty(t *testing.T) {
e := contract.Entity{ID: "go:package:m", Name: "m", Type: contract.EntityGoPackage, FilePath: ""}
b, err := json.Marshal(e)
require.NoError(t, err)
var got map[string]any
require.NoError(t, json.Unmarshal(b, &got))
for _, k := range []string{"line_start", "line_end", "source_url", "author", "captured_at"} {
_, has := got[k]
require.False(t, has, "%s must be absent when empty", k)
}
}
func TestDocEntityTypeConstants(t *testing.T) {
require.Equal(t, "doc_file", contract.EntityDocFile)
require.Equal(t, "doc_section", contract.EntityDocSection)
require.Equal(t, "concept", contract.EntityConcept)
require.Equal(t, "rationale", contract.EntityRationale)
}
-
Run it (expect FAIL).
Expected FAIL:undefined: contract.EntityDocFile(compile error in test). -
Minimal implementation. In
internal/contract/contract.go, replace theEntitystruct (lines 7-14) with:
// Entity is a node in the code graph.
type Entity struct {
ID string `json:"id"`
Name string `json:"name"`
Type string `json:"type"`
Description string `json:"description,omitempty"`
FilePath string `json:"file_path"`
LineStart int `json:"line_start,omitempty"` // Go already computes these
LineEnd int `json:"line_end,omitempty"` //
SourceURL string `json:"source_url,omitempty"` // doc frontmatter
Author string `json:"author,omitempty"` // doc frontmatter
CapturedAt string `json:"captured_at,omitempty"` // doc frontmatter, RFC3339
Properties map[string]string `json:"properties,omitempty"`
}
Then in the Entity types const block (currently lines 106-127), add the four new constants after EntityHelmValue:
EntityHelmValue = "helm_value"
EntityDocFile = "doc_file"
EntityDocSection = "doc_section"
EntityConcept = "concept"
EntityRationale = "rationale"
)
-
Run to pass.
Expected PASS:ok .../internal/contract. -
Commit.
Task 3: Contract — add Hyperedge type + GraphPush.Hyperedges + semantic relations¶
Files - Modify: internal/contract/contract.go - Modify: internal/contract/contract_shape_test.go
Steps
- Write the failing test. Append to
internal/contract/contract_shape_test.go:
func TestHyperedgeJSONShape(t *testing.T) {
h := contract.Hyperedge{
ID: "he:1", Label: "trait impl", Relation: "implement",
ConfidenceScore: 0.9, SrcFile: "x.go",
Members: []string{"go:type:m.A", "go:type:m.B", "go:type:m.C"},
Properties: map[string]string{"k": "v"},
}
b, err := json.Marshal(h)
require.NoError(t, err)
var got map[string]any
require.NoError(t, json.Unmarshal(b, &got))
require.ElementsMatch(t,
[]string{"id", "label", "relation", "confidence_score", "src_file", "members", "properties"},
keys(got))
require.Len(t, got["members"].([]any), 3)
}
func TestGraphPushHyperedgesOmitEmpty(t *testing.T) {
p := contract.GraphPush{Repo: "r", Files: []string{"a.go"}, Entities: []contract.Entity{}, Edges: []contract.Edge{}}
b, err := json.Marshal(p)
require.NoError(t, err)
var got map[string]any
require.NoError(t, json.Unmarshal(b, &got))
_, has := got["hyperedges"]
require.False(t, has, "hyperedges key must be absent when empty")
}
func TestGraphPushHyperedgesPresentWhenSet(t *testing.T) {
p := contract.GraphPush{
Repo: "r", Files: []string{"a.go"}, Entities: []contract.Entity{}, Edges: []contract.Edge{},
Hyperedges: []contract.Hyperedge{{ID: "he:1", Label: "l", Relation: "form", SrcFile: "a.go",
Members: []string{"x", "y", "z"}}},
}
b, err := json.Marshal(p)
require.NoError(t, err)
var got map[string]any
require.NoError(t, json.Unmarshal(b, &got))
require.Len(t, got["hyperedges"].([]any), 1)
}
func TestSemanticRelationConstants(t *testing.T) {
require.Equal(t, "conceptually_related_to", contract.RelConceptuallyRelated)
require.Equal(t, "semantically_similar_to", contract.RelSemanticallySimilar)
require.Equal(t, "rationale_for", contract.RelRationaleFor)
require.Equal(t, "shares_data_with", contract.RelSharesDataWith)
require.Equal(t, "cites", contract.RelCites)
}
-
Run it (expect FAIL).
Expected FAIL:go test ./internal/contract/ -run 'TestHyperedge|TestGraphPushHyperedges|TestSemanticRelation' -count=1undefined: contract.Hyperedge(compile error in test). -
Minimal implementation. In
internal/contract/contract.go, add theHyperedgetype immediately before theGraphPushdefinition (before line 41):
// Hyperedge is an n-ary relationship over 3+ entities (Phase 2 producer; reserved now).
type Hyperedge struct {
ID string `json:"id"`
Label string `json:"label"`
Relation string `json:"relation"` // participate_in|implement|form
ConfidenceScore float64 `json:"confidence_score,omitempty"`
SrcFile string `json:"src_file"`
Members []string `json:"members"` // entity IDs (3+)
Properties map[string]string `json:"properties,omitempty"`
}
Replace the GraphPush struct (lines 41-49) with:
// GraphPush is one /code-graph:bulk request.
type GraphPush struct {
Repo string `json:"repo"`
Commit string `json:"commit,omitempty"`
Files []string `json:"files"`
Entities []Entity `json:"entities"`
Edges []Edge `json:"edges"`
Symbols []SymbolRow `json:"symbols,omitempty"`
Hyperedges []Hyperedge `json:"hyperedges,omitempty"` // empty until Phase 2
}
In the Edge relations const block (currently ending at line 144 with RelSubchart), add the five semantic relations after RelSubchart:
RelSubchart = "subchart"
// Semantic relations (reserved Phase 0, emitted Phase 2).
RelConceptuallyRelated = "conceptually_related_to"
RelSemanticallySimilar = "semantically_similar_to"
RelRationaleFor = "rationale_for"
RelSharesDataWith = "shares_data_with"
RelCites = "cites"
)
-
Run to pass.
Expected PASS:go test ./internal/contract/ -run 'TestHyperedge|TestGraphPushHyperedges|TestSemanticRelation' -count=1ok .../internal/contract. -
Commit.
Task 4: Contract — BulkMemoriesRequest with reconcile_files¶
Files - Modify: internal/contract/contract.go - Modify: internal/contract/contract_shape_test.go
Steps
- Write the failing test. Append to
internal/contract/contract_shape_test.go:
func TestBulkMemoriesRequestShape(t *testing.T) {
req := contract.BulkMemoriesRequest{
ReconcileFiles: []string{"a.go", "b.md"},
Items: []contract.IngestItem{{IdempotencyKey: "k", Text: "t"}},
}
b, err := json.Marshal(req)
require.NoError(t, err)
var got map[string]any
require.NoError(t, json.Unmarshal(b, &got))
require.ElementsMatch(t, []string{"reconcile_files", "items"}, keys(got))
require.Len(t, got["reconcile_files"].([]any), 2)
}
func TestBulkMemoriesRequestReconcileOmitEmpty(t *testing.T) {
req := contract.BulkMemoriesRequest{Items: []contract.IngestItem{{IdempotencyKey: "k", Text: "t"}}}
b, err := json.Marshal(req)
require.NoError(t, err)
var got map[string]any
require.NoError(t, json.Unmarshal(b, &got))
_, has := got["reconcile_files"]
require.False(t, has, "reconcile_files must be absent when empty")
require.ElementsMatch(t, []string{"items"}, keys(got))
}
-
Run it (expect FAIL).
Expected FAIL:undefined: contract.BulkMemoriesRequest(compile error in test). -
Minimal implementation. In
internal/contract/contract.go, add immediately after theIngestItemstruct (after line 64):
// BulkMemoriesRequest is the /memories:bulk request body. ReconcileFiles, when
// set, instructs the server to purge prior memories for each file (by source)
// before inserting Items. Absent ReconcileFiles preserves insert-only behavior.
type BulkMemoriesRequest struct {
ReconcileFiles []string `json:"reconcile_files,omitempty"`
Items []IngestItem `json:"items"`
}
-
Run to pass.
Expected PASS: all contract tests green (ok .../internal/contract). -
Commit.
Task 5: Walk — status-aware Diff: full / since classification + Change/Changes types¶
Files - Modify: internal/walk/walk.go - Modify: internal/walk/walk_test.go
The existing
Changedis removed and replaced byDiff. The existing three tests (TestFullWalkListsTrackedFiles,TestSinceWalkListsOnlyChanged,TestFullFlagOverridesSince) are rewritten in this task to useDiff. All callers (cmd/tatara-ingest/run.go) are updated in Task 8; this task leavesrun.gotemporarily referencingwalk.Changed, so packagewalkbuilds and its own tests pass, whilecmd/...will not build until Task 8. Run only thewalkpackage tests in this task's run commands.
Steps
- Write the failing test. Replace the entire body of
internal/walk/walk_test.go(keep the helpersgitRepo,write,commit) with:
package walk_test
import (
"crypto/sha256"
"encoding/hex"
"os"
"os/exec"
"path/filepath"
"testing"
"github.com/stretchr/testify/require"
"github.com/szymonrychu/tatara-memory-repo-ingester/internal/walk"
)
func gitRepo(t *testing.T) string {
t.Helper()
dir := t.TempDir()
for _, args := range [][]string{
{"init", "-q"}, {"config", "user.email", "t@t"}, {"config", "user.name", "t"},
} {
cmd := exec.Command("git", args...)
cmd.Dir = dir
require.NoError(t, cmd.Run())
}
return dir
}
func write(t *testing.T, dir, rel, content string) {
t.Helper()
p := filepath.Join(dir, rel)
require.NoError(t, os.MkdirAll(filepath.Dir(p), 0o755))
require.NoError(t, os.WriteFile(p, []byte(content), 0o644))
}
func commit(t *testing.T, dir, msg string) string {
t.Helper()
for _, args := range [][]string{{"add", "-A"}, {"commit", "-q", "-m", msg}} {
cmd := exec.Command("git", args...)
cmd.Dir = dir
require.NoError(t, cmd.Run())
}
out, err := exec.Command("git", "-C", dir, "rev-parse", "HEAD").Output()
require.NoError(t, err)
return string(out[:len(out)-1])
}
func sha(content string) string {
sum := sha256.Sum256([]byte(content))
return hex.EncodeToString(sum[:])
}
// byPath indexes a Changes result by new path for assertions.
func byPath(c walk.Changes) map[string]walk.Change {
m := map[string]walk.Change{}
for _, ch := range c.Files {
m[ch.Path] = ch
}
return m
}
func TestFullWalkListsTrackedFiles(t *testing.T) {
dir := gitRepo(t)
write(t, dir, "a.go", "package a")
write(t, dir, "sub/b.py", "x = 1")
commit(t, dir, "init")
got, err := walk.Diff(dir, "", false)
require.NoError(t, err)
require.True(t, got.FullSet)
m := byPath(got)
require.Len(t, m, 2)
require.Equal(t, 'A', m["a.go"].Status)
require.Equal(t, 'A', m["sub/b.py"].Status)
require.Equal(t, sha("package a"), m["a.go"].ContentSHA)
}
func TestFullFlagOverridesSince(t *testing.T) {
dir := gitRepo(t)
write(t, dir, "a.go", "package a")
base := commit(t, dir, "init")
write(t, dir, "c.go", "package c")
commit(t, dir, "add c")
got, err := walk.Diff(dir, base, true)
require.NoError(t, err)
require.True(t, got.FullSet)
m := byPath(got)
require.Len(t, m, 2)
require.Contains(t, m, "a.go")
require.Contains(t, m, "c.go")
}
-
Run it (expect FAIL).
Expected FAIL:undefined: walk.Diff/undefined: walk.Changes(compile error). -
Minimal implementation. Replace the entire contents of
internal/walk/walk.gowith:
// Package walk lists the repository files to ingest.
package walk
import (
"crypto/sha256"
"encoding/hex"
"fmt"
"log/slog"
"os"
"os/exec"
"path/filepath"
"sort"
"strings"
)
// Change is one touched file classified by git status.
type Change struct {
Path string // new path (for renames, the destination)
OldPath string // populated only for renames
Status rune // 'A' added, 'M' modified, 'D' deleted, 'R' renamed
ContentSHA string // sha256 of working-tree content for A/M; empty for D
}
// Changes is the classified diff result.
type Changes struct {
Files []Change // every touched file
FullSet bool // true when produced by ls-files (first/full/fallback)
}
// Diff classifies the files to ingest. With full or empty since, it lists all
// tracked files as additions (FullSet). Otherwise it runs
// `git diff --name-status <since>..HEAD`; if that command fails (since not an
// ancestor: force-push, rebase, GC'd commit) it logs WARN and falls back to the
// full ls-files path so a job never hard-fails on history rewrite.
func Diff(repoRoot, since string, full bool) (Changes, error) {
if full || since == "" {
return fullSet(repoRoot)
}
out, err := exec.Command("git", "-C", repoRoot, "diff", "--name-status", since+"..HEAD").Output() //nolint:gosec
if err != nil {
slog.Warn("git diff failed; falling back to full ls-files",
"since", since, "error", err)
return fullSet(repoRoot)
}
return parseDiff(repoRoot, string(out))
}
// fullSet lists all tracked files as additions.
func fullSet(repoRoot string) (Changes, error) {
out, err := exec.Command("git", "-C", repoRoot, "ls-files").Output() //nolint:gosec
if err != nil {
return Changes{}, fmt.Errorf("git -C %s ls-files: %w", repoRoot, err)
}
var files []Change
for _, line := range strings.Split(strings.TrimSpace(string(out)), "\n") {
if line == "" {
continue
}
files = append(files, Change{Path: line, Status: 'A', ContentSHA: contentSHA(repoRoot, line)})
}
sortChanges(files)
return Changes{Files: files, FullSet: true}, nil
}
// parseDiff turns `git diff --name-status` output into classified changes.
// Rename lines `R<score>\told\tnew` become a single Change with Status 'R',
// OldPath=old, Path=new. Copy lines `C<score>` are treated as additions of new.
func parseDiff(repoRoot, out string) (Changes, error) {
var files []Change
for _, line := range strings.Split(strings.TrimSpace(out), "\n") {
if line == "" {
continue
}
fields := strings.Split(line, "\t")
code := fields[0]
switch code[0] {
case 'A', 'M':
if len(fields) < 2 {
continue
}
p := fields[1]
files = append(files, Change{Path: p, Status: rune(code[0]), ContentSHA: contentSHA(repoRoot, p)})
case 'D':
if len(fields) < 2 {
continue
}
files = append(files, Change{Path: fields[1], Status: 'D'})
case 'R':
if len(fields) < 3 {
continue
}
files = append(files, Change{
Path: fields[2], OldPath: fields[1], Status: 'R',
ContentSHA: contentSHA(repoRoot, fields[2]),
})
case 'C':
if len(fields) < 3 {
continue
}
p := fields[2]
files = append(files, Change{Path: p, Status: 'A', ContentSHA: contentSHA(repoRoot, p)})
}
}
sortChanges(files)
return Changes{Files: files, FullSet: false}, nil
}
// contentSHA returns the sha256 hex of the working-tree file; empty when unreadable.
func contentSHA(repoRoot, rel string) string {
b, err := os.ReadFile(filepath.Join(repoRoot, rel)) //nolint:gosec
if err != nil {
return ""
}
sum := sha256.Sum256(b)
return hex.EncodeToString(sum[:])
}
// sortChanges orders by new path for deterministic output.
func sortChanges(files []Change) {
sort.Slice(files, func(i, j int) bool { return files[i].Path < files[j].Path })
}
Note:
run.gostill referenceswalk.Changedand will not compile after this change. That is expected; Task 8 fixes it. Do not rungo build ./...orgo test ./...in this task — run only./internal/walk/.
-
Run to pass.
Expected PASS:ok .../internal/walk. -
Commit.
Task 6: Walk — A/M/D/R classification + rename pairing on a since diff¶
Files - Modify: internal/walk/walk_test.go
Steps
- Write the failing test. Append to
internal/walk/walk_test.go:
func TestSinceDiffClassifiesAddModifyDelete(t *testing.T) {
dir := gitRepo(t)
write(t, dir, "keep.go", "package a")
write(t, dir, "gone.go", "package g")
base := commit(t, dir, "init")
write(t, dir, "keep.go", "package a // changed")
write(t, dir, "new.go", "package n")
require.NoError(t, os.Remove(filepath.Join(dir, "gone.go")))
commit(t, dir, "mutate")
got, err := walk.Diff(dir, base, false)
require.NoError(t, err)
require.False(t, got.FullSet)
m := byPath(got)
require.Equal(t, 'M', m["keep.go"].Status)
require.Equal(t, sha("package a // changed"), m["keep.go"].ContentSHA)
require.Equal(t, 'A', m["new.go"].Status)
require.Equal(t, sha("package n"), m["new.go"].ContentSHA)
require.Equal(t, 'D', m["gone.go"].Status)
require.Empty(t, m["gone.go"].ContentSHA, "deleted file has no content sha")
}
func TestSinceDiffPairsRename(t *testing.T) {
dir := gitRepo(t)
write(t, dir, "old/name.go", "package x\n\nfunc Stable() {}\n")
base := commit(t, dir, "init")
require.NoError(t, os.MkdirAll(filepath.Join(dir, "new"), 0o755))
require.NoError(t, os.Rename(
filepath.Join(dir, "old/name.go"), filepath.Join(dir, "new/name.go")))
commit(t, dir, "rename")
got, err := walk.Diff(dir, base, false)
require.NoError(t, err)
require.False(t, got.FullSet)
require.Len(t, got.Files, 1)
ch := got.Files[0]
require.Equal(t, 'R', ch.Status)
require.Equal(t, "new/name.go", ch.Path)
require.Equal(t, "old/name.go", ch.OldPath)
require.Equal(t, sha("package x\n\nfunc Stable() {}\n"), ch.ContentSHA)
}
-
Run it (expect FAIL or PASS-after-impl). Implementation from Task 5 already supports A/M/D/R; this task asserts the behavior contractually.
Expected: PASS (rename detection is git's default for a pure move). If the rename test FAILS because git reportsD+Ainstead ofR(rare for tiny files), make rename detection explicit by changing the diff invocation inwalk.goDiffto add-M: Re-run; expect PASS. -
Minimal implementation. No new code beyond the optional
-Mflag above if the rename test required it. (If-Mwas added, includeinternal/walk/walk.goin the commit.) -
Run to pass.
Expected PASS:ok .../internal/walk. -
Commit.
Task 7: Walk — missing-since fallback to full ls-files with WARN¶
Files - Modify: internal/walk/walk_test.go
Steps
- Write the failing test. Append to
internal/walk/walk_test.go:
func TestMissingSinceFallsBackToFull(t *testing.T) {
dir := gitRepo(t)
write(t, dir, "a.go", "package a")
write(t, dir, "b.go", "package b")
commit(t, dir, "init")
// A since SHA that does not exist in this repo: diff must error and fall back.
got, err := walk.Diff(dir, "deadbeefdeadbeefdeadbeefdeadbeefdeadbeef", false)
require.NoError(t, err, "missing since must not hard-fail")
require.True(t, got.FullSet, "fallback uses ls-files full set")
m := byPath(got)
require.Len(t, m, 2)
require.Equal(t, 'A', m["a.go"].Status)
require.Equal(t, 'A', m["b.go"].Status)
}
-
Run it (expect PASS — guards the fallback).
Expected PASS (Task 5 implemented the WARN + fallback). If it FAILS with a non-nil error, the fallback branch is missing/incorrect — re-check theif err != nilbranch inDiffreturnsfullSet(repoRoot), then re-run; expect PASS. -
Minimal implementation. None expected; the fallback path already exists.
-
Run to pass.
Expected PASS:ok .../internal/walk. -
Commit.
Task 8: run.go — consume walk.Diff and split touched files into changed vs purged¶
Files - Modify: cmd/tatara-ingest/run.go - Modify: cmd/tatara-ingest/run_test.go
After Task 5 the
cmd/...package does not build (still callswalk.Changed). This task restores the build and wires the classified diff into analysis. The reconcile_files plumbing into/memories:bulklands in Task 9; the deleted-paths-into-code-graph behavior is asserted in Task 10.
Steps
- Write the failing test. Append to
cmd/tatara-ingest/run_test.go:
func TestRunSendsDeletedFilesInGraphAndReconcile(t *testing.T) {
dir := t.TempDir()
for _, a := range [][]string{{"init", "-q"}, {"config", "user.email", "t@t"}, {"config", "user.name", "t"}} {
c := exec.Command("git", a...)
c.Dir = dir
require.NoError(t, c.Run())
}
require.NoError(t, os.WriteFile(filepath.Join(dir, "keep.go"),
[]byte("package m\n\nfunc Keep() {}\n"), 0o644))
require.NoError(t, os.WriteFile(filepath.Join(dir, "gone.go"),
[]byte("package m\n\nfunc Gone() {}\n"), 0o644))
require.NoError(t, os.WriteFile(filepath.Join(dir, "go.mod"),
[]byte("module example.com/m\n\ngo 1.25\n"), 0o644))
base := commitAll(t, dir, "init")
require.NoError(t, os.Remove(filepath.Join(dir, "gone.go")))
require.NoError(t, os.WriteFile(filepath.Join(dir, "keep.go"),
[]byte("package m\n\nfunc Keep() { _ = 1 }\n"), 0o644))
commitAll(t, dir, "delete gone, modify keep")
var capturedPush contract.GraphPush
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch r.URL.Path {
case "/code-graph:bulk":
body, _ := io.ReadAll(r.Body)
_ = json.Unmarshal(body, &capturedPush)
w.WriteHeader(200)
_, _ = w.Write([]byte(`{"repo":"m"}`))
case "/memories:bulk":
w.WriteHeader(202)
_, _ = w.Write([]byte(`{"id":"j","status":"succeeded"}`))
default:
_, _ = w.Write([]byte(`{"id":"j","status":"succeeded"}`))
}
}))
defer srv.Close()
opts := options{repoRoot: dir, repoName: "m", baseURL: srv.URL, since: base}
require.NoError(t, run(context.Background(), opts, http.DefaultClient))
require.Contains(t, capturedPush.Files, "gone.go", "deleted file must be in code-graph Files")
require.Contains(t, capturedPush.Files, "keep.go", "modified file must be in code-graph Files")
for _, e := range capturedPush.Entities {
require.NotEqual(t, "gone.go", e.FilePath, "deleted file must contribute no entities")
}
}
// commitAll commits all changes and returns HEAD.
func commitAll(t *testing.T, dir, msg string) string {
t.Helper()
for _, a := range [][]string{{"add", "-A"}, {"commit", "-q", "-m", msg}} {
c := exec.Command("git", a...)
c.Dir = dir
require.NoError(t, c.Run())
}
out, err := exec.Command("git", "-C", dir, "rev-parse", "HEAD").Output()
require.NoError(t, err)
return strings.TrimSpace(string(out))
}
Add "strings" to the cmd/tatara-ingest/run_test.go import block if not already present.
-
Run it (expect FAIL).
Expected FAIL: compile errorcl.Changed/walk.Changedundefined (the package no longer builds against the newwalkAPI). -
Minimal implementation. In
cmd/tatara-ingest/run.go, replace therunfunction (lines 36-82) with:
func run(ctx context.Context, o options, hc *http.Client) error {
if o.scipPath != "" {
return runSCIP(ctx, o, hc)
}
start := time.Now()
changes, err := walk.Diff(o.repoRoot, o.since, o.full)
if err != nil {
return err
}
// touched is every file in the diff (code-graph Files + memories reconcile_files).
// analyzeFiles is only A/M/renamed-new (the files we re-analyze and chunk).
var touched, analyzeFiles []string
for _, ch := range changes.Files {
touched = append(touched, ch.Path)
switch ch.Status {
case 'A', 'M', 'R':
analyzeFiles = append(analyzeFiles, ch.Path)
}
}
reg := analyze.Default(o.crossRepoPrefix)
groups := reg.Group(analyzeFiles)
var agg analyze.Result
for _, a := range reg.Analyzers() {
fs := groups[a.Name()]
if len(fs) == 0 {
continue
}
res, err := a.Analyze(ctx, o.repoRoot, fs)
if err != nil {
slog.Warn("analyzer failed", "analyzer", a.Name(), "error", err)
continue
}
agg.Entities = append(agg.Entities, res.Entities...)
agg.Edges = append(agg.Edges, res.Edges...)
agg.Chunks = append(agg.Chunks, res.Chunks...)
agg.Symbols = append(agg.Symbols, res.Symbols...)
}
commit := headCommit(o.repoRoot)
cl := push.New(o.baseURL, hc, pollOr(o.pollInterval))
if _, err := cl.PushGraph(ctx, contract.GraphPush{
Repo: o.repoName, Commit: commit, Files: touched,
Entities: agg.Entities, Edges: agg.Edges, Symbols: agg.Symbols,
Hyperedges: agg.Hyperedges,
}); err != nil {
return err
}
reconcile := touched
if changes.FullSet {
reconcile = nil // first/full ingest is insert-only (no reconcile)
}
if err := cl.PushChunks(ctx, reconcile, push.ItemsFromChunks(o.repoName, agg.Chunks)); err != nil {
return err
}
slog.Info("ingest complete",
"repo", o.repoName, "files", len(touched),
"analyzed", len(analyzeFiles),
"entities", len(agg.Entities), "edges", len(agg.Edges),
"chunks", len(agg.Chunks), "symbols", len(agg.Symbols),
"full", changes.FullSet,
"duration_ms", time.Since(start).Milliseconds())
return nil
}
This references agg.Hyperedges (added in Task 12) and the new PushChunks(ctx, reconcile, items) signature (added in Task 9). Both are introduced in their own tasks; this task changes PushChunks call shape too — so do them together is NOT required because the test in this task only needs the build to pass via the Task 9 signature. To keep this task self-contained and GREEN, also apply the Task 9 PushChunks signature change and the Task 12 analyze.Result.Hyperedges field now, OR sequence: do Task 9 and Task 12 first. Recommended sequencing: execute Task 9 and Task 12 before this task. If sequencing Task 9/12 first, the snippet above compiles directly.
To avoid a forward reference, the executing agent runs Task 9 then Task 12 then this Task 8. (The plan keeps Task numbering for traceability; honor the dependency note.)
-
Run to pass.
Expected PASS:ok .../cmd/tatara-ingest. -
Commit.
Task 9: push — PushChunks sends BulkMemoriesRequest with reconcile_files¶
Files - Modify: internal/push/push.go - Modify: internal/push/push_test.go
Execute this task BEFORE Task 8 (Task 8's
run.gocalls the newPushChunkssignature).
Steps
- Write the failing test. Append to
internal/push/push_test.go:
func TestPushChunksSendsReconcileFiles(t *testing.T) {
var gotReq contract.BulkMemoriesRequest
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch {
case r.URL.Path == "/memories:bulk":
require.NoError(t, json.NewDecoder(r.Body).Decode(&gotReq))
w.WriteHeader(202)
_ = json.NewEncoder(w).Encode(contract.IngestJob{ID: "j1", Status: contract.JobSucceeded, Total: 1, Done: 1})
case strings.HasPrefix(r.URL.Path, "/ingest-jobs/"):
_ = json.NewEncoder(w).Encode(contract.IngestJob{ID: "j1", Status: contract.JobSucceeded, Total: 1, Done: 1})
}
}))
defer srv.Close()
c := push.New(srv.URL, http.DefaultClient, time.Millisecond)
err := c.PushChunks(context.Background(),
[]string{"a.go", "gone.go"},
[]contract.IngestItem{{IdempotencyKey: "k", Text: "t"}})
require.NoError(t, err)
require.ElementsMatch(t, []string{"a.go", "gone.go"}, gotReq.ReconcileFiles)
require.Len(t, gotReq.Items, 1)
}
func TestPushChunksReconcileOnlyDeletion(t *testing.T) {
// A pure deletion: reconcile_files set, no items. Must still POST and reconcile.
var gotReq contract.BulkMemoriesRequest
var posted bool
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch {
case r.URL.Path == "/memories:bulk":
posted = true
require.NoError(t, json.NewDecoder(r.Body).Decode(&gotReq))
w.WriteHeader(202)
_ = json.NewEncoder(w).Encode(contract.IngestJob{ID: "j1", Status: contract.JobSucceeded})
case strings.HasPrefix(r.URL.Path, "/ingest-jobs/"):
_ = json.NewEncoder(w).Encode(contract.IngestJob{ID: "j1", Status: contract.JobSucceeded})
}
}))
defer srv.Close()
c := push.New(srv.URL, http.DefaultClient, time.Millisecond)
err := c.PushChunks(context.Background(), []string{"gone.go"}, nil)
require.NoError(t, err)
require.True(t, posted, "deletion-only reconcile must still POST /memories:bulk")
require.Equal(t, []string{"gone.go"}, gotReq.ReconcileFiles)
require.Empty(t, gotReq.Items)
}
func TestPushChunksNoopWhenNothingToDo(t *testing.T) {
var posted bool
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
posted = true
w.WriteHeader(202)
_ = json.NewEncoder(w).Encode(contract.IngestJob{ID: "j1", Status: contract.JobSucceeded})
}))
defer srv.Close()
c := push.New(srv.URL, http.DefaultClient, time.Millisecond)
require.NoError(t, c.PushChunks(context.Background(), nil, nil))
require.False(t, posted, "no reconcile and no items must not POST")
}
Update the existing TestPushChunksPollsToTerminal and TestPushChunksPartialIsError calls (lines 57 and 73 of push_test.go) to the new signature by inserting a nil reconcile argument: - c.PushChunks(context.Background(), nil, []contract.IngestItem{{IdempotencyKey: "k", Text: "t"}})
And in TestPushChunksPollsToTerminal's handler, replace the bare struct{ Items ... } decode (lines 39-42) with:
var req contract.BulkMemoriesRequest
require.NoError(t, json.NewDecoder(r.Body).Decode(&req))
require.Len(t, req.Items, 1)
-
Run it (expect FAIL).
Expected FAIL:too many arguments in call to c.PushChunks(signature mismatch, compile error). -
Minimal implementation. In
internal/push/push.go, replace thePushChunksmethod (lines 37-63) with:
// PushChunks posts a reconcile-aware bulk and polls the resulting job to a
// terminal state. reconcileFiles, when non-empty, instructs the server to purge
// prior memories for each file before inserting items. When both reconcileFiles
// and items are empty there is nothing to do.
func (c *Client) PushChunks(ctx context.Context, reconcileFiles []string, items []contract.IngestItem) error {
if len(items) == 0 && len(reconcileFiles) == 0 {
return nil
}
var job contract.IngestJob
body := contract.BulkMemoriesRequest{ReconcileFiles: reconcileFiles, Items: items}
if err := c.do(ctx, http.MethodPost, "/memories:bulk", body, http.StatusAccepted, &job); err != nil {
return err
}
for !job.Terminal() {
select {
case <-ctx.Done():
return ctx.Err()
case <-time.After(c.pollInterval):
}
if err := c.do(ctx, http.MethodGet, "/ingest-jobs/"+job.ID, nil, http.StatusOK, &job); err != nil {
return err
}
}
if job.Status != contract.JobSucceeded {
return fmt.Errorf("ingest job %s ended %s (failed=%d)", job.ID, job.Status, job.Failed)
}
return nil
}
-
Run to pass.
Expected PASS:ok .../internal/push. -
Commit.
Task 10: run.go — reconcile_files carries the full touched set on a since diff¶
Files - Modify: cmd/tatara-ingest/run_test.go
Asserts the reconcile_files wiring end-to-end through
run(depends on Tasks 8 + 9 done).
Steps
- Write the failing test. Append to
cmd/tatara-ingest/run_test.go:
func TestRunReconcileFilesMatchTouchedSet(t *testing.T) {
dir := t.TempDir()
for _, a := range [][]string{{"init", "-q"}, {"config", "user.email", "t@t"}, {"config", "user.name", "t"}} {
c := exec.Command("git", a...)
c.Dir = dir
require.NoError(t, c.Run())
}
require.NoError(t, os.WriteFile(filepath.Join(dir, "doc.md"), []byte("# Doc\n\nbody\n"), 0o644))
require.NoError(t, os.WriteFile(filepath.Join(dir, "old.md"), []byte("# Old\n\ngone\n"), 0o644))
base := commitAll(t, dir, "init")
require.NoError(t, os.Remove(filepath.Join(dir, "old.md")))
require.NoError(t, os.WriteFile(filepath.Join(dir, "doc.md"), []byte("# Doc\n\nbody2\n"), 0o644))
commitAll(t, dir, "delete old, modify doc")
var bulkReq contract.BulkMemoriesRequest
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch r.URL.Path {
case "/code-graph:bulk":
w.WriteHeader(200)
_, _ = w.Write([]byte(`{"repo":"m"}`))
case "/memories:bulk":
body, _ := io.ReadAll(r.Body)
_ = json.Unmarshal(body, &bulkReq)
w.WriteHeader(202)
_, _ = w.Write([]byte(`{"id":"j","status":"succeeded"}`))
default:
_, _ = w.Write([]byte(`{"id":"j","status":"succeeded"}`))
}
}))
defer srv.Close()
opts := options{repoRoot: dir, repoName: "m", baseURL: srv.URL, since: base}
require.NoError(t, run(context.Background(), opts, http.DefaultClient))
require.ElementsMatch(t, []string{"doc.md", "old.md"}, bulkReq.ReconcileFiles)
}
func TestRunFullIngestHasNoReconcileFiles(t *testing.T) {
dir := t.TempDir()
for _, a := range [][]string{{"init", "-q"}, {"config", "user.email", "t@t"}, {"config", "user.name", "t"}} {
c := exec.Command("git", a...)
c.Dir = dir
require.NoError(t, c.Run())
}
require.NoError(t, os.WriteFile(filepath.Join(dir, "doc.md"), []byte("# Doc\n\nbody\n"), 0o644))
commitAll(t, dir, "init")
var bulkReq contract.BulkMemoriesRequest
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch r.URL.Path {
case "/code-graph:bulk":
w.WriteHeader(200)
_, _ = w.Write([]byte(`{"repo":"m"}`))
case "/memories:bulk":
body, _ := io.ReadAll(r.Body)
_ = json.Unmarshal(body, &bulkReq)
w.WriteHeader(202)
_, _ = w.Write([]byte(`{"id":"j","status":"succeeded"}`))
default:
_, _ = w.Write([]byte(`{"id":"j","status":"succeeded"}`))
}
}))
defer srv.Close()
opts := options{repoRoot: dir, repoName: "m", baseURL: srv.URL} // full (no since)
require.NoError(t, run(context.Background(), opts, http.DefaultClient))
require.Empty(t, bulkReq.ReconcileFiles, "full/first ingest is insert-only")
}
-
Run it (expect FAIL or PASS). With Tasks 8 + 9 in place this should PASS. If
Expected PASS.TestRunReconcileFilesMatchTouchedSetFAILS becausereconcileis empty, confirmrun.gosetsreconcile = touchedfor non-full diffs. -
Minimal implementation. None expected (behavior added in Task 8). If a fix was needed in
run.go, include it in the commit. -
Run to pass.
Expected PASS:ok .../cmd/tatara-ingest. -
Commit.
Task 11: docs.go — emit a doc entity per file and route docs into the code-graph¶
Files - Modify: internal/analyze/docs.go - Modify: internal/analyze/docs_test.go
Steps
- Write the failing test. Replace the body of
TestDocsAnalyzerininternal/analyze/docs_test.goand add a new test:
func TestDocsAnalyzer(t *testing.T) {
a := analyze.NewDocs()
require.True(t, a.Match("README.md"))
require.True(t, a.Match("docs/guide.txt"))
require.False(t, a.Match("main.go"))
res, err := a.Analyze(context.Background(), "testdata/docs", []string{"README.md"})
require.NoError(t, err)
require.Len(t, res.Chunks, 1)
require.Equal(t, "markdown", res.Chunks[0].Language)
require.Contains(t, res.Chunks[0].Body, "Some prose.")
require.Len(t, res.Entities, 1, "doc files now emit a doc entity")
e := res.Entities[0]
require.Equal(t, contract.EntityDocFile, e.Type)
require.Equal(t, "doc:file:README.md", e.ID)
require.Equal(t, "README.md", e.FilePath)
require.Equal(t, "README.md", e.Name)
require.Empty(t, res.Edges)
}
Add the contract import to internal/analyze/docs_test.go:
-
Run it (expect FAIL).
Expected FAIL:Not equal: expected 1 actual 0forlen(res.Entities)(docs currently emit no entities). -
Minimal implementation. Replace the
Analyzemethod ininternal/analyze/docs.go(lines 27-48) with:
func (docsAnalyzer) Analyze(_ context.Context, repoRoot string, files []string) (Result, error) {
var res Result
for _, f := range files {
b, err := os.ReadFile(filepath.Join(repoRoot, f)) //nolint:gosec
if err != nil {
continue // unreadable doc: skip, do not fail the run
}
lang := "markdown"
if strings.ToLower(filepath.Ext(f)) == ".txt" {
lang = "text"
}
res.Entities = append(res.Entities, contract.Entity{
ID: "doc:file:" + f,
Name: f,
Type: contract.EntityDocFile,
FilePath: f,
})
res.Chunks = append(res.Chunks, contract.Chunk{
EntityID: "doc:file:" + f,
Type: contract.EntityDocFile,
FilePath: f,
Language: lang,
Header: "[doc] " + f,
Body: string(b),
})
}
return res, nil
}
The chunk
EntityIDnow matches the entity ID (doc:file:<path>) so semantic chunks reference the graph node, andTypebecomesEntityDocFilesocode_searchmatches docs by type.
-
Run to pass.
Expected PASS:ok .../internal/analyze. -
Commit.
Task 12: analyze.Result — carry Hyperedges so run.go can forward them¶
Files - Modify: internal/analyze/analyzer.go - Modify: internal/analyze/analyzer_test.go
Execute this task BEFORE Task 8 (run.go forwards
agg.Hyperedges).
Steps
- Write the failing test. Append to
internal/analyze/analyzer_test.go(create the test file's package/imports if reusing the existinganalyze_testpackage — check the file first; it ispackage analyze_test):
func TestResultCarriesHyperedges(t *testing.T) {
var r analyze.Result
r.Hyperedges = append(r.Hyperedges, contract.Hyperedge{
ID: "he:1", Label: "l", Relation: "form", SrcFile: "a.go",
Members: []string{"x", "y", "z"},
})
require.Len(t, r.Hyperedges, 1)
require.Equal(t, "he:1", r.Hyperedges[0].ID)
}
Ensure internal/analyze/analyzer_test.go imports both:
"github.com/szymonrychu/tatara-memory-repo-ingester/internal/analyze"
"github.com/szymonrychu/tatara-memory-repo-ingester/internal/contract"
"github.com/stretchr/testify/require"
-
Run it (expect FAIL).
Expected FAIL:r.Hyperedges undefined (type analyze.Result has no field or method Hyperedges). -
Minimal implementation. In
internal/analyze/analyzer.go, replace theResultstruct (lines 12-18) with:
// Result is what an Analyzer emits for its assigned file set.
type Result struct {
Entities []contract.Entity
Edges []contract.Edge
Chunks []contract.Chunk
Symbols []contract.SymbolRow
Hyperedges []contract.Hyperedge
}
-
Run to pass.
Expected PASS:ok .../internal/analyze. -
Commit.
Task 13: docs.go — capture YAML frontmatter (source_url/author/captured_at) onto the doc entity¶
Files - Modify: internal/analyze/docs.go - Modify: internal/analyze/docs_test.go - Create: internal/analyze/testdata/docs/front.md
Frontmatter is the leading
---\n...\n---\nYAML block of a Markdown file. Parsed withsigs.k8s.io/yaml(already a direct dependency; JSON struct tags). Only available at analyze time, so it is captured here.
Steps
- Write the failing test. First create the fixture
internal/analyze/testdata/docs/front.mdwith exactly:
---
source_url: https://example.com/origin
author: Alice Example
captured_at: 2026-06-09T12:00:00Z
---
# Captured Doc
This document came from elsewhere.
Append to internal/analyze/docs_test.go:
func TestDocsAnalyzerCapturesFrontmatter(t *testing.T) {
a := analyze.NewDocs()
res, err := a.Analyze(context.Background(), "testdata/docs", []string{"front.md"})
require.NoError(t, err)
require.Len(t, res.Entities, 1)
e := res.Entities[0]
require.Equal(t, "https://example.com/origin", e.SourceURL)
require.Equal(t, "Alice Example", e.Author)
require.Equal(t, "2026-06-09T12:00:00Z", e.CapturedAt)
require.Len(t, res.Chunks, 1)
require.Contains(t, res.Chunks[0].Body, "This document came from elsewhere.")
require.NotContains(t, res.Chunks[0].Body, "source_url",
"frontmatter block must be stripped from the chunk body")
}
func TestDocsAnalyzerNoFrontmatter(t *testing.T) {
a := analyze.NewDocs()
res, err := a.Analyze(context.Background(), "testdata/docs", []string{"README.md"})
require.NoError(t, err)
require.Len(t, res.Entities, 1)
e := res.Entities[0]
require.Empty(t, e.SourceURL)
require.Empty(t, e.Author)
require.Empty(t, e.CapturedAt)
require.Contains(t, res.Chunks[0].Body, "Some prose.")
}
-
Run it (expect FAIL).
Expected FAIL:go test ./internal/analyze/ -run 'TestDocsAnalyzerCapturesFrontmatter|TestDocsAnalyzerNoFrontmatter' -count=1Not equal: expected "https://example.com/origin" actual ""(frontmatter not parsed yet). -
Minimal implementation. Replace the entire
internal/analyze/docs.gowith:
package analyze
import (
"context"
"os"
"path/filepath"
"strings"
sigsyaml "sigs.k8s.io/yaml"
"github.com/szymonrychu/tatara-memory-repo-ingester/internal/contract"
)
type docsAnalyzer struct{}
// NewDocs returns the docs analyzer. It emits one doc_file entity per file
// (so docs participate in the code graph) plus a semantic chunk, and captures
// YAML frontmatter (source_url/author/captured_at) onto the entity.
func NewDocs() Analyzer { return docsAnalyzer{} }
func (docsAnalyzer) Name() string { return "docs" }
func (docsAnalyzer) Match(path string) bool {
switch strings.ToLower(filepath.Ext(path)) {
case ".md", ".markdown", ".txt", ".rst":
return true
}
return false
}
func (docsAnalyzer) Analyze(_ context.Context, repoRoot string, files []string) (Result, error) {
var res Result
for _, f := range files {
b, err := os.ReadFile(filepath.Join(repoRoot, f)) //nolint:gosec
if err != nil {
continue // unreadable doc: skip, do not fail the run
}
lang := "markdown"
if strings.ToLower(filepath.Ext(f)) == ".txt" {
lang = "text"
}
fm, body := splitFrontmatter(string(b))
ent := contract.Entity{
ID: "doc:file:" + f,
Name: f,
Type: contract.EntityDocFile,
FilePath: f,
SourceURL: fm.SourceURL,
Author: fm.Author,
CapturedAt: fm.CapturedAt,
}
res.Entities = append(res.Entities, ent)
res.Chunks = append(res.Chunks, contract.Chunk{
EntityID: ent.ID,
Type: contract.EntityDocFile,
FilePath: f,
Language: lang,
Header: "[doc] " + f,
Body: body,
})
}
return res, nil
}
// docFrontmatter is the subset of YAML frontmatter promoted to provenance columns.
type docFrontmatter struct {
SourceURL string `json:"source_url"`
Author string `json:"author"`
CapturedAt string `json:"captured_at"`
}
// splitFrontmatter extracts a leading `---\n...\n---\n` YAML block and returns
// the parsed provenance plus the remaining body. With no frontmatter, it returns
// a zero docFrontmatter and the original content unchanged. A malformed block is
// ignored (zero provenance) and the original content is returned.
func splitFrontmatter(content string) (docFrontmatter, string) {
if !strings.HasPrefix(content, "---\n") {
return docFrontmatter{}, content
}
rest := content[len("---\n"):]
end := strings.Index(rest, "\n---\n")
if end < 0 {
return docFrontmatter{}, content
}
yamlBlock := rest[:end]
body := rest[end+len("\n---\n"):]
var fm docFrontmatter
if err := sigsyaml.Unmarshal([]byte(yamlBlock), &fm); err != nil {
return docFrontmatter{}, content
}
return fm, body
}
-
Run to pass.
Expected PASS: allTestDocsAnalyzer*green (ok .../internal/analyze). -
Commit.
Task 14: golang.go — promote line_start/line_end into typed Entity columns¶
Files - Modify: internal/analyze/golang.go - Modify: internal/analyze/golang_test.go
Phase 0 item 3: Go already computes line numbers into
Properties; promote them into the typedLineStart/LineEndEntity columns (kept inPropertiestoo, for back-compat with anything reading the string form).
Steps
- Write the failing test. Append to
internal/analyze/golang_test.go(it ispackage analyze_test; confirm imports includecontext,testify/require,analyze,contract):
func TestGoEntityHasTypedLineColumns(t *testing.T) {
a := analyze.NewGo("github.com/szymonrychu/")
res, err := a.Analyze(context.Background(), "testdata/go", []string{"pkg/pkg.go"})
require.NoError(t, err)
var fn *contract.Entity
for i := range res.Entities {
if res.Entities[i].Type == contract.EntityGoFunc || res.Entities[i].Type == contract.EntityGoMethod {
fn = &res.Entities[i]
break
}
}
require.NotNil(t, fn, "expected at least one func/method entity")
require.Greater(t, fn.LineStart, 0, "line_start promoted to typed column")
require.GreaterOrEqual(t, fn.LineEnd, fn.LineStart, "line_end promoted to typed column")
}
-
Run it (expect FAIL).
Expected FAIL:line_start promoted to typed column(fn.LineStartis 0). -
Minimal implementation. In
internal/analyze/golang.goprocessPackage, replace the entity append (lines 196-209) with:
lineStart := pkg.Fset.Position(fd.Pos()).Line
lineEnd := pkg.Fset.Position(fd.End()).Line
props := map[string]string{
"line_start": fmt.Sprintf("%d", lineStart),
"line_end": fmt.Sprintf("%d", lineEnd),
"signature": sig,
"exported": fmt.Sprintf("%v", ast.IsExported(fd.Name.Name)),
}
res.Entities = append(res.Entities, contract.Entity{
ID: entityID,
Name: name,
Type: entityType,
FilePath: rel,
LineStart: lineStart,
LineEnd: lineEnd,
Properties: props,
})
-
Run to pass.
Expected PASS:ok .../internal/analyze. -
Commit.
Task 15: golang.go — promote call-edge confidence into typed Edge columns¶
Files - Modify: internal/analyze/golang.go - Modify: internal/analyze/golang_test.go
Phase 0 item 2: the Go analyzer already stamps
properties["confidence"]from the resolution level. Promote that scalar intoEdge.ConfidenceScoreand deriveEdge.ConfidenceTierviacontract.TierForScore.
Steps
- Write the failing test. Append to
internal/analyze/golang_test.go:
func TestGoCallEdgeHasTypedConfidence(t *testing.T) {
a := analyze.NewGo("github.com/szymonrychu/")
res, err := a.Analyze(context.Background(), "testdata/go", []string{"pkg/pkg.go", "pkg/other.go"})
require.NoError(t, err)
var call *contract.Edge
for i := range res.Edges {
if res.Edges[i].Relation == contract.RelCalls {
call = &res.Edges[i]
break
}
}
require.NotNil(t, call, "expected at least one calls edge in the fixture")
require.InDelta(t, 0.98, call.ConfidenceScore, 1e-9, "type_resolved -> 0.98 promoted to column")
require.Equal(t, contract.TierInferred, call.ConfidenceTier, "0.98 maps to INFERRED")
}
-
Run it (expect FAIL).
Expected FAIL:type_resolved -> 0.98 promoted to column(call.ConfidenceScoreis 0). (If the fixture yields nocallsedge with the given file subset, broaden the file list to the package's files; thetestdata/go/pkgpackage contains an intra-package call.) -
Minimal implementation. In
internal/analyze/golang.goemitCallEdges, replace the edge append (lines 461-470) with:
score := scoreFor(contract.ResTypeResolved)
res.Edges = append(res.Edges, contract.Edge{
From: callerID,
To: calleeID,
Relation: contract.RelCalls,
SrcFile: callerRelFile,
ConfidenceScore: score,
ConfidenceTier: contract.TierForScore(score),
Properties: map[string]string{
"resolution": contract.ResTypeResolved,
"confidence": contract.ConfidenceFor(contract.ResTypeResolved),
},
})
Add this helper at the end of internal/analyze/golang.go:
// scoreFor parses the confidence prior string for a resolution level into a float.
func scoreFor(resolution string) float64 {
switch resolution {
case contract.ResTypeResolved:
return 0.98
case contract.ResScopedNameMatch:
return 0.85
case contract.ResImportedNameMatch:
return 0.7
case contract.ResGlobalNameMatch:
return 0.45
case contract.ResAmbiguousMultiDef:
return 0.2
default:
return 0.0
}
}
-
Run to pass.
Expected PASS:ok .../internal/analyze. -
Commit.
Task 16: Full regression sweep, vet, and contract-shape lock confirmation¶
Files - None (verification only). If go vet or a regression surfaces a defect, fix the minimal file and include it in the commit.
Steps
-
Run the whole suite.
Expected PASS: every packageok, includinginternal/contract,internal/walk,internal/analyze,internal/push,cmd/tatara-ingest. Ifcmd/tatara-ingest/e2e_test.goormain_test.goreference the oldwalk.Changedor two-argPushChunks, update those call sites to the new APIs (walk.Diff,PushChunks(ctx, reconcileFiles, items)) — this is the boy-scout fix. -
Vet + tidy check (read-only verify).
Expected:go vetclean;go.mod/go.sumunchanged (sigs.k8s.io/yamlwas already a direct dependency, so no new requires). Ifgit diff --exit-codereports changes, stage them. -
Confirm the contract shape lock matches the spec byte-for-byte. Re-read
internal/contract/contract.goagainst2026-06-09-phase0-contract-lock.md: Edge:confidence_score,confidence_tierboth,omitempty. ✓Entity:line_start,line_end,source_url,author,captured_atall,omitempty;file_pathNOT omitempty. ✓Hyperedge:members(no omitempty),relationnon-omitempty;GraphPush.Hyperedges,omitempty. ✓-
Expected PASS for every named test.BulkMemoriesRequest:reconcile_files,omitempty,itemsnon-omitempty. ✓ Run the shape tests alone to prove the lock: -
Commit any boy-scout fixes.
(If step 1-3 needed no edits, skip the commit.) -
Final state confirmation.
Expected: 14-16 commits on the feature branch, full suite green.
Execution ordering note (dependencies)¶
Run tasks in this order to avoid forward references:
- Task 1 → 2 → 3 → 4 (contract types; no cross-package deps)
- Task 9 (push
PushChunksnew signature) - Task 12 (
analyze.Result.Hyperedges) - Task 8 (run.go wiring — needs Tasks 5+9+12)
- Task 5 (walk
Diff) must precede Task 8's build; run Task 5 → 6 → 7 first, then 9 → 12 → 8 → 10. - Task 11 → 13 (docs)
- Task 14 → 15 (golang typed columns)
- Task 16 (sweep)
Concretely: 1, 2, 3, 4, 5, 6, 7, 9, 12, 8, 10, 11, 13, 14, 15, 16.
Critical Files for Implementation¶
- /Users/szymonri/Documents/tatara/tatara-memory-repo-ingester/internal/contract/contract.go
- /Users/szymonri/Documents/tatara/tatara-memory-repo-ingester/internal/walk/walk.go
- /Users/szymonri/Documents/tatara/tatara-memory-repo-ingester/cmd/tatara-ingest/run.go
- /Users/szymonri/Documents/tatara/tatara-memory-repo-ingester/internal/push/push.go
- /Users/szymonri/Documents/tatara/tatara-memory-repo-ingester/internal/analyze/docs.go