engine: rename pageindex→treewalk + auto strategy default (HAL-106) (#35)

* engine: rename pageindex strategy to treewalk (HAL-106)

Strip the borrowed PageIndex name — treewalk is now the native identifier
for Vectorless's tree-navigation strategy across the strategy id, Go types,
config keys, the /v1/answer/treewalk endpoint, and all tests. Pure rename;
go build, go vet, and package tests all green.

Remaining HAL-106 sub-tasks: promote treewalk to default strategy, dedup
the agentic overlap, and fold the dedicated endpoint into /v1/answer.

* engine: add auto strategy as the default (HAL-106)

Smart default selection: route small documents (total leaf content within
the call budget) to single-pass for one fast cheap call, and larger or more
complex documents to treewalk for agentic navigation. New AutoStrategy in
pkg/retrieval with unit tests; wired into config (default value + validation)
and both binaries' strategy builders + the selectable-strategy set.

go build, go vet, and package tests all green.

* engine: address PR #35 review (auto page-fallback, nil-safety, finish rename)

Addresses the Copilot + CodeRabbit review on #35:
- auto: non-paged docs (no PageStart/PageEnd — Markdown/HTML/DOCX/TXT) now route
  to single-pass, never treewalk (which navigates by page range) + nil-safe
  pick/Select/SelectWithCost so a misconfigured Auto can't panic.
- complete the rename in non-Go files: openapi.yaml + config.example*.yaml
  (incl. the uppercase env-var refs) — zero pageindex remains anywhere.
- query.go: list 'auto' in the strategy-override doc comment.
- config_test: exercise 'auto' as a valid Validate() strategy.
- tests: non-paged fallback + nil-safety.

Out of scope (filed separately): treewalk SSE streams reasoning regardless of
IncludeReasoning (pre-existing behaviour, not introduced here).

* engine: green the CI baseline (HAL-116) — race fix, dep + toolchain bump, lint

- toc_builder: fix the data race in verifyTitlesConcurrent — each goroutine
  now accumulates a local Usage and folds it under the mutex, instead of
  sharing &localUse across errgroup goroutines (concurrent usage.add writes).
- deps: golang.org/x/net v0.53.0 -> v0.55.0 (GO-2026-5030); add toolchain
  go1.25.11 + pin CI go-version (stdlib GO-2026-5039 / GO-2026-5037).
- lint: //lint:ignore U1000 the four staged helpers with reasons; drop the
  genuinely-unused idempotencyCache.mu field.

These CI failures were pre-existing (surfaced on PR #35, not caused by the
rename). go build + staticcheck green locally; -race verified in CI.

Closes HAL-116.

* engine: fix the pdftable parser data race + golangci //nolint (HAL-116)

- pkg/parser: serialize pdftable.OpenBytes behind a package mutex (openPDFBytes).
  pdftable mutates package-level state on open and is not concurrency-safe; the
  -race detector flagged concurrent opens (real in prod — ingest workers parse
  in parallel). Stopgap; the proper fix belongs in pdftable (Foundational Libs).
- convert the unused-helper suppressions from staticcheck //lint:ignore to
  golangci-lint //nolint:unused,staticcheck (CI lint uses golangci-lint).

go build + go vet + parser tests green.

* engine: green the CI lint baseline — errcheck / ineffassign / staticcheck (HAL-117)

Completes the errcheck/ineffassign cleanup across the repo, including the
renamed files (treewalk.go / answer_treewalk.go) the off-main pass couldn't reach:
- errcheck: _, _ = fmt.Fprintf on the SSE writers; unchecked returns handled
  across cmd/benchmark, handlers, connecthandler, parser, storage, queue, db, ingest.
- ineffassign: drop the ineffectual 'written += remaining' before break in
  treewalk_strategy.
- staticcheck: real TLS-disabled assertion (was an empty branch) in config_test;
  De Morgan's rewrite in retrieval_test; drop embedded-field selector in decompose_test.
- misspell: ignore-rule for 'strat' (our 'strategy' abbreviation) in .golangci.yml.
- formatting normalized via the golangci formatters.

golangci-lint run: 0 issues. go build + unit tests green. Closes HAL-117.

* engine: remove dead funcs + guard pdftable ExtractTables race (HAL-116/117)

- lint: the CI 'lint' job runs BOTH golangci-lint and standalone staticcheck
  (honnef), and the latter reads //lint:ignore, not golangci's //nolint. Rather
  than juggle two directive formats, delete the 4 genuinely-unused funcs
  (treeWalkTraceTokenFromCitations, originMatches, isTimeout, isEncryptedPDFError)
  — both linters agree they're dead. (Re-add with callers when HAL-73/HAL-112 land.)
- race: pdftable.ExtractTables also mutates package-level state — a second pdftable
  race distinct from OpenBytes; serialize it on the same mutex. Stopgap; the root
  fix is HAL-118 (make pdftable concurrency-safe).

golangci-lint + standalone staticcheck: 0 issues. go build + unit tests green.

* engine: go mod tidy — blank line before toolchain directive (HAL-116)

The ubuntu-only 'Verify go.mod/go.sum are tidy' CI step requires the canonical
format go mod tidy emits: a blank line between the 'go' and 'toolchain'
directives. My earlier manual insertion of the toolchain line omitted it.

Author: hallelx2

.github/workflows/ci.yml

@@ -17,7 +17,7 @@ jobs:
       fail-fast: false
       matrix:
         os: [ubuntu-latest, macos-latest, windows-latest]
-        go: ["1.25"]
+        go: ["1.25.11"]
     steps:
       - uses: actions/checkout@v6
 
@@ -70,7 +70,7 @@ jobs:
       - name: Set up Go
         uses: actions/setup-go@v6
         with:
-          go-version: "1.25"
+          go-version: "1.25.11"
           cache: true
 
       # MinIO can't run as a `services:` container because GitHub Actions
@@ -115,7 +115,7 @@ jobs:
       - name: Set up Go
         uses: actions/setup-go@v6
         with:
-          go-version: "1.25"
+          go-version: "1.25.11"
           cache: true
 
       - name: staticcheck
@@ -137,7 +137,7 @@ jobs:
       - uses: actions/checkout@v6
       - uses: actions/setup-go@v6
         with:
-          go-version: "1.25"
+          go-version: "1.25.11"
           cache: true
       - name: Install govulncheck
         run: go install golang.org/x/vuln/cmd/govulncheck@latest

.golangci.yml

@@ -30,6 +30,11 @@ linters:
         - (net/http.ResponseWriter).Write
         - (github.com/jackc/pgx/v5.Tx).Rollback
         - (github.com/hallelx2/vectorless-engine/pkg/queue.Queue).Close
+    misspell:
+      # "strat" is our abbreviation for "strategy" (variable + key names),
+      # not a misspelling of "start".
+      ignore-rules:
+        - strat
   exclusions:
     rules:
       - path: _test\.go

cmd/benchmark/main.go

@@ -77,7 +77,7 @@ func main() {
 			ID string `json:"id"`
 		} `json:"sections"`
 	}
-	json.Unmarshal(treeResp, &treeData)
+	_ = json.Unmarshal(treeResp, &treeData) // benchmark best-effort parse
 	sectionID := ""
 	if len(treeData.Sections) > 1 {
 		sectionID = treeData.Sections[1].ID // pick a child section
@@ -87,9 +87,9 @@ func main() {
 	fmt.Printf("Using section ID: %s\n\n", sectionID)
 
 	type result struct {
-		name     string
-		rest     []time.Duration
-		grpc     []time.Duration
+		name string
+		rest []time.Duration
+		grpc []time.Duration
 	}
 
 	var results []result
@@ -101,12 +101,12 @@ func main() {
 		fmt.Print(".")
 		// REST
 		start := time.Now()
-		restGET(base + "/v1/health")
+		_, _ = restGET(base + "/v1/health")
 		r.rest = append(r.rest, time.Since(start))
 
 		// gRPC (Connect)
 		start = time.Now()
-		healthClient.Check(ctx, connect.NewRequest(&v1.HealthCheckRequest{}))
+		_, _ = healthClient.Check(ctx, connect.NewRequest(&v1.HealthCheckRequest{}))
 		r.grpc = append(r.grpc, time.Since(start))
 	}
 	fmt.Println(" done")
@@ -118,11 +118,11 @@ func main() {
 	for i := 0; i < n; i++ {
 		fmt.Print(".")
 		start := time.Now()
-		restGET(base + "/v1/documents")
+		_, _ = restGET(base + "/v1/documents")
 		r.rest = append(r.rest, time.Since(start))
 
 		start = time.Now()
-		docsClient.ListDocuments(ctx, connect.NewRequest(&v1.ListDocumentsRequest{Limit: 10}))
+		_, _ = docsClient.ListDocuments(ctx, connect.NewRequest(&v1.ListDocumentsRequest{Limit: 10}))
 		r.grpc = append(r.grpc, time.Since(start))
 	}
 	fmt.Println(" done")
@@ -134,11 +134,11 @@ func main() {
 	for i := 0; i < n; i++ {
 		fmt.Print(".")
 		start := time.Now()
-		restGET(base + "/v1/documents/" + *docID + "/tree")
+		_, _ = restGET(base + "/v1/documents/" + *docID + "/tree")
 		r.rest = append(r.rest, time.Since(start))
 
 		start = time.Now()
-		docsClient.GetDocumentTree(ctx, connect.NewRequest(&v1.GetDocumentTreeRequest{
+		_, _ = docsClient.GetDocumentTree(ctx, connect.NewRequest(&v1.GetDocumentTreeRequest{
 			DocumentId: *docID,
 		}))
 		r.grpc = append(r.grpc, time.Since(start))
@@ -153,11 +153,11 @@ func main() {
 		for i := 0; i < n; i++ {
 			fmt.Print(".")
 			start := time.Now()
-			restGET(base + "/v1/sections/" + sectionID)
+			_, _ = restGET(base + "/v1/sections/" + sectionID)
 			r.rest = append(r.rest, time.Since(start))
 
 			start = time.Now()
-			docsClient.GetSection(ctx, connect.NewRequest(&v1.GetSectionRequest{
+			_, _ = docsClient.GetSection(ctx, connect.NewRequest(&v1.GetSectionRequest{
 				SectionId: sectionID,
 			}))
 			r.grpc = append(r.grpc, time.Since(start))
@@ -182,12 +182,12 @@ func main() {
 			"query":       q,
 		})
 		start := time.Now()
-		restPOST(base+"/v1/query", body)
+		_, _ = restPOST(base+"/v1/query", body)
 		r.rest = append(r.rest, time.Since(start))
 
 		// gRPC (Connect)
 		start = time.Now()
-		queryClient.Query(ctx, connect.NewRequest(&v1.QueryRequest{
+		_, _ = queryClient.Query(ctx, connect.NewRequest(&v1.QueryRequest{
 			DocumentId: *docID,
 			Query:      q,
 		}))
@@ -246,7 +246,7 @@ func restGET(url string) ([]byte, error) {
 	if err != nil {
 		return nil, err
 	}
-	defer resp.Body.Close()
+	defer func() { _ = resp.Body.Close() }() // best-effort close
 	return io.ReadAll(resp.Body)
 }
 
@@ -255,7 +255,7 @@ func restPOST(url string, body []byte) ([]byte, error) {
 	if err != nil {
 		return nil, err
 	}
-	defer resp.Body.Close()
+	defer func() { _ = resp.Body.Close() }() // best-effort close
 	return io.ReadAll(resp.Body)
 }
 

cmd/engine/main.go

@@ -90,7 +90,7 @@ func run() error {
 	if err != nil {
 		return fmt.Errorf("init queue: %w", err)
 	}
-	defer q.Close()
+	defer func() { _ = q.Close() }() // best-effort close
 
 	llmClient, err := buildLLM(cfg.LLM)
 	if err != nil {
@@ -206,41 +206,41 @@ func run() error {
 	}
 	q.Register(queue.KindIngestDocument, pipeline.Handler())
 
-	// /v1/answer/pageindex gets its OWN PageIndexStrategy instance,
+	// /v1/answer/treewalk gets its OWN TreeWalkStrategy instance,
 	// independent of whatever selection strategy is configured in
 	// retrieval.strategy. This way the endpoint is always available
-	// (gated by retrieval.pageindex.enabled), even on a deployment
+	// (gated by retrieval.treewalk.enabled), even on a deployment
 	// using chunked-tree as its default selection path.
-	var pageIndexStrategy *retrieval.PageIndexStrategy
-	if cfg.Retrieval.PageIndex.Enabled && llmClient != nil {
-		pageIndexStrategy = buildPageIndexStrategy(cfg.Retrieval, llmClient, store)
-		logger.Info("retrieval: pageindex answer endpoint enabled",
-			"max_hops", pageIndexStrategy.MaxHops,
-			"page_content_limit", pageIndexStrategy.PageContentLimit,
-			"model_override", cfg.Retrieval.PageIndex.Model,
+	var treeWalkStrategy *retrieval.TreeWalkStrategy
+	if cfg.Retrieval.TreeWalk.Enabled && llmClient != nil {
+		treeWalkStrategy = buildTreeWalkStrategy(cfg.Retrieval, llmClient, store)
+		logger.Info("retrieval: treewalk answer endpoint enabled",
+			"max_hops", treeWalkStrategy.MaxHops,
+			"page_content_limit", treeWalkStrategy.PageContentLimit,
+			"model_override", cfg.Retrieval.TreeWalk.Model,
 		)
 	}
 
 	deps := api.Deps{
-		Logger:            logger,
-		DB:                pool,
-		Storage:           store,
-		Queue:             q,
-		Strategy:          strategy,
-		Version:           version,
-		MultiDoc:          multiDoc,
-		LLM:               llmClient,
-		LLMModel:          modelFor(cfg.LLM),
-		AnswerSpan:        cfg.Retrieval.AnswerSpan,
-		Answer:            cfg.Retrieval.Answer,
-		Planner:           planner,
-		Planning:          cfg.Retrieval.Planning,
-		ReRanker:          reRanker,
-		ReRank:            cfg.Retrieval.ReRank,
-		Replay:            replayStore,
-		Abstain:           cfg.Retrieval.Abstain,
-		PageIndexStrategy: pageIndexStrategy,
-		PageIndex:         cfg.Retrieval.PageIndex,
+		Logger:           logger,
+		DB:               pool,
+		Storage:          store,
+		Queue:            q,
+		Strategy:         strategy,
+		Version:          version,
+		MultiDoc:         multiDoc,
+		LLM:              llmClient,
+		LLMModel:         modelFor(cfg.LLM),
+		AnswerSpan:       cfg.Retrieval.AnswerSpan,
+		Answer:           cfg.Retrieval.Answer,
+		Planner:          planner,
+		Planning:         cfg.Retrieval.Planning,
+		ReRanker:         reRanker,
+		ReRank:           cfg.Retrieval.ReRank,
+		Replay:           replayStore,
+		Abstain:          cfg.Retrieval.Abstain,
+		TreeWalkStrategy: treeWalkStrategy,
+		TreeWalk:         cfg.Retrieval.TreeWalk,
 	}
 
 	srv := &http.Server{
@@ -393,36 +393,38 @@ func buildStrategy(c config.RetrievalConfig, client llmgate.Client, store storag
 		}
 		a.ModelOverride = c.Agentic.Model
 		return a
-	case "pageindex":
-		return buildPageIndexStrategy(c, client, store)
+	case "treewalk":
+		return buildTreeWalkStrategy(c, client, store)
+	case "auto":
+		return retrieval.NewAuto(retrieval.NewSinglePass(client), buildTreeWalkStrategy(c, client, store))
 	default:
 		return retrieval.NewChunkedTree(client)
 	}
 }
 
-// buildPageIndexStrategy constructs the page-based agentic
+// buildTreeWalkStrategy constructs the page-based agentic
 // strategy with the storage-backed PageLoader and the configured
-// caps. Used by buildStrategy when retrieval.strategy=pageindex AND
-// by the /v1/answer/pageindex endpoint setup (which wires its own
+// caps. Used by buildStrategy when retrieval.strategy=treewalk AND
+// by the /v1/answer/treewalk endpoint setup (which wires its own
 // instance regardless of the selection strategy).
 //
 // The TOCProvider is left nil here. PR-A (toc-tree-builder) adds
 // documents.toc_tree + a DB-backed provider; until it lands the
 // strategy degrades to its synthesised view, which is the
 // documented fallback path.
-func buildPageIndexStrategy(c config.RetrievalConfig, client llmgate.Client, store storage.Storage) *retrieval.PageIndexStrategy {
-	p := retrieval.NewPageIndexStrategy(client)
+func buildTreeWalkStrategy(c config.RetrievalConfig, client llmgate.Client, store storage.Storage) *retrieval.TreeWalkStrategy {
+	p := retrieval.NewTreeWalkStrategy(client)
 	p.PageLoader = storagePageLoader{s: store}
-	if c.PageIndex.MaxHops > 0 {
-		p.MaxHops = c.PageIndex.MaxHops
+	if c.TreeWalk.MaxHops > 0 {
+		p.MaxHops = c.TreeWalk.MaxHops
 	}
-	if c.PageIndex.PageContentLimit > 0 {
-		p.PageContentLimit = c.PageIndex.PageContentLimit
+	if c.TreeWalk.PageContentLimit > 0 {
+		p.PageContentLimit = c.TreeWalk.PageContentLimit
 	}
-	if c.PageIndex.MaxCitations > 0 {
-		p.MaxCitations = c.PageIndex.MaxCitations
+	if c.TreeWalk.MaxCitations > 0 {
+		p.MaxCitations = c.TreeWalk.MaxCitations
 	}
-	p.ModelOverride = c.PageIndex.Model
+	p.ModelOverride = c.TreeWalk.Model
 	return p
 }
 
@@ -437,14 +439,14 @@ func (sf storageFetcher) Get(ctx context.Context, ref string) ([]byte, error) {
 	if err != nil {
 		return nil, err
 	}
-	defer rc.Close()
+	defer func() { _ = rc.Close() }() // best-effort close
 	return io.ReadAll(rc)
 }
 
 // storagePageLoader adapts a storage.Storage to
 // retrieval.PageContentLoader. Mirrors storageFetcher but lives
 // behind a separate interface so the two callers (agentic /
-// pageindex) can be wired independently. The PageIndex strategy
+// treewalk) can be wired independently. The TreeWalk strategy
 // materialises section bodies once per get_pages observation, so
 // reading the full reader into a []byte is the right shape.
 type storagePageLoader struct{ s storage.Storage }
@@ -454,7 +456,7 @@ func (l storagePageLoader) Load(ctx context.Context, ref string) ([]byte, error)
 	if err != nil {
 		return nil, err
 	}
-	defer rc.Close()
+	defer func() { _ = rc.Close() }() // best-effort close
 	return io.ReadAll(rc)
 }