refactor: Replace progressStore with file-based IndexStatus

[doITmagic]

Description

Replaces the old, complex progressStore indexing progress mechanism with a simple file-based IndexStatus system.

Problem

The previous system had two separate file walks (one for counting, one for indexing), an in-memory progressStore with preRegister/update/carry-over/flusher logic, and multiple intermediate structs (IndexingProgressSummary, LangProgressItem). This made progress reporting confusing, inaccurate, and hard to maintain.

Solution

• Single source of truth: {workspaceRoot}/.ragcode/index_status.json — a simple JSON file written by the indexer during processing
• Direct write from indexer: The Progress callback in pkg/indexer/service.go writes OnDisk, Changed, Processed counts per language directly to the status file
• Direct read by tools: MCP tools read the status file via indexer.LoadIndexStatus() — no intermediate transformations
• Lifecycle: starting → running (first file processed) → completed/failed

What was removed

• progressStore (preRegister, update, carry-over, flusher goroutine)
• IndexingProgressSummary, LangProgressItem, BuildIndexingProgress, formatAge, buildIndexingMessage
• Auto-resume logic from SearchCode/HybridSearchCode (redundant with DetectContext which already triggers re-indexing on first tool call)
• resumeAttempts field from Engine struct

What was added

• pkg/indexer/index_status.go — IndexStatus, LangStatus, SaveIndexStatus, LoadIndexStatus
• pkg/indexer/index_status_test.go — round-trip and missing file tests
• Progress callback wiring in engine.IndexWorkspace

Files changed (22 files, +208 / -1182 lines)

• Moved: engine/index_progress.go → pkg/indexer/index_status.go
• Modified: engine.go, response.go, smart_search.go, smart_search_pipeline.go, index_workspace.go, list_package_exports.go, find_usages.go, call_hierarchy.go, evaluate_ragcode.go, skills.go, read_file_context.go
• Tests updated: engine_searchcode_test.go, engine_*_test.go, health_metrics_test.go, detector_test.go

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation update

Checklist:

• I have performed a self-review of my own code
• I have formatted my code with go fmt ./...
• I have run tests go test ./... and they pass
• I have verified integration with Ollama/Qdrant (if applicable)
• I have updated the documentation accordingly

[Copilot]

Pull request overview

Replaces the in-memory progressStore indexing progress mechanism (with preRegister, flusher goroutine, carry-over logic) with a simpler file-based IndexStatus system in pkg/indexer/index_status.go. MCP tools now read status directly from {workspaceRoot}/.ragcode/index_status.json via indexer.LoadIndexStatus(). This is a significant simplification that removes ~1000 lines of complex concurrency code.

Changes:

• New pkg/indexer/index_status.go with IndexStatus/LangStatus structs and SaveIndexStatus/LoadIndexStatus functions, replacing the old engine/index_progress.go with its progressStore, flusher goroutine, and deep-copy logic
• All MCP tools updated to use the new IndexingStatus field (backed by indexer.IndexStatus) instead of IndexingProgress (backed by IndexingProgressSummary), with the JSON tag preserved as "indexing_progress" for backward compatibility
• Removed auto-resume logic from SearchCode/HybridSearchCode and the resumeAttempts throttle, as workspace re-indexing is now triggered via DetectContext

Reviewed changes

Copilot reviewed 22 out of 22 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
pkg/indexer/index_status.go	New file: IndexStatus/LangStatus types, SaveIndexStatus, LoadIndexStatus
pkg/indexer/index_status_test.go	New file: round-trip and missing-file tests
internal/service/engine/engine.go	Remove progressStore/resumeAttempts, add GetIndexStatus, wire Progress callback with file-based status
internal/service/engine/index_progress.go	Deleted: old progressStore and all related types/functions
internal/service/engine/index_progress_test.go	Deleted: tests for removed progressStore
internal/service/tools/response.go	Replace IndexingProgressSummary with IndexingStatus, rename helper to ContextFromWorkspaceWithStatus
internal/service/tools/smart_search_pipeline.go	Use LoadIndexStatus, replace dynamic fallback note with static string
internal/service/tools/smart_search.go	Simplify indexing error messages, remove progress attachment
internal/service/tools/find_usages.go	Switch to GetIndexStatus/ContextFromWorkspaceWithStatus
internal/service/tools/call_hierarchy.go	Switch to GetIndexStatus/ContextFromWorkspaceWithStatus
internal/service/tools/list_package_exports.go	Switch to ContextFromWorkspaceWithStatus, remove idx-based status override
internal/service/tools/index_workspace.go	Use LoadIndexStatus directly
internal/service/tools/skills.go	Set IndexingStatus: nil
internal/service/tools/evaluate_ragcode.go	Remove unused wctxID/wctxRoot, set IndexingStatus: nil
internal/service/tools/read_file_context.go	Set IndexingStatus: nil
internal/service/tools/tests/health_metrics_test.go	Remove tests for deleted types/functions
internal/service/engine/engine_searchcode_test.go	Remove auto-resume test
internal/service/engine/engine_nonblocking_search_test.go	Remove progress.stop() cleanup
internal/service/engine/engine_fallback_search_test.go	Remove progress.stop() cleanup
internal/service/engine/engine_sticky_test.go	Remove progress.stop() cleanup
pkg/workspace/detector/detector_test.go	Isolate test to avoid picking up .ragcode markers from parent dirs
cmd/rag-code-mcp/main.go	Version bump to 2.1.63

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

[Copilot]

Pull request overview

Copilot reviewed 23 out of 23 changed files in this pull request and generated 9 comments.

Comments suppressed due to low confidence (2)

internal/service/tools/call_hierarchy.go:136

• The check if idx != nil at line 133 is now semantically incorrect. With the old in-memory progressStore, GetIndexProgress returned nil when no indexing was active. With the new file-based system, GetIndexStatus reads from disk and returns non-nil even for a completed previous run (the file persists). This means the response will always be indexing_in_progress when no collections exist but a previous run left an index_status.json file — even if indexing completed hours ago.

This needs to check the actual state (e.g., EndedAt is empty and Error is empty) to determine if indexing is truly in progress. Without a State field on IndexStatus, you'd need something like: if idx != nil && idx.EndedAt == "" && idx.Error == "".

			if idx != nil {
				resp.Status = "indexing_in_progress"
				resp.Data = map[string]any{"indexing": idx}
			}

internal/service/tools/find_usages.go:105

• Same issue as in call_hierarchy.go: if idx != nil at line 103 will now be true even for a completed previous indexing run (the file persists on disk), incorrectly changing the status from indexing_required to indexing_in_progress. This check needs to verify that indexing is actually ongoing (e.g., idx.EndedAt == "").

			if idx != nil {
				resp.Status = "indexing_in_progress"
			}

[Copilot]

Pull request overview

Copilot reviewed 28 out of 28 changed files in this pull request and generated 8 comments.

Comments suppressed due to low confidence (2)

internal/service/tools/call_hierarchy.go:136

• Same bug as in find_usages.go: idx is loaded from the persisted index_status.json file. Once any indexing run has completed, the file exists and idx != nil is always true — causing the response status to be incorrectly set to "indexing_in_progress" even when no indexing job is active.

The condition on line 133 should check whether indexing is actually in progress (e.g., via ActiveIndexingJobs() or checking idx.EndedAt == "") rather than just checking file existence.

	idx := t.engine.GetIndexStatus(wctx.Root)

	visited := make(map[string]bool)

	rootNode := &CallNode{Name: symbolName}

	// Try to find root symbol info
	rootRes := t.findSymbolInfo(ctx, wctx.ID, symbolName)
	if rootRes != nil {
		rootNode.Type, _ = rootRes.Point.Payload["type"].(string)
		rootNode.FilePath, _ = rootRes.Point.Payload["file_path"].(string)
		rootNode.Package, _ = rootRes.Point.Payload["package"].(string)
	} else {
		// If nothing is indexed yet, ExactSearchPolyglot will return ErrNoCollectionsFound.
		// Signal indexing status instead of returning an empty hierarchy.
		_, sErr := t.engine.ExactSearchPolyglot(ctx, wctx.ID, map[string]interface{}{"name": symbolName}, 1)
		var noCollections *engine.ErrNoCollectionsFound
		if errors.As(sErr, &noCollections) {
			resp := ToolResponse{
				Status:  "indexing_required",
				Message: fmt.Sprintf("⏳ Workspace '%s' is not indexed yet. Indexing is required for complete call hierarchy results.", wctx.Root),
				Context: ContextFromWorkspaceWithStatus(wctx, t.engine),
			}
			if idx != nil {
				resp.Status = "indexing_in_progress"
				resp.Data = map[string]any{"indexing": idx}
			}

internal/service/tools/find_usages.go:105

• Bug: idx is now loaded from the persisted index_status.json file via GetIndexStatus(). Unlike the old GetIndexProgress() which returned non-nil only when an active in-memory job was running, the file persists after indexing completes. This means idx != nil will always be true once any indexing run has occurred, causing the status to incorrectly change from "indexing_required" to "indexing_in_progress" even when indexing finished long ago.

To fix: either check if an indexing job is currently active (via ActiveIndexingJobs() or indexingJobs.Load(wctx.ID)), or check a specific field on the status (e.g., s.EndedAt == "") before treating it as "in progress".

	idx := t.engine.GetIndexStatus(wctx.Root)
	allResults, err := t.engine.ExactSearchPolyglot(ctx, wctx.ID, filter, 100)
	if err != nil {
		var noCollections *engine.ErrNoCollectionsFound
		if errors.As(err, &noCollections) {
			resp := ToolResponse{
				Status:  "indexing_required",
				Message: fmt.Sprintf("⏳ Workspace '%s' is not indexed yet. Indexing is required for complete results.", wctx.Root),
				Context: ContextFromWorkspaceWithStatus(wctx, t.engine),
			}
			if idx != nil {
				resp.Status = "indexing_in_progress"
			}

[Copilot]

Pull request overview

Copilot reviewed 34 out of 34 changed files in this pull request and generated 9 comments.

[Copilot]

Pull request overview

Copilot reviewed 35 out of 36 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (2)

internal/service/tools/find_usages.go:105

• In the ErrNoCollectionsFound branch, resp.Status is switched to "indexing_in_progress" whenever idx != nil. With the new file-based IndexStatus, idx will be non-nil for any workspace that has ever written index_status.json (even if indexing already completed), so this can misreport "indexing_in_progress". Consider keying this on a real in-progress signal (e.g., idx.EndedAt == "" / idx.Error == "" or checking Engine.ActiveIndexingJobs for wctx.ID) instead of mere file existence.

	idx := t.engine.GetIndexStatus(wctx.Root)
	allResults, err := t.engine.ExactSearchPolyglot(ctx, wctx.ID, filter, 100)
	if err != nil {
		var noCollections *engine.ErrNoCollectionsFound
		if errors.As(err, &noCollections) {
			resp := ToolResponse{
				Status:  "indexing_required",
				Message: fmt.Sprintf("⏳ Workspace '%s' is not indexed yet. Indexing is required for complete results.", wctx.Root),
				Context: ContextFromWorkspaceWithStatus(wctx, t.engine),
			}
			if idx != nil {
				resp.Status = "indexing_in_progress"
			}

internal/service/tools/call_hierarchy.go:136

• The "indexing_in_progress" status is currently set whenever idx != nil, but IndexStatus will be non-nil for any workspace that has an index_status.json from a previous run. This can incorrectly report indexing as in progress when indexing is actually completed (or stale). Consider switching this condition to something that reflects an active run (e.g., idx.EndedAt == "" and idx.Error == "" / checking Engine.ActiveIndexingJobs for wctx.ID).

	idx := t.engine.GetIndexStatus(wctx.Root)

	visited := make(map[string]bool)

	rootNode := &CallNode{Name: symbolName}

	// Try to find root symbol info
	rootRes := t.findSymbolInfo(ctx, wctx.ID, symbolName)
	if rootRes != nil {
		rootNode.Type, _ = rootRes.Point.Payload["type"].(string)
		rootNode.FilePath, _ = rootRes.Point.Payload["file_path"].(string)
		rootNode.Package, _ = rootRes.Point.Payload["package"].(string)
	} else {
		// If nothing is indexed yet, ExactSearchPolyglot will return ErrNoCollectionsFound.
		// Signal indexing status instead of returning an empty hierarchy.
		_, sErr := t.engine.ExactSearchPolyglot(ctx, wctx.ID, map[string]interface{}{"name": symbolName}, 1)
		var noCollections *engine.ErrNoCollectionsFound
		if errors.As(sErr, &noCollections) {
			resp := ToolResponse{
				Status:  "indexing_required",
				Message: fmt.Sprintf("⏳ Workspace '%s' is not indexed yet. Indexing is required for complete call hierarchy results.", wctx.Root),
				Context: ContextFromWorkspaceWithStatus(wctx, t.engine),
			}
			if idx != nil {
				resp.Status = "indexing_in_progress"
				resp.Data = map[string]any{"indexing": idx}
			}

[Copilot]

Pull request overview

Copilot reviewed 36 out of 37 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (2)

internal/service/tools/find_usages.go:106

• idx != nil is treated as “indexing_in_progress”, but IndexStatus is loaded from disk and will remain non-nil even after indexing completed (EndedAt set) or failed. This can incorrectly label the workspace as indexing when ErrNoCollectionsFound occurs for other reasons (e.g., collections deleted). Consider deriving “in progress” from the status fields (e.g., StartedAt set AND EndedAt empty), and only then switch to indexing_in_progress / attach indexing data.

	idx := t.engine.GetIndexStatus(wctx.Root)
	allResults, err := t.engine.ExactSearchPolyglot(ctx, wctx.ID, filter, 100)
	if err != nil {
		var noCollections *engine.ErrNoCollectionsFound
		if errors.As(err, &noCollections) {
			resp := ToolResponse{
				Status:  "indexing_required",
				Message: fmt.Sprintf("⏳ Workspace '%s' is not indexed yet. Indexing is required for complete results.", wctx.Root),
				Context: ContextFromWorkspaceWithStatus(wctx, t.engine),
			}
			if idx != nil {
				resp.Status = "indexing_in_progress"
			}
			return resp.JSON()

internal/service/tools/call_hierarchy.go:137

• The tool sets status=indexing_in_progress whenever an IndexStatus file exists (idx != nil), but IndexStatus persists after completion. In the ErrNoCollectionsFound branch this can misreport state if collections are missing for other reasons. Prefer a deterministic “in progress” check (e.g., StartedAt present and EndedAt empty) before reporting indexing_in_progress and returning indexing data.

	idx := t.engine.GetIndexStatus(wctx.Root)

	visited := make(map[string]bool)

	rootNode := &CallNode{Name: symbolName}

	// Try to find root symbol info
	rootRes := t.findSymbolInfo(ctx, wctx.ID, symbolName)
	if rootRes != nil {
		rootNode.Type, _ = rootRes.Point.Payload["type"].(string)
		rootNode.FilePath, _ = rootRes.Point.Payload["file_path"].(string)
		rootNode.Package, _ = rootRes.Point.Payload["package"].(string)
	} else {
		// If nothing is indexed yet, ExactSearchPolyglot will return ErrNoCollectionsFound.
		// Signal indexing status instead of returning an empty hierarchy.
		_, sErr := t.engine.ExactSearchPolyglot(ctx, wctx.ID, map[string]interface{}{"name": symbolName}, 1)
		var noCollections *engine.ErrNoCollectionsFound
		if errors.As(sErr, &noCollections) {
			resp := ToolResponse{
				Status:  "indexing_required",
				Message: fmt.Sprintf("⏳ Workspace '%s' is not indexed yet. Indexing is required for complete call hierarchy results.", wctx.Root),
				Context: ContextFromWorkspaceWithStatus(wctx, t.engine),
			}
			if idx != nil {
				resp.Status = "indexing_in_progress"
				resp.Data = map[string]any{"indexing": idx}
			}
			return resp.JSON()

[Copilot]

Pull request overview

Copilot reviewed 36 out of 37 changed files in this pull request and generated 2 comments.

---

You can also share your feedback on Copilot code review. Take the survey.

[Copilot]

Pull request overview

Copilot reviewed 44 out of 45 changed files in this pull request and generated 4 comments.

---

You can also share your feedback on Copilot code review. Take the survey.

[Copilot]

Pull request overview

Copilot reviewed 59 out of 62 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (2)

internal/service/tools/find_usages.go:105

• In the ErrNoCollectionsFound branch, the tool sets status="indexing_in_progress" whenever GetIndexStatus returns non-nil. With the new file-based IndexStatus, a non-nil status file can exist even when indexing is completed/failed, so this can incorrectly report “in progress”. Consider checking something like idx.EndedAt == "" && idx.StartedAt != "" (and/or idx.Error=="") before switching the status.
  internal/service/tools/call_hierarchy.go:136
• Same issue as in FindUsagesTool: idx := GetIndexStatus(...) is used as a boolean to decide status="indexing_in_progress". Since IndexStatus persists after completion, this can mislabel a workspace as “in progress” even when it’s done. Gate this on an “active” condition (e.g., EndedAt == "").

	idx := t.engine.GetIndexStatus(wctx.Root)

	visited := make(map[string]bool)

	rootNode := &CallNode{Name: symbolName}

	// Try to find root symbol info
	rootRes := t.findSymbolInfo(ctx, wctx.ID, symbolName)
	if rootRes != nil {
		rootNode.Type, _ = rootRes.Point.Payload["type"].(string)
		rootNode.FilePath, _ = rootRes.Point.Payload["file_path"].(string)
		rootNode.Package, _ = rootRes.Point.Payload["package"].(string)
	} else {
		// If nothing is indexed yet, ExactSearchPolyglot will return ErrNoCollectionsFound.
		// Signal indexing status instead of returning an empty hierarchy.
		_, sErr := t.engine.ExactSearchPolyglot(ctx, wctx.ID, map[string]interface{}{"name": symbolName}, 1)
		var noCollections *engine.ErrNoCollectionsFound
		if errors.As(sErr, &noCollections) {
			resp := ToolResponse{
				Status:  "indexing_required",
				Message: fmt.Sprintf("⏳ Workspace '%s' is not indexed yet. Indexing is required for complete call hierarchy results.", wctx.Root),
				Context: ContextFromWorkspaceWithStatus(wctx, t.engine),
			}
			if idx != nil {
				resp.Status = "indexing_in_progress"
				resp.Data = map[string]any{"indexing": idx}
			}

---

You can also share your feedback on Copilot code review. Take the survey.

[Copilot]

Pull request overview

Copilot reviewed 75 out of 78 changed files in this pull request and generated 14 comments.

---

You can also share your feedback on Copilot code review. Take the survey.