fix(performance): prevent OOM and system freezes during indexing

This addresses issues where indexing large files (e.g., barou.sql) caused the host system to freeze due to host CPU/GPU starvation and excessive GC pressure.

- Fix Ollama throttling bug in indexer service by correctly using a 150ms delay instead of 10ms.
- Prevent GC thrashing in treesitter parser by evaluating byte sizes instead of allocating strings for every AST node.
- Truncate massive leaf nodes (>8KB) to prevent crashing the Ollama embedding API.

Author: doITmagic

.github/PULL_REQUEST_TEMPLATE.md

@@ -269,3 +269,44 @@ In `AnalyzePackage` in `pkg/parser/go/analyzer.go`, add iteration over `typ.Func
 
 *Last updated: 2026-03-09 — BUG-001 fixed, BUG-003 added*
 
+---
+
+## BUG-004: AST Fallback Search and Indexer do not exclude unconfigured directories like `inspirations/`
+
+**Status:** Open  
+**Date confirmed:** 2026-03-09  
+**Affected component:** `FallbackDirectSearch` (`internal/service/engine/engine_fallback_search.go`) and `IndexWorkspace` (`pkg/indexer/service.go`)  
+**Severity:** Medium — causes irrelevant, old, or draft code to pollute semantic and fallback search results.
+
+### Description
+
+When performing a search that falls back to the AST (e.g. while `go` files are `processed: 0`), RAGCode can return results from the `inspirations/` directory (or other directories that should logically be ignored). This happens because `filepath.WalkDir` relies entirely on a hardcoded list of `excludePatterns` loaded from `config.Workspace.ExcludePatterns`, alongside a basic check for `.`, `vendor`, and `node_modules`.
+
+### Example
+
+Searching for the processing of `state.json` via `rag_search` returned a fallback result pointing to:
+`/home/razvan/go/src/github.com/doITmagic/rag-code-mcp/inspirations/rag-code-mcp/internal/workspace/state.go` 
+instead of the actual code in `pkg/indexer/state.go`.
+
+### Root Cause
+In both `internal/service/engine/engine_fallback_search.go` (lines 88-103) and `pkg/indexer/service.go` (lines 72-88), the exclusion logic is implemented manually:
+```go
+if d.IsDir() {
+    name := d.Name()
+    if strings.HasPrefix(name, ".") || name == "vendor" || name == "node_modules" {
+        return filepath.SkipDir
+    }
+    for _, p := range excludePatterns {
+        if name == p {
+            return filepath.SkipDir
+        }
+    }
+    return nil
+}
+```
+If `inspirations` or other custom draft folders are not explicitly provided in the YAML config `exclude_patterns`, they are scanned by the fallback module and indexer. The system **does not automatically parse `.ragcodeignore` or `.gitignore`**, nor does it have a default ignore list for common draft/backup directories like `inspirations`.
+
+### Proposed Fix
+1. Ensure that `.gitignore` or `.ragcodeignore` files are parsed and respected during the `filepath.WalkDir` traversal.
+2. Consider adding `inspirations` and `drafts` strings to the default hardcoded exclusions if they represent common anti-patterns for this specific repo, or automatically bundle `.gitignore` rules into the `excludePatterns` array at startup.
+

.github/copilot-instructions.md

@@ -147,3 +147,15 @@ Indexează fișierele `.md` din workspace (README, guides, API docs) în aceeaș
 - [ ] **[P2]** `.txt` — split pe paragrafe cu `RecursiveCharacterSplitter`.
 - [ ] **[P2]** `.json` / `.yaml` — flatten keys ca text și indexare ca documentație structurată.
 - [ ] **[P2]** `.rst` / `.adoc` — convertor la markdown + chunking standard.
+
+## Task 9: UX / Metrics Simplification & Indexing Priority
+
+### Goal
+- Simplify the indexing progress metrics visible to the AI (MCP output) to prevent confusion and encourage semantic tool usage.
+- Prioritize indexing the project's majority language first (e.g., if it's a Go project with 177 files and 10 markdown files, index Go files before Docs).
+
+### Subtasks
+- [ ] **[P0]** Refactor `index_status.json` structure or the MCP response envelope to only expose `total_files` and `indexed_files` per language, dropping task-specific states like `changed` and `processed`.
+- [ ] **[P0]** Provide an explicit `"status": "up_to_date"` string when `indexed_files == total_files` to build AI trust.
+- [ ] **[P0]** In `internal/service/engine/engine.go` during `IndexWorkspace`, dynamically sort the `languages` slice (e.g. `docs`, `go`, etc.) descending based on their `fileCounts` value before entering the core processing loop. This guarantees the highest coverage language completes first.
+- [ ] **[P1]** Ensure `IndexFiles` (incremental logic triggered on single file edits) safely patches `index_status.json` by delta (e.g., adding +1 to total/indexed) without resetting or zeroing out the existing statistics built by the main `IndexWorkspace` routine.

BUGS.md

@@ -17,7 +17,7 @@ import (
 )
 
 var (
-	Version = "2.1.63"
+	Version = "2.1.65"
 	Commit  = "none"
 	Date    = "24.10.2025"
 )