Lower memory use in the cross-file reference index

Author: AJenbo

docs/CHANGELOG.md

@@ -42,6 +42,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - **Continuous progress reporting.** The indexing progress bar now advances file by file with live counts (e.g. "Scanning vendor packages (3201/8544 files)") instead of jumping between a few fixed milestones. This covers single-project, monorepo, and non-Composer workspaces. Go to Implementation, Find References, and Type Hierarchy show the same live progress while they scan, including when one of them triggers the first full workspace index.
 - **Property hover now shows effective types as a `var` detail line.** Property hovers now mirror method hovers by displaying the resolved/effective property type above the PHP snippet as `**var**`, while the snippet itself shows only the native PHP property declaration. This keeps docblock-inferred, virtual, and schema-derived property types out of the generated signature block. Contributed by @calebdw.
 - **Updated the bundled mago toolchain to 1.43.0.** The parser, docblock parser, formatter, and supporting crates are refreshed to the latest upstream release. Contributed by @enwi in https://github.com/PHPantom-dev/phpantom_lsp/pull/234.
+- **Lower memory use in the cross-file reference index.** The index backing Find References and the reference-count inlay hints now keeps only the distinct files and counts each symbol actually needs, instead of one entry per matching location plus never-read position data. On large projects this removes millions of short-lived allocations and shrinks the index to a fraction of its previous size, with no change to Find References or inlay hint results.
 
 ### Removed
 

docs/todo.md

@@ -26,7 +26,6 @@ within the same impact tier.
 | #   | Item                                                                                                                  | Impact     | Effort |
 | --- | --------------------------------------------------------------------------------------------------------------------- | ---------- | ------ |
 | P36 | [Diagnostic-path call sites re-merge inheritance per call](todo/performance.md#p36-diagnostic-path-call-sites-re-merge-inheritance-per-call-instead-of-reading-the-resolved-class-cache)          | Medium      | Low         |
-| P31 | [Reference index stores per-span entries when consumers only read distinct URIs](todo/performance.md#p31-reference-index-stores-per-span-entries-when-consumers-only-read-distinct-uris)          | Medium      | Low-Medium  |
 | P38 | [Resolved type values are duplicated ~34x](todo/performance.md#p38-resolved-type-values-are-duplicated-34x)                                                          | High        | High        |
 | P39 | [`SymbolKind` stores owned strings per span](todo/performance.md#p39-symbolkind-stores-owned-strings-per-span)                                                       | Medium      | Low-Medium  |
 | P40 | [`method_index` is a per-class `HashMap` even when the member vec is shared](todo/performance.md#p40-method_index-is-a-per-class-hashmap-even-when-the-member-vec-is-shared) | Low-Medium  | Low         |
@@ -217,6 +216,7 @@ unlikely to move the needle for most users.
 | P21 | [Offset-shifting for cached diagnostics on partial edits](todo/performance.md#p21-offset-shifting-for-cached-diagnostics-on-partial-edits)                                  | Medium      | Medium      |
 | P23 | [`workspace/symbol` lowercases every symbol name per request](todo/performance.md#p23-workspacesymbol-allocates-a-lowercase-copy-of-every-symbol-name-per-request)          | Low-Medium  | Low         |
 | P28 | [`process_assert_narrowing` clones the scope per variable for every statement](todo/performance.md#p28-process_assert_narrowing-clones-the-top-level-scope-once-per-variable-for-every-statement) | Low-Medium  | Low         |
+| P42 | [Reference index is built during the headless `analyze` pipeline but never queried there](todo/performance.md#p42-reference-index-is-built-during-the-headless-analyze-pipeline-but-never-queried-there) | Low-Medium  | Low         |
 |     | **[Indexing](todo/indexing.md)**                                                                                                                                            |             |             |
 | X3  | Completion item detail on demand (`completionItem/resolve`)                                                                                                                 | Medium      | Medium      |
 | X7  | [Recency tracking](todo/indexing.md#x7-recency-tracking)                                                                                                                    | Medium      | Medium      |

docs/todo/performance.md

@@ -831,56 +831,6 @@ waiting on the triggers above.
 
 ---
 
-## P31. Reference index stores per-span entries when consumers only read distinct URIs
-
-**Impact: Medium · Effort: Low-Medium**
-
-The cross-file reference candidate index (`reference_index`) stores
-one entry per matching span, each with its own cloned URI `String`
-plus `start`/`end` offsets and an `is_declaration` flag. It has two
-consumers: `reference_candidate_uris_for_keys` reads only the
-distinct URIs per key, and the inlay-hint reference counts
-(`ref_count` in `src/inlay_hints.rs`) count the non-declaration
-entries per key. The `start`/`end` offsets are never read
-anywhere. Class and function spans additionally fan out into up
-to three name keys each (FQN, source spelling, short name), each with
-its own entry copy. On a large project this multiplies out to
-millions of heap-allocated URI strings held for the whole session —
-against the full-index memory aim — and the index is built on every
-`update_ast` in *all* indexing strategies and in the headless
-`analyze` pipeline, where it is never queried.
-
-Two directions, pick one:
-
-1. **Shrink to what is consumed.** Store per key the deduplicated
-   URIs plus a non-declaration span count (what the inlay-hint
-   consumer needs), e.g. `HashMap<Key, HashMap<Arc<str>, u32>>`
-   with per-file URI interning so each file's URI is allocated
-   once. Drop the unused offsets. Smallest change, keeps the
-   current candidate-pruning design.
-2. **Make the spans pay for themselves.** Keep the per-span entries
-   but have find-references consume them directly (the original
-   O(k)-lookup goal), skipping the per-file symbol-map re-scan for
-   keys that cannot produce false positives (constants, Laravel
-   string keys). More work, bigger payoff.
-
-Also worth fixing while in there: building entries calls
-`file_context_at` (which clones the file's full use map) once per
-`self`/`static`/`parent` span (and in fallback paths for class and
-function spans); only the enclosing class and namespace are needed.
-Skipping the index build entirely in the `analyze` pipeline is a
-free win under either direction.
-
-**Quantified by the memory audit (see "Memory baseline" below):** on a
-large Laravel app the index holds 218,167 entries behind 74,454 distinct
-`(key, uri)` pairs — **2.9x span duplication** — costing 43 MB across
-330 K allocations. Direction 1 (dedup + count) collapses that to
-roughly the distinct-pair count, an estimated ~5 MB. Still worth doing,
-and now the third-largest single store after the resolved-class cache
-and `symbol_maps`.
-
----
-
 ## P32. Vendor package scan reads every file twice for origin classification
 
 **Impact: Medium · Effort: Low-Medium**
@@ -916,7 +866,7 @@ set after the pass is ~1.16 GB, of which the resolved-class cache is
 **~748 MB — about 65%**. Ranked by the RSS each structure releases
 when dropped: resolved-class cache ~748 MB, class indexes
 (`method_store` + `fqn_class_index` + `uri_classes_index`) ~99 MB,
-`symbol_maps` ~26 MB, `reference_index` ~14 MB (see P31). The cache
+`symbol_maps` ~26 MB, `reference_index` ~14 MB. The cache
 holds ~247 KB per resolved class.
 
 Two compounding causes:
@@ -1303,6 +1253,28 @@ the count itself has come down.
 
 ---
 
+## P42. Reference index is built during the headless `analyze` pipeline but never queried there
+
+**Impact: Low-Medium · Effort: Low**
+
+`reindex_references_for_symbol_maps_batch` (`src/reference_index.rs`)
+runs on every `update_ast`, including from the parallel index workers
+in `src/analyse/run.rs` that back the headless `analyze` CLI
+subcommand. That pipeline never calls
+`reference_candidate_uris_for_keys` or any other reader of
+`reference_index` (it has no find-references, rename, or inlay-hints
+request to serve), so every span the workers walk to build index
+entries there is wasted CPU and short-lived allocation.
+
+Skip the build in that mode: add a per-`Backend` flag (not a
+process-global static or thread-local, so test isolation between
+`Backend` instances is preserved) that `Backend::new_headless` sets
+and `reindex_references_for_symbol_maps_batch` checks before doing any
+work, so the skip applies uniformly to every caller of `update_ast` in
+headless mode rather than special-casing the `analyze` call site.
+
+---
+
 # Remaining anti-pattern fixes
 
 Most remaining depth-cap issues were addressed by eager class

src/backend/file_access.rs

@@ -7,6 +7,7 @@
 //! lock-and-unwrap boilerplate that used to be duplicated across the
 //! completion handler, definition resolver, and other consumers.
 
+use std::collections::HashMap;
 use std::sync::Arc;
 
 use tower_lsp::lsp_types::Url;
@@ -195,6 +196,44 @@ impl Backend {
         }
     }
 
+    /// Subset of [`file_context_at`](Self::file_context_at) for callers
+    /// that only need the enclosing class list and namespace (e.g.
+    /// resolving `self`/`static`/`parent`). Skips the `use`-map and
+    /// `resolved_names` clones.
+    pub(crate) fn classes_and_namespace_at(
+        &self,
+        uri: &str,
+        byte_offset: u32,
+    ) -> (Vec<Arc<ClassInfo>>, Option<String>) {
+        let classes = self
+            .symbols
+            .uri_classes_index
+            .read()
+            .get(uri)
+            .cloned()
+            .unwrap_or_default();
+        let namespace = self.namespace_at_offset(uri, byte_offset);
+        (classes, namespace)
+    }
+
+    /// Subset of [`file_context_at`](Self::file_context_at) for callers
+    /// that only need the `use`-map and namespace (e.g. resolving a bare
+    /// name to its FQN). Skips the class-list and `resolved_names` clones.
+    pub(crate) fn use_map_and_namespace_at(
+        &self,
+        uri: &str,
+        byte_offset: u32,
+    ) -> (HashMap<String, String>, Option<String>) {
+        let use_map = self
+            .file_imports
+            .read()
+            .get(uri)
+            .cloned()
+            .unwrap_or_default();
+        let namespace = self.namespace_at_offset(uri, byte_offset);
+        (use_map, namespace)
+    }
+
     /// Return the namespace that contains the given byte offset in a file.
     ///
     /// For single-namespace files (the common case) this returns the file's

src/inlay_hints.rs

@@ -403,7 +403,7 @@ impl Backend {
         self.reference_index
             .read()
             .get(key)
-            .map(|entries| entries.iter().filter(|e| !e.is_declaration).count())
+            .map(|entries| entries.values().map(|&count| count as usize).sum())
             .unwrap_or(0)
     }
 

src/mem_audit.rs

@@ -1311,58 +1311,46 @@ pub(crate) fn report(backend: &Backend, runner_content_bytes: usize) {
     );
 
     // ── 6. Reference index ──────────────────────────────────────────
+    // Per key, only the distinct contributing URIs are stored (as a
+    // shared per-file `Arc<str>`) mapped to a non-declaration span
+    // count, so there is no per-span duplication left to account for.
     let mut refs = Sz::default();
-    let mut n_entries = 0usize;
+    let mut n_key_uri_pairs = 0usize;
     let n_keys;
+    let mut distinct_uris: HashSet<*const u8> = HashSet::new();
     {
         let idx = backend.reference_index.read();
         let (by_key, uri_keys) = idx.audit_maps();
         n_keys = by_key.len();
-        refs +=
-            map_buckets::<crate::reference_index::ReferenceIndexKey, Vec<u8>>(by_key.capacity());
-        for (k, v) in by_key {
+        refs += map_buckets::<crate::reference_index::ReferenceIndexKey, HashMap<Arc<str>, u32>>(
+            by_key.capacity(),
+        );
+        for (k, inner) in by_key {
             refs.add(k.audit_heap());
-            n_entries += v.len();
-            refs.add(v.capacity() * size_of::<crate::reference_index::ReferenceIndexEntry>());
-            for e in v {
-                refs.add(e.uri.capacity());
+            n_key_uri_pairs += inner.len();
+            refs += map_buckets::<Arc<str>, u32>(inner.capacity());
+            for uri in inner.keys() {
+                distinct_uris.insert(Arc::as_ptr(uri).cast::<u8>());
             }
         }
-        refs += map_buckets::<String, Vec<crate::reference_index::ReferenceIndexKey>>(
+        refs += map_buckets::<Arc<str>, Vec<crate::reference_index::ReferenceIndexKey>>(
             uri_keys.capacity(),
         );
         for (uri, keys) in uri_keys {
-            refs.add(uri.capacity());
+            refs.add(ARC + uri.len());
             refs.add(keys.capacity() * size_of::<crate::reference_index::ReferenceIndexKey>());
             for k in keys {
                 refs.add(k.audit_heap());
             }
         }
     }
-    // How much of the index survives P31 direction 1: per key, keep the
-    // distinct URIs plus a non-declaration count, drop the never-read
-    // offsets and the per-span duplication.
-    let mut distinct_pairs = 0usize;
-    {
-        let idx = backend.reference_index.read();
-        let (by_key, _) = idx.audit_maps();
-        let mut uris: HashSet<&str> = HashSet::new();
-        for entries in by_key.values() {
-            uris.clear();
-            for e in entries {
-                uris.insert(e.uri.as_str());
-            }
-            distinct_pairs += uris.len();
-        }
-    }
     eprintln!(
-        "── reference_index: {} keys, {} entries, {:.1} MB ({} allocs) | distinct (key, uri) pairs: {} ({:.1}x span duplication)",
+        "── reference_index: {} keys, {} (key, uri) pairs, {} distinct uris, {:.1} MB ({} allocs)",
         n_keys,
-        n_entries,
+        n_key_uri_pairs,
+        distinct_uris.len(),
         mb(refs.bytes),
         refs.allocs,
-        distinct_pairs,
-        n_entries as f64 / distinct_pairs.max(1) as f64,
     );
 
     // ── 7. Remaining session stores ─────────────────────────────────
@@ -1569,7 +1557,7 @@ pub(crate) fn report(backend: &Backend, runner_content_bytes: usize) {
     probe("file_imports", &mut || backend.file_imports.write().clear());
     probe("symbol_maps", &mut || backend.symbol_maps.write().clear());
     probe("reference_index", &mut || {
-        let uris: Vec<String> = backend
+        let uris: Vec<Arc<str>> = backend
             .reference_index
             .read()
             .audit_maps()