Vendor package scanning no longer reads every file twice

Author: AJenbo

docs/CHANGELOG.md

@@ -53,6 +53,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - **Faster diagnostics on large projects.** Several diagnostic checks (by-reference parameter detection, the `Stringable`-to-`string` acceptance check, and `model-property<Model>` literal validation) now read a class's inheritance from the resolved-class cache instead of re-merging traits, parent classes, and generics from scratch on every call. On large Laravel projects this removed a measurable share of the diagnostic pass's CPU time, and as a side effect these checks now also see interface-declared members (e.g. a `__toString` declared only on an implemented interface).
 - **Laravel string keys are collected once instead of once per CPU core.** Checking a `route()`, `config()`, `view()`, or `__()` key needs the project's full list of valid keys, and building each list walks the project directory. The diagnostic pass runs one worker per core, so all of them used to reach the empty list at the same moment and every one repeated the same walk, leaving most cores waiting on the disk rather than analysing. Each list is now built once and shared, which roughly doubles the number of cores the diagnostic pass keeps busy and cuts whole-project analysis time by around 15% on large Laravel projects.
 - **Repeated class lookups are remembered instead of searched again.** Answering "which class does this name refer to?" is the single most frequent thing PHPantom does: analysing a large Laravel project asks it millions of times, over only a few thousand distinct classes. Every question used to hash the name and take a read lock on two shared indexes, so with one worker per CPU core the workers spent much of their time queueing behind each other instead of analysing. Each worker now remembers the answers it has already looked up and drops them the moment the class indexes change, so a repeated question costs a single pointer comparison. Whole-project analysis is 8-12% faster on large Laravel projects and uses up to a third less CPU, with identical results and no measurable change in memory use. Hover, completion, and go-to-definition resolve names through the same path and get the same saving.
+- **Vendor package scanning no longer reads every file twice.** Startup used to scan each vendor file once to find its classes, functions, and constants, then read and scan it again just to classify which package it came from for completion ranking. Each file's package is now known before it is scanned, so both are done in a single parallel pass, roughly halving the I/O and CPU cost of the vendor scan.
 
 ### Removed
 

docs/todo.md

@@ -209,7 +209,6 @@ unlikely to move the needle for most users.
 | P15 | [Two-phase stub index construction (eliminate `RwLock` on stub maps)](todo/performance.md#p15-two-phase-stub-index-construction-eliminate-rwlock-on-stub-maps)              | Low         | Medium      |
 | P17 | [`mago-names` resolution on the parse hot path](todo/performance.md#p17-mago-names-resolution-on-the-parse-hot-path)                                                        | Medium      | Low         |
 | P18 | [Subtype result caching](todo/performance.md#p18-subtype-result-caching) (per-request HashMap for hierarchy walks)                                                          | Medium      | Low         |
-| P32 | [Vendor package scan reads every file twice for origin classification](todo/performance.md#p32-vendor-package-scan-reads-every-file-twice-for-origin-classification)        | Medium      | Low-Medium  |
 | P20 | [Content-hash gated resolution cache persistence](todo/performance.md#p20-content-hash-gated-resolution-cache-persistence)                                                  | Medium      | Medium      |
 | P21 | [Offset-shifting for cached diagnostics on partial edits](todo/performance.md#p21-offset-shifting-for-cached-diagnostics-on-partial-edits)                                  | Medium      | Medium      |
 | P23 | [`workspace/symbol` lowercases every symbol name per request](todo/performance.md#p23-workspacesymbol-allocates-a-lowercase-copy-of-every-symbol-name-per-request)          | Low-Medium  | Low         |

docs/todo/performance.md

@@ -816,30 +816,6 @@ waiting on the triggers above.
 
 ---
 
-## P32. Vendor package scan reads every file twice for origin classification
-
-**Impact: Medium · Effort: Low-Medium**
-
-`scan_vendor_packages_with_skip` (`src/classmap_scanner.rs`) scans the
-collected vendor files twice on every startup:
-
-1. `scan_files_parallel_full(&all_files, …)` reads and byte-scans every
-   file in parallel to build the classmap/function/constant indices.
-2. A sequential origin-classification pass then **re-reads and
-   re-scans the same files** — `read_for_scan` + `scan_content` per
-   PSR-4 file, and a full `scan_file_full` per plain file — solely to
-   map each discovered symbol to its package's completion-origin tier.
-
-That doubles the I/O and scan cost of the vendor scan, and the second
-pass is single-threaded. The origin is known per *file* before either
-pass runs (it comes from the package entry the file was collected
-under), so the classification can be produced during the parallel scan:
-thread the per-file origin through `scan_files_parallel_full` (e.g.
-scan `(PathBuf, origin)` pairs and emit origins alongside symbols)
-and delete the second pass entirely.
-
----
-
 ## P34. Eager class population is single-threaded
 
 **Impact: Medium · Effort: Medium**
@@ -1061,10 +1037,9 @@ Init is ~40% of the run at barely two cores. It covers composer
 reading, autoload/classmap scanning, stub setup, and the vendor
 package scan, and it gates everything after it, so the same stretch is
 in front of the LSP's time-to-usable as well as the CLI's. Worth a
-per-phase breakdown inside init before choosing a fix — P32 (vendor
-scan reads every file twice) and P34 (eager population is
-single-threaded, the 1.0-core row above) are both already-filed pieces
-of the same window.
+per-phase breakdown inside init before choosing a fix — P34 (eager
+population is single-threaded, the 1.0-core row above) is an
+already-filed piece of the same window.
 
 Reproduce with the CPU-sampling loop in the Appendix, or by reading
 `utime + stime` from `/proc/self/stat` at each phase boundary and

src/classmap_scanner/discovery.rs

@@ -403,48 +403,20 @@ pub fn scan_vendor_packages_with_skip(
         }
     }
 
-    // Phase 2: scan all collected files in parallel
-    let mut all_files: Vec<PathBuf> = psr4_files.iter().map(|(path, _, _)| path.clone()).collect();
-    all_files.extend(plain_files.iter().map(|(path, _)| path.clone()));
-
-    // The origin classification pass below re-reads every file, so it
-    // counts as its own work units.
-    progress_add_total(
-        progress,
-        all_files.len() + psr4_files.len() + plain_files.len(),
-    );
-
-    let mut result = scan_files_parallel_full(&all_files, progress);
-    let mut class_origins = HashMap::new();
-    let mut function_origins = HashMap::new();
-    let mut constant_origins = HashMap::new();
-    for (path, expected_fqn, origin) in psr4_files {
-        progress_add_done(progress);
-        if let Ok(content) = read_for_scan(&path) {
-            for fqn in scan_content(&content) {
-                if fqn == expected_fqn {
-                    class_origins.entry(fqn).or_insert(origin);
-                }
-            }
-        }
-    }
-    for (path, origin) in plain_files {
-        progress_add_done(progress);
-        let symbols = super::scan_file_full(&path);
-        for fqn in symbols.classes {
-            class_origins.entry(fqn).or_insert(origin);
-        }
-        for fqn in symbols.functions {
-            function_origins.entry(fqn).or_insert(origin);
-        }
-        for name in symbols.constants {
-            constant_origins.entry(name).or_insert(origin);
-        }
-    }
-    result.class_origins = class_origins;
-    result.function_origins = function_origins;
-    result.constant_origins = constant_origins;
-    result
+    // Phase 2: scan all collected files in parallel. Each file's origin is
+    // already known from the package it was collected under (phase 1), so
+    // it travels alongside the path and gets attached to the functions and
+    // constants discovered in the same read, instead of re-reading every
+    // file in a second sequential pass just to classify it.
+    let mut all_files: Vec<(PathBuf, crate::ClassCompletionOrigin)> = psr4_files
+        .into_iter()
+        .map(|(path, _, origin)| (path, origin))
+        .collect();
+    all_files.extend(plain_files);
+
+    progress_add_total(progress, all_files.len());
+
+    scan_files_parallel_full(&all_files, progress)
 }
 
 /// Scan all `.php` files under the workspace root using the PSR-4
@@ -609,8 +581,14 @@ fn scan_files_parallel_psr4(
 
 /// Scan a batch of files for all symbols (classes, functions, constants)
 /// in parallel and return a [`WorkspaceScanResult`].
+///
+/// Each file carries its own completion-origin tier, which is attached to
+/// any function or constant discovered in it. This lets callers that know
+/// a file's package provenance up front (e.g. vendor package scanning)
+/// classify symbols in the same read/scan pass instead of re-reading every
+/// file afterwards just to determine its origin.
 fn scan_files_parallel_full(
-    files: &[PathBuf],
+    files: &[(PathBuf, crate::ClassCompletionOrigin)],
     progress: Option<&ScanProgress>,
 ) -> WorkspaceScanResult {
     if files.is_empty() {
@@ -620,7 +598,7 @@ fn scan_files_parallel_full(
     // Small batches: sequential
     if files.len() <= 4 {
         let mut result = WorkspaceScanResult::default();
-        for path in files {
+        for (path, origin) in files {
             progress_add_done(progress);
             if let Ok(content) = read_for_scan(path) {
                 let scan = super::find_symbols(&content);
@@ -642,14 +620,16 @@ fn scan_files_parallel_full(
                 for fqn in scan.functions {
                     result
                         .function_index
-                        .entry(fqn)
+                        .entry(fqn.clone())
                         .or_insert_with(|| path.clone());
+                    result.function_origins.entry(fqn).or_insert(*origin);
                 }
                 for name in scan.constants {
                     result
                         .constant_index
-                        .entry(name)
+                        .entry(name.clone())
                         .or_insert_with(|| path.clone());
+                    result.constant_origins.entry(name).or_insert(*origin);
                 }
             }
         }
@@ -659,42 +639,44 @@ fn scan_files_parallel_full(
     let n_threads = thread_count().min(files.len());
     let chunk_size = files.len().div_ceil(n_threads);
 
-    let results: Vec<Vec<(ScanResult, PathBuf)>> = std::thread::scope(|s| {
-        let handles: Vec<_> = files
-            .chunks(chunk_size)
-            .map(|chunk| {
-                s.spawn(move || {
-                    let mut local: Vec<(ScanResult, PathBuf)> = Vec::new();
-                    for path in chunk {
-                        progress_add_done(progress);
-                        if let Ok(content) = read_for_scan(path) {
-                            let scan = super::find_symbols(&content);
-                            if !scan.classes.is_empty()
-                                || !scan.functions.is_empty()
-                                || !scan.constants.is_empty()
-                            {
-                                local.push((scan, path.clone()));
+    let results: Vec<Vec<(ScanResult, PathBuf, crate::ClassCompletionOrigin)>> =
+        std::thread::scope(|s| {
+            let handles: Vec<_> = files
+                .chunks(chunk_size)
+                .map(|chunk| {
+                    s.spawn(move || {
+                        let mut local: Vec<(ScanResult, PathBuf, crate::ClassCompletionOrigin)> =
+                            Vec::new();
+                        for (path, origin) in chunk {
+                            progress_add_done(progress);
+                            if let Ok(content) = read_for_scan(path) {
+                                let scan = super::find_symbols(&content);
+                                if !scan.classes.is_empty()
+                                    || !scan.functions.is_empty()
+                                    || !scan.constants.is_empty()
+                                {
+                                    local.push((scan, path.clone(), *origin));
+                                }
                             }
                         }
-                    }
-                    local
+                        local
+                    })
                 })
-            })
-            .collect();
-        handles
-            .into_iter()
-            .map(|h| {
-                h.join().unwrap_or_else(|_| {
-                    tracing::error!("PHPantom: thread panic in scan_files_parallel_full");
-                    Vec::new()
+                .collect();
+            handles
+                .into_iter()
+                .map(|h| {
+                    h.join().unwrap_or_else(|_| {
+                        tracing::error!("PHPantom: thread panic in scan_files_parallel_full");
+                        Vec::new()
+                    })
                 })
-            })
-            .collect()
-    });
+                .collect()
+        });
 
     let mut result = WorkspaceScanResult::default();
     for batch in results {
-        for (scan, path) in batch {
+        for (scan, path, origin) in batch {
             for fqcn in scan.classes {
                 let class_short_name = fqcn_short_name(&fqcn).to_owned();
                 result
@@ -719,14 +701,16 @@ fn scan_files_parallel_full(
             for fqn in scan.functions {
                 result
                     .function_index
-                    .entry(fqn)
+                    .entry(fqn.clone())
                     .or_insert_with(|| path.clone());
+                result.function_origins.entry(fqn).or_insert(origin);
             }
             for name in scan.constants {
                 result
                     .constant_index
-                    .entry(name)
+                    .entry(name.clone())
                     .or_insert_with(|| path.clone());
+                result.constant_origins.entry(name).or_insert(origin);
             }
         }
     }
@@ -778,11 +762,11 @@ pub fn scan_workspace_fallback_full(
         })
         .build();
 
-    let mut php_files: Vec<PathBuf> = Vec::new();
+    let mut php_files: Vec<(PathBuf, crate::ClassCompletionOrigin)> = Vec::new();
     for entry in walker.flatten() {
         let path = entry.path();
         if path.is_file() && path.extension().is_some_and(|ext| ext == "php") {
-            php_files.push(path.to_path_buf());
+            php_files.push((path.to_path_buf(), crate::ClassCompletionOrigin::Project));
         }
     }
 
@@ -822,7 +806,7 @@ pub fn scan_drupal_directories(
         "sites",
     ];
 
-    let mut php_files: Vec<PathBuf> = Vec::new();
+    let mut php_files: Vec<(PathBuf, crate::ClassCompletionOrigin)> = Vec::new();
 
     for rel in &drupal_dirs {
         let dir = web_root.join(rel);
@@ -855,7 +839,7 @@ pub fn scan_drupal_directories(
         for entry in walker.flatten() {
             let path = entry.path();
             if path.is_file() && is_drupal_php_file(path) {
-                php_files.push(path.to_path_buf());
+                php_files.push((path.to_path_buf(), crate::ClassCompletionOrigin::Project));
             }
         }
     }

src/classmap_scanner/mod.rs

@@ -152,8 +152,6 @@ pub struct ScanResult {
 pub struct WorkspaceScanResult {
     /// FQN → file path for classes, interfaces, traits, and enums.
     pub classmap: HashMap<String, PathBuf>,
-    /// FQN → completion origin tier.
-    pub(crate) class_origins: HashMap<String, crate::ClassCompletionOrigin>,
     /// FQN → file path for standalone functions.
     pub function_index: HashMap<String, PathBuf>,
     /// FQN → completion origin tier for standalone functions.