Project startup is roughly three times faster

Author: AJenbo

docs/CHANGELOG.md

@@ -55,6 +55,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - **Repeated class lookups are remembered instead of searched again.** Answering "which class does this name refer to?" is the single most frequent thing PHPantom does: analysing a large Laravel project asks it millions of times, over only a few thousand distinct classes. Every question used to hash the name and take a read lock on two shared indexes, so with one worker per CPU core the workers spent much of their time queueing behind each other instead of analysing. Each worker now remembers the answers it has already looked up and drops them the moment the class indexes change, so a repeated question costs a single pointer comparison. Whole-project analysis is 8-12% faster on large Laravel projects and uses up to a third less CPU, with identical results and no measurable change in memory use. Hover, completion, and go-to-definition resolve names through the same path and get the same saving.
 - **Vendor package scanning no longer reads every file twice.** Startup used to scan each vendor file once to find its classes, functions, and constants, then read and scan it again just to classify which package it came from for completion ranking. Each file's package is now known before it is scanned, so both are done in a single parallel pass, roughly halving the I/O and CPU cost of the vendor scan.
 - **Whole-workspace class pre-resolution now spreads across cores.** Resolving every known class in dependency order after indexing ran on a single core, leaving a single-threaded pause between the parallel index and the parallel diagnostic pass that grew with the number of classes in the project. That work is now shared across a pool of workers, which cuts the phase to roughly a quarter of its previous time on large Laravel projects and takes close to 20% off whole-project analysis on the largest of them. The editor's post-startup pre-resolution goes through the same path, so a large project becomes fully warm sooner. Results are unchanged.
+- **Project startup is roughly three times faster.** Building the class index at startup was the least parallel part of a run: it read every project and vendor file through a memory map, and mapping a file takes a lock that the whole process shares, so the more cores read files at once the longer each of them waited. Files are now read normally (only genuinely large ones are still mapped), the per-package directory walks run in parallel instead of one package after another, and both the file scan and the scan of a bundled `.phar` tool hand work out in small batches so no core sits idle behind a batch of large files. On large Laravel projects the startup phase drops from around 0.6s to 0.2s, taking 15-37% off whole-project analysis time across every project measured and up to 19% off its CPU use. The editor goes through the same pipeline, so it becomes usable sooner. Results are unchanged.
 - **The `analyze` and `fix` CLI subcommands no longer build the cross-file reference index.** That index only serves Find References, Rename, and reference-count inlay hints, none of which the CLI subcommands ever query, so every file they parsed was still paying to populate it. Skipping the build removes that wasted CPU and short-lived allocation from whole-project `analyze` and `fix` runs; the editor's LSP session is unaffected.
 
 ### Removed

docs/todo.md

@@ -197,7 +197,7 @@ unlikely to move the needle for most users.
 |     | **[Performance](todo/performance.md)**                                                                                       |             |             |
 | P46 | [`mago-phpdoc-syntax` cannot parse `@method static (…) name()`](todo/performance.md#p46-mago-phpdoc-syntax-cannot-parse-method-static--name)                                | Low         | Low         |
 | P30 | [Evaluate migrating parse/resolve/docblock pipeline to `mago-hir`](todo/performance.md#p30-evaluate-migrating-parseresolvedocblock-pipeline-to-mago-hir) (parked — re-evaluated at mago 1.45.0, still no `mago-hir` consumers upstream) | Medium-High | High        |
-| P43 | [`init_single_project` is the longest single-threaded stretch of a run](todo/performance.md#p43-init_single_project-is-the-longest-single-threaded-stretch-of-a-run)        | Medium-High | Medium      |
+| P48 | [Project init's remaining serial stretches](todo/performance.md#p48-project-inits-remaining-serial-stretches)                                                                | Low-Medium  | Medium      |
 | P47 | [The resolved-class cache lock caps concurrent class resolution](todo/performance.md#p47-the-resolved-class-cache-lock-caps-concurrent-class-resolution)                     | Medium      | Medium-High |
 | P16 | [Pre-parsed stub format (eliminate raw PHP embedding)](todo/performance.md#p16-pre-parsed-stub-format-eliminate-raw-php-embedding)                                          | High        | Medium-High |
 | P25 | [`type_mismatch_argument` / `argument_count_mismatch` slow on large single files](todo/performance.md#p25-type_mismatch_argument-argument_count_mismatch-slow-on-large-single-files) | Medium      | Medium      |

docs/todo/performance.md

@@ -1049,34 +1049,60 @@ real ceiling than the hashing was.
 
 ---
 
-## P43. `init_single_project` is the longest single-threaded stretch of a run
+## P48. Project init's remaining serial stretches
 
-**Impact: Medium-High · Effort: Medium**
+**Impact: Low-Medium · Effort: Medium**
 
-With the diagnostic pass now reaching ~22 of 32 cores (P35), project
-init is the least parallel phase left and the largest remaining share
-of wall clock. Per-phase timing of `analyze` on a large Laravel
-project (32-core machine, release build, warm page cache):
+`init_single_project` is down from 0.60 s at 3.2 of 32 cores to 0.21 s
+at 6.2 (mmap contention, per-package collection, and the batch scanners'
+static chunking are fixed), so the diagnostic pass is once again the
+largest share of wall clock. What is left inside init, measured on a
+large Laravel project (32-core machine, release build, warm page cache,
+`utime + stime` from `/proc/self/stat` at each boundary):
 
-| Phase | Wall | Avg cores |
+| Step | Wall | Avg cores |
 | --- | --- | --- |
-| `init_single_project` | 1.09 s | 2.3 |
-| `discover_user_files` | 0.03 s | 0.8 |
-| Phase 1 index | 0.10 s | 7.1 |
-| Phase 1.5 eager populate | 0.22 s | 1.0 |
-| Phase 2 diagnostics | 1.31 s | 22.6 |
-
-Init is ~40% of the run at barely two cores. It covers composer
-reading, autoload/classmap scanning, stub setup, and the vendor
-package scan, and it gates everything after it, so the same stretch is
-in front of the LSP's time-to-usable as well as the CLI's. Worth a
-per-phase breakdown inside init before choosing a fix — P34 (eager
-population is single-threaded, the 1.0-core row above) is an
-already-filed piece of the same window.
+| vendor package collection (`collect_package_files`) | 0.060 s | 6.8 |
+| vendor file scan (`scan_files_parallel_full`) | 0.033 s | 20.5 |
+| `scan_autoload_files` | 0.038 s | 3.9 |
+| `classify_class_origin` + `fqn_uri_index` fill | 0.019 s | 1.6 |
+| workspace walk (`ignore` crate) | 0.018 s | 1.1 |
+| `vendor_package_roots` | 0.006 s | 1.0 |
+
+Each is small on its own; together they are most of what is left:
+
+- **Vendor collection is per-package parallel, but one package is the
+  straggler.** `laravel/framework`'s PSR-4 tree is a single work item, so
+  the tail of the walk runs on one core. Splitting a package's PSR-4
+  roots (or its subdirectories) into separate work items would even it
+  out, but the concatenation has to stay in `installed.json` order for
+  duplicate FQNs to resolve to the same file.
+- **Class origins are derived twice.** The vendor scan knows which
+  package each file came from, and records it for functions and constants
+  (`WorkspaceScanResult::function_origins` / `constant_origins`), but not
+  for classes. Init then re-derives every vendor class's origin by
+  prefix-matching its path against ~400 package roots, single-threaded.
+  Carrying class origins out of the scan the way the other two already
+  are would delete the step. The Composer-classmap path (`strategy =
+  "composer"`) has no origin for its entries, so the prefix match has to
+  stay as the fallback there.
+- **`vendor_package_roots` re-reads and re-parses `installed.json`**
+  moments after `scan_vendor_packages_with_skip` parsed the same file,
+  and canonicalizes every package path a second time.
+- **The workspace walk is single-threaded.** `ignore::WalkBuilder` has
+  `build_parallel`, but the walk order feeds the duplicate-FQN
+  tie-break, so a parallel walk needs its results sorted to stay
+  deterministic.
+- **Reading and manifest-parsing a bundled `.phar` is serial.** With
+  phpstan installed this is 17 ms of the 24 ms `require_once` sweep; the
+  class scan inside the archive is already parallel.
 
 Reproduce with the CPU-sampling loop in the Appendix, or by reading
 `utime + stime` from `/proc/self/stat` at each phase boundary and
-dividing by the phase's wall time.
+dividing by the phase's wall time. Per-phase numbers for the run as a
+whole (same project): init 0.21 s at 6.2 cores, `discover_user_files`
+0.02 s at 1.1, Phase 1 index 0.02 s at 21.5, Phase 1.5 eager populate
+0.05 s at 4.4, Phase 2 diagnostics 0.47 s at 20.9.
 
 ---