Merge composer and self scanner, less misses and fater

Author: AJenbo

docs/ARCHITECTURE.md

@@ -567,7 +567,7 @@ At resolution time, `merge_traits_into` loads the `UnitEnum` or `BackedEnum` stu
 
 PSR-4 mappings come exclusively from the project's own `composer.json`. Vendor PSR-4 (`vendor/composer/autoload_psr4.php`) is not loaded. The Composer classmap is the sole source of truth for vendor code.
 
-**Design principle:** if the classmap is missing or stale, vendor classes fail to resolve visibly rather than being silently papered over by PSR-4. This makes the problem obvious to the user (fix: run `composer dump-autoload`). User PSR-4 roots are walked by Go-to-implementation (Phase 5) and Find References because user files change between `dump-autoload` runs.
+**Design principle:** vendor PSR-4 mappings are not loaded separately. The Composer classmap (augmented by the self-scanner, see below) is the sole source of truth for vendor code. User PSR-4 roots are walked by Go-to-implementation (Phase 5) and Find References because user files change between `dump-autoload` runs.
 
 ### Self-Generated Classmap
 
@@ -577,16 +577,18 @@ Scanning is parallelised using a two-phase approach: directory walks collect fil
 
 The indexing strategy is configurable via `[indexing] strategy` in `.phpantom.toml`:
 
-- **`"composer"`** (default) — use Composer's classmap when available and complete; fall back to self-scan when it is missing or incomplete (i.e. the project's own PSR-4 namespaces have no entries in the classmap).
-- **`"self"`** — always self-scan, ignoring Composer's classmap entirely.
+- **`"composer"`** (default) — merged classmap + self-scan. Load Composer's classmap (if it exists) as a skip set, then self-scan all PSR-4 and vendor directories for anything the classmap missed. Whatever the classmap already covers is a free performance win; whatever it's missing, we find ourselves. No completeness heuristic needed.
+- **`"self"`** — always self-scan, ignoring Composer's classmap entirely. Equivalent to the merged approach with an empty skip set.
 - **`"full"`** — same as `"self"` for now; reserved for future background indexing.
-- **`"none"`** — no proactive scanning; uses Composer's classmap if present but never falls back to self-scan.
+- **`"none"`** — no proactive scanning; uses Composer's classmap if present but never self-scans to fill gaps.
+
+The merged pipeline works in three steps: (1) load `autoload_classmap.php` into a `HashMap<String, PathBuf>`, (2) collect the classmap's file paths into a `HashSet<PathBuf>` skip set, (3) self-scan all PSR-4 and vendor directories, skipping files already in the skip set. The result is a merged index: classmap entries for everything Composer already knew about, plus self-scanned entries for everything it missed. When the classmap is complete (the common case), the self-scanner walks directories but skips every file, finishing almost instantly. When the classmap is empty or absent, it falls back to a full self-scan. When the classmap is partial (e.g. vendor classes only), vendor files are skipped and only user code is scanned. Every state of the classmap helps.
 
 When self-scanning with a `composer.json` present, the scanner reads `autoload.psr-4`, `autoload-dev.psr-4`, `autoload.classmap`, and `autoload-dev.classmap` to determine which directories to walk. PSR-4 directories are filtered: only classes whose FQN matches the namespace prefix plus the relative file path are included. Vendor packages are discovered from `vendor/composer/installed.json` (both Composer 1 and 2 formats); the JSON packages array is borrowed rather than cloned to avoid allocating a copy of the entire vendor manifest. All directory walkers (full-scan, PSR-4 scanner, vendor package scanner, and go-to-implementation file collector) use the `ignore` crate for gitignore-aware traversal. Hidden directories are skipped automatically, and `.gitignore` rules are respected at every level. When no `composer.json` exists at all, the scanner falls back to walking all `.php` files under the workspace root.
 
 The result is a `HashMap<String, PathBuf>` in the same format as the existing `Backend.classmap`. Everything downstream (resolution, diagnostics, go-to-definition) works unchanged.
 
-**Redundant I/O elimination:** `init_single_project` parses `composer.json` once and passes the pre-parsed `serde_json::Value` to both `build_self_scan_composer` and `is_classmap_incomplete_with_json`. Previously each function re-read and re-parsed the file independently.
+**Redundant I/O elimination:** `init_single_project` parses `composer.json` once and passes the pre-parsed `serde_json::Value` to `build_self_scan_composer`. Previously each function re-read and re-parsed the file independently.
 
 **Vendor dir detection:** the `config.vendor-dir` setting is read from `composer.json` once during `initialized` (via `parse_composer_json`, which returns both the PSR-4 mappings and the vendor dir name). The absolute vendor directory path is cached on `Backend.vendor_dir_paths` and a `file://` URI prefix is stored in `Backend.vendor_uri_prefixes` for fast vendor-file detection at runtime. Both fields are `Vec`s to support monorepo workspaces with multiple subprojects (see below). For single-project workspaces, each collection has exactly one entry.
 

docs/CHANGELOG.md

@@ -52,9 +52,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - **Two-phase diagnostic publishing.** Cheap diagnostics (unused import dimming, deprecated strikethrough) are published immediately. Expensive diagnostics (unknown classes, unknown members, unknown functions) arrive in a second pass. Previously all diagnostics waited for the slowest collector to finish, so unused imports could take several seconds to grey out in large files.
 - **Concurrent read access to shared state.** All read-heavy maps now use `parking_lot::RwLock` instead of `std::sync::Mutex`, allowing multiple requests to read in parallel.
 - **Parallel workspace indexing.** Find References and other workspace-wide operations now parse files across multiple CPU cores. The workspace walk respects `.gitignore` rules instead of hardcoding directory names to skip. Self-scan classmap building, PSR-4 directory scanning, and vendor package scanning now read and scan files in parallel across all CPU cores. The byte-level PHP scanner uses `memchr` SIMD acceleration to skip comments, strings, and heredocs instead of scanning byte-by-byte. Redundant `composer.json` reads during initialization have been eliminated.
+- **Merged classmap + self-scan pipeline.** The Composer classmap and the self-scanner are no longer mutually exclusive. PHPantom always loads the Composer classmap (when present) and then self-scans all PSR-4 and vendor directories, skipping files the classmap already covers. Stale or incomplete classmaps are no longer a problem: whatever Composer already knew about is a free performance win, and whatever it missed is discovered automatically. The fragile completeness heuristic that decided whether to trust the classmap has been removed. Monorepo subprojects use the same merged pipeline.
 
 ### Fixed
 
+- **Parallel file scanner panics no longer crash the server.** If a thread panics during workspace indexing (e.g. due to a malformed PHP file), the panic is caught and logged instead of killing the entire LSP process. Previously a single thread failure during class, PSR-4, or full-symbol scanning would propagate the panic and silently terminate PHP intelligence in the editor.
 - **Type alias array shape diagnostics no longer fire on object values.** When a method returns a `@phpstan-type` alias that expands to an array shape containing object values (e.g. `array{pen: Pen}`), accessing a method on the object value no longer triggers a false diagnostic. Type aliases are now expanded before walking array shape segments in the resolution pipeline.
 - **Inline array-element function calls resolve correctly in diagnostics.** `end($obj->items)->method()` no longer produces a false "unknown member" diagnostic. Previously the argument text for property access expressions was lost during symbol map extraction, causing the resolver to fall back to the native `mixed|false` return type instead of extracting the element type from the array's generic annotation.
 - **Eloquent Builder scope chain diagnostics no longer flicker.** Scope methods on Builder chains (e.g. `Bakery::where(...)->fresh()->get()`) no longer produce false diagnostics that flip on and off between edits. The diagnostic now checks the pre-resolved class returned by the completion pipeline before re-resolving through the cache, preventing stale cache entries from hiding context-specific virtual members.

docs/todo.md

@@ -32,17 +32,6 @@ with each step.
 
 ---
 
-## Sprint 2 — Stabilise for 0.5.0 release
-
-| # | Item | Effort | Domain | Doc Link |
-|---|---|---|---|---|
-| 106 | Thread panic in parallel file scanners crashes server | Low | Bug Fixes | [bugs.md §0](todo/bugs.md#0-thread-panic-in-parallel-file-scanners-crashes-the-server) |
-| 107 | Analyse sequential write-lock acquisition in `parse_and_cache_content_versioned` | Low | Bug Fixes | [bugs.md §1](todo/bugs.md#1-sequential-write-lock-acquisition-in-parse_and_cache_content_versioned) |
-
-**After Sprint 2:** Tag 0.5.0, write release notes, publish.
-
----
-
 ## Sprint 3 — Quick wins: close the visible gaps
 
 Every item here is Low or Low-Medium effort and directly removes
@@ -64,7 +53,6 @@ feature surface grows.
 | 22 | Selection Ranges (`textDocument/selectionRange`) | Low | LSP Features | [lsp-features.md §13](todo/lsp-features.md#13-selection-ranges-textdocumentselectionrange) |
 | 81 | Work-done progress for GTI and Find References | Low | LSP Features | [lsp-features.md §18](todo/lsp-features.md#18-work-done-progress-for-gti-and-find-references) |
 | 101 | Argument count diagnostic | Low | Diagnostics | [diagnostics.md §7](todo/diagnostics.md#7-argument-count-diagnostic) |
-| 105 | Merged classmap + self-scan | Low | Indexing | [indexing.md §1.5](todo/indexing.md#phase-15-merged-classmap--self-scan) |
 | 88 | Early-exit and `Cow` return in `apply_substitution` | Low | Performance | [performance.md §7](todo/performance.md#7-recursive-string-substitution-in-apply_substitution) |
 | 87 | Reference-counted `ClassInfo` (`Arc<ClassInfo>`) | Medium | Performance | [performance.md §2](todo/performance.md#2-reference-counted-classinfo-arcclassinfo) |
 

docs/todo/bugs.md

@@ -14,71 +14,7 @@ Items are ordered by **impact** (descending), then **effort** (ascending).
 
 ---
 
-## 0. Thread panic in parallel file scanners crashes the server
-
-**Impact: High — Effort: Low**
-
-The three `scan_files_parallel_*` functions in `classmap_scanner.rs`
-join spawned threads with `.join().unwrap()`. If any spawned thread
-panics (e.g. due to a malformed file encountered during the byte-level
-scan), the `unwrap()` propagates the panic to the caller. Because
-these functions run during server initialization (outside any
-`catch_unwind` guard), a single thread panic kills the entire LSP
-process. The user's editor silently loses PHP intelligence with no
-error message.
-
-The `scan_content`, `find_classes`, and `find_symbols` functions
-process arbitrary user and vendor files from disk. While they are
-designed to be robust, a panic in any of them (or in future code
-changes) would be fatal.
-
-### Fix
-
-Replace `.join().unwrap()` with `.join().unwrap_or_default()` (or
-`.join().ok()` with appropriate flattening) and log an error for
-any thread that panicked. The affected scan results are simply
-skipped, matching the existing behaviour of `fs::read` failures
-which are silently ignored.
-
-Three call sites:
-- `scan_files_parallel_classes` (L407)
-- `scan_files_parallel_psr4` (L467)
-- `scan_files_parallel_full` (L537)
-
----
-
-## 1. Sequential write-lock acquisition in `parse_and_cache_content_versioned`
-
-**Impact: Low-Medium — Effort: Low**
-
-`parse_and_cache_content_versioned` in `resolution.rs` acquires write
-locks on `ast_map`, `use_map`, `namespace_map`, `fqn_index`, and
-`resolved_class_cache` in sequence. Each lock is acquired and released
-individually (not nested), so there is no deadlock risk with the
-current `parking_lot` implementation. However, holding one write lock
-while acquiring the next creates brief windows where readers of the
-earlier-released lock see updated data but readers of the
-not-yet-written lock see stale data.
-
-In practice this is unlikely to cause user-visible issues because
-all five writes complete within microseconds and all consumers
-re-read the maps on each request. But the pattern is fragile: if a
-future change adds a reader that checks two of these maps for
-consistency within the same request, it could observe a partial
-update.
-
-### Analysis needed
-
-Audit the code paths that read multiple maps in a single request
-to confirm no consumer relies on cross-map consistency. If none
-does, document the invariant ("these maps are eventually consistent
-within a single `update_ast` call but not atomically consistent")
-and close the item. If a consumer is found, batch the writes under
-a single coordination mechanism.
-
----
-
-## 2. Native type hints not considered in virtual property specificity ranking
+## 0. Native type hints not considered in virtual property specificity ranking
 
 **Impact: Low-Medium — Effort: Medium**
 
@@ -100,4 +36,4 @@ but to the general type resolution pipeline. Fixing it here would not help
 because the native vs docblock decision happens upstream in the parser.
 
 This is out of scope for the virtual member specificity work but worth
-tracking as a separate improvement to `resolve_effective_type`.
+tracking as a separate improvement to `resolve_effective_type`.

docs/todo/indexing.md

@@ -59,7 +59,7 @@ Four indexing strategies, selectable via `.phpantom.toml`:
 
 ```toml
 [indexing]
-# "composer" (default) - use composer classmap, self-scan on fallback
+# "composer" (default) - merged classmap + self-scan
 # "self"    - always self-scan, ignore composer classmap
 # "full"    - background-parse all project files for rich intelligence
 # "none"    - no proactive scanning
@@ -68,17 +68,18 @@ strategy = "composer"
 
 ### `"composer"` (default)
 
-Use Composer's classmap when available and complete. Fall back to
-self-scan when the classmap is missing or incomplete. This is the
-zero-config experience. Phase 1.5 will merge these into a single
-pipeline where the classmap acts as a skip set for the self-scanner.
+Merged classmap + self-scan.  Load Composer's classmap (if it exists)
+as a skip set, then self-scan all PSR-4 and vendor directories for
+anything the classmap missed.  Whatever the classmap already covers is
+a free performance win; whatever it's missing, we find ourselves.  No
+completeness heuristic needed.  This is the zero-config experience.
 
 ### `"self"`
 
 Always build the classmap ourselves. Ignores `autoload_classmap.php`
-entirely. For users who prefer PHPantom's own scanner or who are
-actively editing `composer.json` dependencies. After Phase 1.5, this
-is equivalent to the merged approach with an empty skip set.
+entirely. Equivalent to the merged approach with an empty skip set.
+For users who prefer PHPantom's own scanner or who are actively
+editing `composer.json` dependencies.
 
 ### `"full"`
 
@@ -94,8 +95,7 @@ completion items. Memory usage grows proportionally to project size.
 No proactive file scanning. Still uses Composer's classmap if present,
 still resolves classes on demand when the user triggers completion or
 hover, still has embedded stubs. The only difference from `"composer"`
-is that it never falls back to self-scan when the classmap is missing
-or incomplete. This is essentially what PHPantom does today.
+is that it never self-scans to fill gaps.
 
 ---
 
@@ -109,70 +109,13 @@ configuration, and user feedback messages.
 ---
 
 ## Phase 1.5: Merged classmap + self-scan
-**Impact: High · Effort: Low**
-
-Instead of treating the Composer classmap and the self-scanner as
-mutually exclusive strategies (use the classmap when it looks complete,
-fall back to self-scan when it doesn't), merge them into a single
-pipeline that always runs:
-
-1. **Load the Composer classmap** if it exists. Don't validate it,
-   don't check whether it's complete, don't check whether it's stale.
-   Just parse `autoload_classmap.php` into the `HashMap<String, PathBuf>`
-   as today.
-2. **Self-scan all PSR-4 directories** (and vendor packages), but
-   **skip every file that already appears in the classmap**. The
-   classmap values are absolute paths, so the skip check is a
-   `HashSet::contains` on each walked path.
-3. The result is a merged index: classmap entries for everything
-   Composer already knew about, plus self-scanned entries for
-   everything it missed.
-
-### Why this is better
-
-- **No staleness problem.** If the classmap is out of date (new
-  classes added, old classes removed, files moved), the self-scanner
-  picks up the difference. We never serve stale data.
-- **No completeness check.** The current code inspects the classmap
-  to decide whether it's "complete enough" to trust. That heuristic
-  is fragile and was the source of the old `composer dump-autoload -o`
-  prompts. With the merged approach, we don't care. Whatever the
-  classmap has is a free performance win; whatever it's missing, we
-  find ourselves.
-- **Faster in every scenario.** When the classmap is complete (the
-  common case), the self-scanner walks directories but skips every
-  file, finishing almost instantly. When the classmap is empty or
-  absent, we fall back to a full self-scan exactly as today. When
-  the classmap is partial (e.g. vendor classes only), we skip vendor
-  files and scan only user code. Every state of the classmap helps.
-- **Less code.** The `"composer"` and `"self"` strategies collapse
-  into one path. The completeness-checking logic in `initialized`
-  can be removed. The `"none"` strategy remains for users who want
-  no proactive scanning at all.
-
-### Implementation
-
-1. In `initialized`, always load `autoload_classmap.php` when it
-   exists and collect the file paths into a `HashSet<PathBuf>`.
-2. Pass the skip set to `scan_psr4_directories` and
-   `scan_vendor_packages`. Before reading and scanning a file,
-   check whether its canonical path is in the skip set. If so,
-   skip it.
-3. Remove the "is the classmap complete?" branching. The pipeline
-   is always: load classmap → self-scan with skip set → merge.
-4. Remove the `"self"` strategy mode (or keep it as an alias for
-   the merged approach with an empty skip set, ignoring the
-   classmap entirely). Update `"composer"` to mean "merged."
-5. Update the strategy mode documentation and log messages.
-
-### Effect on Phase 2
-
-Phase 2 (staleness detection and auto-refresh) becomes much simpler.
-Single-file `did_save` refresh still applies (re-scan the saved file,
-update its classmap entries). But the expensive "did the whole
-classmap go stale?" question disappears. On `composer.lock` change,
-re-running the merged pipeline with the new classmap as the skip set
-is sufficient.
+
+No outstanding items. Implemented: the Composer classmap and the
+self-scanner run as a single merged pipeline. The classmap file paths
+serve as a skip set for the self-scanner, so whatever Composer already
+covers is a free performance win and whatever it missed is discovered
+automatically. The completeness heuristic has been removed. Monorepo
+subprojects use the same merged pipeline.
 
 ---