Switch all linux release builds to musl + mimalloc, add memory profiling

features, plan memory optimizations

Author: AJenbo

Cargo.toml

@@ -47,11 +47,16 @@ zip = "2"
 # musl's built-in malloc is very slow under the multi-threaded,
 # allocation-heavy load of parallel indexing and parse workers (4-6x
 # slower whole-project analysis), so the static musl binaries we ship
-# for Linux use mimalloc instead. Scoped to musl only: glibc, macOS,
-# and Windows system allocators perform fine, and mimalloc retains
-# freed pages more aggressively than they do, so enabling it where it
-# isn't needed just raises resident memory.
-[target.'cfg(target_env = "musl")'.dependencies]
+# for Linux use mimalloc instead.
+#
+# Scoped to Linux rather than musl alone so that a local `cargo build`
+# on a glibc workstation gets the same allocator as the binaries we
+# release: every Linux artifact we publish is musl, so no shipped build
+# is affected by widening this, while performance and memory figures
+# measured locally now match what users actually run. macOS and Windows
+# release from their own targets and use the system allocator both in
+# CI and locally, so they are already representative and are left alone.
+[target.'cfg(target_os = "linux")'.dependencies]
 mimalloc = { version = "0.1.47", optional = true }
 libmimalloc-sys = { version = "0.1", optional = true, features = ["extended"] }
 libc = { version = "0.2", optional = true }
@@ -67,6 +72,11 @@ flate2 = "1"
 # can be toggled off for benchmarking or debugging without editing code.
 default = ["mimalloc"]
 mimalloc = ["dep:mimalloc", "dep:libmimalloc-sys", "dep:libc"]
+# Deep-size accounting of the long-lived symbol stores, printed at the
+# end of `analyze` when PHPANTOM_MEM_AUDIT is set. Off by default: it
+# installs a counting global allocator that adds two atomics to every
+# allocation, so it must never be enabled in a shipped build.
+mem-audit = []
 
 [dev-dependencies]
 datatest-stable = "0.3"

docs/BUILDING.md

@@ -18,6 +18,34 @@ The stubs are downloaded on first build and cached in `stubs/`. To update to the
 
 For details on how symbol resolution and stub loading work, see [ARCHITECTURE.md](ARCHITECTURE.md).
 
+### Matching the released binary
+
+The Linux binaries we publish are static musl builds using mimalloc as
+the allocator (see the comment on the `mimalloc` dependency in
+`Cargo.toml`). A plain `cargo build --release` on a glibc workstation
+already uses the same allocator — mimalloc is enabled for Linux, not
+just musl — so day-to-day performance and memory testing on any Linux
+dev machine is representative without extra setup.
+
+To build the exact target we ship (e.g. to reproduce a memory or
+performance report bit-for-bit), build for musl:
+
+```bash
+rustup target add x86_64-unknown-linux-musl
+sudo apt install musl-tools   # provides musl-gcc; adjust for your distro
+CARGO_BUILD_TARGET=x86_64-unknown-linux-musl cargo build --release
+```
+
+This produces `target/x86_64-unknown-linux-musl/release/phpantom_lsp`
+alongside your normal build (it doesn't touch `target/release/`), so it
+doesn't disturb incremental builds of your everyday work.
+
+If you're benchmarking or profiling memory and get numbers that look
+off from what's documented in `docs/todo/performance.md`, check which
+allocator built the binary before assuming a regression — macOS and
+Windows use the system allocator in both CI and locally already, so
+they need no special steps.
+
 ## Testing
 
 Run the full test suite:

docs/todo.md

@@ -27,6 +27,11 @@ within the same impact tier.
 | --- | --------------------------------------------------------------------------------------------------------------------- | ---------- | ------ |
 | P36 | [Diagnostic-path call sites re-merge inheritance per call](todo/performance.md#p36-diagnostic-path-call-sites-re-merge-inheritance-per-call-instead-of-reading-the-resolved-class-cache)          | Medium      | Low         |
 | P31 | [Reference index stores per-span entries when consumers only read distinct URIs](todo/performance.md#p31-reference-index-stores-per-span-entries-when-consumers-only-read-distinct-uris)          | Medium      | Low-Medium  |
+| P37 | [`PhpType` is 64 bytes and is embedded ~3 M times](todo/performance.md#p37-phptype-is-64-bytes-and-is-embedded-3-m-times)                                            | High        | Medium      |
+| P38 | [Resolved type values are duplicated ~33x](todo/performance.md#p38-resolved-type-values-are-duplicated-33x)                                                          | High        | High        |
+| P39 | [`SymbolKind` stores owned strings per span](todo/performance.md#p39-symbolkind-stores-owned-strings-per-span)                                                       | Medium      | Low-Medium  |
+| P40 | [`method_index` is a per-class `HashMap` even when the member vec is shared](todo/performance.md#p40-method_index-is-a-per-class-hashmap-even-when-the-member-vec-is-shared) | Low-Medium  | Low         |
+| P41 | [3.8 M live allocations cost ~130 MB in allocator overhead](todo/performance.md#p41-38-m-live-allocations-cost-130-mb-in-allocator-overhead)                          | Medium      | Medium      |
 | P33 | [Workspace diagnostics leaves the whole project fully resolved in memory](todo/performance.md#p33-workspace-diagnostics-leaves-the-whole-project-fully-resolved-in-memory) | High        | Medium-High |
 | X10 | [Interactive requests block on the workspace index lock during initial indexing](todo/indexing.md#x10-interactive-requests-block-on-the-workspace-index-lock-during-initial-indexing) | Medium      | Medium      |
 | L21 | [Tighten the supertype-where-subtype comparison escape hatch (blocked on resolver precision)](todo/laravel.md#l21-tighten-the-supertype-where-subtype-comparison-escape-hatch-blocked-on-resolver-precision) | Medium      | High        |