perf: kernel-scale indexing campaign (-61% wall, deterministic output)#921
Merged
Conversation
Two wall-clock fixes from a linux-kernel (8.5M nodes) profiling run: 1. Extraction back-pressure futility latch: when a full collect+nap cycle ends still over the RSS budget, the resident floor (accumulated graph + retained sources) -- not in-flight transients -- holds the memory, so napping again on every subsequent pull reclaims nothing and only idles workers (measured: one full cycle per pull, ~390 s at 79% avg CPU across 63k pulls). Latch bp_futile after the first futile cycle, proceed with the designed soft overshoot, and re-arm via the cheap per-pull probe once RSS drains under budget. WARN mem.backpressure.futile exactly once per latch event (0->1 transition). 2. Supervised-worker fast exit: the index worker piecemeal-freed a multi-GB graph (and store) right before process death (measured: minutes of free() in the tail). In worker role, skip cbm_pipeline_free and _Exit after the response is written and flushed; the OS reclaims memory wholesale. In-process paths (tests, kill switch, degrade) free normally, keeping ASan/LSan meaningful. Reproduce-first: pipeline_backpressure_futile_nap_disengages (64-file fixture > MIN_FILES_FOR_PARALLEL so the parallel gate actually runs, CBM_MEM_BUDGET_MB=1, deterministic nap-cycle counter; engagement guard cycles>=1 prevents a vacuous pass). RED on the old gate: 64 cycles (one per pull) > bound cores+2. GREEN with the latch. Suites: pipeline 216, mcp 140, subprocess 17 -- all pass. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
…BM_PROFILE The worker's msg=prof pass/sub-phase report is written only to its supervisor logfile, which was deleted on every clean exit -- profiling a successful run was impossible (the log survived only on failure). Keep it when profiling is active and echo the path via index.supervisor.profile_log. Default behavior unchanged: clean-run logs are still pruned when CBM_PROFILE is off. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Kernel-scale profiling put ~87% of the 361 s cross-file resolve phase inside
c_lookup_member_depth's miss cascade: in macro-heavy C nearly every member
lookup misses, identical (type_qn, member) pairs miss millions of times, and
each miss re-paid an arena_sprintf("%s.%s") (the strlen storm) plus the
O(type_count) short-name registry scan.
Fix, semantics preserved exactly:
- Per-file negative memo (open-addressing uint64 set) on (type_qn, member),
active only at depth 0 under the shared READ-ONLY Tier-2 registry, where the
registry-only cascades are pure functions of (type_qn, registry). The direct
method lookup always runs first (collision/staleness guard) and the
context-dependent scope-alias path is always evaluated; only the provably-
NULL module-prefix/base-class/short-name cascades are skipped on a memo hit.
- Module-prefixed QN retries build into a stack buffer via cached
module_qn_len (arena_sprintf only as truncation fallback).
Soundness: cbm_c_build_cross_registry finalizes then sets read_only=true;
cbm_registry_add_func/add_type hard-return when read_only, so no miss can
flip to a hit during resolve. Suites: c_lsp 748/0 (incl. the tier2 read-only
guards), pipeline+registry+scope+semantic 656/0. Graph equivalence verified
on fs/xfs: nodes 13341 / edges 58559 byte-identical with and without the fix.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
py_register_instance_field mutated the shared Tier-2 registry through direct struct writes that bypass the read_only seal: it overwrote field-type pointers in place and reassigned the sealed type entry's field arrays to memory owned by the per-file arena. In the fused parallel resolve path (one sealed registry shared by all resolve workers) that is a cross-thread data race, and it leaves the shared registry pointing into arenas that die with their file -- a cross-file use-after-free. The count-based seal guard could not see it because type/func counts never change. Python extraction emits no instance fields, so resolve-time discovery is the ONLY source of self.x / PEP-526 fields; a plain read_only gate would silently drop real attribute-chain CALLS edges. Fix instead routes discovery into a per-file field overlay on PyLSPContext (arena-backed, dies with the file) and has py_lookup_field_depth consult shared-base + overlay at the exact precedence point of the direct field scan. Mutable per-file registries keep the original in-place write, byte-identical. Reproduce-first, both halves pinned: - seal_py_shared_registry_readonly_fields: RED on the bug (field count 0->1 on a sealed registry), GREEN with the overlay. - pylsp_fused_self_attr_chain_via_overlay: GREEN with the overlay, RED under a simulated plain gate (self.b.work() CALLS edge lost) -- proven both ways. Suites: c_lsp 749, py_lsp 63+45 stress/scale/bench, pipeline 216, registry 54, scope 10, type_rep 19 -- zero failures. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The Rust cross-file resolver's trait-method, sole-trait-impl, and free-func fallback loops scanned the ENTIRE type/func registry per call site on a miss. On kernel-scale registries (~10^6 entries, macro-inflated) four tiny trait-heavy rust files burned ~96 s EACH (~200k eval-capped call sites x full-registry strcmp scans). Fix: two hash indexes built at registry finalize, mirroring the existing bucket+entries idiom -- fnv1a(bare(embedded_type)) -> chain of declaring type indices (trait -> implementers), and fnv1a(short_name) -> chain of free-func indices. Chains preserve ascending registry order (first-match tie-break unchanged); allocation-free iterators fall back to the post-finalize tail and to a full scan on unfinalized registries, so the candidate set is identical in every mode. The four rust loops now walk the iterator and re-apply their exact original predicates -- byte-identical results, loop bodies untouched. Verified: graph equivalence on the kernel rust/ subtree (nodes 9654, edges 44798 identical with and without the change); registry_short_name_indexes unit test pins iterator contents/order/dedup; suites c_lsp 750, rust_lsp 505, registry 54, pipeline 216 -- zero failures. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
dump_and_persist_hashes upserted file hashes one statement per file -- ~89k autocommit transactions (one fsync each) on the linux kernel. Switch to the existing cbm_store_upsert_file_hash_batch (same cached upsert inside one begin/commit, rollback on error): identical tuples and replace semantics, one fsync. OOM fallback to the per-file path retained. Also instrument the whole post-dump tail with CBM_PROFILE spans (persist/1_reopen .. 6_artifact_export) -- this stretch was ~60 s unattributed on kernel-scale runs; the FTS5 backfill over all node names is now measurable before any optimization decision. New deterministic guard: store_file_hash_batch_equals_loop asserts the batched path produces byte-identical file_hashes rows to the per-file loop. Suites: pipeline 216, store_nodes 56, store_edges 25, store_search 65, store_arch 52, store_bulk 3, store_pragmas 7, store_checkpoint 1, incremental 161 -- zero failures. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Multi-threaded indexing loses edges vs single-threaded and is run-to-run non-deterministic (fs/xfs: ST 58573 edges vs MT 58561 on the first divergent run; rust subtree flickers 44445-44857 across runs). Discovered while hardening perf-change equivalence checks; also the reason raw node/edge counts are not a valid equivalence gate. The guard indexes a corpus once single-threaded (reference) and K=4x multi-threaded (fresh store each run) and asserts the SORTED (source_qn|type|target_qn) edge sets are all identical. RED today; GREEN if and only if MT output is deterministic and equals ST. Calibration note (false-guard avoidance): three escalating synthetic C corpora (300/500/600 files, dense cross-file calls, token-identical bodies) stayed fully deterministic -- the similarity pass emits zero edges on synthetic C, and the race needs real-code edge diversity. A synthetic guard would pass on buggy code, so this repro uses the smallest real corpus that reproduces (kernel fs/xfs) and SKIPs when the corpus is absent. Suspected root causes documented in the test header (per-worker edge-buffer merge, gbuf edge dedup check-then-insert, symmetric SIMILAR_TO emission, resolve ordering). Fix deferred; this test is its guard. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
…files Profiling the kernel showed 4 tiny trait-heavy rust files at ~90 s EACH in cross-file resolve. Live call trees put the time not in lookups but in REGISTRY CONSTRUCTION: cbm_pxc_filter_defs_for_file returns NULL for these files (their path-derived module QNs never match the def_module_qn index keys), so each one built its own registry over the ENTIRE project def set (millions of entries at kernel scale). Original premise corrected during implementation: only 4/87 rust files (on the kernel rust subtree) hit the NULL-filter path; 83/87 resolve against proper import-filtered subsets. A registry shared across ALL rust files would hand the subset files a superset and could add edges. The fix is therefore CONDITIONAL, byte-identical for every file by construction: - filter NULL -> ONE shared all_defs registry (built once in pipeline.c, finalized, read_only-sealed; module_qn=NULL is safe: an instrumented probe found 0 NULL def_module_qn across all_defs, so the per-file def_mod fallback never fired anyway). Kills the amplifier. - filter SUBSET -> today's per-file subset build, unchanged. The cross-crate Cargo manifest reaches the shared path through cbm_pxc_get_rust_manifest() (same worker-thread thread-local the fallback read), preserving cross-crate resolution exactly. Byte-identity verified empirically: single-threaded sorted edge-set diff on the kernel rust subtree, fix off vs on -- zero difference outside SEMANTICALLY_RELATED edges, which self-differ between two runs of the SAME unfixed binary (the pre-existing semantic non-determinism guarded by the repro_parallel_edge_determinism RED test). New suite test rustlsp_shared_registry_resolves_like_per_file exercises the shared path. Suites: rust_lsp 506, c_lsp 750, registry 54, pipeline 216, java_lsp 94, kotlin_lsp 78, php_lsp 278, store_nodes 56 -- zero failures. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Follow-up to the conditional shared registry: the eager pipeline.c build paid the O(all_defs) construction and a multi-million-entry registry RSS on every run that contains rust files, even when every rust file filters to a proper import subset and the shared registry is never consulted. Build it lazily instead: the first worker that hits a NULL-filter rust file takes a mutex, double-checks, builds into a dedicated arena, seals and atomically publishes; later null-filter files reuse it and subset files never trigger it. Verified the build fires exactly once (scale log: types=2320 funcs=8860 on the kernel rust subtree) and the shared-path unit test still resolves identically to the per-file build. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Multi-threaded indexing produced a different graph on every run (xfs: node labels flipping Class/Function/Macro, ~2k edge lines differing across USAGE/ CALLS/IMPORTS/SIMILAR_TO/SEMANTICALLY_RELATED) and lost edges vs repeated runs. Five root causes, all order- or scheduling-dependence: 1. gbuf QN collisions resolved last-wins in MERGE ORDER: in C a struct, a function and a macro can share one name, and the QN scheme does not encode entity kind, so whichever entity a worker merged last overwrote the node's label/name/lines/props (the flickering node set that fed every pass). Both cbm_gbuf_upsert_node and the parallel-merge path now keep the SAME entity's refresh semantics but resolve cross-entity collisions by a canonical content rule -- lexicographically smallest (label, file_path, start_line, name) -- and keep the label/name secondary indexes consistent when the survivor changes (the old overwrite left nodes listed under their original label, which is how the flicker reached find_by_label consumers). 2. Semantic-edges admission raced per-node max_edges budgets on shared atomic counters under dynamic scheduling: scoring stays parallel and pure; pairs carry (func index, candidate rank); one sequential pass admits in canonical order. funcs[] itself is sorted by qualified name up front (the label-index scan order is merge order). 3. Import resolution's symbol-name fallback returned the FIRST same-named node in registration order; now the lexicographically-smallest-QN candidate. 4. Similarity entries inherited merge order into LSH chains + per-node cap truncation, and pair ownership keyed on run-varying node ids; entries are sorted by QN and ownership compares QNs (lsh entries carry the QN). 5. Call-neighbor context (semantic vectors + tokens) truncated MAX_CALLEES subsets in edge-array order; neighbor names are sorted first. Verified: four consecutive multi-threaded kernel-xfs runs produce byte- identical sorted node AND edge sets (13341/58598, zero diff). The repro guard repro_parallel_edge_determinism (MT x K mutual equality) is GREEN; a new deliberately-RED repro_seq_parallel_equivalence tracks the remaining OPEN bug that the sequential and parallel pipelines systematically disagree (~3459 USAGE / ~1666 WRITES lines on xfs) -- separate code paths, fix deferred. Suites: pipeline 216, c_lsp 750, rust_lsp 506, py_lsp 63, java 94, kotlin 78, php 278, registry 54, store_nodes 56, incremental 161, semantic 32, simhash 24 (one test updated: SIMILAR_TO pair direction is now canonical by QN), mcp 140, subprocess 17 -- zero failures. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Macro-expanded kernel rust asks the same failing (receiver, method) / (trait, method) questions thousands of times per file; each repeat re-ran the full registry-pure cascade (embedded-impl walk + trait-default tail) against the shared 1.4M-entry registry -- ~63 s per trait-heavy file even after the build-once registry and the trait indexes. New internal/cbm/lsp/lsp_neg_memo.h: a generic, arena-backed negative-lookup memo for LSP resolve cascades (open-addressing u64 set, site-tagged FNV pair keys, lazy alloc, grow-by-rehash). Gated on a SEALED registry (read_only), so a miss is a pure fact of (registry, query); the cheap direct lookup stays before the memo check so a hash collision can never hide a real hit. Any language wires it in a few lines -- rust first (rust_resolve_trait_method and rust_find_sole_trait_impl, both proven to read nothing but the registry and their query strings); the C resolver's bespoke memo can migrate, and the audit's php/c#/java/kotlin cascades are wiring candidates. Only the full miss is memoized (zero impls and no trait default), preserving out_impl_count/out_n fidelity for the ambiguous >=2-impl cases. Verified: kernel rust subtree indexed before/after -- sorted node AND edge sets byte-identical (0 diff; exact comparison, now that indexing is deterministic). rust_lsp suite 506 passed. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
… dedup
The four kernel rust files' wall time was a CONSTANT ~63.5 s that no
resolution-side fix moved -- because it was not resolution. The shared
all_defs registry build had two lookups INSIDE its registration loop
(receiver auto-registration probe via cbm_registry_lookup_type, and the
trait-linkage scan), and before finalize the registry has no buckets: both
probes were full linear scans over the types registered so far. On the
1.4M-def shared build that is ~10^10 strcmps ~= 63 s, paid once by the
building file while the sibling null-filter files waited on the once-guard
for exactly the same wall time. This is the checklist's own
lookup-in-registration-loop pattern -- invisible on small per-file builds.
Fix: a CBMIdxMemo (new in lsp_neg_memo.h: exact-match, strcmp-verified
string-key -> index map, arena-backed, first-put-wins to mirror the linear
scans' first-match semantics) maintained alongside every add_type in
rust_populate_cross_registry; both probes become O(1).
Also rides along: a per-root-invocation macro-expansion memo. Kernel
macro_rules are self-recursive (define_sizes! re-invokes itself) and the
expansion fallback ("no pattern matched -- still expand the first rule")
made recursion non-convergent: unbounded BREADTH under the depth-8 guard
turned 2-44 source invocations into ~200k expansions (each a fresh
tree-sitter parse of the substituted body). Identical (macro, substituted
text) now expands once per chain; the memo resets at each top-level source
site so per-site attribution is untouched, and the arg re-parse dedup only
applies at recursion depth > 0. Side effect: the eval-step cap no longer
exhausts, so a few previously-truncated real calls now resolve (kernel:
+25 edges).
Verified: kernel rust subtree byte-identical before/after (sorted node and
edge sets, exact); rust_lsp suite 506 passed.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
… dead field Three memory quick-wins from the gbuf compaction research (graph output verified identical; all delete/incremental paths suite-covered): 1. Release ALL lookup indexes right after build_dump_nodes instead of after edge building: nothing later in the dump reads them, and at kernel scale they are ~3.8 GB (hash buckets + key strings + pointer arrays) that previously coexisted with every dump-side transient — the dump-phase RSS peak. At 2x-kernel scale this is ~7.5 GB at exactly the OOM point. 2. node_by_id was a hash table keyed on STRDUP'D DECIMAL STRINGS of the int64 id — ~0.44 GB of buckets and key strings, plus a snprintf + strdup + hash on every hot find_by_id call. Ids are dense and sequential; a plain grow-on-demand pointer array replaces all of it with one bounds-checked load. 3. The per-node and per-edge `project` field was write-only (every reader uses the store structs or gb->project directly; -Werror confirms) — 8 bytes x 24M instances ~= 192 MB of dead pointers, deleted. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
… SQ)
The semantic pass kept four dense 768-float vectors resident per function
(ri/api/type/deco = 12.3 KB per func, ~9.4 GB transient on the linux kernel)
solely so candidate pairs could be cosine-scored after LSH. Replace the
resident floats with 4-bit quantized codes (~0.5 KB each, ~6x): new
src/semantic/rotsq.{h,c} — rotated scalar quantization from the RaBitQ
family of algorithms (Gao et al., SIGMOD 2024/2025), implemented from the
papers in plain C11 (no Eigen/library dependency): a deterministic
XXH3-seeded +-1 diagonal followed by a Fast Walsh-Hadamard rotation
(768 -> 1024 padded) spreads vector mass so per-vector scalar quantization
is near-optimal; the inner product is recovered from the codes with the
exact SQ expansion (one u8 integer dot + four multiplies) — a deterministic
pure function of the codes. Named rotsq, not rabitq: this is the family
core, not the papers' exact codebook construction.
Producers build each dense vector in a worker-local buffer, encode, and
discard; the int8 qvec export is unchanged. LSH signatures are computed
from DEQUANTIZED codes against hyperplanes in the rotated basis — random
hyperplanes are basis-agnostic, so no dense originals are retained and the
phase structure is untouched.
Estimator quality gated by sem_rotsq_ip_error_bounds (seeded vectors:
mean error < 1%, max < 4% of unit scale, ASan-green). Effect on output:
multi-threaded determinism preserved (xfs: two runs byte-identical);
SEMANTICALLY_RELATED edges shift once as scores move within ~1% of the
0.75 threshold (xfs: ~130 -> 203; sampled new pairs are high quality —
alloc/free, read-verify and collapse/insert families).
Suites: semantic 33 (incl. the new estimator gate), simhash 24,
pipeline 216 — zero failures.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
…rned
Two borrow-don't-copy wins from the layer-duplication analysis (a hot string
was resident 4-5x across extraction arenas, gbuf, registries, defs and
tokens). Graph output verified byte-identical.
1. Resolution registry (registry.c): every registered definition strdup'd its
qualified name TWICE (exact-map key + by_name array item) and its label
once (~8.5M label strdups). The by_name items now borrow the exact map's
single owned key (same lifetime, freed in one place) and labels intern
into a small per-registry pool (<= ~30 distinct). ~280 MB + ~17M
allocations on the kernel.
2. Semantic tokens (pass_semantic_edges.c): all_tokens held one strdup per
token INSTANCE — identical tokens ("xfs", "error", ...) duplicated across
hundreds of thousands of functions. Each tokenize worker now interns into
a per-worker pool (key == value == the single owned copy; no contention,
content-identical strings so downstream corpus/tfidf output is unchanged);
all_tokens slots borrow. Ownership moves from per-func frees to pool
frees at pass end.
Suites: semantic 33, simhash 24, pipeline 216, registry 54, incremental 161.
xfs: two MT runs byte-identical, and byte-identical to the pre-change graph.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Phase-aligned RSS attribution located the global peak (24.4 GB at t=47s, ~14 GB transient above the retained floor) INSIDE parallel extraction: the size-descending queue put the kernel's 12 largest files into 12 workers simultaneously, each holding a multi-MB source + its 10-50x TSTree + arena. Every post-extraction optimization sat below this waterline. Replace the single biggest-first cursor with a dual-ended claim queue over the same size-sorted list: big_workers = clamp(workers/6, 1, 3) claim from the BIG end while the rest rush the small end, so at most big_workers giant working-sets coexist -- a structural bound, not a scheduling heuristic (a static interleave cannot bound concurrency under dynamic pulls: smalls finish in ms and workers converge back onto the giants). Giants still start at t=0, preserving the LPT no-straggler property; workers never idle while either end has files. Per-file claim flags (atomic exchange) make the crossover race-free; end-of-queue rescans are O(n) bytes once. Intra-file parallel parsing was researched and ruled out: tree-sitter has no such concept (incremental != parallel; one parser per thread is the model), and data-parallel parsing theory (PAPAGENO through SLE 2025 cyclic OPG work) requires operator-precedence grammars with local parsability -- none of the 159 vendored GLR grammars qualify, and C's preprocessor makes byte-chunking unsound. Output is schedule-independent since the determinism fix and verified so: xfs sorted edge set byte-identical to the previous commit. pipeline 216 passed. Worst case (all-giant repo) unchanged by design. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
…parses" This reverts commit 0475a880e5735ef5cb0b0d99532e552096e5a2d5. Measured on the linux kernel: peak RSS unchanged (23.95 vs 24.4 GB, within run noise) at +22 s wall -- the extraction peak turned out to be monotonic retained-tree/source accumulation, which is schedule-independent, not the concurrent giant transients this queue bounded. Net negative as measured; the retention-side fix is parked for post-release. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Performance and determinism campaign, measured end-to-end on a linux-kernel-scale corpus (8.5M nodes / 15.5M edges):
Highlights per commit: extraction back-pressure futility latch; supervised-worker fast exit; C-LSP negative-lookup memo; Python instance-field overlay (fixes a sealed-registry data race / cross-file UAF); registry short-name + trait indexes; file-hash batch upsert + persist-phase profiling spans; conditional shared rust registry (lazy) + registration-loop O(1) type map + macro-expansion dedup; parallel determinism fixes (canonical collision winners, sequential canonical admission, canonical import/similarity winners); semantic vectors quantized via rotated 4-bit scalar quantization (RaBitQ-family, from-paper, dependency-free) with a permanent estimator-error gate; registry/token borrow-don't-copy; gbuf dense by-id array + dead-field removal.
Includes one deliberately-RED reproduction (repro_seq_parallel_equivalence, non-gating board) tracking the pre-existing sequential-vs-parallel pipeline divergence, and a reverted experiment (dual-ended extraction queue) kept in history with its falsifying measurement.
All suites green locally (ASan/UBSan); graph equivalence verified by sorted edge-set byte-diffs at each step.