Skip to content

Commit 7743f86

Browse files
apstenku123apstenku123
committed
discover(crossrepo): ranked base-lib usage-breadth list + cross-link candidates
1 parent f5b6288 commit 7743f86

1 file changed

Lines changed: 206 additions & 0 deletions

File tree

‎outputs/crossrepo/base_lib_crosslink_candidates.md‎

Lines changed: 206 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,206 @@
1+
# Base-Library Cross-Link Candidates — Ranked Usage-Breadth Census
2+
3+
**Goal.** We already infer platform from cmake/make/ninja. We have full-history base-lib repos
4+
(linux, glibc, boost, ACE, libstdc++/gcc, openssl, zlib, Windows base libs, etc.). We want to
5+
*selectively* cross-repo link **only the base libraries that are used in many places**, so a
6+
function in repo X that calls e.g. a boost/glibc/std symbol can pull that definition cross-repo.
7+
This report ranks base-lib usage breadth across the entire corpus and recommends a careful cut.
8+
9+
**RULE #1 (fail-loud, real measurement).** The ranking below is a **full census**, not a sample
10+
and not a guess. Honest classifier-coverage gaps are called out explicitly rather than papered over.
11+
12+
## Method (read-only, real measurement)
13+
14+
Single read-only streaming pass over the entire corpus tarball:
15+
16+
```
17+
nice -15 zstd -dc /Users/dave/sources/parquet/data-cpp_all/data-cpp_all.tar.zst \
18+
| nice -15 python3 /tmp/rank_usage2.py # tarfile mode 'r|' — stream, no extraction
19+
```
20+
21+
For each source file (ext c/cc/cpp/cxx/h/hpp/hh/cu/cuh/m/mm/tcc/inl/ipp/...) we read only the
22+
top 49152 bytes (includes live at file top) and run:
23+
1. ONE `#include` regex capturing every include path; each path is classified to a base-lib family
24+
by a dispatch function (first-path-segment prefix + exact-basename tables for STL/libc/Windows
25+
headers) — O(includes), not O(families×bytes).
26+
2. ONE combined symbol-prefix regex (`std:: boost:: absl:: Eigen:: folly:: fmt::
27+
google::protobuf:: gsl:: thrust::`) for the prefixed families.
28+
29+
**Self-references excluded** via per-family provider subtree sets (a file inside `boost/` does not
30+
count toward boost). **Breadth metric** = count of DISTINCT referencing repos (top-level
31+
`cpp_all/<repo>`); **volume** = total includes + total symbol refs. Classifier unit-tested 28/28
32+
before the run. v1 (per-family findall over decoded text) ran at 73 files/s and was discarded for
33+
v2 (include-centric) at ~3,100 files/s — a 42× speedup.
34+
35+
**Coverage / completeness (this is a complete census, not a sample):**
36+
37+
| metric | value |
38+
|---|---|
39+
| run mode | FULL — entire ~235 GiB tarball streamed end-to-end |
40+
| elapsed | 884.9 s (vs 14,400 s budget) |
41+
| tar members streamed | 7,703,235 |
42+
| subtrees seen | 594 / 594 (100% of manifest) |
43+
| source files scanned | 2,749,108 |
44+
| file-content bytes scanned | 25,069,556,276 (top-48 KiB-per-file cap) |
45+
| files skipped (>16 MiB) | 15 |
46+
| rate | 3,106.7 files/sec |
47+
48+
Concurrency safety: the unified conveyor (pid 8697) reads the same tarball + repo_list.json and was
49+
confirmed alive and unharmed before/after; the tarball is read-only; no writes to
50+
`outputs/reindexed*`. Membership manifest:
51+
`/Volumes/external/sources/cppmega/outputs/pr_ingest/repo_list.json`. Source artifact (ranked JSON):
52+
`/Volumes/external/sources/cppmega/outputs/crossrepo/base_lib_usage_ranked.json` (27,500 bytes).
53+
54+
## Presence in corpus
55+
56+
**All base-lib families measured were confirmed present in the corpus (`present_in_corpus=true` for
57+
every family).** Providers are drawn from the 594 subtrees in the manifest, all members of the
58+
tarball under layout `cpp_all/<repo>/...`. Key providers → what they provide:
59+
60+
- **libstdcxx_STL** (C++ std lib): gcc-mirror (libstdc++), llvm-project (libc++), STL/stl
61+
(microsoft/STL), stlport, sgi-stl, libcudacxx, cccl(+.bare), rocm-llvm, intel-llvm-dpcpp.
62+
- **libc_posix** (C stdlib/POSIX/syscalls): glibc, musl, apple-libc, gcc-mirror, llvm-project, plus
63+
kernels linux / freebsd-src / openbsd / netbsd / src (NetBSD) / xnu / illumos-gate / 4.4bsd-lite2.
64+
- **boost**: boost (boostorg). **abseil**: abseil-cpp. **folly**: folly. **eigen**: eigen (gitlab
65+
libeigen). **fmt**: fmt (+ spdlog bundles it). **spdlog**: spdlog.
66+
- **openssl_crypto**: openssl, boringssl, portable (libressl). **zlib**: zlib. **lz4/zstd/xz**:
67+
lz4, zstd, xz. mbedtls/wolfssl/libsodium present.
68+
- **protobuf, flatbuffers, nlohmann_json (json), rapidjson, msgpack-c, capnproto, simdjson, thrift**
69+
(serialization).
70+
- **googletest, Catch2, benchmark** (test). **gsl_microsoft (GSL/gsl), range-v3, wil** (utility hdrs).
71+
- **windows_base** (CRT/SDK/MFC/ATL/DDK): windows_2000_source_code, windows_nt_4_source_code,
72+
nt5src, windows-research-kernel, ddk_wdk, windows_10_shared_source_kit, STL/stl.
73+
- **libuv, libevent, asio, curl, c-ares, nghttp2** (event/io/net). **sqlite, leveldb, lmdb**
74+
(embedded db). **glib, cpython, mimalloc** (runtime/alloc). **libpng, libjpeg-turbo, freetype,
75+
harfbuzz, openexr** (image/text codec).
76+
77+
**RULE #1 caveats found:**
78+
- `bionic` subtree = nickg/nvc (VHDL compiler), **NOT** Android libc; AOSP libc is present as
79+
aosp-system-core / aosp-frameworks-*.
80+
- 82 `.bare` subtrees are bare git mirrors (no working-tree source to scan, so they show 0 scannable
81+
files but carry full history). The 512 non-bare subtrees provided the working-tree source scanned.
82+
83+
## Ranked usage table (by breadth = distinct referencing repos; self-refs excluded)
84+
85+
Columns: family | distinct_repos (union of include+symbol) | inc_repos | sym_repos |
86+
total_includes | total_symbol_refs | provider subtree(s).
87+
88+
| # | family | repos∪ | inc_repos | sym_repos | includes | symbol_refs | provider(s) |
89+
|---|---|---:|---:|---:|---:|---:|---|
90+
| 1 | libc_posix | 481 | 481 | 0 | 864,640 | 0 | glibc, musl, apple-libc, gcc-mirror, llvm-project, linux, freebsd-src, openbsd, netbsd, src, xnu, illumos-gate, 4.4bsd-lite2 |
91+
| 2 | libstdcxx_STL | 423 | 414 | 418 | 978,244 | 8,310,086 | gcc-mirror, llvm-project, STL, stl, stlport, sgi-stl, libcudacxx, cccl(.bare), rocm-llvm, intel-llvm-dpcpp |
92+
| 3 | windows_base | 416 | 416 | 0 | 63,111 | 0 | windows_2000_source_code, windows_nt_4_source_code, nt5src, windows-research-kernel, ddk_wdk, windows_10_shared_source_kit, STL/stl |
93+
| 4 | zlib | 200 | 200 | 0 | 3,031 | 0 | zlib |
94+
| 5 | googletest | 191 | 191 | 0 | 62,365 | 0 | googletest |
95+
| 6 | boost | 166 | 107 | 165 | 238,917 | 338,402 | boost |
96+
| 7 | openssl_crypto | 152 | 152 | 0 | 78,494 | 0 | openssl, boringssl, portable (libressl) |
97+
| 8 | abseil | 105 | 78 | 105 | 81,923 | 377,062 | abseil-cpp |
98+
| 9 | cpython | 99 | 99 | 0 | 1,021 | 0 | cpython |
99+
| 10 | zstd | 95 | 95 | 0 | 766 | 0 | zstd |
100+
| 11 | fmt | 75 | 58 | 73 | 3,637 | 59,163 | fmt, spdlog |
101+
| 12 | cuda_cccl | 72 | 72 | 49 | 18,774 | 46,217 | libcudacxx, cccl(.bare) |
102+
| 13 | libevent | 71 | 71 | 0 | 4,427 | 0 | libevent |
103+
| 13 | curl | 71 | 71 | 0 | 1,119 | 0 | curl |
104+
| 15 | eigen | 68 | 58 | 68 | 3,773 | 55,944 | eigen |
105+
| 16 | libpng | 66 | 66 | 0 | 525 | 0 | libpng |
106+
| 17 | libjpeg | 65 | 65 | 0 | 2,211 | 0 | libjpeg-turbo |
107+
| 18 | protobuf | 61 | 51 | 56 | 27,449 | 23,217 | protobuf |
108+
| 19 | glib | 60 | 60 | 0 | 4,242 | 0 | glib |
109+
| 19 | nlohmann_json | 60 | 60 | 0 | 3,369 | 0 | json |
110+
| 19 | xz | 60 | 60 | 0 | 220 | 0 | xz |
111+
| 22 | mbedtls | 54 | 54 | 0 | 16,881 | 0 | mbedtls |
112+
| 22 | lz4 | 54 | 54 | 0 | 394 | 0 | lz4 |
113+
| 24 | sqlite | 53 | 53 | 0 | 285 | 0 | sqlite |
114+
| 25 | freetype | 52 | 52 | 0 | 6,641 | 0 | freetype |
115+
| 26 | catch2 | 49 | 49 | 0 | 1,396 | 0 | Catch2 |
116+
| 27 | gsl_microsoft | 45 | 27 | 32 | 902 | 13,388 | GSL, gsl |
117+
| 28 | rapidjson | 42 | 42 | 0 | 1,464 | 0 | rapidjson |
118+
| 29 | wolfssl | 35 | 35 | 0 | 6,548 | 0 | wolfssl |
119+
| 30 | folly | 28 | 17 | 28 | 2,124 | 12,038 | folly |
120+
| 30 | libuv | 28 | 28 | 0 | 1,766 | 0 | libuv |
121+
| 32 | asio | 26 | 26 | 0 | 20,339 | 0 | asio, boost |
122+
| 33 | flatbuffers | 23 | 23 | 0 | 1,495 | 0 | flatbuffers |
123+
| 34 | benchmark | 21 | 21 | 0 | 77 | 0 | benchmark |
124+
| 35 | spdlog | 19 | 19 | 0 | 1,614 | 0 | spdlog |
125+
| 35 | c_ares | 19 | 19 | 0 | 90 | 0 | c-ares |
126+
| 37 | harfbuzz | 18 | 18 | 0 | 193 | 0 | harfbuzz |
127+
| 38 | wil_microsoft | 14 | 14 | 0 | 775 | 0 | wil |
128+
| 39 | mimalloc | 11 | 11 | 0 | 192 | 0 | mimalloc |
129+
| 40 | lmdb | 9 | 9 | 0 | 51 | 0 | lmdb |
130+
| 41 | simdjson | 4 | 4 | 0 | 13 | 0 | simdjson |
131+
| 42 | range_v3 | 3 | 3 | 0 | 4,078 | 0 | range-v3 |
132+
| 42 | libsodium | 3 | 3 | 0 | 181 | 0 | libsodium |
133+
| 42 | capnproto | 3 | 3 | 0 | 25 | 0 | capnproto |
134+
| 45 | leveldb | 2 | 2 | 0 | 214 | 0 | leveldb |
135+
| 46 | msgpack | 1 | 1 | 0 | 1 | 0 | msgpack-c |
136+
137+
**Honest coverage gaps (RULE #1 — these are NOT zero real usage):**
138+
- `thrift` (uses `thrift/` dir prefix — unmapped by classifier), `openexr` (`OpenEXR/` prefix —
139+
unmapped), `nghttp2` (`nghttp2/` dir form — only the bare `.h` was mapped) all show 0 in this run
140+
purely from classifier-coverage gaps. All three are long-tail and **below the recommended cut**, so
141+
the gap does not change the recommendation; flagged here for transparency.
142+
143+
## Recommended cross-link cut
144+
145+
Selection criterion: **distinct-repos-using breadth × value / cost**. Cost is dominated by (a)
146+
provider-tree size, (b) provider multiplicity (multiply-defined symbols), and (c) symbol-namespace
147+
cleanliness (prefixed → safe symbol cross-link; unprefixed → high false-link risk).
148+
149+
### TIER 1 — cross-link (highest value, used almost everywhere)
150+
151+
- **libstdcxx_STL** (423 repos, 978k includes, 8.31M `std::` refs) — **HIGHEST VALUE.**
152+
Cost MEDIUM: pick **ONE canonical provider** (gcc-mirror libstdc++ is the cleanest single tree)
153+
rather than linking all 11 STL providers; do NOT cross-link every STL impl or you multiply
154+
definitions. Risk: header-only templates resolve to many candidate defs — **canonicalize to one
155+
provider**.
156+
- **boost** (166 repos, 239k includes, 338k `boost::` refs) — HIGH VALUE, header-heavy, single clean
157+
provider (boost). Cost MEDIUM (large tree). Recommended.
158+
- **libc_posix** (481 repos — widest of all, 865k includes) — cross-link but **CAREFULLY.**
159+
Cost HIGH: providers include linux+glibc (huge) and libc symbols are unprefixed, so symbol-level
160+
linking is ambiguous across 13 providers. Recommendation: cross-link **DECLARATIONS** from ONE
161+
canonical libc (glibc headers) for include→header resolution; do NOT attempt full cross-repo symbol
162+
resolution into the linux kernel tree (definitions are unprefixed, multiply-defined across
163+
kernels/libcs — high false-link risk).
164+
165+
### TIER 2 — cross-link (high breadth, single clean provider, low cost)
166+
167+
- **googletest** (191), **zlib** (200), **openssl_crypto** (152), **abseil** (105), **fmt** (75),
168+
**eigen** (68), **protobuf** (61). Each has 1–3 well-namespaced providers; low ambiguity; strong
169+
value. abseil / eigen / fmt / protobuf carry clear symbol prefixes (`absl:: Eigen:: fmt::
170+
google::protobuf::`) making **symbol** cross-link safe (not just include-resolution).
171+
172+
### TIER 3 — optional (60–99 breadth, cheap, include-resolution only)
173+
174+
cpython (99), zstd (95), cuda_cccl (72), libevent (71), curl (71), libpng (66), libjpeg (65),
175+
glib (60), nlohmann_json (60), xz (60).
176+
177+
### SKIP (long tail < 55 repos, or ambiguous)
178+
179+
mbedtls / lz4 / sqlite / freetype / catch2 / gsl / rapidjson / wolfssl and everything below
180+
(folly 28 … msgpack 1).
181+
182+
Also **SKIP windows_base for symbol-linking** despite its 416-repo breadth: it is almost entirely
183+
`<windows.h>` / `<atlbase.h>` includes against leaked SDK kits with no clean symbol namespace —
184+
include-resolution only, **no symbol cross-link**.
185+
186+
### Net recommendation
187+
188+
Cross-link **~10 high-value libs** for the bulk of cross-repo benefit:
189+
**STL, boost, libc (decls-only), googletest, zlib, openssl, abseil, protobuf, eigen, fmt** —
190+
and **skip the ~30-lib long tail**. Biggest cost/risk items are **linux+glibc** (huge, unprefixed)
191+
and the **11-way STL provider multiplicity** — mitigate by canonicalizing each family to a **SINGLE
192+
provider tree**.
193+
194+
| decision | families | rationale |
195+
|---|---|---|
196+
| TIER 1 cross-link | libstdcxx_STL, boost, libc_posix (decls-only) | top-3 breadth; canonicalize to one provider; libc symbol-link off |
197+
| TIER 2 cross-link | googletest, zlib, openssl_crypto, abseil, fmt, eigen, protobuf | high breadth, clean single provider, prefixed symbols safe |
198+
| TIER 3 optional | cpython, zstd, cuda_cccl, libevent, curl, libpng, libjpeg, glib, nlohmann_json, xz | 60–99 breadth, cheap, include-resolution only |
199+
| SKIP | mbedtls, lz4, sqlite, freetype, catch2, gsl_microsoft, rapidjson, wolfssl, folly, libuv, asio, flatbuffers, benchmark, spdlog, c_ares, harfbuzz, wil_microsoft, mimalloc, lmdb, simdjson, range_v3, libsodium, capnproto, leveldb, msgpack | long tail < 55 repos or ambiguous namespace |
200+
| SKIP (symbol) — include-only | windows_base | 416 breadth but no clean symbol namespace |
201+
202+
## Artifacts
203+
204+
- Ranked JSON (authoritative): `/Volumes/external/sources/cppmega/outputs/crossrepo/base_lib_usage_ranked.json`
205+
- Membership manifest: `/Volumes/external/sources/cppmega/outputs/pr_ingest/repo_list.json`
206+
- This report: `/Volumes/external/sources/cppmega.mlx/outputs/crossrepo/base_lib_crosslink_candidates.md`

0 commit comments

Comments
 (0)