Merge origin/main into mh/trunk-pin-on-cachestack

Bring the cache stack current with main (parallel/streaming commitment
correctness fixes #22184/#22113, nibblized-keccak cache #22185, erigondb.toml
commitment referencing #21452, trie io.Writer trace #21859, etc.). One
conflict in commitment_context.go's trieContext: keep both this branch's
probeSd/probeTx (adaptive trunk-pin probe) and main's traceW (io.Writer trace).

Author: mh0lt

.claude/rules/comments.md

@@ -19,6 +19,6 @@ Default to **no comment**. Clear names and small functions should carry the mean
 
 ## When a comment is warranted
 
-Keep it short and *why*-focused. If a reader could delete it without losing information, it shouldn't exist. Two tight lines beat a six-line essay.
+Keep it concise and *why*-focused, but err on the side of clarity: cut what only adds length, keep what saves the reader a wrong guess. If a reader could delete it without losing information, it shouldn't exist. Two clear sentences beat both a cryptic one-liner and a six-line essay.
 
 Test docstrings get slightly more latitude (explaining a non-obvious scenario the test pins), but the same rules apply — no incident/scope/task narration.

.claude/skills/erigondb-sync-integration-test-plan/SKILL.md

@@ -51,7 +51,7 @@ All scenarios use `--prune.mode=archive`.
    ```bash
    cat <clone-path>/snapshots/erigondb.toml
    ```
-   Expected: `step_size = 1562500`
+   Expected: `step_size = 1562500` and `references_in_commitment_branches = true`
 6. Kill the process and clean up the ephemeral datadir.
 
 ## Scenario B: Fresh sync with downloader
@@ -102,16 +102,34 @@ All scenarios use `--prune.mode=archive`.
    ```bash
    cat <empty-path>/snapshots/erigondb.toml
    ```
-   Expected: `step_size = 1562500` (code defaults, since no downloader to provide network settings)
+   Expected: `step_size = 1562500` and `references_in_commitment_branches = true` (code defaults, since no downloader to provide network settings)
 5. Kill the process and clean up the ephemeral datadir.
 
 ## Success Criteria
 
-| Scenario | Condition | Expected step_size | File timing |
-|----------|-----------|-------------------|-------------|
-| A (legacy) | `Creating erigondb.toml with LEGACY settings` log | 1,562,500 | Written immediately on startup |
-| B (fresh+downloader) | `erigondb.toml not found, using defaults` then `erigondb stepSize changed, propagating` | Code default 1,562,500 then network value (chain-dependent) | After downloader delivers it |
-| C (fresh+no-downloader) | `Initializing erigondb.toml with DEFAULT settings (nodownloader)` log | 1,562,500 | Written immediately on startup |
+| Scenario | Condition | Expected step_size | Expected references_in_commitment_branches | File timing |
+|----------|-----------|-------------------|--------------------------------------------|-------------|
+| A (legacy) | `Creating erigondb.toml with LEGACY settings` log | 1,562,500 | `true` (written explicitly) | Written immediately on startup |
+| B (fresh+downloader) | `erigondb.toml not found, using defaults` then `erigondb stepSize changed, propagating` | Code default 1,562,500 then network value (chain-dependent) | Resolves to `true` in memory until the downloader delivers the file; whatever the network publishes thereafter (a producer-published plain snapshot set ships `false`) | After downloader delivers it |
+| C (fresh+no-downloader) | `Initializing erigondb.toml with DEFAULT settings (nodownloader)` log | 1,562,500 | `true` (written explicitly) | Written immediately on startup |
+
+### references_in_commitment_branches per scenario
+
+The `references_in_commitment_branches` field governs whether new commitment merges write referenced (short-key) `.kv` files. It resolves identically to `step_size` across the three scenarios — absent field normalizes to `true` in memory, and a downloaded `erigondb.toml` is never rewritten (it is synced snapshot metadata, so a producer's published `false` survives). The resolution logic is unit-tested in `db/state/erigondb_settings_test.go` (`TestResolveErigonDBSettings*`); these runtime scenarios confirm the same behavior end-to-end.
+
+- **A (legacy):** the generated file contains `references_in_commitment_branches = true`.
+- **B (fresh+downloader):** no file until the downloader delivers it; the in-memory resolved value is `true`. After delivery the field reflects the network's published value (a plain-snapshot producer ships `false`).
+- **C (fresh+no-downloader):** the immediately-written file contains `references_in_commitment_branches = true`.
+
+## Producer workflow: publishing plain (unreferenced) snapshots
+
+`references_in_commitment_branches` governs only *new* commitment merges; reads are version-aware and stay correct regardless of the flag (a `v2.0` commitment `.kv` whose range ≥ threshold is referenced/short-key; a `v2.1` file is plain). To publish a snapshot set whose commitment files carry no plain-key references:
+
+1. In the producer datadir's `snapshots/erigondb.toml`, set `references_in_commitment_branches = false` **before** running merges.
+2. Run the node / merge so commitment merges produce plain `v2.2` `.kv` files. Existing `v2.1` referenced files convert to plain lazily as they are merged — no upfront migration.
+3. Ship the datadir's `erigondb.toml` with the snapshot set. Consumers read the file and inherit `false`; a downloaded `erigondb.toml` is never rewritten, so the producer's `false` survives.
+
+This supersedes the build-time const flip (`awskii/36exp`): no binary divergence and no CLI flag. Default nodes (field absent → `true`) keep writing byte-compatible `v2.1` referenced files with no behavior change.
 
 ## Cleanup
 

.github/CODEOWNERS

@@ -1,6 +1,6 @@
 *                      @AskAlexSharov @yperbasis
 /.github/workflows     @mriccobene @lystopad
-/cl/                   @domiwei @Giulio2002
+/cl/                   @domiwei @Giulio2002 @sudeepdino008
 /db/                   @AskAlexSharov @sudeepdino008
 /docs/                 @bloxster @AskAlexSharov @yperbasis
 /execution/            @yperbasis @mh0lt

.github/workflows/ci-gate.yml

@@ -2,11 +2,6 @@ name: CI Gate
 
 on:
   pull_request:
-    branches:
-      - main
-      - 'release/**'
-      - performance
-      - performance-stable
     types:
       - opened
       - reopened

.github/workflows/scripts/rpc_version.env

@@ -1,2 +1 @@
-
-RPC_VERSION=fix_expected_bn_as_hex_string
+RPC_VERSION=v2.21.0

ChangeLog.md

@@ -36,6 +36,12 @@ Aligns Erigon with the `eth_simulateV1` error code specification ([NethermindEth
 
 - If your tooling matches on error code `-32602` to detect base-fee-too-low conditions in `eth_simulateV1` responses, update it to match `-38012` instead.
 
+### Added
+
+#### CLI & Operations
+
+- `--prune.distance.blocks` now accepts readable policy names — `keep-post-merge` and `keep-all` — instead of the raw `MaxUint64`-based magic numbers (`18446744073709551615` / `18446744073709551614`); `--prune.distance` likewise accepts `keep-all`. Numeric values still work (#22119) — by @yperbasis
+
 ---
 
 # Erigon v3.5.0 — Tidal Tails — 2026-06-26

agents.md

@@ -132,7 +132,7 @@ A comment may be warranted for: a non-obvious invariant the types don't enforce;
 
 When a comment is genuinely required, it MUST be:
 
-- **One sentence; rarely two; never a paragraph.** No bulleted lists inside `//`. No multi-section block comments with `// Concurrency:` / `// Why:` / `// How to apply:` sub-headings. If the explanation doesn't fit in two sentences, the rest belongs in the commit message, the PR description, or a design doc — not in source, where it goes stale.
+- **Concise, but err on the side of clarity.** Most comments fit in a sentence or two; keep an extra sentence when cutting it would cost understanding, and cut it when it only adds length — two clear sentences beat both a cryptic one-liner and a six-line essay. An explanation that needs sub-headings (`// Concurrency:` / `// Why:`) or bullet lists to stay organized has outgrown source: put the full story in the commit message, the PR description, or a design doc, and keep the distilled why in the code.
 - **High-level, not scenario-specific.** Explain the invariant or gotcha in general terms. Don't walk through specific call sites, sequences of operations, or particular situations a reader could find with `grep`. The right level of abstraction is "what must remain true," not "what happened to me last Tuesday."
 - **Free of forensic detail.** Strip dates (`// found on 2026-05-21`), devnet/branch names (`// seen on bal-devnet-7`), PR/issue/review references (`// flagged in #21314 round-4 review`), incident anecdotes, and "used by X, Y, Z" callsite lists. That history belongs in the commit message and PR description, where it survives intact; in source it rots, misleads later readers, and bloats the file.
 - **Not a restatement of the code.** If a reader could delete the comment without losing information, delete it. Standard Go idioms and well-known library behavior don't need annotation.
@@ -168,7 +168,7 @@ Function docstrings follow the same rule: a one-line summary, plus param/return
 
 `// TODO` notes are only acceptable with a linked tracking issue and an owner. Better: file the issue and don't add the TODO; or fix it now.
 
-**For automated agents specifically:** previous iterations of this guidance were not enough — agents kept producing multi-paragraph block comments enumerating call sites and incident history. Treat the rules above as hard limits. If you catch yourself writing a third sentence in a comment, stop and either delete it, condense to one sentence, or move the content into the commit message.
+**For automated agents specifically:** previous iterations of this guidance were not enough — agents kept producing multi-paragraph block comments enumerating call sites and incident history. The forensic-detail and scenario rules above are hard limits; length is a judgment call with clarity as the tiebreaker. When a comment grows, look at what the growth is made of: call-site inventories and incident history move to the commit message; a sentence that saves the reader a wrong guess stays.
 
 ## Pull Requests & Workflows
 

cmd/commitment-regime/README.md

@@ -0,0 +1,33 @@
+# commitment-regime
+
+Reports whether a commitment domain `.kv` file stores **plain** full keys or
+**referenced** (shortened file-offset) keys in its branch values.
+
+It reads every key/value pair in order and stops at the first branch that
+carries a shortened key:
+
+- `referenced` — at least one shortened key found (prints the pair index where).
+- `plain` — every pair scanned, all keys are full account (20B) / storage (52B) keys.
+
+A malformed branch or corrupt word stream is reported conservatively as
+`referenced`. Files are opened read-only, so a running node need not be stopped.
+
+## Run
+
+```sh
+go run ./cmd/commitment-regime <path/to/vX.Y-commitment.A-B.kv> [more.kv ...]
+```
+
+or build once and point it at a datadir:
+
+```sh
+go build -o /tmp/commitment-regime ./cmd/commitment-regime
+/tmp/commitment-regime <datadir>/snapshots/domain/*commitment*.kv
+```
+
+Example output:
+
+```
+v1.1-commitment.0-512.kv      referenced (first shortened key at pair 4 of 4)
+v2.1-commitment.576-640.kv    plain (scanned all 57763381 pairs)
+```

cmd/commitment-regime/main.go

@@ -0,0 +1,71 @@
+// Command commitment-regime classifies a commitment .kv file as plain or referenced
+// (shortened keys). Temporary diagnostic tooling as-is — remove once the regime work
+// settles, or promote into cmd/integration.
+package main
+
+import (
+	"bytes"
+	"fmt"
+	"os"
+	"path/filepath"
+
+	"github.com/erigontech/erigon/db/seg"
+	"github.com/erigontech/erigon/execution/commitment"
+	"github.com/erigontech/erigon/execution/commitment/commitmentdb"
+)
+
+// scan reads every key/value pair greedily and stops at the first branch that
+// carries a shortened key. referenced=true means at least one shortened key was
+// found; firstAt is the 1-based pair index where it was found.
+func scan(path string) (referenced bool, firstAt, pairs uint64, err error) {
+	d, derr := seg.NewDecompressor(path)
+	if derr != nil {
+		return false, 0, 0, derr
+	}
+	defer d.Close()
+
+	defer func() {
+		if rec := recover(); rec != nil {
+			referenced, err = true, fmt.Errorf("corrupt word stream: %v", rec)
+		}
+	}()
+
+	r := seg.NewReader(d.MakeGetter(), seg.CompressKeys)
+	var keyBuf, valBuf []byte
+	for r.HasNext() {
+		keyBuf, _ = r.Next(keyBuf[:0])
+		if !r.HasNext() {
+			break
+		}
+		valBuf, _ = r.Next(valBuf[:0])
+		pairs++
+		if bytes.Equal(keyBuf, commitmentdb.KeyCommitmentState) {
+			continue
+		}
+		if commitment.BranchData(valBuf).HasShortenedKeys() {
+			return true, pairs, pairs, nil
+		}
+	}
+	return false, 0, pairs, nil
+}
+
+func main() {
+	if len(os.Args) < 2 {
+		fmt.Fprintln(os.Stderr, "usage: commitment-regime <commitment.kv> [more.kv ...]")
+		os.Exit(2)
+	}
+	for _, path := range os.Args[1:] {
+		referenced, firstAt, pairs, err := scan(path)
+		name := filepath.Base(path)
+		switch {
+		case err != nil && referenced:
+			fmt.Printf("%-40s referenced (%v, scanned %d)\n", name, err, pairs)
+		case err != nil:
+			fmt.Printf("%-40s ERROR %v\n", name, err)
+		case referenced:
+			fmt.Printf("%-40s referenced (first shortened key at pair %d of %d)\n", name, firstAt, pairs)
+		default:
+			fmt.Printf("%-40s plain (scanned all %d pairs)\n", name, pairs)
+		}
+	}
+}

cmd/integration/commands/commitment.go

@@ -394,7 +394,7 @@ func commitmentRebuild(db kv.TemporalRwDB, ctx context.Context, logger log.Logge
 	}
 
 	blockSnapBuildSema := semaphore.NewWeighted(int64(runtime.NumCPU()))
-	agg.ForTestReplaceKeysInValues(kv.CommitmentDomain, false)
+	agg.ForTestReferencesInCommitmentBranches(kv.CommitmentDomain, false)
 	agg.SetSnapshotBuildSema(blockSnapBuildSema)
 	agg.SetErigondbDomainStepsInFrozenFile(config3.UnboundedDomainMerge)
 	agg.PresetOfflineMerge()
@@ -557,14 +557,6 @@ func commitmentConvert(db kv.TemporalRwDB, ctx context.Context, logger log.Logge
 	agg.SetSnapshotBuildSema(semaphore.NewWeighted(int64(runtime.NumCPU())))
 	agg.DisableAllDependencies()
 	defer agg.MadvNormal().DisableReadAhead()
-	// commitmentValTransformDomain (the squeeze path's value transformer)
-	// short-circuits to pass-through when the live runtime flag is false.
-	// Force-enable it for the duration of the converter run so --squeeze=true
-	// works regardless of the operator's current schema config. Matches the
-	// pattern in RebuildCommitmentFiles / RebuildCommitmentFilesWithHistory.
-	if opts.TargetSqueeze {
-		agg.ForTestReplaceKeysInValues(kv.CommitmentDomain, true)
-	}
 
 	acRo := agg.BeginFilesRo()
 	defer acRo.Close()