feat: fork mainnet/testnet chain data into the local devnet#453
feat: fork mainnet/testnet chain data into the local devnet#453humble-little-bear wants to merge 2 commits into
Conversation
Add 'offckb devnet fork' implementing the Devnet From Existing Data flow: copy an existing mainnet/testnet data directory into the local devnet, import and patch the source chain spec for local Dummy mining, verify the genesis hash, and boot the first run with --skip-spec-check --overwrite-spec automatically. - src/devnet/fork.ts: fork command (source validation, chain detection, spec fetch/cache or --spec-file, ckb init --import-spec, dev.toml and ckb.toml alignment with offckb's devnet, fork.json state) - src/cmd/node.ts: first run of a fork adds the spec flags and clears them once the node answers RPC - system scripts resolve from the chain's own list-hashes and, when the genesis hash identifies a mainnet/testnet fork, supplement post-genesis deployments (sudt/xudt/omnilock/spore/...) from the static records; the devnet ccc client uses the ckb prefix on a mainnet fork - debug: fall back to get_transaction when the tx json is not in the local proxy cache; fix buildTxFileOptionBy checking the wrong path - tx dumper: embed full header objects in mock_info.header_deps instead of bare hashes (ckb-debugger rejects the latter) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Important Review skippedAuto incremental reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
📝 WalkthroughWalkthroughThis change adds ChangesDevnet fork workflow
Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant User
participant OffckbCLI
participant ForkWorkflow
participant CKBNode
participant Devnet
User->>OffckbCLI: offckb devnet fork --from ...
OffckbCLI->>ForkWorkflow: forkDevnet(options)
ForkWorkflow->>CKBNode: copy and initialize source data
CKBNode-->>ForkWorkflow: genesis hash
ForkWorkflow->>Devnet: write fork.json
User->>OffckbCLI: offckb node
OffckbCLI->>Devnet: start with fork overrides
Devnet-->>OffckbCLI: get_tip_block_number
OffckbCLI->>Devnet: clear firstRunPending
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 6
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/cmd/node.ts`:
- Around line 94-97: Update the first-run readiness flow around ckbProcess and
clearForkFirstRunWhenNodeUp so readiness is tied to the newly spawned fork, not
merely any node at rpcUrl. Abort the readiness polling when ckbProcess exits,
verify the spawned node reports the expected genesis, and only then clear
firstRunPending; preserve the pending flag when startup fails or the port
belongs to an unrelated node.
In `@src/devnet/fork.ts`:
- Around line 322-343: Extend the fork validation around runCkbInit and
genesisHash to read the copied source database’s genesis hash using the
available CKB database inspection command, such as ckb list-hashes. Compare that
value with the imported spec’s genesisHash and reject mismatches before
committing the fork, while preserving the existing source-based expected hash
validation.
- Around line 317-319: Move the copySourceData call into the existing
try/rollback scope so failures from fs.cpSync are handled by the same cleanup
logic. Ensure partial configPath data is removed when copying fails, while
preserving the current behavior for successful copies.
- Around line 252-257: Update runCkbInit to avoid shell interpolation by
replacing execSync with execFileSync or spawnSync and passing the CKB binary
path plus init arguments as a separate argv array. Do not use
encodeBinPathForTerminal for command arguments; preserve the existing dev chain,
import-spec, force flags, UTF-8 output, buffer limit, and debug command logging
without executing user-supplied paths through a shell.
In `@src/scripts/private.ts`:
- Around line 140-151: Update buildDevnetCCCClient to require an explicit unsafe
opt-in when resolved.forkedFrom is 'mainnet', and reject that mode otherwise.
For the opted-in mainnet fork, configure a fork-only replay-protection marker or
cell dependency in the constructed CCC client/known scripts so transactions
cannot validate on mainnet; preserve existing testnet behavior and
address-prefix selection.
In `@src/util/json-rpc.ts`:
- Around line 30-45: Update the HTTP response handling callback around the
IncomingMessage `res` stream to register a one-time `error` listener that
rejects the promise and a `close` listener that rejects when `!res.complete`.
Keep the existing `data`/`end` parsing behavior, and do not rely on an `aborted`
event or flag.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: cd51abf8-7217-4a3c-966f-f5e095e32c2a
📒 Files selected for processing (19)
.changeset/devnet-fork.mdREADME.mdsrc/cli.tssrc/cmd/debug.tssrc/cmd/devnet-fork.tssrc/cmd/node.tssrc/cmd/system-scripts.tssrc/devnet/fork.tssrc/scripts/const.tssrc/scripts/private.tssrc/scripts/util.tssrc/sdk/ckb.tssrc/tools/ckb-tx-dumper.tssrc/util/json-rpc.tstests/ckb-tx-dumper.test.tstests/debug-tx-file.test.tstests/devnet-fork.test.tstests/sdk/ckb.udt.test.tstests/system-scripts.test.ts
| const initOutput = runCkbInit(ckbBinPath, configPath, specFile); | ||
| const genesisHash = parseGenesisHashFromInitOutput(initOutput); | ||
| if (!genesisHash) { | ||
| throw new Error(`Could not parse the genesis hash from ckb init output:\n${initOutput}`); | ||
| } | ||
|
|
||
| // A custom spec may still be a well-known chain; let the chain data | ||
| // self-identify via its genesis hash. | ||
| if (source == null) { | ||
| source = identifyPublicChainByGenesisHash(genesisHash) ?? 'custom'; | ||
| } | ||
| if (source !== 'custom') { | ||
| const expected = expectedGenesisHash(source); | ||
| if (genesisHash !== expected) { | ||
| throw new Error( | ||
| `Genesis hash mismatch: expected ${expected} for ${source}, got ${genesisHash}. ` + | ||
| `This usually means the chain spec does not match the source data. ` + | ||
| `(Importing a testnet spec with a CKB older than v0.207.0 sets a wrong genesis_epoch_length, ` + | ||
| `see nervosnetwork/ckb#5205.)`, | ||
| ); | ||
| } | ||
| } |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift
Verify the copied database genesis, not only the imported spec.
genesisHash comes from ckb init, so --source mainnet with testnet data and a mainnet spec passes this check. The mismatch is only discovered when the fork starts. Read the source database’s genesis hash, for example through ckb list-hashes, and compare it with the imported spec before committing the fork.
🧰 Tools
🪛 ast-grep (0.44.1)
[warning] Importing child_process exposes a command-execution surface; ensure any command/argument built from input is validated, and prefer execFile/spawn with an argument array over exec.
Context: import { execSync } from 'child_process';
Note: [CWE-78] Improper Neutralization of Special Elements used in an OS Command ('OS Command Injection').
(detect-child-process-typescript)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/devnet/fork.ts` around lines 322 - 343, Extend the fork validation around
runCkbInit and genesisHash to read the copied source database’s genesis hash
using the available CKB database inspection command, such as ckb list-hashes.
Compare that value with the imported spec’s genesisHash and reject mismatches
before committing the fork, while preserving the existing source-based expected
hash validation.
There was a problem hiding this comment.
Partially addressed in 1f9019f, with one correction: ckb list-hashes is spec-driven — it resolves the config dir's chain spec and never opens the database, so it cannot literally read the copied db's genesis.
What it does catch is the realistic mislabel case: we now cross-check the source dir's own configured genesis (ckb list-hashes -C <sourceDir>) against the imported spec's genesis and reject before committing the fork. Verified e2e: --source testnet + the real testnet spec over dev-chain data now fails at fork time with a clear error and rolls back.
The residual gap (a db not matching its own config dir) is caught at first boot — ckb refuses to start on a genesis mismatch, and firstRunPending stays set so the flags retry with a clear error.
There was a problem hiding this comment.
@humble-little-bear, thanks for the correction. You’re right: ckb list-hashes validates the resolved chain spec, not the RocksDB contents, so my suggested database-genesis check via that command was inaccurate.
The source-configured-genesis versus imported-spec check in 1f9019f correctly catches the realistic mislabel/spec mismatch before the fork is retained. For the remaining database/config mismatch, CKB’s startup rejection plus retaining firstRunPending provides a safe, recoverable failure path.
🐇🔍
✏️ Learnings added
Learnt from: humble-little-bear
Repo: ckb-devrel/offckb PR: 453
File: src/devnet/fork.ts:349-370
Timestamp: 2026-07-17T15:11:18.222Z
Learning: In `src/devnet/fork.ts`, `ckb list-hashes -C <configDir>` is spec-driven: it resolves the chain spec configured in that directory and does not inspect the RocksDB database’s genesis. It can validate source configuration/spec consistency, but not prove that copied database data matches that configuration.
Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure review instructions. You can manage existing learnings in the app.
- fork: run ckb init via execFileSync argv (no shell interpolation) - fork: roll back partial configPath when the data copy fails - fork: cross-check the source dir's own genesis (list-hashes) against the imported spec so a mislabeled --source/--spec-file is rejected before committing the fork - node: clear firstRunPending only when the spawned node answers with the fork's genesis; abort the poll when the process exits, and never trust an unrelated node occupying the port - json-rpc: reject on response stream error / premature close instead of hanging until the request timeout - docs: caution that fork transactions spending mainnet cells are replayable on mainnet (CKB has no chain id) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 participant
What
Adds
offckb devnet fork— fork an existing Mainnet/Testnet data directory into the local devnet, implementing the Devnet From Existing Data flow (docs.nervos.org#857, tracked in #451) natively in offckb:The result is a locally minable dev chain that carries the source chain's real state (deployed contracts, cells) — useful for reproducing/debugging mainnet transactions and integration-testing against real deployments.
How it works
Fork setup (
src/devnet/fork.ts)data/dbpresent), refuses when a ckb process (source or running offckb devnet, incl. daemon) is alive — copying a live RocksDB corrupts data.ckb.tomlbundled spec;--source/--spec-fileoverride (spec file also enables offline use). Specs are fetched per CKB version and cached.data/(never hardlinks — RocksDB appends WAL/MANIFEST in place), runsckb init --chain dev --import-spec --force, and verifies the genesis hash against the well-known constants, which turns the ckb#5205genesis_epoch_lengthtrap into a loud error.specs/dev.tomlper the guide:pow = Dummy,cellbase_maturity = 0,permanent_difficulty_in_dummy = true; Mainnet keepsgenesis_epoch_length = 1743, Testnet must not have the key (falls back to the default 1000).ckb.toml/ckb-miner.tomlwith offckb's built-in devnet (miner account as block assembler, RPC 8114 + Indexer) so mining rewards flow to the miner account anddepositetc. work on the fork.fork.json;offckb nodeappends--skip-spec-check --overwrite-specon the first run only and clears the flag once the node answers RPC (failed first boots keep the flag and retry).offckb cleanremoves the fork entirely.Fork-aware system scripts (one pipeline, no fork special-casing)
ckb list-hashesof the actual chain spec — on a fork that is the source chain's spec, so genesis scripts are correct with zero changes.system-scripts(now titled e.g.DEVNET (fork of TESTNET)), the SDK'sgetSystemScripts, and ccc known scripts, so transfer/deploy/fee estimation keep working on a fork.ckbaddress prefix (previouslyckb1addresses were rejected on devnet).Debug: local-first
offckb debug --tx-hashnow falls back to fetching the transaction from the node (get_transaction) when it's not in the local proxy cache — historical txs on a fork debug fully offline. Also fixesbuildTxFileOptionBychecking the wrong path (tx json missing previously crashed with ENOENT).mock_info.header_deps; ckb-debugger requires full header objects (ReprMockInfo.header_deps: Vec<HeaderView>), so any tx with header deps (DAO withdrawals, timelocks — the typical mainnet debug case) produced an invalid mock file. Headers are now fetched and embedded.Verification
ckb init --chain testnetsource →offckb devnet fork(auto-detect, genesis hash verified) →offckb node(first-run flags applied and auto-cleared after boot; second run plain) → chain mines on testnet genesis →system-scriptsshowsDEVNET (fork of TESTNET)with supplemented contracts →depositworks (miner rewards mature instantly) →debug --tx-hashon a cache-deleted tx (fallback fires, debugger runs) and on a tx with header deps (full header embedded, debugger runs) → refusal cases (existing devnet w/o--force, node running) →--force+--spec-file→offckb cleanrestores a pureckb_devdevnet.Not in this PR (follow-ups)
reflink/CoW fast copy + disk-space precheck,
devnet fork --status,debug --replay(node-nativeestimate_cyclesreplay), docs cross-linking with the official guide.