fix(rivetkit-core): orphan engine process and detect crashes

[NathanFlurry]

Description

Please include a summary of the changes and the related issue. Please also include relevant motivation and context.

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

How Has This Been Tested?

Please describe the tests that you ran to verify your changes.

Checklist:

• My code follows the style guidelines of this project
• I have performed a self-review of my code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes

[NathanFlurry]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• fix(rivetkit-core): gate startup until runtime is ready #4736 : 2 dependent PRs (#4737 , #4739 )
• fix(rivetkit-sqlite): preserve text with embedded nul bytes #4735
• chore(rivetkit): add child spans for logging #4734
• fix(rivetkit-core): orphan engine process and detect crashes #4733 👈 (View in Graphite)
• fix(rivetkit-core): return None for empty inspector state to avoid frontend CBOR decode error #4732
• fix(rivetkit-core): don't install tokio::signal::ctrl_c() handler inside serve_with_config #4731
• refactor(rivetkit-core): use subtle::ConstantTimeEq for inspector token verify #4730
• fix(rivetkit): drop is_read_only_sql classifier, surface RETURNING rows #4728
• docs: require close-code rejection for websocket auth/policy failures #4727
• fix(rivetkit): inspector reports actual config state + real queue messages #4726
• fix(rivetkit-napi): plumb ctx through serializeState TSF and stop panicking on runtime_state Ref drop #4725
• chore(rivetkit): remove legacy metrics & prometheus auth #4724
• chore(rivetkit): remove legacy metrics #4723
• chore(rivetkit): impl inspector token #4722
• chore: disable auth on serverless #4721
• chore: fix cors for envoys #4720
• chore: fix serverless in rivetkit-core #4719
• chore(rivetkit): impl follow up review #4711
• chore: rivetkit core/napi/typescript follow up review #4702
• chore: fix remaining issues with rivetkit-core #4701
• chore: sqlite v1 to v2 data migration #4692 : 1 other dependent PR (#4640 )
• chore: move rivetkit to task model #4680
• chore: rivetkit to rust #4679
• feat(rivetkit): sqlite vfs v2 #4673
• chore: rename sandbox -> kitchen sink #4671 : 1 other dependent PR (#4672 )
• chore: configure TLS trust roots for rustls clients #4667
• fix(rivetkit): update config to match envoys & remove manager references #4663
• chore(publish): pin docker base image refs #4652
• chore(engine): publish engine bases in ci #4649 : 2 other dependent PRs (#4650 , #4666 )
• chore: lockfile lefthook #4647
• fix(rivetkit): surface native sqlite kv errors to callers #4645
• fix(rivetkit): prevent sleep races during disconnect and db work #4644
• fix(rivetkit): isolate engine envoys and propagate startup failures #4643
• test(rivetkit): remove wasm sqlite-only fixtures and coverage #4642
• chore(sqlite-vfs): remove wasm sqlite packages and workspace wiring #4641
• chore(rivetkit): remove wasm sqlite runtime #4614
• fix(sqlite-native): restore native startup kv preload #4639
• fix(sqlite-native): delete metadata before chunk range #4638
• fix(sqlite-native): keep truncate cache coherent #4637
• fix(sqlite-vfs): use delete range for truncate cleanup #4636
• fix(sqlite-native): restore kv error hook #4635
• perf(sqlite-native): avoid cloning cached read chunks #4634
• perf(sqlite-native): gate kv operation labels #4633
• perf(sqlite-native): reuse read cache for partial writes #4632
• perf(sqlite-native): remove delete file existence probe #4631
• fix(sqlite-native): gate native read cache behind env var #4630
• fix(pegboard): isolate runner config dc lookup failures #4625 : 1 other dependent PR (#4626 )
• docs(pegboard): note runner config upsert split write risk #4624
• fix(pegboard): unpack global runner config keys correctly #4623
• fix(pegboard-gateway): enforce tunnel message state #4622
• fix(pegboard-runner): clear terminal tunnel routes #4621
• fix(pegboard): skip protocol version keys in runner pool backfill #4620
• fix(pegboard): create missing normal runner configs #4619
• fix(epoxy): reuse cached optimistic value #4618
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[claude]

PR Review: fix(rivetkit-core): orphan engine process and detect crashes

Overview

This PR refactors engine_process.rs to implement intentional process orphaning of the rivet-engine subprocess. The goal is for the engine to outlive the rivetkit host so in-flight actor state survives dev-server hot reloads. The design is architecturally sound and the module-level doc comment explains the contract well.

Key changes:

• Removes kill_on_drop(true) - engine now survives host death
• Routes engine stdio to timestamped log files (file fds survive host pipe closure)
• Uses process_group(0) on Unix to isolate engine from terminal signals (Ctrl+C, SIGHUP)
• Adds probe_existing_engine() to detect and reuse a running engine
• Introduces persistent stable storage at ~/.rivetkit/ (replaces random UUID temp dirs)
• Adds crash-detection watcher task (spawn_engine_watcher)
• Removes the shutdown() method entirely
• Adds PortOccupied error variant

---

Issues

Bug: terminate_failed_spawn() leaks the process on timeout

When the 5-second timeout fires, the function returns Ok(()) after dropping child - but with kill_on_drop=false (the new default), dropping Child does NOT kill the process. The half-started engine leaks and a subsequent start() call will hit a port conflict. child.start_kill() should be called before returning Ok(()) here.

Bug: probe_existing_engine() can spuriously block start when runtime field is absent

If an older engine version does not include the runtime field in its health response (it is Option), this produces a false PortOccupied error against a healthy engine that should be reused. Consider treating None the same as Some(ENGINE_RUNTIME), or document the minimum engine version that includes this field.

Unintuitive env var behavior in storage_root()

Setting RIVETKIT_STORAGE_PATH=~/data puts storage at ~/data/.rivetkit, not ~/data, because .rivetkit is unconditionally joined onto the env var path. Either treat the env var as the direct storage root or rename it to make the parent-dir semantics explicit.

Surprising current_dir() fallback in storage_root()

If HOME is unset (containers, CI), state silently lands in whatever the cwd is at startup. Failing with an actionable error pointing to RIVETKIT_STORAGE_PATH would be more predictable.

---

Suggestions

Collapse redundant match arms in probe_existing_engine - Some(RIVETKIT_RUNTIME) and Some(other) do identical things and can be collapsed with a wildcard arm that uses health.runtime.as_deref().unwrap_or("unknown") for the PortOccupied runtime field.

Add pid to terminate_failed_spawn logs - capture child.id() before start_kill and include it in every log line for easier correlation with ps/journalctl. The original code logged pid here.

Use sub-second precision in log_timestamp() - two spawns within the same second produce identically named log files. With append mode they interleave silently. Using milliseconds or including the process pid in the filename would make filenames unique.

Log file permissions - log files are created with default OpenOptions permissions (mode 0666 before umask). Engine stderr may contain credentials or error details. Consider creating the logs directory with mode 0700 on Unix.

---

Correctness

The core orphaning contract is sound: kill_on_drop=false, process_group(0), and file-fd stdio together correctly implement the intentional-orphan pattern. The zombie-reaping comment in spawn_engine_watcher is accurate - calling child.wait() in the task reaps the process via waitpid.

Windows gap: process_group(0) is #[cfg(unix)] only. On Windows the engine may still be terminated through a job object when the parent exits. Worth documenting.

---

Test Coverage

No tests are added. Since this is a draft PR that may be intentional, but high-value cases to cover before merge:

• probe_existing_engine reattaches to a healthy engine (Some(ENGINE_RUNTIME))
• probe_existing_engine errors on unknown runtime (PortOccupied)
• storage_root() env var override produces the expected path

---

Draft PR - the two most important issues to address before marking ready are the timeout-path process leak in terminate_failed_spawn and the None runtime false-positive in probe_existing_engine.

[NathanFlurry]

Landed in main via stack-merge fast-forward push. Commits are in main; closing to match.