Merge remote-tracking branch 'origin/pr/559' into release/v0.10.0-rc7

Author: bussyjd

CLAUDE.md

@@ -176,6 +176,10 @@ obol sell delete ollama-gated -n llm
 
 Two-stage templating: `values.yaml.gotmpl` with `@enum/@default/@description` annotations → CLI flags → rendered `values.yaml` (Stage 1), then `helmfile sync --state-values-file values.yaml --state-values-set id=<id>` (Stage 2). Unique namespaces: `<network>-<id>` where ID is petname or `--id <name>`. Local Ethereum nodes auto-registered as priority upstream in eRPC via `RegisterERPCUpstream()` (write methods blocked on local → routed to remote).
 
+**Ethereum `--mode full|archive`** (default `full`): controls whether reth runs as a pruned full node (~500 GB mainnet / ~100 GB testnet) or an archive node retaining all historical state (~4 TB+ mainnet / ~300 GB testnet). Archive mode is for state replay (block explorers, historical `eth_call`, indexers); full mode is the right default for everything else. The mode flows through to (a) the reth `--full` arg in `internal/embed/networks/ethereum/helmfile.yaml.gotmpl`, (b) PVC sizing in `templates/pvc.yaml`, and (c) the `helmfile` `persistence.size` request. `obol network install ethereum` runs a disk-space preflight via `internal/network/preflight.go` — it warns when `cfg.DataDir` has less free disk than `(network, mode)` is expected to need, prompts the user, and auto-continues in non-interactive mode (no TTY / JSON output) so scripted installs don't deadlock. Other execution clients (geth, nethermind, besu, erigon) ignore the mode flag for now.
+
+**Ethereum `--since` (partial archive)**: when `--mode=archive` would otherwise mean genesis-to-tip, `--since` bounds the archive at a known historical point and translates to reth's `--prune.account-history.{before,distance}` + `--prune.storage-history.{before,distance}` flags (plus `--prune.receipts.pre-merge` / `--prune.bodies.pre-merge` when the cutoff is at or before the merge). Accepted forms: EL hardfork names (`merge`, `shanghai`, `cancun`, `prague`, `osaka` — mainnet only, verified block numbers in `internal/network/hardforks.go`); durations (`365d`, `1y`, `6mo` — resolved against the post-merge 12s slot rate as a `--prune.*.distance` value); raw block numbers (`22500000`); or `genesis`/`all` (no extra args). Resolution happens in `resolveEthereumArchiveScope` in `internal/network/picker.go`: `--since` wins outright; if `--mode` is unset and a TTY is attached, a `full vs archive` picker runs; if `--mode=archive` is set without `--since` on a TTY, an `Archive scope` picker offers the hardfork presets + custom block + 365 days + genesis. Non-TTY defaults to `mode=full` (mode unset) or `since=genesis` (mode=archive). Resolved scope is appended to `values.yaml` as `pruneKind` / `pruneBlock` / `pruneDistance` and consumed by the helmfile. Partial archive is wired only for reth; geth/besu/erigon/nethermind emit a warning and run with chart defaults. Hardfork-name presets are rejected on testnets (mainnet block numbers don't apply).
+
 ## Stack Lifecycle
 
 | Command | Action |

docs/getting-started.md

@@ -216,6 +216,64 @@ obol network sync ethereum/demo
 
 This creates the `ethereum-demo` namespace with an execution client (reth) and a consensus client (lighthouse).
 
+### Full vs archive mode
+
+`obol network install ethereum` defaults to `--mode=full`, which prunes
+historical state and needs ~500 GB on mainnet (~100 GB on testnets). Pass
+`--mode=archive` if you need to replay state across history (block
+explorers, historical `eth_call`, indexers); archive nodes hold the full
+state trie and grow to ~4 TB+ on mainnet.
+
+```bash
+# Default: pruned full node
+obol network install ethereum --network=mainnet
+
+# Archive node for state replay (requires ~4-5 TB free)
+obol network install ethereum --network=mainnet --mode=archive
+```
+
+When `--mode` is omitted on a TTY, the installer prompts. The disk-space
+preflight warns when the data directory has less free disk than the
+chosen mode is likely to need.
+
+### Partial archive (`--since`)
+
+A full mainnet archive from genesis is ~4 TB+, but most archive use cases
+(indexers, recent-state replay) only need history back to a known point.
+`--since` keeps an archive bounded — reth gets the right `--prune.*`
+flags wired through the chart:
+
+```bash
+# Archive back to the merge (Sep 2022, ~1.5 TB)
+obol network install ethereum --network=mainnet --mode=archive --since=merge
+
+# Archive back to Cancun (Mar 2024, ~800 GB)
+obol network install ethereum --network=mainnet --mode=archive --since=cancun
+
+# Archive of the last 365 days (~600 GB)
+obol network install ethereum --network=mainnet --mode=archive --since=365d
+
+# Archive from a specific block forward
+obol network install ethereum --network=mainnet --mode=archive --since=22500000
+```
+
+Accepted `--since` values:
+
+| Form | Example | Meaning |
+|---|---|---|
+| EL fork name | `merge`, `shanghai`, `cancun`, `prague`, `osaka` | Prune state before that mainnet hardfork |
+| Duration | `365d`, `1y`, `6mo` | Keep last N blocks (~12s slot rate) |
+| Block number | `22500000` | Prune state before that block |
+| `genesis` / `all` | `genesis` | Full archive from genesis |
+
+When `--mode=archive` is set without `--since` on a TTY, the installer
+shows an interactive picker. On non-TTY (scripts, CI), the default is
+`all history`. `--since` is currently fine-tuned for **reth**; other
+execution clients fall back to their chart-default behavior with a warning.
+
+Fork-name presets reference mainnet block numbers — use a raw block
+number or a duration on testnets.
+
 Verify:
 
 ```bash

internal/embed/infrastructure/values/erpc.yaml.gotmpl

@@ -29,7 +29,10 @@ config: |-
         - id: memory-cache
           driver: memory
           memory:
-            maxItems: 10000
+            # LRU cap on cached RPC responses. The connector is memory-only
+            # (no disk), so the upper bound is enforced by the pod memory
+            # limit below; this cap keeps cache RAM well under that limit.
+            maxItems: 5000
       policies:
         - network: "*"
           method: "*"

internal/embed/infrastructure/values/monitoring.yaml.gotmpl

@@ -11,7 +11,12 @@ prometheus:
       matchLabels:
         release: monitoring
     podMonitorNamespaceSelector: {}
-    retention: 8d
+    # Time-based retention is still honored, but retentionSize is the hard
+    # cap that prevents the TSDB from filling the k3d node's writable layer
+    # overnight and triggering DiskPressure → cascading pod evictions.
+    # 2GB on emptyDir is plenty for a local single-node stack.
+    retention: 2d
+    retentionSize: 2GB
     resources:
       requests:
         cpu: 100m

internal/embed/networks/aztec/helmfile.yaml.gotmpl

@@ -9,7 +9,8 @@ releases:
     namespace: aztec-{{ .Values.id }}
     createNamespace: true
     chart: obol/aztec-node
-    version: 0.2.0
+    # renovate: datasource=helm depName=aztec-node registryUrl=https://obolnetwork.github.io/helm-charts
+    version: 2.3.0
     values:
       - role: sequencer
         sequencer:
@@ -19,7 +20,8 @@ releases:
         networkName: '{{ .Values.id }}'
         image:
           repository: aztecprotocol/aztec
-          tag: 2.1.5
+          # renovate: datasource=docker depName=aztecprotocol/aztec
+          tag: 4.3.0
           pullPolicy: Always
         node:
           replicas: 1

internal/embed/networks/ethereum/helmfile.yaml.gotmpl

@@ -24,9 +24,20 @@ releases:
     namespace: ethereum-{{ .Values.id }}
     createNamespace: true
     chart: ethereum-helm-charts/ethereum-node
+    # renovate: datasource=helm depName=ethereum-node registryUrl=https://ethpandaops.github.io/ethereum-helm-charts
+    version: 0.2.14
     needs: [ethereum-pvcs]
     values:
       # Network and checkpoint sync configuration
+      #
+      # global.clientArgs.networks.<network>.execution.reth is the upstream
+      # umbrella chart's per-network reth args list. We APPEND to it rather
+      # than overriding reth.extraArgs directly, because reth.extraArgs in
+      # the umbrella chart is a templated string that looks up this very
+      # list — overriding it would silently drop the --chain=<testnet> arg
+      # that testnets depend on. Mainnet's upstream list is empty; testnets
+      # are ["--chain=<network>"]. We carry those over verbatim and add
+      # --full only when mode != archive.
       - global:
           main:
             network: '{{ .Values.network }}'
@@ -36,46 +47,91 @@ releases:
               mainnet: https://mainnet-checkpoint-sync.attestant.io
               sepolia: https://checkpoint-sync.sepolia.ethpandaops.io
               hoodi: https://checkpoint-sync.hoodi.ethpandaops.io
+          {{- if eq .Values.executionClient "reth" }}
+          clientArgs:
+            networks:
+              {{ .Values.network }}:
+                execution:
+                  reth:
+                    {{- if ne .Values.network "mainnet" }}
+                    - --chain={{ .Values.network }}
+                    {{- end }}
+                    {{- if ne .Values.mode "archive" }}
+                    - --full
+                    {{- else if eq (.Values.pruneKind | default "") "before" }}
+                    # Partial archive: prune state history before block N.
+                    # Receipts/bodies are pruned to the same cutoff via the
+                    # pre-merge presets where applicable; otherwise reth
+                    # uses the same `before` value.
+                    - --prune.account-history.before={{ .Values.pruneBlock }}
+                    - --prune.storage-history.before={{ .Values.pruneBlock }}
+                    {{- if le (int .Values.pruneBlock) 15537394 }}
+                    - --prune.receipts.pre-merge
+                    - --prune.bodies.pre-merge
+                    {{- else }}
+                    - --prune.receipts.before={{ .Values.pruneBlock }}
+                    - --prune.bodies.before={{ .Values.pruneBlock }}
+                    {{- end }}
+                    {{- else if eq (.Values.pruneKind | default "") "distance" }}
+                    # Partial archive: keep last N blocks of history.
+                    - --prune.account-history.distance={{ .Values.pruneDistance }}
+                    - --prune.storage-history.distance={{ .Values.pruneDistance }}
+                    {{- end }}
+          {{- end }}
 
-      # Execution client (pinned versions — update periodically)
+      # Execution client (pinned versions — Renovate-tracked)
+      # Reth defaults to archive (~4TB+ mainnet). The --mode flag controls
+      # whether we pass --full to prune historical state down to ~500GB,
+      # wired through global.clientArgs above (NOT reth.extraArgs — see
+      # comment above).
       - {{ .Values.executionClient }}:
           enabled: true
           image:
             {{- if eq .Values.executionClient "reth" }}
-            tag: v1.11.1
+            # renovate: datasource=github-releases depName=paradigmxyz/reth
+            tag: v2.2.0
             {{- else if eq .Values.executionClient "geth" }}
-            tag: v1.17.0
+            # renovate: datasource=github-releases depName=ethereum/go-ethereum
+            tag: v1.17.3
             {{- else if eq .Values.executionClient "nethermind" }}
-            tag: "1.36.0"
+            # renovate: datasource=github-releases depName=NethermindEth/nethermind
+            tag: "1.37.2"
             {{- else if eq .Values.executionClient "besu" }}
-            tag: "26.2.0"
+            # renovate: datasource=github-releases depName=hyperledger/besu
+            tag: "26.5.0"
             {{- else if eq .Values.executionClient "erigon" }}
-            tag: v3.3.8
+            # renovate: datasource=github-releases depName=erigontech/erigon
+            tag: v3.4.2
             {{- end }}
           persistence:
             enabled: true
-            size: 500Gi
+            size: {{ if eq .Values.network "mainnet" }}{{ if eq .Values.mode "archive" }}4500Gi{{ else }}500Gi{{ end }}{{ else }}{{ if eq .Values.mode "archive" }}300Gi{{ else }}100Gi{{ end }}{{ end }}
             existingClaim: execution-{{ .Values.executionClient }}-{{ .Values.network }}
 
-      # Consensus client (pinned versions — update periodically)
+      # Consensus client (pinned versions — Renovate-tracked)
       # The upstream chart wires --execution-endpoint and --network automatically.
       - {{ .Values.consensusClient }}:
           enabled: true
           image:
             {{- if eq .Values.consensusClient "lighthouse" }}
-            tag: v8.1.1
+            # renovate: datasource=github-releases depName=sigp/lighthouse
+            tag: v8.1.3
             {{- else if eq .Values.consensusClient "prysm" }}
-            tag: v7.1.2
+            # renovate: datasource=github-releases depName=OffchainLabs/prysm
+            tag: v7.1.4
             {{- else if eq .Values.consensusClient "teku" }}
-            tag: "26.2.0"
+            # renovate: datasource=github-releases depName=Consensys/teku
+            tag: "26.4.0"
             {{- else if eq .Values.consensusClient "nimbus" }}
-            tag: multiarch-v26.3.0
+            # renovate: datasource=docker depName=statusim/nimbus-eth2
+            tag: multiarch-v26.5.0
             {{- else if eq .Values.consensusClient "lodestar" }}
-            tag: v1.40.0
+            # renovate: datasource=github-releases depName=ChainSafe/lodestar
+            tag: v1.43.0
             {{- end }}
           persistence:
             enabled: true
-            size: 200Gi
+            size: {{ if eq .Values.network "mainnet" }}{{ if eq .Values.mode "archive" }}500Gi{{ else }}200Gi{{ end }}{{ else }}{{ if eq .Values.mode "archive" }}100Gi{{ else }}50Gi{{ end }}{{ end }}
             existingClaim: consensus-{{ .Values.consensusClient }}-{{ .Values.network }}
 
   # Metadata ConfigMap for frontend discovery

internal/embed/networks/ethereum/templates/pvc.yaml

@@ -1,4 +1,32 @@
 {{- if eq .Release.Name "ethereum-pvcs" }}
+{{- /*
+  PVC sizing is a function of (network, mode). Sizes are estimates with
+  ~30% headroom for chain growth. local-path storage does not pre-allocate,
+  so these requests primarily document intent and serve as soft caps when
+  a sized storage class is swapped in later.
+*/ -}}
+{{- $mode := default "full" .Values.mode -}}
+{{- $isArchive := eq $mode "archive" -}}
+{{- $execSize := "500Gi" -}}
+{{- $consensusSize := "200Gi" -}}
+{{- if eq .Values.network "mainnet" -}}
+  {{- if $isArchive -}}
+    {{- $execSize = "4500Gi" -}}
+    {{- $consensusSize = "500Gi" -}}
+  {{- else -}}
+    {{- $execSize = "500Gi" -}}
+    {{- $consensusSize = "200Gi" -}}
+  {{- end -}}
+{{- else -}}
+  {{- /* sepolia, hoodi and other testnets */ -}}
+  {{- if $isArchive -}}
+    {{- $execSize = "300Gi" -}}
+    {{- $consensusSize = "100Gi" -}}
+  {{- else -}}
+    {{- $execSize = "100Gi" -}}
+    {{- $consensusSize = "50Gi" -}}
+  {{- end -}}
+{{- end -}}
 ---
 # Ethereum Execution Client PVC
 apiVersion: v1
@@ -12,7 +40,7 @@ spec:
   storageClassName: local-path
   resources:
     requests:
-      storage: 500Gi
+      storage: {{ $execSize }}
 ---
 # Ethereum Consensus Client PVC
 apiVersion: v1
@@ -26,5 +54,5 @@ spec:
   storageClassName: local-path
   resources:
     requests:
-      storage: 200Gi
+      storage: {{ $consensusSize }}
 {{- end }}

internal/embed/networks/ethereum/values.yaml.gotmpl

@@ -15,3 +15,12 @@ executionClient: {{.ExecutionClient}}
 # @default lighthouse
 # @description Consensus layer client
 consensusClient: {{.ConsensusClient}}
+
+# @enum full,archive
+# @default full
+# @description Node mode. 'full' prunes historical state (~500GB mainnet); 'archive' keeps all state for history replay (~4TB+ mainnet)
+mode: {{.Mode}}
+
+# @default
+# @description Archive scope (only used with --mode=archive). Accepts: 'merge', 'shanghai', 'cancun', 'prague', 'osaka' (EL fork name); duration like '365d', '1y', '6mo'; raw block number; or 'genesis' for all history. Prompts interactively on TTY if omitted.
+since: {{.Since}}