Add better processing behavior and tests#134
Merged
Merged
Conversation
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
…lock Adds a `block_processing` table populated by three writers — chaintracks header subscription, the merkle-service callback handler, and the bump-builder consumer — so we can answer "which blocks reached which milestone?" without scanning transactions or compound BUMPs. - Models: `BlockProcessingStatus` with active/orphaned status and pointer-time milestones so JSON cleanly distinguishes "not yet" from zero. - Store: 6 new methods (Upsert/MarkProcessed/MarkBUMPBuilt/MarkOrphaned/ Get/List) implemented across Postgres (column-level ON CONFLICT), Pebble (inverted-uint64 height index for descending scans), and Aerospike (per-bin Operate to avoid clobbering concurrent writers). Drops dead `setProcessedBlocks` helpers in Aerospike along the way. - Writers: chaintracks tip + reorg subscription in api-server, `MarkBlockProcessed` in `handleBlockProcessed` before Kafka publish (log-and-continue on store error), `MarkBlockBUMPBuilt` in bump-builder after `InsertBUMP`. - API: `GET /api/v1/blocks/processing-status` (paginated, descending height) and `GET /api/v1/blocks/processing-status/:blockHash`. - Tests: 8 Pebble + 7 Postgres backend tests + 7 api-server handler tests + 2 bump-builder integration tests; existing test mocks extended to satisfy the larger interface. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`buildServices` and the dependency wiring around it lived as private helpers inside `cmd/arcade/main.go`, which meant test harnesses that want to boot arcade in-process either had to launch the binary or duplicate every kafka/store/teranode/merkle-client setup line. Moves both into a new top-level `app` package with two entry points: `app.Bootstrap(ctx, cfg, logger)` returns a `*Deps` plus a cleanup func (closing publisher → teranode client → store → producer in reverse order), and `app.BuildServices(deps)` returns the slice of services to run for the configured mode. `cmd/arcade/main.go` becomes a thin supervisor that calls the two and handles signals. No behavior change. The full default test suite passes; this lands ahead of the e2e harness which will reuse the same boot path. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Lays the foundation for a reusable end-to-end test harness that boots real backing services and runs against `ghcr.io/bsv-blockchain/ merkle-service:latest` instead of mocks. - Adds testcontainers-go (postgres + redpanda modules) to go.mod. - New `tests/e2e/harness` package, gated behind the `e2e` build tag so the default `go test ./...` run is unaffected. - `harness.New(t, ...Option)` brings up Postgres, Redpanda, and the merkle-service container on a shared user-defined bridge network. merkle-service is wired with `STORE_BACKEND=sql` (Postgres), `KAFKA_BROKERS` (Redpanda alias), `BLOB_STORE_URL=memory:`, and the SSRF/private-IP guards relaxed so callbacks back to `host.docker.internal` succeed. Wait strategy accepts both 200 (healthy) and 503 (degraded — peer count zero before libp2p host attaches) on `/health`. - Container teardown handles partial-start failures and runs via `t.Cleanup`. Helpers expose Postgres DSN / Kafka broker addresses and a `WaitForMerkleLogLine` poll that reads container stdout — used by later steps to assert deterministic events without poking at internal state. - `TestContainers_BootAndTearDown` skips when no container runtime is reachable (so devs without docker/podman aren't blocked) and otherwise takes ~16s warm to boot the stack and verify /health. Verified against rootless podman with `DOCKER_HOST=unix:///run/user/$(id -u)/podman/podman.sock`. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The smoke test needs to drive synthetic SubtreeMessage / BlockMessage announcements that merkle-service consumes via libp2p. Adds a `LibP2PHost` that wraps go-p2p-message-bus and: - Listens on a random TCP port and advertises `/dns4/host.docker.internal/tcp/<port>` so the merkle-service container can dial back via the host gateway. - Subscribes to the regtest block + subtree topics so gossipsub forms mesh links with merkle-service (publish-only peers don't get picked up by the mesh). A 250ms grace after Subscribe avoids a msgbus close-of-closed race when tests tear down quickly. - Exposes `BootstrapMultiaddr()` for feeding to merkle-service via `P2P_BOOTSTRAP_PEERS`, and `PublishBlock` / `PublishSubtree` that JSON-encode `teranode.BlockMessage` / `teranode.SubtreeMessage` and send on the right pubsub topic. `TestLibP2PHost_MerkleServicePeersWithHost` brings up the harness host plus the full container stack and asserts merkle-service logs `[CONNECTED] Topic peer <harness-peer-id>` within 90s. Using the container log line as the peering signal — rather than `msgbus.GetPeers()` — because msgbus's peerTracker only counts peers we've received messages from, which stays empty when merkle-service is silent. Round-trip peering verified end-to-end against rootless podman in ~5s warm. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The smoke test needs to feed arcade and merkle-service the same block + subtree binaries that they would normally pull from a teranode datahub. Adds two pieces: - `harness.Datahub` is an httptest.Server bound on 0.0.0.0:auto-port exposing GET /block/<hash> and GET /subtree/<hash>. Tests Stage* payloads keyed by hash; the merkle-service container reaches the server via the host.docker.internal URL. - `txbuilder.go` mints synthetic transactions (using LockTime to vary txids), packs them into the concatenated-32-byte-hash subtree binary merkle-service expects, and constructs full model.Block.Bytes() payloads via teranode's `model` package. Bitcoin merkle-tree helpers (SubtreeRoot, MerkleRootFromCoinbaseAndSubtree) compose a header- valid layout for callers that need ValidateCompoundRoot to pass downstream. `TestDatahub_StageAndServe` round-trips a synthetic block through arcade's `bump.FetchBlockDataForBUMP`, which exercises the same parseBlockBinary path the bump-builder runs. This verifies our format is wire-compatible with both arcade (unchanged) and merkle-service (model.NewBlockFromBytes uses identical byte layout). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The harness now boots arcade via the same `app.Bootstrap` → `app.BuildServices` path the production binary uses, with config overridden in-memory: - regtest network (chaintracks auto-disabled) - pebble store on a t.TempDir() - in-process memory kafka broker - merkle_service.url = merkle-service container's host-mapped URL - callback_url = host.docker.internal so the container can reach back - datahub_urls seeded with the harness datahub fake - p2p bootstrap_peers = harness libp2p multiaddr (datahub_discovery off) - callback.allow_private_ips = true (RFC1918 loopback by definition) `StartArcade(t, opts)` returns an ArcadeRuntime exposing the bound port, host-side base URL, and merkle-service-reachable host URL. All services start in goroutines bound to the test context; t.Cleanup cancels and waits with a 15s bound. `poll.go` adds the helpers later steps lean on: - BroadcastTx: POST /tx with raw bytes; returns server-side txid. - GetTxStatus / WaitForMined: poll /tx/:txid until every supplied txid reaches MINED with a non-empty merklePath, or fail with the first stuck txid's last-observed status. `TestArcadeRuntime_BootsAndServesHealth` brings up the full stack — containers + libp2p + datahub + arcade — and asserts /health returns 200 with the harness datahub URL listed as healthy. Verified against rootless podman in ~5s warm. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The first cross-service smoke scenario. It boots the full harness (Postgres + Redpanda + merkle-service container, plus in-process libp2p host, datahub fake, and arcade), submits a single transaction to arcade via POST /tx, and asserts the txid lands in merkle-service's registration store via GET /api/lookup/<txid>. That walks every piece of wiring this PR adds: - arcade's tx-validator parses + structurally validates the tx - propagation calls merkleClient.Register over HTTP - merkle-service inside its container reaches Postgres and persists the (txid, callbackUrl, callbackToken) entry Adds harness.BuildValidatableTxs which produces the minimum-valid transactions arcade's structural validator accepts (1 input + 1 output + non-data scripts + LockTime-driven txid uniqueness). TxBuilder unit tests confirm the validator path on these synthetic txs without any container involvement. Round-trips cleanly against rootless podman in ~5s warm. Full MINED status (which requires merkle-tree-valid block construction so arcade's ValidateCompoundRoot accepts the compound BUMP) is the next step — tracked as a gap in tests/e2e/MERKLE_SERVICE_GAPS.md in a follow-up commit. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds .github/workflows/e2e-smoke.yml — pre-pulls the merkle-service image so registry latency stays out of the test's wait budgets, disables Ryuk (flaky on rootless container runtimes), and runs the full e2e suite under the e2e build tag with a 20-min hard timeout. Triggers on every PR and push to main; configured to be the required gate via repo branch-protection settings. tests/e2e/README.md documents the local-run recipe for both Docker and rootless podman (DOCKER_HOST socket override), the file layout, and how to add a new scenario. tests/e2e/MERKLE_SERVICE_GAPS.md captures six friction points the harness work surfaced — backend-import drift, mandatory external Kafka, private-IP guard defaults, health-endpoint signal aliasing, no deterministic block-replay, and missing private-network docs — each with a suggested fix and a note about how the harness works around it today. Worth filing upstream against /git/merkle-service. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The harness now reaches the host from inside containers via the docker bridge gateway IP (auto-discovered) with host.docker.internal as a fallback. README updates to reflect the new flow: - Document the gateway-IP-first announce strategy and the host.docker.internal fallback. - Call out the rootless-podman + pasta limitation: when pasta runs with --no-map-gw (the security-conscious default on recent Fedora/Ubuntu), the host is not reachable from inside containers via either the gateway IP or host.docker.internal. Tests that require merkle-service to dial back into the harness will time out under this configuration. CI uses Docker so the required PR gate is unaffected. - Add a one-liner reachability check developers can run before trying the libp2p-peering tests locally. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
… between integer types' Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request introduces a significant internal refactor to the service initialization logic and adds infrastructure to support end-to-end smoke testing. The main changes are the extraction of shared dependency wiring into a new
apppackage to enable easier reuse and testing, updates to the main entrypoint to use this new structure, and the addition of a GitHub Actions workflow for automated E2E smoke tests. It also adds new dependencies required for containerized integration testing.Refactor: Service Initialization and Dependency Management
cmd/arcade/main.gointo a newapppackage (app/app.go). This allows both the main binary and E2E test harnesses to reuse the same bootstrapping logic without duplication. ([app/app.goR1-R207](https://github.com/bsv-blockchain/arcade/pull/134/files#diff-0f1d2976054440336a576d47a44a37b80cdf6701dd9113012bce0e3c425819b7R1-R207))cmd/arcade/main.goto useapp.Bootstrapandapp.BuildServicesinstead of its own wiring logic, greatly simplifying the main entrypoint and removing now-redundant code. ([[1]](https://github.com/bsv-blockchain/arcade/pull/134/files#diff-ba5f579cff02f2078827e8044948fa803efbd58494f1233bc6697f0d92359fe1L10-L30),[[2]](https://github.com/bsv-blockchain/arcade/pull/134/files#diff-ba5f579cff02f2078827e8044948fa803efbd58494f1233bc6697f0d92359fe1L57-R53),[[3]](https://github.com/bsv-blockchain/arcade/pull/134/files#diff-ba5f579cff02f2078827e8044948fa803efbd58494f1233bc6697f0d92359fe1L199-L269))Testing Infrastructure
.github/workflows/e2e-smoke.yml, a GitHub Actions workflow that runs end-to-end smoke tests by spinning up dependencies (Postgres, Redpanda, Merkle Service) in containers and running a representative transaction flow through the system. This ensures every PR is validated by a real integration test before merging. ([.github/workflows/e2e-smoke.ymlR1-R55](https://github.com/bsv-blockchain/arcade/pull/134/files#diff-739f0c67f2e629274fad22c30bcd0fdfe9e225961341630db183b16f70537ce9R1-R55))Dependency Updates
testcontainers-goand related modules togo.modto facilitate containerized integration testing in E2E scenarios. Also added Docker and related dependencies required for testcontainers. ([[1]](https://github.com/bsv-blockchain/arcade/pull/134/files#diff-33ef32bf6c23acb95f5902d7097b7a1d5128ca061167ec0716715b0b9eeaa5f6R14-R35),[[2]](https://github.com/bsv-blockchain/arcade/pull/134/files#diff-33ef32bf6c23acb95f5902d7097b7a1d5128ca061167ec0716715b0b9eeaa5f6R69),[[3]](https://github.com/bsv-blockchain/arcade/pull/134/files#diff-33ef32bf6c23acb95f5902d7097b7a1d5128ca061167ec0716715b0b9eeaa5f6R78-R91),[[4]](https://github.com/bsv-blockchain/arcade/pull/134/files#diff-33ef32bf6c23acb95f5902d7097b7a1d5128ca061167ec0716715b0b9eeaa5f6R100),[[5]](https://github.com/bsv-blockchain/arcade/pull/134/files#diff-33ef32bf6c23acb95f5902d7097b7a1d5128ca061167ec0716715b0b9eeaa5f6R110),[[6]](https://github.com/bsv-blockchain/arcade/pull/134/files#diff-33ef32bf6c23acb95f5902d7097b7a1d5128ca061167ec0716715b0b9eeaa5f6R190-R191),[[7]](https://github.com/bsv-blockchain/arcade/pull/134/files#diff-33ef32bf6c23acb95f5902d7097b7a1d5128ca061167ec0716715b0b9eeaa5f6R200-R208))