Migrate query-service availability endpoints by ss-es · Pull Request #4270 · EspressoSystems/espresso-network

ss-es · 2026-05-07T14:56:08Z

Migrates remaining query-service availability endpoints to axum.

Notably, this PR does not include new v2 endpoints for any of these. I will add those in a separate PR.

There are tests in crates/espresso/node/src/api.rs that should ensure parity between the axum and tide APIs for all endpoints.

ss-es · 2026-05-08T05:38:05Z

+            axum_msgs.len(),
+        );
+        Ok(())
+    }


it's a bit hard to test streaming websocket endpoints for parity conceptually, since there may be subtle timing differences between them (neither returns deterministic data)

to make this test a bit more resilient, we wait for 10 messages from each and just check that the streams had at least 2 in common

github-actions · 2026-05-08T15:26:07Z

Unable to build without Cargo.lock file.

This means that after this change 3rd party projects may have
difficulties using crates in this repo as a dependency. If this
isn't easy to fix please open an issue so we can fix it later.

For the first failing build see: https://github.com/EspressoSystems/espresso-network/actions/runs/25563643402

To reproduce locally run

cargo generate-lockfile
cargo check --all-targets

This PR can still be merged.

ss-es · 2026-05-11T05:48:17Z

+        std::thread::Builder::new()
+            .stack_size(32 * 1024 * 1024)
+            .spawn(move || {
+                tokio::runtime::Builder::new_multi_thread()
+                    .enable_all()
+                    .build()
+                    .unwrap()
+                    .block_on(async move {
+                        const EPOCH_HEIGHT: u64 = 10;
+                        const NUM_NODES: usize = 5;


it took a lot of experimentation to get this test to not hit a stack overflow. to me this seems innocuous so I didn't try to dig much further to figure out exactly where the line was

claude · 2026-05-11T05:49:48Z

Claude finished @ss-es's task in 3m 42s —— View job

Code Review: Migrate query-service availability endpoints

Read repository guidelines (CLAUDE.md)
Read existing review comments
Analyze PR diff and changed files
Review code quality, security, performance, and testing
Post detailed review feedback

Summary

This PR migrates the remaining query-service availability endpoints from tide-disco to Axum, adding a new HotShotAvailabilityApi trait with ~30 endpoints (leaves, headers, blocks, payloads, VID common, transactions, block summaries, limits, cert2) plus 8 WebSocket streaming endpoints. The trait design, handler wiring, and parity tests are solid overall.

Key Finding

Missing range limit enforcement on HotShotAvailabilityApi range endpoints — The original hotshot-query-service enforces enforce_range_limit on all range queries (small_object_range_limit=500 for leaves/vid_common, large_object_range_limit=100 for blocks/headers/payloads/summaries). The new Axum implementation passes from..until directly to the data source without any validation. A client can request an arbitrarily large range (e.g. /v1/availability/leaf/0/1000000) and the server will try to load and serialize the entire range into a Vec, risking OOM or excessive DB load. See inline comment.

Minor Notes

Inconsistent height type in stream_namespace_proofs handler (u64 vs usize used by all other stream handlers). See inline comment.
Hardcoded /limits response reports limits that aren't enforced server-side. See inline comment.

What looks good

Clean trait separation: AvailabilityApi for espresso-specific endpoints vs HotShotAvailabilityApi for generic hotshot mirroring.
drive_ws_stream is a clean, reusable helper for WebSocket streaming.
block_id_to_hs / payload_id_to_hs conversion helpers are well-structured.
Thorough parity tests comparing tide-disco and axum responses across all new endpoints, including WebSocket streams.
The RawAvidMShare custom Debug impl avoids log spam from large payloads — nice cleanup.
The 32 MiB stack thread workaround for the test is well-documented with the reasoning.

github-actions · 2026-05-12T18:37:55Z

Nextest failures (1) in this run

Test	Attempts	Time (s)	Main history
`hotshot-testing::test_epoch_upgrade::test_epoch_upgrade::testtypes_::memoryimpl_::test_epoch_upgrade`	1	15.38	passing

See the step summary for flaky tests and slowest tests.

sveitser · 2026-05-13T08:28:47Z

 parking_lot     = { workspace = true }
 rand            = "0.10.0"
+
+# optional:


sveitser · 2026-05-13T10:24:46Z

+        from: usize,
+        namespace: u32,
+    ) -> anyhow::Result<futures::stream::BoxStream<'static, Self::NamespaceProofQueryData>> {
+        use espresso_types::{NamespaceId, NamespaceProofQueryData, NsProof};


Move imports to top?

sveitser · 2026-05-13T10:25:40Z

+    type BlockSummary =
+        hotshot_query_service::availability::BlockSummaryQueryData<espresso_types::SeqTypes>;
+    type Limits = hotshot_query_service::availability::Limits;
+    type Cert2 = hotshot_new_protocol::message::Certificate2<espresso_types::SeqTypes>;


import some of these types for readability?

sveitser · 2026-05-13T10:30:29Z

 async-trait = { workspace = true }
+
+# Testing dependencies
+bitvec = { workspace = true, optional = true }


accidental cargo sort without -g?

sveitser · 2026-05-13T14:02:57Z

@ss-es I think my main concern is that we should do some equivalence checks for common errors where they are informative, like range limits / not found. To ensure we are returning reasonable errors from Axum for clients. At least for errors that they probably care about.

ss-es added 30 commits April 23, 2026 02:26

add availability endpoint

eae4c87

update proto definitions with more detail

b5c2851

remove types defined in axum.rs

9f26eab

remove unhelpful comments

4bf2129

cleanup

e1a08f0

collapse swagger by default

1192448

add example values

be65c6f

cleanup

d94ea9e

cleanup examples

592fea6

update readme

3fc97f3

Merge branch 'main' into ss/axum-2

318ebee

fix formatting

e4ec214

update with serialization fixes/cleanup

33e6ca2

fix inconsistency

a6d2f64

move descriptions into routes

846fe8d

clippy

bdbb33d

cargo sort

67c708c

update

178357d

add new v1 API tests

408e4dc

fix reward merkle tree

06c508a

fix reward balance endpoint

f94e2a4

refactor test

ab8103e

fix latest reward balance

96ac79a

fix routes

7d86f58

fix test

dafaccd

rustfmt

d6f02f5

fix reward-amounts

f28b106

Merge branch 'ss/axum-tests-1' into ss/axum-2

5a678c8

add parity tests

9d917b1

update incorrect_encoding_proof

9864160

clippy

c7298e8

ss-es commented May 8, 2026

View reviewed changes

ss-es added 4 commits May 8, 2026 02:59

fix stack overflow

79cf683

start websocket stream earlier

e3e0a30

clean up log

45d645b

Merge branch 'main' into ss/axum-4

6662fb5

ss-es added 5 commits May 8, 2026 11:44

fix

d126eac

spawn thread with 16MB stack

6e54a63

fix?

f4ed102

fix

b7517c5

fix?

bf1a4c1

ss-es commented May 11, 2026

View reviewed changes

ss-es marked this pull request as ready for review May 11, 2026 05:49

ss-es requested review from bfish713, imabdulbasit, jbearer, lukaszrzasik and sveitser as code owners May 11, 2026 05:49

claude Bot reviewed May 11, 2026

View reviewed changes

Comment thread crates/espresso/node/src/api/state.rs

claude Bot reviewed May 11, 2026

View reviewed changes

Comment thread crates/espresso/node/src/api/state.rs

claude Bot reviewed May 11, 2026

View reviewed changes

Comment thread crates/espresso/api/src/axum.rs Outdated

fix

492847d

sveitser reviewed May 13, 2026

View reviewed changes

Comment thread crates/cliquenet/Cargo.toml

parking_lot = { workspace = true }

rand = "0.10.0"

# optional:

Copy link
Copy Markdown

Collaborator

sveitser May 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

undo?

sveitser reviewed May 13, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Migrate query-service availability endpoints#4270

Migrate query-service availability endpoints#4270
ss-es wants to merge 52 commits into
mainfrom
ss/axum-4

ss-es commented May 7, 2026 •

edited

Loading

Uh oh!

ss-es May 8, 2026

Uh oh!

github-actions Bot commented May 8, 2026

Uh oh!

ss-es May 11, 2026

Uh oh!

claude Bot commented May 11, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented May 12, 2026

Uh oh!

sveitser May 13, 2026

Uh oh!

sveitser May 13, 2026

Uh oh!

sveitser May 13, 2026

Uh oh!

sveitser May 13, 2026

Uh oh!

sveitser commented May 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

ss-es commented May 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ss-es May 8, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented May 8, 2026

Uh oh!

ss-es May 11, 2026

Choose a reason for hiding this comment

Uh oh!

claude Bot commented May 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Review: Migrate query-service availability endpoints

Summary

Key Finding

Minor Notes

What looks good

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented May 12, 2026

Nextest failures (1) in this run

Uh oh!

sveitser May 13, 2026

Choose a reason for hiding this comment

Uh oh!

sveitser May 13, 2026

Choose a reason for hiding this comment

Uh oh!

sveitser May 13, 2026

Choose a reason for hiding this comment

Uh oh!

sveitser May 13, 2026

Choose a reason for hiding this comment

Uh oh!

sveitser commented May 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ss-es commented May 7, 2026 •

edited

Loading

claude Bot commented May 11, 2026 •

edited

Loading