Skip to content

refactor: Wave 3 — typed SubscriptionTier + ArcSwap streaming state + layout moves + LOW batch (refs #500)#507

Merged
userFRM merged 1 commit into
mainfrom
refactor/wave3-mega
May 7, 2026
Merged

refactor: Wave 3 — typed SubscriptionTier + ArcSwap streaming state + layout moves + LOW batch (refs #500)#507
userFRM merged 1 commit into
mainfrom
refactor/wave3-mega

Conversation

@userFRM
Copy link
Copy Markdown
Owner

@userFRM userFRM commented May 7, 2026

Summary

Wave 3 of the #500 refactor sweep. Four items, one squashable commit, version
bump 8.0.36 → 8.0.37, CHANGELOG entry mirrored to docs-site/. No behavior
change at the wire / FPSS / MDDS layer — every modification is internal
shape, public re-exports preserve the existing surface.

E. Typed SubscriptionTier enum

crates/thetadatadx/src/mdds/tier.rs introduces a four-variant enum
(Free, Value, Standard, Pro) replacing the prior Option<i32>
representation of stock_tier and options_tier on MddsClient.

Item Before After
Field type Option<i32> Option<SubscriptionTier>
Wire decode implicit comparison SubscriptionTier::from_wire(i32) -> Option<Self>
Concurrency math scattered 2^tier literal in two places SubscriptionTier::max_concurrent_requests(self)
Public surface none thetadatadx::SubscriptionTier

The wire-side auth::nexus::AuthUser keeps its raw Option<i32> fields so
deserialization stays infallible for unknown future tiers; the typed enum is
the post-decode in-memory shape callers see. Unit tests cover the round-trip,
the unknown-value rejection (from_wire(99) == None), and the powers-of-two
semaphore math (1, 2, 4, 8).

F. ArcSwap<StreamingSlot> state machine

Streaming side of ThetaDataDx collapses to a single ArcSwap<StreamingSlot>
cell walking Idle → Live → Stopped → Live → Stopped.

Item Before After
State fields Mutex<Option<FpssClient>> + Mutex<Option<StreamingDispatcher>> + AtomicBool was_streaming one ArcSwap<StreamingSlot>
is_streaming takes a Mutex one atomic load + matches!
connection_status takes a Mutex + AtomicBool load one atomic load + match
with_streaming takes a Mutex one atomic load + borrow
stop_streaming two Mutex locks one atomic swap; old slot drops shutdown sequence
Drop idempotency implicit documented invariant — only Live triggers shutdown

Adds arc-swap 1.7 as a runtime dep. A state-machine walk through
Idle → Live → Stopped → Live → Stopped and a CAS-race test covering the
rejected double-install live alongside the existing reconnect_streaming
regression tests.

G. Layout moves

Top-level mdds-specific modules move under mdds/:

Old New
crates/thetadatadx/src/endpoint.rs crates/thetadatadx/src/mdds/endpoint_args.rs
crates/thetadatadx/src/macros.rs crates/thetadatadx/src/mdds/macros.rs
crates/thetadatadx/src/registry.rs crates/thetadatadx/src/mdds/registry.rs
crates/thetadatadx/src/validate.rs merged into crates/thetadatadx/src/mdds/validate.rs
crates/thetadatadx/src/wire_semantics.rs crates/thetadatadx/src/mdds/wire_semantics.rs
crates/thetadatadx/src/unified.rs crates/thetadatadx/src/client.rs
crates/thetadatadx/src/frames.rs + frames_generated.rs crates/thetadatadx/src/frames/{mod,generated}.rs
crates/tdbe/src/types/{enums_endpoint,tick,tick_layout_asserts}_generated.rs crates/tdbe/src/types/generated/{enums_endpoint,tick,tick_layout_asserts}.rs

mdds/validate.rs now hosts the canonical two-arg validators
(date / expiration / strike / symbol / interval / right / year) plus a
validate_date_required single-arg adapter the parsed_endpoint! macro
expansion needs. The macro switches to the fully-qualified
$crate::mdds::validate::validate_date_required path so it expands cleanly
from generated endpoint bodies; the generator emit
(build_support/endpoints/render/mdds.rs) follows.

tdbe::types::generated/ ships a documenting mod.rs; the hand-written
siblings keep their include! sites (now include!("generated/...rs")).
mod generated; is declared privately in types/mod.rs so the directory is
part of the module tree without re-exposing the generator output.

Re-exports in lib.rs preserve the public surface:

pub use mdds::endpoint_args as endpoint;          // → thetadatadx::endpoint::*
pub use mdds::registry::{..., EndpointMeta, ENDPOINTS, ...};  // → thetadatadx::EndpointMeta / ENDPOINTS
pub use client::{ConnectionStatus, SubscriptionInfo, ThetaDataDx};

git mv was used everywhere file blame is meaningful. Build-script
[path = ...] mounts (build_support/mod.rs,
crates/thetadatadx/src/bin/generate_sdk_surfaces.rs) follow the file moves;
the sdk-surfaces generator (build_support/ticks/,
build_support/endpoints/render/sdk_files.rs) emits to the new paths.

H. LOW batch

Item Status Notes
3.1 mdds.connect_timeout_secs Already shipped Wave 2 landed it
3.2 extract_*_column iterator return Kept Vec<Option<T>> Public surface, exercised by benches / list-endpoint macros / Polars / Arrow projections; iterator conversion would force every caller to handle the iterator shape and lose the missing-header early return
3.5 FpssConnectArgs struct Deferred to Wave 4 B4 Breaking signature change
3.9 Drop::drop idempotency on ThetaDataDx Done in F State machine guarantees stop_streaming runs the FPSS / dispatcher shutdown at most once; Drop::drop's doc comment records the contract
3.10 #[allow(dead_code)] on flatfiles::framing::msg Removed All ten u16 wire-code constants are genuinely used by flatfiles::request and flatfiles::session; the attribute was a false positive

Test plan

  • cargo fmt --all -- --check
  • cargo clippy --workspace --all-targets -- -D warnings
  • cargo test --workspace (every test suite green; 6 new unit tests added — 4 for SubscriptionTier, 2 for the ArcSwap state machine)
  • cargo deny check
  • cargo run -p thetadatadx --bin generate_sdk_surfaces --features config-file -- --check (no surface drift)
  • cargo check --manifest-path tools/{cli,mcp,server}/Cargo.toml --locked
  • cargo check --manifest-path sdks/{python,typescript}/Cargo.toml --locked
  • cargo clippy --manifest-path tools/mcp/Cargo.toml --all-targets -- -D warnings
  • cargo test --manifest-path tools/mcp/Cargo.toml --no-run
  • python3 scripts/check_version_sync.pyversion sync: ok

… layout moves + LOW batch (refs #500)

E. Typed `SubscriptionTier` enum
================================

`crates/thetadatadx/src/mdds/tier.rs` introduces a four-variant
`SubscriptionTier` enum (`Free`, `Value`, `Standard`, `Pro`) replacing
the prior `Option<i32>` representation of the `stock_tier` and
`options_tier` fields on `MddsClient`. The discriminant mirrors the
wire byte; `from_wire(i32)` decodes (returning `None` for unknown
values rather than silently coercing); `max_concurrent_requests`
codifies the `2^tier` semaphore semantics. Re-exported as
`thetadatadx::SubscriptionTier`. The wire-side `auth::nexus::AuthUser`
keeps its raw `Option<i32>` fields so deserialization stays
infallible for unknown future tiers; the typed enum is the post-
decode in-memory shape callers see. Unit tests cover the
round-trip, the unknown-value rejection, and the powers-of-two
semaphore math (1, 2, 4, 8).

F. `ArcSwap<StreamingSlot>` state machine
=========================================

The streaming side of `ThetaDataDx` collapses to a single
`ArcSwap<StreamingSlot>` cell walking
`Idle -> Live -> Stopped -> Live -> Stopped`. The previous trio of
coordinated fields — `Mutex<Option<FpssClient>>`,
`Mutex<Option<StreamingDispatcher>>`, `AtomicBool was_streaming` —
went away. Read paths (`is_streaming`, `connection_status`,
`with_streaming`, every per-subscription forwarder) collapse to one
atomic load on the slot and a borrow of the contained
`Arc<FpssClient>`. Lifecycle paths retain serial semantics through
an rcu-CAS install and an atomic swap-to-`Stopped`, with the previous
`Live` payload (if any) running the FPSS-then-dispatcher shutdown
sequence. The `dispatcher` field sits in a `Mutex<Option<...>>`
inside `Live` because `StreamingDispatcher::shutdown` consumes by
value; the lock is hit only by `stop_streaming` and
`dropped_event_count`, never by hot-path readers. Adds `arc-swap`
1.7 to the runtime dependencies. A state-machine walk through
`Idle -> Live -> Stopped -> Live -> Stopped` and a CAS-race test
covering the rejected double-install live alongside the existing
`reconnect_streaming` regression tests. The `Drop` contract is
documented in place: stop_streaming is idempotent because only a
`Live` slot triggers the shutdown sequence.

G. Layout moves
===============

Top-level mdds-specific modules move under `mdds/`:

  endpoint.rs       -> mdds/endpoint_args.rs (rename clarifies args-only)
  macros.rs         -> mdds/macros.rs
  registry.rs       -> mdds/registry.rs
  validate.rs       merged into mdds/validate.rs
  wire_semantics.rs -> mdds/wire_semantics.rs

`mdds/validate.rs` now hosts the canonical two-arg validators
(date / expiration / strike / symbol / interval / right / year)
plus a `validate_date_required` single-arg adapter the
`parsed_endpoint!` macro expansion needs. The macro switches to the
fully-qualified `$crate::mdds::validate::validate_date_required`
path so it expands cleanly from generated endpoint bodies; the
generator emit (`build_support/endpoints/render/mdds.rs`) follows.

`unified.rs` is renamed to `client.rs` so the filename matches the
primary type (`ThetaDataDx`). The frames module folds into a
directory: `frames.rs` + `frames_generated.rs` -> `frames/mod.rs` +
`frames/generated.rs`. The build-time generator paths
(`build_support/ticks/mod.rs`, `build_support/ticks/rust_frames.rs`)
follow.

The `tdbe::types::*_generated.rs` trio segregates under
`tdbe::types::generated/`: `enums_endpoint.rs`, `tick.rs`,
`tick_layout_asserts.rs`, with a `mod.rs` documenting the
directory. The hand-written siblings keep their `include!` sites
(now `include!("generated/...rs")`); `mod generated;` is declared
privately in `types/mod.rs`. The build-time generator paths
(`build_support/endpoints/render/sdk_files.rs`,
`build_support/ticks/tdbe_structs.rs`,
`build_support/ticks/mod.rs`) follow.

Re-exports in `lib.rs` keep the public surface stable:

  pub use mdds::endpoint_args as endpoint;
  pub use mdds::registry::{by_category, find, ..., EndpointMeta, ENDPOINTS, ...};
  pub use client::{ConnectionStatus, SubscriptionInfo, ThetaDataDx};

`thetadatadx::endpoint::*` and `thetadatadx::ENDPOINTS` /
`EndpointMeta` continue to resolve.

Build-script `[path = ...]` mounts (`build_support/mod.rs`,
`crates/thetadatadx/src/bin/generate_sdk_surfaces.rs`) follow the
file moves. `git mv` was used everywhere file blame is meaningful so
history follows the rename.

H. LOW batch
============

* 3.1 (`mdds.connect_timeout_secs`) — already shipped in Wave 2.
* 3.2 (`extract_*_column` return type) — kept as `Vec<Option<T>>`.
  The three helpers are public surface, exercised by benches,
  integration tests, the macro-driven list endpoints, and the
  Polars / Arrow column projections. Switching to
  `impl Iterator<Item = Option<T>>` would force every caller
  (including generated FFI bindings) to deal with the iterator
  shape and would lose the "missing-header -> empty result" early
  return the warn-log path relies on. The Vec allocation is one
  per column per call, never on a per-tick path.
* 3.5 (`FpssConnectArgs` struct) — deferred to Wave 4 B4 as a
  breaking signature change. Not landed here.
* 3.9 (`Drop::drop` idempotency on `ThetaDataDx`) — addressed in F.
  The state machine guarantees `stop_streaming` runs the FPSS /
  dispatcher shutdown at most once; `Drop::drop`'s doc comment now
  records the contract in-place.
* 3.10 (`#[allow(dead_code)]` on `flatfiles::framing::msg`) —
  removed. Every one of the ten `u16` wire-code constants is
  genuinely used by `flatfiles::request` and `flatfiles::session`.
  The attribute was a false positive; clippy stays green.

Version bump: 8.0.36 -> 8.0.37 across every crate manifest,
`sdks/typescript/package.json` (+ optionalDependencies, + the three
per-platform `npm/<platform>/package.json` files), Cargo.lock, and
the matching CHANGELOG entry mirrored to `docs-site/docs/changelog.md`.
@userFRM userFRM merged commit a115170 into main May 7, 2026
32 checks passed
@userFRM userFRM deleted the refactor/wave3-mega branch May 7, 2026 05:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant