Skip to content

Commit af30a62

Browse files
balegasclaude
balegas
and
claude
authored
feat(server-rust): Rust Durable Streams server (#4652)
## What's here - **The server** — a single-node, crash-durable append-only log server (sharded WAL with group-commit `fdatasync`, a `memory` zero-copy mode, optional S3 tiering, a hand-rolled HTTP/1.1 engine with `sendfile`/`splice`). Imported **with its git history** (via `git subtree`, so `git blame` follows the original commits). - **npm packaging** re-scoped to `@electric-ax/durable-streams-server-rust` (+ 4 platform packages). - **Changesets release** — the `package.json` anchor is `private` (Changesets bumps the version; CI publishes the real artifacts). A release currently publishes the **`durable-streams` crate** (crates.io) and a multi-arch **`electricax/durable-streams-server-rust`** Docker image (distroless). **npm publishing is intentionally disabled for now** (gated off in `server_rust_publish.yml`). - **CI** — `server_rust_tests.yml`: `cargo build`/`test`/`clippy -D warnings` + the conformance matrix (wal-default, wal-resident-cache, wal-read-offload-always, memory). Conformance consumes the published `@durable-streams/server-conformance-tests`. ## Notes for review - **128 commits** = the imported server-rust history (~120) + ~12 integration commits; the net diff is ~16.3k of imported crate + **~600 lines** of integration. Review the integration via the non-import commits. - The `durable-streams` crate name is **reserved** on crates.io (yanked `0.0.0`, owned by `electric-sql:core`) and its **crates.io Trusted Publishing is configured**. ## Verified locally - `cargo build` (default/`tier`/`telemetry`) + `clippy -D warnings` clean. - `cargo test` (release + debug, ±`tier`): 87 passed. - Conformance: **326 passed / 6 skipped** on all 4 matrix configs — including `memory` (Linux-only) run inside the Docker image. - Docker image builds and serves (`/health` 200, `PUT` 201, `HEAD` 200). ## Publishing status - ✅ **crate** (crates.io) and ✅ **Docker image** publish on a release (OIDC; DockerHub `electricax` creds via `secrets: inherit`). - ⏸️ **npm** disabled — re-enable the `npm-publish` job (search `DISABLED:`) and configure npm trusted publishers when ready. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_019zZcfPyDkSmxpkcZJvi379 --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
1 parent 68349b8 commit af30a62

42 files changed

Lines changed: 16822 additions & 0 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.changeset/durable-streams-server-rust.md‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
---
2+
'@electric-ax/durable-streams-server-rust': patch
3+
---
4+
5+
Import the high-performance Rust Durable Streams server as
6+
`@electric-ax/durable-streams-server-rust`. Released via Changesets (the anchor is
7+
`private`, so Changesets bumps the version and CI publishes the binary packages):
8+
npm (main + four platform packages), the `durable-streams` crate on crates.io, and
9+
a multi-arch `electricax/durable-streams-server-rust` Docker image. Adds a Rust
10+
build/test/clippy + conformance-matrix CI workflow and a distroless Dockerfile.

‎.github/workflows/changesets_release.yml‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,7 @@ jobs:
2525
desktop_release_tag: ${{ steps.desktop_release.outputs.tag }}
2626
mobile_release_version: ${{ steps.mobile_release.outputs.version }}
2727
mobile_release_tag: ${{ steps.mobile_release.outputs.tag }}
28+
server_rust_needs_release: ${{ steps.server_rust_release.outputs.needs_release }}
2829
steps:
2930
- uses: actions/checkout@v4
3031
with:
@@ -86,6 +87,24 @@ jobs:
8687
echo "version=$VERSION" >> "$GITHUB_OUTPUT"
8788
echo "tag=@electric-ax/agents-mobile@$VERSION" >> "$GITHUB_OUTPUT"
8889
fi
90+
- name: Detect a server-rust release (private anchor → idempotent crates.io check)
91+
id: server_rust_release
92+
if: steps.changesets.outputs.published == 'true'
93+
run: |
94+
# @electric-ax/durable-streams-server-rust is `private: true` (changesets
95+
# bumps it but never publishes it), so it is not in publishedPackages.
96+
# The npm packages are disabled, so the crate is what actually publishes —
97+
# check crates.io (the published artifact) for idempotency. Publish only if
98+
# this version is not already on crates.io: idempotent across re-runs (and
99+
# across unrelated releases that leave the server-rust version unchanged),
100+
# and correct on the first run. The crate version is synced to this anchor
101+
# at publish time by sync-cargo-version.mjs.
102+
VERSION="$(node -p "require('./packages/server-rust/package.json').version")"
103+
if curl -sf -H 'User-Agent: electric-sql-release-ci' "https://crates.io/api/v1/crates/durable-streams/$VERSION" >/dev/null 2>&1; then
104+
echo "needs_release=false" >> "$GITHUB_OUTPUT"
105+
else
106+
echo "needs_release=true" >> "$GITHUB_OUTPUT"
107+
fi
89108
- name: Add latest tag to published packages
90109
if: steps.changesets.outputs.published == 'true'
91110
run: node scripts/tag-latest.mjs
@@ -112,6 +131,25 @@ jobs:
112131
with:
113132
git_ref: ${{ needs.changesets.outputs.agent_server_release_tag }}
114133

134+
publish-server-rust:
135+
needs: changesets
136+
# Publishes the `durable-streams` crate (crates.io); the npm packages are
137+
# disabled inside server_rust_publish.yml. crates.io Trusted Publishing is
138+
# configured for the crate.
139+
if: ${{ needs.changesets.outputs.published == 'true' && needs.changesets.outputs.server_rust_needs_release == 'true' }}
140+
uses: ./.github/workflows/server_rust_publish.yml
141+
secrets: inherit
142+
with:
143+
git_ref: ${{ github.sha }}
144+
145+
publish-server-rust-to-dockerhub:
146+
needs: [changesets, publish-server-rust]
147+
if: ${{ needs.changesets.outputs.published == 'true' && needs.changesets.outputs.server_rust_needs_release == 'true' }}
148+
uses: ./.github/workflows/server_rust_dockerhub_image.yml
149+
secrets: inherit
150+
with:
151+
git_ref: ${{ github.sha }}
152+
115153
update-cloud-agents-server:
116154
name: Update Agents Server version used by Cloud
117155
runs-on: ubuntu-latest

‎.github/workflows/server_rust_dockerhub_image.yml‎

Lines changed: 109 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,109 @@
1+
name: Publish durable-streams-server-rust image to Docker Hub
2+
3+
# If you change the triggers, also update the conditional logic in derive_build_vars.
4+
on:
5+
# DISABLED: auto-publishing a canary image on merge to main is off until we're
6+
# ready to publish. Re-enable by restoring this push trigger:
7+
# push:
8+
# branches: ['main']
9+
# paths:
10+
# - 'packages/server-rust/**'
11+
# Called by the Changesets release workflow after a version bump.
12+
workflow_call:
13+
inputs:
14+
git_ref:
15+
description: 'Git ref to build from (branch name, tag, or commit SHA)'
16+
required: true
17+
type: string
18+
docker_tag:
19+
description: 'Optional Docker Hub tag override (derived from package.json if omitted)'
20+
required: false
21+
type: string
22+
workflow_dispatch:
23+
inputs:
24+
git_ref:
25+
description: 'Git ref to build from (branch name, tag, or commit SHA)'
26+
required: true
27+
type: string
28+
docker_tag:
29+
description: 'Optional Docker Hub tag override (derived from package.json if omitted)'
30+
required: false
31+
type: string
32+
33+
env:
34+
DOCKERHUB_REPO: electricax/durable-streams-server-rust
35+
36+
jobs:
37+
derive_build_vars:
38+
name: Derive build variables from the source code
39+
runs-on: blacksmith-2vcpu-ubuntu-2404
40+
outputs:
41+
git_ref: ${{ steps.git_ref.outputs.git_ref }}
42+
image_tags: ${{ steps.vars.outputs.image_tags }}
43+
44+
steps:
45+
- name: Determine the ref to check out
46+
id: git_ref
47+
env:
48+
INPUT_GIT_REF: ${{ inputs.git_ref }}
49+
INPUT_DOCKER_TAG: ${{ inputs.docker_tag }}
50+
COMMIT_SHA: ${{ github.sha }}
51+
run: |
52+
if [ -n "$INPUT_DOCKER_TAG" ]; then
53+
ref="$INPUT_GIT_REF"
54+
build_mode=custom
55+
elif [ -n "$INPUT_GIT_REF" ]; then
56+
ref="$INPUT_GIT_REF"
57+
build_mode=release
58+
else
59+
ref="$COMMIT_SHA"
60+
build_mode=canary
61+
fi
62+
63+
echo "git_ref=$ref" >> "$GITHUB_OUTPUT"
64+
echo "build_mode=$build_mode" >> "$GITHUB_OUTPUT"
65+
66+
- uses: actions/checkout@v4
67+
with:
68+
ref: ${{ steps.git_ref.outputs.git_ref }}
69+
fetch-depth: 0
70+
71+
- name: Determine final image tags
72+
id: vars
73+
env:
74+
BUILD_MODE: ${{ steps.git_ref.outputs.build_mode }}
75+
INPUT_DOCKER_TAG: ${{ inputs.docker_tag }}
76+
run: |
77+
# The crate version lives in the package.json anchor (changeset-managed),
78+
# which is private — so there is no git tag to describe; read it directly.
79+
VERSION=$(node -p "require('./packages/server-rust/package.json').version")
80+
81+
if [ -n "$INPUT_DOCKER_TAG" ]; then
82+
IMAGE_TAGS="${DOCKERHUB_REPO}:${INPUT_DOCKER_TAG}"
83+
elif [ "$BUILD_MODE" = "release" ]; then
84+
IMAGE_TAGS=$(
85+
printf '%s\n%s\n' \
86+
"${DOCKERHUB_REPO}:latest" \
87+
"${DOCKERHUB_REPO}:${VERSION}"
88+
)
89+
else
90+
IMAGE_TAGS="${DOCKERHUB_REPO}:canary"
91+
fi
92+
93+
{
94+
echo "image_tags<<EOF"
95+
echo "$IMAGE_TAGS"
96+
echo "EOF"
97+
} >> "$GITHUB_OUTPUT"
98+
99+
build_and_publish_image:
100+
needs: [derive_build_vars]
101+
uses: ./.github/workflows/docker_multiarch_image.yml
102+
secrets: inherit
103+
with:
104+
git_ref: ${{ needs.derive_build_vars.outputs.git_ref }}
105+
image_repo: electricax/durable-streams-server-rust
106+
artifact_prefix: durable-streams-server-rust
107+
context: .
108+
file: packages/server-rust/Dockerfile
109+
tags: ${{ needs.derive_build_vars.outputs.image_tags }}

‎.github/workflows/server_rust_publish.yml‎

Lines changed: 152 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,152 @@
1+
name: Publish durable-streams-server-rust (npm + crates.io)
2+
3+
# Reusable: called by changesets_release.yml after a version bump, OR manually.
4+
# Publishes the binary npm packages (main + 4 platform) and the crate. The Docker
5+
# image is published separately by server_rust_dockerhub_image.yml.
6+
#
7+
# The version is read from packages/server-rust/package.json (the changeset-managed
8+
# anchor, `private: true` so changesets bumps it but does not publish it). Both
9+
# registries use OIDC Trusted Publishing — no tokens (one-time setup required on
10+
# crates.io and npm before the first release).
11+
on:
12+
workflow_call:
13+
inputs:
14+
git_ref:
15+
description: 'Git ref to build/publish from'
16+
required: false
17+
type: string
18+
workflow_dispatch:
19+
inputs:
20+
git_ref:
21+
description: 'Git ref to build/publish from'
22+
required: false
23+
type: string
24+
25+
permissions:
26+
contents: read
27+
id-token: write # OIDC for npm --provenance and crates.io trusted publishing
28+
29+
jobs:
30+
build:
31+
name: build ${{ matrix.target }}
32+
runs-on: ${{ matrix.os }}
33+
strategy:
34+
fail-fast: false
35+
matrix:
36+
include:
37+
- {
38+
target: x86_64-unknown-linux-gnu,
39+
os: blacksmith-4vcpu-ubuntu-2404,
40+
}
41+
- {
42+
target: aarch64-unknown-linux-gnu,
43+
os: blacksmith-4vcpu-ubuntu-2404-arm,
44+
}
45+
- { target: x86_64-apple-darwin, os: macos-13 }
46+
- { target: aarch64-apple-darwin, os: macos-latest }
47+
defaults:
48+
run:
49+
working-directory: packages/server-rust
50+
steps:
51+
- uses: actions/checkout@v4
52+
with:
53+
ref: ${{ inputs.git_ref }}
54+
55+
- uses: dtolnay/rust-toolchain@stable
56+
with:
57+
targets: ${{ matrix.target }}
58+
59+
- uses: Swatinem/rust-cache@v2
60+
with:
61+
workspaces: packages/server-rust
62+
63+
- name: Build
64+
run: cargo build --release --locked --target ${{ matrix.target }}
65+
66+
- name: Smoke test (/health)
67+
shell: bash
68+
run: |
69+
bin="target/${{ matrix.target }}/release/durable-streams-server"
70+
"$bin" --port 4599 --data-dir "$RUNNER_TEMP/smoke" &
71+
pid=$!
72+
ok=
73+
for _ in $(seq 1 30); do
74+
if curl -fsS http://127.0.0.1:4599/health | grep -q ok; then ok=1; break; fi
75+
sleep 0.3
76+
done
77+
kill "$pid" 2>/dev/null || true
78+
test -n "$ok" || { echo "server did not answer /health"; exit 1; }
79+
80+
- name: Upload binary artifact
81+
uses: actions/upload-artifact@v4
82+
with:
83+
name: bin-${{ matrix.target }}
84+
path: packages/server-rust/target/${{ matrix.target }}/release/durable-streams-server
85+
if-no-files-found: error
86+
87+
cargo-publish:
88+
needs: build
89+
runs-on: blacksmith-4vcpu-ubuntu-2404
90+
steps:
91+
- uses: actions/checkout@v4
92+
with:
93+
ref: ${{ inputs.git_ref }}
94+
- uses: dtolnay/rust-toolchain@stable
95+
- name: Sync Cargo.toml version from the package.json anchor
96+
run: node packages/server-rust/scripts/sync-cargo-version.mjs
97+
- name: Authenticate to crates.io (Trusted Publishing, OIDC)
98+
uses: rust-lang/crates-io-auth-action@v1
99+
id: auth
100+
- name: cargo publish
101+
env:
102+
CARGO_REGISTRY_TOKEN: ${{ steps.auth.outputs.token }}
103+
# --allow-dirty: sync-cargo-version edited Cargo.toml in the working tree.
104+
run: cargo publish --allow-dirty --manifest-path packages/server-rust/Cargo.toml
105+
106+
npm-publish:
107+
needs: build
108+
# DISABLED: not publishing the npm packages yet (the crate + Docker image still
109+
# publish). Re-enable by removing this `if: false`.
110+
if: ${{ false }}
111+
runs-on: blacksmith-4vcpu-ubuntu-2404
112+
steps:
113+
- uses: actions/checkout@v4
114+
with:
115+
ref: ${{ inputs.git_ref }}
116+
- uses: actions/setup-node@v4
117+
with:
118+
node-version-file: '.tool-versions'
119+
registry-url: 'https://registry.npmjs.org'
120+
- name: Update npm for OIDC trusted publishing (>= 11.5.1)
121+
run: npm install -g npm@latest
122+
- name: Download built binaries
123+
uses: actions/download-artifact@v4
124+
with:
125+
pattern: bin-*
126+
path: ${{ runner.temp }}/bins
127+
- name: Arrange binaries by rust target
128+
run: |
129+
# download-artifact unpacks each as <path>/bin-<target>/durable-streams-server
130+
mkdir -p "$RUNNER_TEMP/arranged"
131+
for d in "$RUNNER_TEMP"/bins/bin-*; do
132+
target="$(basename "$d" | sed 's/^bin-//')"
133+
mkdir -p "$RUNNER_TEMP/arranged/$target"
134+
cp "$d/durable-streams-server" "$RUNNER_TEMP/arranged/$target/"
135+
done
136+
- name: Assemble npm packages
137+
run: |
138+
version="$(node -p "require('./packages/server-rust/package.json').version")"
139+
node packages/server-rust/npm/assemble.mjs \
140+
--version "$version" \
141+
--bins "$RUNNER_TEMP/arranged" \
142+
--out "$RUNNER_TEMP/npm-dist"
143+
- name: Publish platform packages, then main (OIDC, public)
144+
run: |
145+
set -e
146+
for d in "$RUNNER_TEMP"/npm-dist/*/; do
147+
[ "$(basename "$d")" = "main" ] && continue
148+
echo "publishing $d"
149+
npm publish "$d" --access public --provenance
150+
done
151+
echo "publishing main"
152+
npm publish "$RUNNER_TEMP/npm-dist/main" --access public --provenance

0 commit comments

Comments
 (0)