Skip to content

Commit 6ff4b80

Browse files
jy-tanjy-tan
committed
Commit
1 parent 612d1e9 commit 6ff4b80

17 files changed

Lines changed: 2397 additions & 1752 deletions

File tree

‎.dockerignore‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,8 @@ __pycache__/
2020
# Build artifacts
2121
build/
2222
dist/
23+
target/
24+
**/target/
2325
*.egg-info/
2426
.eggs/
2527
eggs/

‎.gitignore‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -229,4 +229,6 @@ __marimo__/
229229
# Coverage
230230
coverage.lcov
231231
coverage.xml
232-
.coverage
232+
.coverage
233+
234+
experimental/

‎docs/environment-variables.md‎

Lines changed: 28 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -130,6 +130,33 @@ TUSK_SAMPLING_RATE=0.1 python app.py
130130

131131
For more details on sampling rate configuration methods and precedence, see the [Initialization Guide](./initialization.md#configure-sampling-rate).
132132

133+
## Rust Core Flags
134+
135+
These variables control optional Rust-accelerated paths in the SDK.
136+
137+
| Variable | Description | Default |
138+
| --- | --- | --- |
139+
| `TUSK_USE_RUST_CORE` | Enables Rust binding usage when available (`1`, `true`, `yes`) | `0` (disabled) |
140+
| `TUSK_SKIP_PROTO_VALIDATION` | Skips expensive protobuf validation in hot path (`1`, `true`, `yes`) | `0` (disabled) |
141+
142+
**Notes:**
143+
144+
- The SDK is fail-open: if Rust bindings are unavailable or a Rust call fails, it falls back to Python implementation.
145+
- `TUSK_USE_RUST_CORE` does not install Rust bindings automatically. The `drift-core-python` package still must be installed in your environment.
146+
- `TUSK_SKIP_PROTO_VALIDATION` is performance-focused and should be used with confidence in parity tests and serialization correctness.
147+
148+
See [`rust-core-bindings.md`](./rust-core-bindings.md) for more details.
149+
150+
**Example usage:**
151+
152+
```bash
153+
# Enable Rust path (if drift-core-python is installed)
154+
TUSK_USE_RUST_CORE=1 python app.py
155+
156+
# Enable Rust path and skip proto validation
157+
TUSK_USE_RUST_CORE=1 TUSK_SKIP_PROTO_VALIDATION=1 python app.py
158+
```
159+
133160
## Connection Variables
134161

135162
These variables configure how the SDK connects to the Tusk CLI during replay:
@@ -142,9 +169,7 @@ These variables configure how the SDK connects to the Tusk CLI during replay:
142169

143170
These are typically set automatically by the Tusk CLI and do not need to be configured manually.
144171

145-
---
146-
147-
## Related Documentation
172+
## Related Docs
148173

149174
- [Initialization Guide](./initialization.md) - SDK initialization parameters and config file settings
150175
- [Quick Start Guide](./quickstart.md) - Record and replay your first trace

‎docs/rust-core-bindings.md‎

Lines changed: 91 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,91 @@
1+
# Rust Core Bindings
2+
3+
This document explains how Rust acceleration works in the Python SDK, how to enable it, and what fallback behavior to expect.
4+
5+
## Overview
6+
7+
The SDK can offload selected hot-path logic to Rust through optional Python bindings (`drift-core-python`), defined in the [`drift-core`](https://github.com/Use-Tusk/drift-core) repository. This is controlled by environment flags and is designed to fail open.
8+
9+
At a high level:
10+
11+
- Python SDK logic remains the source of truth.
12+
- Rust paths are opportunistic optimizations.
13+
- If Rust is unavailable or fails at runtime, SDK behavior falls back to Python equivalents.
14+
15+
## Enablement
16+
17+
Set:
18+
19+
```bash
20+
TUSK_USE_RUST_CORE=1
21+
```
22+
23+
Truthy values are `1`, `true`, and `yes` (case-insensitive). Any other value is treated as disabled.
24+
25+
## Installation Requirements
26+
27+
Rust acceleration requires the `drift-core-python` package to be installed in the runtime environment.
28+
29+
Notes:
30+
31+
- The SDK does not auto-install this package at runtime.
32+
- If the package is missing or cannot be imported on a machine, the SDK continues on Python code paths.
33+
34+
You can install the SDK with Rust bindings via extras:
35+
36+
```bash
37+
pip install "tusk-drift-python-sdk[rust]"
38+
```
39+
40+
## Wheel Platform Coverage
41+
42+
Based on the current `drift-core` publish workflow, prebuilt wheels are built for:
43+
44+
- Linux `x86_64-unknown-linux-gnu`
45+
- Linux `aarch64-unknown-linux-gnu`
46+
- macOS Apple Silicon `aarch64-apple-darwin`
47+
- Windows `x86_64-pc-windows-msvc`
48+
49+
Likely missing prebuilt wheels (source build fallback required) include:
50+
51+
- macOS Intel (`x86_64-apple-darwin`)
52+
- Linux musl targets (e.g. Alpine)
53+
- Windows ARM64
54+
- Other uncommon Python/platform combinations not covered by release artifacts
55+
56+
If no wheel matches the environment, `pip` may attempt a source build of `drift-core-python`, which typically requires a Rust toolchain and native build prerequisites.
57+
58+
## Fallback Behavior
59+
60+
The bridge module is fail-open:
61+
62+
- Rust calls are guarded.
63+
- On import failures or call exceptions, the corresponding helper returns `None`.
64+
- Calling code then uses the existing Python implementation.
65+
66+
This means users do not need Rust installed to run the SDK when Rust acceleration is disabled or unavailable.
67+
68+
## Optional Performance Flag
69+
70+
```bash
71+
TUSK_SKIP_PROTO_VALIDATION=1
72+
```
73+
74+
This skips expensive protobuf validation checks in hot paths.
75+
76+
Use with care:
77+
78+
- Recommended only when parity/smoke tests are healthy.
79+
- Keep it off in environments where strict serialization verification is preferred.
80+
81+
## Practical Guidance
82+
83+
- Default production-safe posture: leave Rust disabled unless you have tested your deployment matrix.
84+
- Performance posture: enable Rust + benchmark on your workloads before broad rollout.
85+
- Reliability posture: keep parity tests and smoke tests in CI to detect drift between Python and Rust paths.
86+
87+
## Related Docs
88+
89+
- [Environment Variables](./environment-variables.md)
90+
- [Initialization Guide](./initialization.md)
91+
- [Context Propagation](./context-propagation.md)

‎drift/core/json_schema_helper.py‎

Lines changed: 37 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,8 @@
1010
from enum import Enum
1111
from typing import Any
1212

13+
from .rust_core_binding import deterministic_hash_jsonable, normalize_and_hash_jsonable, normalize_json_jsonable
14+
1315

1416
class JsonSchemaType(Enum):
1517
UNSPECIFIED = 0
@@ -132,11 +134,39 @@ def generate_schema(data: Any, schema_merges: SchemaMerges | None = None) -> Jso
132134

133135
@staticmethod
134136
def generate_schema_and_hash(data: Any, schema_merges: SchemaMerges | None = None) -> SchemaComputationResult:
135-
normalized = JsonSchemaHelper._normalize_data(data)
136-
decoded = JsonSchemaHelper._decode_with_merges(normalized, schema_merges)
137+
# Convert non-JSON primitives once. Rust/Python paths consume this shared shape.
138+
sanitized = JsonSchemaHelper._to_jsonable(data)
139+
140+
normalized: Any
141+
decoded: Any
142+
decoded_value_hash: str
143+
144+
# If there are no merges, use one coarse Rust call for normalize+hash.
145+
if not schema_merges:
146+
rust_normalized = normalize_and_hash_jsonable(sanitized)
147+
if rust_normalized is not None:
148+
normalized, decoded_value_hash = rust_normalized
149+
else:
150+
normalized = json.loads(json.dumps(sanitized))
151+
decoded_value_hash = JsonSchemaHelper.generate_deterministic_hash(normalized)
152+
decoded = normalized
153+
else:
154+
# Merges require decode before value-hash. Let Rust handle normalize only.
155+
rust_normalized_only = normalize_json_jsonable(sanitized)
156+
if rust_normalized_only is not None:
157+
normalized = rust_normalized_only
158+
else:
159+
normalized = json.loads(json.dumps(sanitized))
160+
161+
decoded = JsonSchemaHelper._decode_with_merges(normalized, schema_merges)
162+
rust_decoded_hash = deterministic_hash_jsonable(decoded)
163+
decoded_value_hash = rust_decoded_hash or JsonSchemaHelper.generate_deterministic_hash(decoded)
164+
137165
schema = JsonSchemaHelper.generate_schema(decoded, schema_merges)
138-
decoded_value_hash = JsonSchemaHelper.generate_deterministic_hash(decoded)
139-
decoded_schema_hash = JsonSchemaHelper.generate_deterministic_hash(schema.to_primitive())
166+
167+
schema_primitive = schema.to_primitive()
168+
rust_schema_hash = deterministic_hash_jsonable(schema_primitive)
169+
decoded_schema_hash = rust_schema_hash or JsonSchemaHelper.generate_deterministic_hash(schema_primitive)
140170
return SchemaComputationResult(
141171
schema=schema,
142172
decoded_value_hash=decoded_value_hash,
@@ -145,6 +175,9 @@ def generate_schema_and_hash(data: Any, schema_merges: SchemaMerges | None = Non
145175

146176
@staticmethod
147177
def generate_deterministic_hash(data: Any) -> str:
178+
rust_hash = deterministic_hash_jsonable(data)
179+
if rust_hash is not None:
180+
return rust_hash
148181
sorted_data = JsonSchemaHelper._sort_object_keys(data)
149182
payload = json.dumps(sorted_data, ensure_ascii=False, separators=(",", ":"))
150183
return hashlib.sha256(payload.encode("utf-8")).hexdigest()

0 commit comments

Comments
 (0)