Skip to content

Commit cd7847d

Browse files
taddestaddes
authored
chore(claude): create Claude.md (#2270)
* first pass at Claude.md * clarify branch convention
1 parent aea42fa commit cd7847d

1 file changed

Lines changed: 286 additions & 0 deletions

File tree

‎CLAUDE.md‎

Lines changed: 286 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,286 @@
1+
# syncstorage-rs
2+
3+
Firefox Sync storage and token server, written in Rust. The workspace contains a
4+
Rust multi-crate server plus Python utilities and integration/end-to-end tests.
5+
6+
## Agent rules
7+
8+
Security takes absolute precedence. This repository handles synchronization of
9+
user data and token management to access storage nodes.
10+
11+
### 1) Scope & writes
12+
13+
- Operate strictly within this repo root; normalize paths; do not follow
14+
symlinks outside the repo.
15+
- Writes are allowed to the working tree. Always present a diff for review
16+
before any staging or commit.
17+
- Do not modify files adjacent to a requested change. Mention issues found
18+
but do not fix them unless asked.
19+
- Ask before running any command (build, test, install, DB ops, git,
20+
service management). Do not run `git add`, `git commit`, `git push`, or
21+
`git rebase` unless explicitly instructed.
22+
23+
### 2) Non-negotiables
24+
25+
- **Secrets:** never read, print, summarize, or transmit secret files or
26+
values. Use placeholders (`YOUR_API_KEY_HERE`) in examples.
27+
- **External network:** only with explicit approval to a trusted, documented
28+
endpoint.
29+
- **Pipelines & contracts:** flag breaking API or contract changes. Do not
30+
alter GitHub Actions defined in `.github/`, any configuration for CI/CD workflows or git hooks without explicit, reviewed
31+
justification.
32+
- **Published DB migrations:** never edit existing published migration files or schema definitions (ex. `syncstorage-spanner/src/schema.ddl`), whether for MySQL, Postgres, or Spanner.
33+
Always add a new forward migration and a separate rollback migration.
34+
- **Workspace recommendations:** the repo deliberately ships no
35+
`.vscode/extensions.json`. Do not re-add one. Useful extensions are listed
36+
in `.vscode/README.md` for contributors to install at their own discretion.
37+
38+
### 3) Do-not-touch paths
39+
40+
Never read or write these paths. This list mirrors `.gitignore` — if there is
41+
a discrepancy, `.gitignore` is the source of truth.
42+
43+
```
44+
service-account.json # GCP credentials
45+
.sentryclirc # Sentry auth token
46+
.envrc # direnv secrets
47+
config/local.toml # local server config (may contain secrets)
48+
tools/tokenserver/loadtests/*.pem
49+
tools/tokenserver/loadtests/*.pub
50+
target/ # Rust build artifacts
51+
*.xml # test result files
52+
openapi.json # generated artifact
53+
book/ # generated docs
54+
docs/output/ # generated docs
55+
venv/ # Python virtual environments
56+
.install.stamp # Poetry install marker
57+
workspace/ # CI artifact directory
58+
```
59+
60+
### 4) Git commit messages
61+
62+
Follow the commit message format defined in [CONTRIBUTING.md](CONTRIBUTING.md) Git Commit Guidelines.
63+
64+
- Format: `type(scope): subject` — `type` is one of `feat`, `fix`, `docs`,
65+
`style`, `refactor`, `perf`, `test`, `chore`
66+
- Subject: imperative present tense, no capital first letter, no trailing period
67+
- Body: explain motivation and contrast with previous behavior
68+
- Footer: `Closes STOR-1234` (Mozilla engineers — matches the Jira ticket) or
69+
`Closes #N` / `Issue #N` (community contributors — matches the GitHub issue number);
70+
`BREAKING CHANGE:` for breaking changes
71+
- Branch naming (Mozilla engineers): `type/description-STOR-1234` where `1234` must match the numeric suffix of the associated Jira ticket
72+
- Branch naming (community): `type/description-1112`
73+
- All commits must be GPG/SSH signed (enforced by CI)
74+
75+
Example:
76+
77+
```
78+
feat: add node reassignment on capacity overflow
79+
80+
Nodes at 100% capacity previously returned 503 rather than
81+
assigning the user to an available node.
82+
83+
Closes #42.
84+
```
85+
86+
## Project layout
87+
88+
```
89+
syncserver/ Main Actix-web server (entry point)
90+
syncstorage-db/ Syncstorage database abstraction layer
91+
syncstorage-db-common/ Syncstorage shared database types and traits
92+
syncstorage-mysql/ MySQL backend for syncstorage
93+
syncstorage-postgres/ PostgreSQL backend for syncstorage
94+
syncstorage-spanner/ Google Cloud Spanner backend for syncstorage
95+
syncstorage-settings/ Syncstorage configuration types
96+
tokenserver-db/ Tokenserver database abstraction layer
97+
tokenserver-db-common/ Tokenserver shared database types
98+
tokenserver-mysql/ MySQL backend for tokenserver
99+
tokenserver-postgres/ PostgreSQL backend for tokenserver
100+
tokenserver-auth/ HAWK token generation and verification
101+
tokenserver-common/ Shared tokenserver utilities
102+
tokenserver-settings/ Tokenserver configuration types
103+
syncserver-common/ Shared server utilities
104+
syncserver-db-common/ Shared database utilities
105+
syncserver-settings/ Syncserver configuration types
106+
glean/ Glean telemetry crate
107+
tools/
108+
integration_tests/ Pytest-based integration + e2e tests
109+
tokenserver/ Tokenserver utilities and load tests
110+
spanner/ Spanner utilities
111+
hawk/ HAWK token generation utility
112+
syncstorage-loadtest/ Molotov load tests
113+
docker/ Docker Compose files for each backend
114+
config/ Local TOML configuration files
115+
docs/ mdBook documentation source
116+
.github/ GitHub CI Actions and configuration
117+
```
118+
119+
## Rust builds
120+
121+
The default feature set targets MySQL, however production uses the Spanner database with MySQL Tokenserver. Backend features are mutually exclusive.
122+
123+
```bash
124+
# MySQL (default)
125+
cargo build
126+
127+
# PostgreSQL
128+
cargo build --no-default-features --features=syncstorage-db/postgres --features=tokenserver-db/postgres --features=py_verifier
129+
130+
# Spanner
131+
cargo build --no-default-features --features=syncstorage-db/spanner --features=py_verifier
132+
```
133+
134+
## Rust tests
135+
136+
Tests require `RUST_TEST_THREADS=1` (many tests share a real database).
137+
138+
```bash
139+
# MySQL unit tests (default DATABASE_URL envs set in Makefile)
140+
make test
141+
142+
# Specific package
143+
make test ARGS="--package syncstorage-db"
144+
145+
# With coverage
146+
make test_with_coverage # MySQL
147+
make postgres_test_with_coverage
148+
make spanner_test_with_coverage
149+
```
150+
151+
## Clippy / formatting / audit
152+
153+
Formatting checks are necessary before applying any code changes, as they
154+
can capture problematic patterns or compiler issues in advance.
155+
156+
```bash
157+
cargo fmt -- --check # format check
158+
cargo fmt # auto-format
159+
160+
make clippy_mysql
161+
make clippy_postgres
162+
make clippy_spanner
163+
164+
make clippy_release_mysql # release-mode (catches dead code etc.)
165+
make clippy_release_postgres
166+
make clippy_release_spanner
167+
168+
cargo audit # check dependencies for known CVEs (run by CI on every push)
169+
```
170+
171+
## Python tooling
172+
173+
All Python tooling uses [Poetry](https://python-poetry.org/). The root
174+
`pyproject.toml` covers the `tools/` tree. Each subdirectory under `tools/` has
175+
its own `pyproject.toml` for its specific dependencies.
176+
177+
All of these Python checks must be run with any related change to a Python
178+
utility or test.
179+
180+
```bash
181+
make install # install root dependencies
182+
make integration-test # install integration test dependencies
183+
make tokenserver # install tokenserver utility dependencies
184+
```
185+
186+
Lint / format / type-check:
187+
188+
```bash
189+
make ruff-lint # ruff check tools/
190+
make ruff-format # ruff format tools/
191+
make ruff-fmt-chk # format diff check
192+
make mypy # mypy type checking
193+
make pydocstyle # docstring validation
194+
make bandit # security linting
195+
```
196+
197+
## Integration / E2E tests
198+
199+
### Local (against a running server)
200+
201+
```bash
202+
# Run all integration tests except real FxA e2e
203+
PYTHONPATH=$(pwd)/tools \
204+
SYNC_MASTER_SECRET=secret0 \
205+
SYNC_TOKENSERVER__FXA_OAUTH_SERVER_URL=http://localhost:6000 \
206+
TOKENSERVER_HOST=http://localhost:8000 \
207+
poetry -C tools/integration_tests run pytest . --ignore=tokenserver/test_e2e.py
208+
```
209+
210+
Or via the Makefile shortcut (sets the env vars above):
211+
212+
```bash
213+
make run_local_e2e_tests
214+
```
215+
216+
### Docker-based E2E (full stack)
217+
218+
```bash
219+
make docker_run_mysql_e2e_tests
220+
make docker_run_postgres_e2e_tests
221+
make docker_run_spanner_e2e_tests
222+
```
223+
224+
Each command spins up two compose stacks (with and without JWK cache), copies
225+
JUnit XML results out of the container, then tears everything down.
226+
227+
## Running the server locally
228+
229+
```bash
230+
# MySQL
231+
make run_mysql
232+
233+
# Spanner (requires GCP service account key at ./service-account.json)
234+
make run_spanner
235+
```
236+
237+
Server listens on `http://localhost:8000`. Health check: `GET /__heartbeat__`.
238+
239+
## Key environment variables
240+
241+
| Variable | Purpose |
242+
|---|---|
243+
| `SYNC_MASTER_SECRET` | HAWK token signing secret |
244+
| `SYNC_SYNCSTORAGE__DATABASE_URL` | Syncstorage DB connection string |
245+
| `SYNC_TOKENSERVER__DATABASE_URL` | Tokenserver DB connection string |
246+
| `SYNC_TOKENSERVER__NODE_TYPE` | `mysql` / `postgres` / `spanner` |
247+
| `SYNC_TOKENSERVER__RUN_MIGRATIONS` | Auto-migrate on startup (`true`/`false`) |
248+
| `SYNC_TOKENSERVER__FXA_OAUTH_SERVER_URL` | FxA OAuth endpoint |
249+
| `RUST_TEST_THREADS` | Set to `1` for all test runs |
250+
| `RUST_LOG` | Log level (`debug`, `info`, etc.) |
251+
| `SYNC_TEST_LOG_HTTP` | Set any value to log all HTTP req/resp in integration tests |
252+
| `SYNC_SERVER_URL` | Integration test base URL (default: `http://localhost:8000`) |
253+
| `TOKENSERVER_HOST` | Tokenserver host for integration tests |
254+
| `PYTHONPATH` | Must point to `$(pwd)/tools` when running integration tests |
255+
256+
## Docker Compose files
257+
258+
| File | Purpose |
259+
|---|---|
260+
| `docker/docker-compose.mysql.yaml` | MySQL syncstorage + MySQL tokenserver |
261+
| `docker/docker-compose.postgres.yaml` | PostgreSQL syncstorage + PostgreSQL tokenserver |
262+
| `docker/docker-compose.spanner.yaml` | Spanner emulator + MySQL tokenserver |
263+
| `docker/docker-compose.e2e.{mysql,postgres,spanner}.yaml` | E2E test runner overlay |
264+
| `docker/docker-compose.e2e.jwk-cache.yaml` | Enable JWK caching overlay |
265+
| `docker/docker-compose.e2e.no-jwk-cache.yaml` | Disable JWK caching overlay |
266+
267+
## Documentation
268+
269+
```bash
270+
make doc-install-deps # install mdBook + mdBook-mermaid
271+
make doc-watch # live preview at localhost
272+
make doc-prev # build and serve
273+
make doc-test # validate markdown
274+
275+
make api-prev # generate OpenAPI spec + serve Swagger UI on :8080
276+
```
277+
278+
279+
## Test structure
280+
281+
- `tools/integration_tests/conftest.py` — pytest fixtures only (`st_ctx`)
282+
- `tools/integration_tests/helpers.py` — retry helpers, auth state, `switch_user`
283+
- `tools/integration_tests/test_storage.py` — storage protocol tests
284+
- `tools/integration_tests/tokenserver/conftest.py` — tokenserver fixtures only
285+
- `tools/integration_tests/tokenserver/helpers.py` — tokenserver DB/auth helpers
286+
- `tools/integration_tests/tokenserver/test_*.py` — tokenserver tests

0 commit comments

Comments
 (0)