Skip to content

Commit 9daf33e

Browse files
jeffmakijeffmaki
authored
Merge pull request #66 from TaskarCenterAtUW/jeffmaki-patch-1
Reordering of README with setup instructions on top
2 parents f332c1d + bf3b4f3 commit 9daf33e

1 file changed

Lines changed: 121 additions & 123 deletions

File tree

‎README.md‎

Lines changed: 121 additions & 123 deletions
Original file line numberDiff line numberDiff line change
@@ -6,54 +6,101 @@ This is a combination API backend for workspaces, providing /workspaces* methods
66
the OSM API ("openstreetmap website") plus OSM CGI-map (the C-accelerated methods) that enforces
77
authorization and authentication based on a TDEI/Keycloak JWT token (see main.py for this proxy logic).
88

9-
## What the proxy must provide for osm-rails / osm-web
9+
## Repo Branch Index
1010

11-
This backend is the **only entry point** to the OSM tier (osm-rails + cgimap,
12-
behind `osm-web`): in the deployment, the public OSM host routes to this
13-
container, which proxies `/api/0.6/*` to `WS_OSM_HOST` (default
14-
`http://osm-web`). For the OSM services to work, the proxy must uphold the
15-
following contract. `CLAUDE.md` has the full rationale.
11+
* ```develop``` merge your work here; keep this up to date with the "development" environment / dev tag
12+
* ```staging``` keep this up to date with the "staging" environment / stage tag
13+
* ```production``` keep this up to date with the "production" environment / prod tag
1614

17-
1. **Bridge TDEI auth into OSM's OAuth2.** osm-rails authenticates the API *only*
18-
via doorkeeper OAuth2 (`oauth_access_tokens`); it has no TDEI/JWT path. So on
19-
token validation the backend mirrors the TDEI JWT into `oauth_access_tokens`
20-
in the OSM DB (the "token bridge" in `api/core/security.py`), and forwards the
21-
incoming `Authorization: Bearer <token>` header unchanged. Then osm-rails and
22-
cgimap authenticate the token via plain OAuth2. Controlled by
23-
`WS_OSM_TOKEN_BRIDGE_ENABLED` (on by default) — with it off, osm-rails returns
24-
**401** for TDEI tokens. The backend auto-creates the doorkeeper application
25-
(and a system user to own it), so no manual OSM setup is required.
15+
## Development with local environment
2616

27-
2. **Provision valid OSM `users` rows.** The backend creates OSM `users` rows for
28-
TDEI users (`auth_provider='TDEI'`, `auth_uid` = the JWT `sub`). These must
29-
satisfy OSM's `User` validations — in particular `pass_crypt` length 8..255.
30-
A too-short value is invisible to cgimap but makes osm-rails operations that
31-
re-validate the user fail (e.g. posting a changeset comment or a note),
32-
surfacing as *"Unable to serialize … without an id"*. The
33-
`alembic_osm` migration `*_heal_short_tdei_pass_crypt` repairs legacy rows.
17+
Use the file `docker-compose.local.yml` to build and deploy local code changes. This allows you to run the entire system at once instead of
18+
connecting to existing Databases.
3419

35-
3. **Carry workspace tenancy.** Workspace-scoped OSM requests must include an
36-
`X-Workspace: <id>` header. The proxy authorizes it against the caller's
37-
workspaces and forwards it (it is *not* stripped) so cgimap/osm-rails scope to
38-
the `workspace-<id>` schema. A few paths are exempt (`TENANT_BYPASSES` in
39-
`api/main.py`): workspace create/delete (`PUT`/`DELETE /api/0.6/workspaces/{id}`)
40-
and user provisioning during sign-in (`PUT /api/0.6/user/{uid}`).
20+
### To start on your local machine for dev work (no docker)
4121

42-
4. **Set the proxy headers.** The proxy rewrites `Host` to the OSM host and sets
43-
`X-Real-IP` / `X-Forwarded-For` / `-Host` / `-Proto`, while stripping
44-
hop-by-hop headers and any spoofed forwarding headers from the client. It does
45-
*not* strip `Authorization` or `X-Workspace`.
22+
```
23+
cp .env.example .env # edit this file for your config
24+
uv sync
25+
uv run uvicorn api.main:app
26+
```
4627

47-
5. **Connectivity.** `WS_OSM_HOST` must reach `osm-web`, and the backend needs
48-
**both** `OSM_DATABASE_URL` and `TASK_DATABASE_URL` — the token bridge and
49-
user provisioning write to the OSM database.
28+
### Initial setup for development local environment
5029

51-
## Branch Index
30+
Step 1: Login to azure docker
5231

53-
* ```develop``` merge your work here; keep this up to date with the "development" environment / dev tag
54-
* ```staging``` keep this up to date with the "staging" environment / stage tag
55-
* ```production``` keep this up to date with the "production" environment / prod tag
32+
The docker compose relies on images in `opensidewalksdev` azure container registry. Make sure your docker system is logged into it before pulling the images and trying to run the containers.
33+
34+
Docker login command:
5635

36+
`docker login opensidewalksdev.azurecr.io -u opensidewalksdev `
37+
38+
Password needs to be obtained from Azure portal
39+
40+
Step 2: Run docker compose for the first time
41+
42+
Use the following command to start the containers first time
43+
44+
`docker compose --file docker-compose.local.yml up --build`
45+
46+
Step 3: Run the migration scripts.
47+
48+
You will observe that only `osm-rails` component seems to work but the backend and other services may be down. this is because the database migrations on the base osm database are not done. To do the base migrations, do the following:
49+
50+
- Connect to the `osm-rails` container. If you are using docker hub for desktop, just go to the exec section of the container.
51+
If you want to use command line, execute the command `docker exec -it <container_name_or_id> /bin/bash` where `container_name` is the name of osm-rails container
52+
- Execute the migration script in the /bin/bash with `bundle exec rails db:migrate`
53+
- The above command runs the migration script for databases
54+
55+
Step 4: Add `workspaces-tasks-local` database in postgresql
56+
57+
Workspaces backend relies on an additional database. This is needed for some older migrations code.
58+
59+
- Connect to `database` container.
60+
- Run the following set of commands one by one
61+
62+
```shell
63+
psql --username postgres
64+
create database "workspaces-tasks-local";
65+
exit;
66+
psql --username postgres --dbname "workspaces-tasks-local";
67+
create extension if not exists postgis;
68+
69+
```
70+
71+
Step 5: Restart the docker compose again
72+
73+
- `docker compose --file docker-compose.local.yml down`
74+
- `docker compose --file docker-compose.local.yml up --build`
75+
76+
### Commands to start and stop the docker compose
77+
78+
`docker compose --file docker-compose.local.yml up --build -d`
79+
80+
`docker compose --file docker-compose.local.yml down`
81+
82+
Backend code will be available at `http://localhost:8000`
83+
84+
85+
## Running the tests
86+
87+
Tests are fast and require no database, Docker, or network (see
88+
`tests/README.md` for the design, and `CLAUDE.md` for conventions).
89+
90+
```
91+
uv run pytest # full suite with coverage (configured in pyproject.toml)
92+
uv run pytest --no-cov -q # quick run, no coverage
93+
uv run pytest tests/unit # unit tests only
94+
uv run pytest tests/integration # integration tests only
95+
uv run pytest -k workspaces # filter by keyword
96+
```
97+
98+
Type-check and format (matches the pre-commit hooks):
99+
100+
```
101+
uvx pyright --pythonpath .venv/bin/python api tests
102+
uv run black api tests && uv run isort api tests
103+
```
57104
## Deployment architecture
58105
59106
The deployed system is defined by [`docker-compose.az.yml`](docker-compose.az.yml). It runs the
@@ -125,94 +172,45 @@ Every image tag, database name/user, and server host is parameterized by `${ENV}
125172
(`dev` / `stage` / `prod`), and secrets are injected from the shell environment
126173
(`${WS_TASKS_DB_PASS}`, `${WS_OSM_DB_PASS}`, `${WS_OSM_SECRET_KEY_BASE}`). Branches map to these
127174
environments — see the Branch Index below.
128-
129-
## To start on your local machine for dev work
130-
131-
```
132-
cp .env.example .env # edit this file for your config
133-
uv sync
134-
uv run uvicorn api.main:app
135-
```
136-
137-
## Running the tests
138-
139-
Tests are fast and require no database, Docker, or network (see
140-
`tests/README.md` for the design, and `CLAUDE.md` for conventions).
141-
142-
```
143-
uv run pytest # full suite with coverage (configured in pyproject.toml)
144-
uv run pytest --no-cov -q # quick run, no coverage
145-
uv run pytest tests/unit # unit tests only
146-
uv run pytest tests/integration # integration tests only
147-
uv run pytest -k workspaces # filter by keyword
148-
```
149-
150-
Type-check and format (matches the pre-commit hooks):
151-
152-
```
153-
uvx pyright --pythonpath .venv/bin/python api tests
154-
uv run black api tests && uv run isort api tests
155-
```
156-
157-
## Development with local environment
158-
159-
Use the file `docker-compose.local.yml` to build and deploy local code changes. This allows you to run the entire system at once instead of
160-
connecting to existing Databases.
161-
162-
### Initial setup for development local environment
163-
164-
Step 1: Login to azure docker
165-
166-
The docker compose relies on images in `opensidewalksdev` azure container registry. Make sure your docker system is logged into it before pulling the images and trying to run the containers.
167-
168-
Docker login command:
169-
170-
`docker login opensidewalksdev.azurecr.io -u opensidewalksdev `
171-
172-
Password needs to be obtained from Azure portal
173-
174-
Step 2: Run docker compose for the first time
175-
176-
Use the following command to start the containers first time
177-
178-
`docker compose --file docker-compose.local.yml up --build`
179-
180-
Step 3: Run the migration scripts.
181-
182-
You will observe that only `osm-rails` component seems to work but the backend and other services may be down. this is because the database migrations on the base osm database are not done. To do the base migrations, do the following:
183-
184-
- Connect to the `osm-rails` container. If you are using docker hub for desktop, just go to the exec section of the container.
185-
If you want to use command line, execute the command `docker exec -it <container_name_or_id> /bin/bash` where `container_name` is the name of osm-rails container
186-
- Execute the migration script in the /bin/bash with `bundle exec rails db:migrate`
187-
- The above command runs the migration script for databases
188-
189-
Step 4: Add `workspaces-tasks-local` database in postgresql
190-
191-
Workspaces backend relies on an additional database. This is needed for some older migrations code.
192-
193-
- Connect to `database` container.
194-
- Run the following set of commands one by one
195-
196-
```shell
197-
psql --username postgres
198-
create database "workspaces-tasks-local";
199-
exit;
200-
psql --username postgres --dbname "workspaces-tasks-local";
201-
create extension if not exists postgis;
202175
203-
```
204-
205-
Step 5: Restart the docker compose again
206-
207-
- `docker compose --file docker-compose.local.yml down`
208-
- `docker compose --file docker-compose.local.yml up --build`
176+
### What the proxy must provide for osm-rails / osm-web
209177
178+
This backend is the **only entry point** to the OSM tier (osm-rails + cgimap,
179+
behind `osm-web`): in the deployment, the public OSM host routes to this
180+
container, which proxies `/api/0.6/*` to `WS_OSM_HOST` (default
181+
`http://osm-web`). For the OSM services to work, the proxy must uphold the
182+
following contract. `CLAUDE.md` has the full rationale.
210183
184+
1. **Bridge TDEI auth into OSM's OAuth2.** osm-rails authenticates the API *only*
185+
via doorkeeper OAuth2 (`oauth_access_tokens`); it has no TDEI/JWT path. So on
186+
token validation the backend mirrors the TDEI JWT into `oauth_access_tokens`
187+
in the OSM DB (the "token bridge" in `api/core/security.py`), and forwards the
188+
incoming `Authorization: Bearer <token>` header unchanged. Then osm-rails and
189+
cgimap authenticate the token via plain OAuth2. Controlled by
190+
`WS_OSM_TOKEN_BRIDGE_ENABLED` (on by default) — with it off, osm-rails returns
191+
**401** for TDEI tokens. The backend auto-creates the doorkeeper application
192+
(and a system user to own it), so no manual OSM setup is required.
211193
212-
### Commands to start and stop the docker compose
194+
2. **Provision valid OSM `users` rows.** The backend creates OSM `users` rows for
195+
TDEI users (`auth_provider='TDEI'`, `auth_uid` = the JWT `sub`). These must
196+
satisfy OSM's `User` validations — in particular `pass_crypt` length 8..255.
197+
A too-short value is invisible to cgimap but makes osm-rails operations that
198+
re-validate the user fail (e.g. posting a changeset comment or a note),
199+
surfacing as *"Unable to serialize … without an id"*. The
200+
`alembic_osm` migration `*_heal_short_tdei_pass_crypt` repairs legacy rows.
213201
214-
`docker compose --file docker-compose.local.yml up --build -d`
202+
3. **Carry workspace tenancy.** Workspace-scoped OSM requests must include an
203+
`X-Workspace: <id>` header. The proxy authorizes it against the caller's
204+
workspaces and forwards it (it is *not* stripped) so cgimap/osm-rails scope to
205+
the `workspace-<id>` schema. A few paths are exempt (`TENANT_BYPASSES` in
206+
`api/main.py`): workspace create/delete (`PUT`/`DELETE /api/0.6/workspaces/{id}`)
207+
and user provisioning during sign-in (`PUT /api/0.6/user/{uid}`).
215208
216-
`docker compose --file docker-compose.local.yml down`
209+
4. **Set the proxy headers.** The proxy rewrites `Host` to the OSM host and sets
210+
`X-Real-IP` / `X-Forwarded-For` / `-Host` / `-Proto`, while stripping
211+
hop-by-hop headers and any spoofed forwarding headers from the client. It does
212+
*not* strip `Authorization` or `X-Workspace`.
217213
218-
Backend code will be available at `http://localhost:8000`
214+
5. **Connectivity.** `WS_OSM_HOST` must reach `osm-web`, and the backend needs
215+
**both** `OSM_DATABASE_URL` and `TASK_DATABASE_URL` — the token bridge and
216+
user provisioning write to the OSM database.

0 commit comments

Comments
 (0)