Skip to content

Commit 72a1f62

Browse files
filippomcfilippomc
authored
Merge pull request #849 from MetaCell/fix-thread-safety-cache-invalidation
Fix thread safety and cache invalidation
2 parents 635b719 + 443032c commit 72a1f62

5 files changed

Lines changed: 360 additions & 22 deletions

File tree

‎infrastructure/common-images/cloudharness-django/libraries/cloudharness-django/README.md‎

Lines changed: 322 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,322 @@
1+
# cloudharness-django
2+
3+
A Django library that integrates [CloudHarness](https://github.com/MetaCell/cloud-harness) infrastructure into Django applications. It handles Keycloak OIDC authentication, automatically synchronises Keycloak users/groups/organisations into Django's database, and listens to Kafka events to keep that data up to date in real time.
4+
5+
## Table of contents
6+
7+
- [How it works](#how-it-works)
8+
- [Installation](#installation)
9+
- [Configuration](#configuration)
10+
- [Startup](#startup)
11+
- [Models](#models)
12+
- [Authentication](#authentication)
13+
- [Services](#services)
14+
- [Authorization levels](#authorization-levels)
15+
- [Event-driven synchronisation](#event-driven-synchronisation)
16+
- [Management commands](#management-commands)
17+
- [Admin UI](#admin-ui)
18+
- [Local development](#local-development)
19+
20+
---
21+
22+
## How it works
23+
24+
```
25+
HTTP request
26+
│
27+
▼
28+
BearerTokenMiddleware
29+
│ extracts JWT from Authorization header or kc-access cookie
30+
│ decodes token → gets Keycloak user ID (sub)
31+
│ checks Django cache (TTL 60 s)
32+
│ hit ──────────────────────────────────► request.user
33+
│ miss ──► _get_user()
34+
│ │
35+
│ ├─ User.objects.get(member__kc_id=…)
36+
│ │ found ─► sync_kc_user() (updates fields)
37+
│ │ not found ─► sync_kc_user() (creates User + Member)
38+
│ │
39+
│ └─► SAFETY CHECK: User must always have a Member
40+
│ ok ─► cache user ─► request.user
41+
│ fail ─► keep anonymous
42+
▼
43+
Django view / Django Ninja
44+
45+
46+
Keycloak admin events (via Kafka)
47+
│
48+
▼
49+
KeycloakMessageService.event_handler()
50+
│
51+
├─ USER ─► invalidate_user_cache() + sync_kc_user()
52+
├─ GROUP ─► sync_kc_group()
53+
├─ CLIENT_ROLE_MAPPING ─► sync_kc_user()
54+
├─ GROUP_MEMBERSHIP ─► sync_kc_user() (groups re-synced inside)
55+
└─ ORGANIZATION_MEMBERSHIP ─► sync_kc_user() (orgs re-synced inside)
56+
```
57+
58+
Every authenticated request syncs the Keycloak user into the Django `User` table. A `Member` record (holding the Keycloak UUID `kc_id`) is always created alongside it. The middleware caches the resolved `User` object for 60 seconds to avoid a database round-trip on every request. Kafka events invalidate that cache and trigger a full re-sync when Keycloak data changes.
59+
60+
---
61+
62+
## Installation
63+
64+
```bash
65+
pip install cloudharness-django
66+
```
67+
68+
Add the library's settings module to the **top** of your `settings.py`:
69+
70+
```python
71+
PROJECT_NAME = "my-app" # must be set before the import
72+
73+
from cloudharness_django.settings import * # noqa: F401, F403
74+
```
75+
76+
This import:
77+
78+
- Appends `cloudharness_django`, `admin_extra_buttons`, and `django_prometheus` to `INSTALLED_APPS`.
79+
- Inserts `BearerTokenMiddleware` (and Prometheus middleware) into `MIDDLEWARE`.
80+
- Configures the `DATABASES` setting from the CloudHarness deployment values (PostgreSQL in Kubernetes, SQLite on a developer machine when no `allvalues.yaml` is found).
81+
- Sets `CSRF_TRUSTED_ORIGINS` from the CloudHarness domain.
82+
- Redirects `ROOT_URLCONF` through `cloudharness_django.urls` (which mounts Prometheus metrics and then delegates to your original URL config).
83+
84+
Run migrations once after installation:
85+
86+
```bash
87+
python manage.py migrate
88+
```
89+
90+
---
91+
92+
## Configuration
93+
94+
Add the following to your `settings.py` after importing the CloudHarness settings:
95+
96+
```python
97+
# Keycloak client name (usually the application name in lower case)
98+
KC_CLIENT_NAME = PROJECT_NAME.lower()
99+
100+
# Role names that exist (or will be created) in the Keycloak client
101+
KC_ADMIN_ROLE = f"{KC_CLIENT_NAME}-administrator"
102+
KC_MANAGER_ROLE = f"{KC_CLIENT_NAME}-manager"
103+
KC_USER_ROLE = f"{KC_CLIENT_NAME}-user"
104+
105+
KC_ALL_ROLES = [KC_ADMIN_ROLE, KC_MANAGER_ROLE, KC_USER_ROLE]
106+
KC_PRIVILEGED_ROLES = [KC_MANAGER_ROLE]
107+
108+
# Role automatically assigned to every new Keycloak user via the default realm role.
109+
# Set to None to disable.
110+
KC_DEFAULT_USER_ROLE = KC_USER_ROLE
111+
```
112+
113+
### Optional settings
114+
115+
| Setting | Default | Description |
116+
|---|---|---|
117+
| `BEARER_TOKEN_USER_CACHE_TTL` | `60` | Seconds a resolved `User` object is cached in Django's cache backend. |
118+
| `USER_CHANGE_ENABLED` | `False` | Allow editing users directly in the Django admin (bypasses Keycloak). |
119+
120+
---
121+
122+
## Startup
123+
124+
Call `init_services()` during application startup (e.g. in `apps.py` `ready()`):
125+
126+
```python
127+
# apps.py
128+
from django.apps import AppConfig
129+
130+
class MyAppConfig(AppConfig):
131+
name = "my_app"
132+
133+
def ready(self):
134+
from cloudharness_django.services import init_services
135+
from cloudharness_django.services.events import init_listener_in_background
136+
137+
# Initialise auth + user services (creates the Keycloak client and roles if absent)
138+
init_services()
139+
140+
# Start the Kafka consumer in a background thread
141+
init_listener_in_background()
142+
```
143+
144+
If Keycloak may not be reachable immediately (e.g. during container startup), use the background variant for services as well:
145+
146+
```python
147+
from cloudharness_django.services import init_services_in_background
148+
init_services_in_background()
149+
```
150+
151+
Both background helpers retry with a 5-second interval until they succeed.
152+
153+
---
154+
155+
## Models
156+
157+
| Model | Purpose |
158+
|---|---|
159+
| `Member` | Links a Django `User` to its Keycloak UUID (`kc_id`). Always present; a `User` without a `Member` is treated as anonymous. |
160+
| `Team` | Links a Django `Group` to a Keycloak group (`kc_id`). Tracks the group owner. |
161+
| `Organization` | Mirrors a Keycloak organisation. Can be pre-created in Django (without `kc_id`) and will be matched by name when Keycloak syncs. |
162+
| `OrganizationMember` | Many-to-many membership table between `User` and `Organization`. |
163+
164+
Extending `Organization` with application-specific fields:
165+
166+
```python
167+
from cloudharness_django.models import Organization
168+
169+
class MyOrganization(models.Model):
170+
organization = models.OneToOneField(Organization, on_delete=models.CASCADE, related_name="my_org")
171+
logo = models.ImageField(upload_to="org_logos/", blank=True)
172+
```
173+
174+
---
175+
176+
## Authentication
177+
178+
### Middleware: `BearerTokenMiddleware`
179+
180+
Activated automatically via the settings import. On every request it:
181+
182+
1. Reads the bearer token from the `Authorization` header or `kc-access` cookie.
183+
2. Decodes and validates the JWT (via `cloudharness.auth.keycloak.AuthClient`).
184+
3. Looks up the Django `User` by `member__kc_id`, creating it if absent.
185+
4. Syncs user attributes and group memberships from Keycloak.
186+
5. Caches the result and sets `request.user`.
187+
188+
If the token is invalid, the user is logged out and the `kc-access` cookie is deleted.
189+
190+
### Cache invalidation
191+
192+
When a user is deleted or deactivated in Keycloak the Kafka event handler calls `invalidate_user_cache()` to remove the stale cache entry before syncing. You can also call it directly:
193+
194+
```python
195+
from cloudharness_django.middleware import invalidate_user_cache
196+
invalidate_user_cache(kc_user_id)
197+
```
198+
199+
---
200+
201+
## Services
202+
203+
Services are initialised per execution context (thread or coroutine) using `contextvars.ContextVar`. They are created lazily on first access.
204+
205+
```python
206+
from cloudharness_django.services import get_auth_service, get_user_service
207+
208+
auth_service = get_auth_service() # AuthService
209+
user_service = get_user_service() # UserService
210+
```
211+
212+
### `AuthService`
213+
214+
| Method | Description |
215+
|---|---|
216+
| `get_auth_client()` | Returns the underlying `cloudharness.auth.AuthClient`. |
217+
| `create_client()` | Creates the Keycloak client and its roles if they do not exist. |
218+
| `get_auth_level(kc_user, kc_roles)` | Returns the `AuthorizationLevel` for the given user. |
219+
220+
### `UserService`
221+
222+
| Method | Description |
223+
|---|---|
224+
| `sync_kc_user(kc_user, delete=False)` | Create or update the Django `User` + `Member` for a Keycloak user. Atomic. |
225+
| `sync_kc_user_groups(kc_user)` | Replace the user's Django groups with their current Keycloak groups. |
226+
| `sync_kc_user_organizations(kc_user)` | Sync `OrganizationMember` rows for the user. |
227+
| `sync_kc_group(kc_group)` | Create or update the Django `Group` + `Team` for a Keycloak group. |
228+
| `sync_kc_groups()` | Sync all Keycloak groups. |
229+
| `sync_kc_users_groups()` | Full sync: all users, their groups, and their memberships. |
230+
| `create_team(group_name)` | Create a group in both Keycloak and Django. |
231+
| `add_user_to_team(user, team_name)` | Add a user to a group in Keycloak (Django side follows via Kafka). |
232+
| `rm_user_from_team(user, team_name)` | Remove a user from a group in Keycloak. |
233+
234+
Accessing the current user from within a view:
235+
236+
```python
237+
from cloudharness_django.services import get_auth_service
238+
239+
auth_service = get_auth_service()
240+
auth_client = auth_service.get_auth_client()
241+
kc_user = auth_client.get_current_user()
242+
auth_level = auth_service.get_auth_level(kc_user)
243+
```
244+
245+
---
246+
247+
## Authorization levels
248+
249+
`AuthorizationLevel` is an enum returned by `AuthService.get_auth_level()`:
250+
251+
| Level | Condition |
252+
|---|---|
253+
| `NO_AUTHORIZATION` | User has no client roles. |
254+
| `NON_PRIVILEGED` | User has at least one role in `KC_ALL_ROLES`. |
255+
| `PRIVILEGED` | User has at least one role in `KC_PRIVILEGED_ROLES`. |
256+
| `ADMIN` | User has the `KC_ADMIN_ROLE`. Also sets `is_superuser` and `is_staff` on the Django `User`. |
257+
258+
```python
259+
from cloudharness_django.services.auth import AuthorizationLevel
260+
261+
if auth_level == AuthorizationLevel.ADMIN:
262+
...
263+
```
264+
265+
---
266+
267+
## Event-driven synchronisation
268+
269+
`KeycloakMessageService` consumes the `keycloak.fct.admin` Kafka topic. On each Keycloak admin event it dispatches to the appropriate sync method:
270+
271+
| Keycloak event | Action |
272+
|---|---|
273+
| `USER` (any operation) | `invalidate_user_cache()` + `sync_kc_user()` |
274+
| `USER` with `DELETE` | `sync_kc_user(delete=True)` — deactivates the Django user |
275+
| `GROUP` | `sync_kc_group()` |
276+
| `CLIENT_ROLE_MAPPING` | `sync_kc_user()` — re-evaluates superuser status |
277+
| `GROUP_MEMBERSHIP` | `sync_kc_user()` — groups re-synced inside |
278+
| `ORGANIZATION_MEMBERSHIP` | `sync_kc_user()` — organisations re-synced inside |
279+
280+
Use `init_listener_in_background()` (see [Startup](#startup)) to start the consumer. If Kafka is unavailable (e.g. on a developer machine without `allvalues.yaml`), the function exits silently.
281+
282+
---
283+
284+
## Management commands
285+
286+
Manually trigger a full sync of all Keycloak users and groups:
287+
288+
```bash
289+
python manage.py cloudharness sync
290+
```
291+
292+
Useful after bulk changes in Keycloak or to bootstrap a fresh database.
293+
294+
---
295+
296+
## Admin UI
297+
298+
The Django admin provides:
299+
300+
- **User list** with a "Sync from Keycloak" button per user.
301+
- **Group list** with team management.
302+
- **Organization list** with member overview.
303+
304+
Direct user editing in the admin is disabled by default (`USER_CHANGE_ENABLED = False`). To allow it (e.g. in a development environment):
305+
306+
```python
307+
# settings.py
308+
USER_CHANGE_ENABLED = True
309+
```
310+
311+
---
312+
313+
## Local development
314+
315+
Without a running Kubernetes cluster, `cloudharness_django.settings` falls back to SQLite and logs a warning. You still need a reachable Keycloak instance. To test Kafka event handling locally, copy your deployment's `allvalues.yaml` to the path referenced by `ALLVALUES_PATH` and ensure Kafka is accessible.
316+
317+
Environment variables used by the library:
318+
319+
| Variable | Description |
320+
|---|---|
321+
| `ACCOUNTS_ADMIN_USERNAME` | Keycloak admin username for the `AuthClient`. |
322+
| `ACCOUNTS_ADMIN_PASSWORD` | Keycloak admin password for the `AuthClient`. |

‎infrastructure/common-images/cloudharness-django/libraries/cloudharness-django/cloudharness_django/middleware.py‎

Lines changed: 11 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,15 @@
1919
USER_CACHE_TTL = getattr(settings, "BEARER_TOKEN_USER_CACHE_TTL", 60)
2020

2121

22+
def _get_bearer_cache_key(kc_user_id: str) -> str:
23+
return f"bearer_token_user:{kc_user_id}"
24+
25+
26+
def invalidate_user_cache(kc_user_id: str) -> None:
27+
"""Remove a user from the bearer-token cache (e.g. after deletion/deactivation)."""
28+
cache.delete(_get_bearer_cache_key(kc_user_id))
29+
30+
2231
def _get_user(kc_user_id: str) -> User:
2332
"""
2433
Get or create a Django user for the given Keycloak user ID.
@@ -137,7 +146,7 @@ def __call__(self, request):
137146
return response
138147

139148
if kc_user_id:
140-
cache_key = f"bearer_token_user:{kc_user_id}"
149+
cache_key = _get_bearer_cache_key(kc_user_id)
141150
cached_user = cache.get(cache_key)
142151
if cached_user:
143152
request.user = cached_user
@@ -166,7 +175,7 @@ def __call__(self, request):
166175
# elif not request.path.startswith('/admin/'):
167176
# logout(request)
168177
if kc_user_id and user:
169-
cache.set(cache_key, user, timeout=USER_CACHE_TTL)
178+
cache.set(_get_bearer_cache_key(kc_user_id), user, timeout=USER_CACHE_TTL)
170179
return self.get_response(request)
171180

172181

0 commit comments

Comments
 (0)