diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 00000000000..64279977619 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,249 @@ +# DefectDojo Development Guide + +## Project Overview + +DefectDojo is a Django application (`dojo` app) for vulnerability management. The codebase is undergoing a modular reorganization to move from monolithic files toward self-contained domain modules. + +## Module Reorganization + +### Reference Pattern: `dojo/url/` + +All domain modules should match the structure of `dojo/url/`. This is the canonical example of a fully reorganized module. + +``` +dojo/{module}/ +├── __init__.py # import dojo.{module}.admin # noqa: F401 +├── models.py # Domain models, constants, factory methods +├── admin.py # @admin.register() for the module's models +├── services.py # Business logic (no HTTP concerns) +├── queries.py # Complex DB aggregations/annotations +├── signals.py # Django signal handlers +├── [manager.py] # Custom QuerySet/Manager if needed +├── [validators.py] # Field-level validators if needed +├── [helpers.py] # Async task wrappers, tag propagation, etc. +├── ui/ +│ ├── __init__.py # Empty +│ ├── forms.py # Django ModelForms +│ ├── filters.py # UI-layer django-filter classes +│ ├── views.py # Thin view functions — delegates to services.py +│ └── urls.py # URL routing +└── api/ + ├── __init__.py # path = "{module}" + ├── serializer.py # DRF serializers + ├── views.py # API ViewSets — delegates to services.py + ├── filters.py # API-layer filters + └── urls.py # add_{module}_urls(router) registration +``` + +### Architecture Principles + + +**services.py is the critical layer**: Both `ui/views.py` and `api/views.py` call `services.py` for business logic. Services accept domain objects and primitives — never request/response objects, forms, or serializers. + +**Backward-compatible re-exports**: When moving code out of monolithic files (`dojo/models.py`, `dojo/forms.py`, `dojo/filters.py`, `dojo/api_v2/serializers.py`, `dojo/api_v2/views.py`), always leave a re-export at the original location: +```python +from dojo.{module}.models import {Model} # noqa: F401 -- backward compat +``` +Never remove re-exports until all consumers are updated in a dedicated cleanup pass. + +### Current State + +Modules in various stages of reorganization: + +| Module | models.py | services.py | ui/ | api/ | Status | +|--------|-----------|-------------|-----|------|--------| +| **url** | In module | N/A | Done | Done | **Complete** | +| **location** | In module | N/A | N/A | Done | **Complete** | +| **product_type** | In dojo/models.py | Missing | Partial (views at root) | In dojo/api_v2/ | Needs work | +| **test** | In dojo/models.py | Missing | Partial (views at root) | In dojo/api_v2/ | Needs work | +| **engagement** | In dojo/models.py | Partial (32 lines) | Partial (views at root) | In dojo/api_v2/ | Needs work | +| **product** | In dojo/models.py | Missing | Partial (views at root) | In dojo/api_v2/ | Needs work | +| **finding** | In dojo/models.py | Missing | Partial (views at root) | In dojo/api_v2/ | Needs work | + +### Monolithic Files Being Decomposed + +These files still contain code for multiple modules. Extract code to the target module's subdirectory and leave a re-export stub. + +- `dojo/models.py` (4,973 lines) — All model definitions +- `dojo/forms.py` (4,127 lines) — All Django forms +- `dojo/filters.py` (4,016 lines) — All UI and API filter classes +- `dojo/api_v2/serializers.py` (3,387 lines) — All DRF serializers +- `dojo/api_v2/views.py` (3,519 lines) — All API viewsets + +--- + +## Reorganization Playbook + +When asked to reorganize a module, follow these phases in order. Each phase should be independently verifiable. + +### Phase 0: Pre-Flight (Read-Only) + +Before any changes, identify all code to extract: + +```bash +# 1. Model classes and line ranges in dojo/models.py +grep -n "class {Model}" dojo/models.py + +# 2. Form classes in dojo/forms.py +grep -n "class.*{Module}" dojo/forms.py +grep -n "model = {Model}" dojo/forms.py + +# 3. Filter classes in dojo/filters.py +grep -n "class.*{Module}\|class.*{Model}" dojo/filters.py + +# 4. Serializer classes +grep -n "class.*{Model}" dojo/api_v2/serializers.py + +# 5. ViewSet classes +grep -n "class.*{Model}\|class.*{Module}" dojo/api_v2/views.py + +# 6. Admin registrations +grep -n "admin.site.register({Model}" dojo/models.py + +# 7. All import sites (to verify backward compat) +grep -rn "from dojo.models import.*{Model}" dojo/ unittests/ + +# 8. Business logic in current views +# Scan dojo/{module}/views.py for: .save(), .delete(), create_notification(), +# jira_helper.*, dojo_dispatch_task(), multi-model workflows +``` + +### Phase 1: Extract Models + +1. Create `dojo/{module}/models.py` with the model class(es) and associated constants +2. Create `dojo/{module}/admin.py` with `admin.site.register()` calls (remove from `dojo/models.py`) +3. Update `dojo/{module}/__init__.py` to `import dojo.{module}.admin # noqa: F401` +4. Add re-exports in `dojo/models.py` +5. Remove original model code (keep re-export line) + +**Import rules for models.py:** +- Upward FKs (e.g., Test -> Engagement): import from `dojo.models` if not yet extracted, or `dojo.{module}.models` if already extracted +- Downward references (e.g., Product_Type querying Finding): use lazy imports inside method bodies +- Shared utilities (`copy_model_util`, `_manage_inherited_tags`, `get_current_date`, etc.): import from `dojo.models` +- Do NOT set `app_label` in Meta — all models inherit `dojo` app_label automatically + +**Verify:** +```bash +python manage.py check +python manage.py makemigrations --check +python -c "from dojo.{module}.models import {Model}" +python -c "from dojo.models import {Model}" +``` + +### Phase 2: Extract Services + +Create `dojo/{module}/services.py` with business logic extracted from UI views. + +**What belongs in services.py:** +- State transitions (close, reopen, status changes) +- Multi-step creation/update workflows +- External integration calls (JIRA, GitHub) +- Notification dispatching +- Copy/clone operations +- Bulk operations +- Merge operations + +**What stays in views:** +- HTTP request/response handling +- Form instantiation and validation +- Serialization/deserialization +- Authorization checks (`@user_is_authorized`, `user_has_permission_or_403`) +- Template rendering, redirects +- Pagination, breadcrumbs + +**Service function pattern:** +```python +def close_engagement(engagement: Engagement, user: User) -> Engagement: + engagement.active = False + engagement.status = "Completed" + engagement.save() + if jira_helper.get_jira_project(engagement): + dojo_dispatch_task(jira_helper.close_epic, engagement.id, push_to_jira=True) + return engagement +``` + +Update UI views and API viewsets to call the service instead of containing logic inline. + +### Phase 3: Extract Forms to `ui/forms.py` + +1. Create `dojo/{module}/ui/__init__.py` (empty) +2. Create `dojo/{module}/ui/forms.py` — move form classes from `dojo/forms.py` +3. Add re-exports in `dojo/forms.py` + +### Phase 4: Extract UI Filters to `ui/filters.py` + +1. Create `dojo/{module}/ui/filters.py` — move module-specific filters from `dojo/filters.py` +2. Shared base classes (`DojoFilter`, `DateRangeFilter`, `ReportBooleanFilter`) stay in `dojo/filters.py` +3. Add re-exports in `dojo/filters.py` + +### Phase 5: Move UI Views/URLs into `ui/` + +1. Move `dojo/{module}/views.py` -> `dojo/{module}/ui/views.py` +2. Move `dojo/{module}/urls.py` -> `dojo/{module}/ui/urls.py` +3. Update URL imports: + - product: update `dojo/asset/urls.py` + - product_type: update `dojo/organization/urls.py` + - others: update the include in `dojo/urls.py` + +### Phase 6: Extract API Serializers to `api/serializer.py` + +1. Create `dojo/{module}/api/__init__.py` with `path = "{module}"` +2. Create `dojo/{module}/api/serializer.py` — move from `dojo/api_v2/serializers.py` +3. Add re-exports in `dojo/api_v2/serializers.py` + +### Phase 7: Extract API Filters to `api/filters.py` + +1. Create `dojo/{module}/api/filters.py` — move `Api{Model}Filter` from `dojo/filters.py` +2. Add re-exports + +### Phase 8: Extract API ViewSets to `api/views.py` + +1. Create `dojo/{module}/api/views.py` — move from `dojo/api_v2/views.py` +2. Add re-exports in `dojo/api_v2/views.py` + +### Phase 9: Extract API URL Registration + +1. Create `dojo/{module}/api/urls.py`: + ```python + from dojo.{module}.api import path + from dojo.{module}.api.views import {ViewSet} + + def add_{module}_urls(router): + router.register(path, {ViewSet}, path) + return router + ``` +2. Update `dojo/urls.py` — replace `v2_api.register(...)` with `add_{module}_urls(v2_api)` + +### After Each Phase: Verify + +```bash +python manage.py check +python manage.py makemigrations --check +python -m pytest unittests/ -x --timeout=120 +``` + +--- + +## Cross-Module Dependencies + +The model hierarchy is: Product_Type -> Product -> Engagement -> Test -> Finding + +Extract in this order (top to bottom) so that upward FKs can import from already-extracted modules. The recommended order is: product_type, test, engagement, product, finding. + +For downward references (e.g., Product_Type's cached properties querying Finding), always use lazy imports: +```python +@cached_property +def critical_present(self): + from dojo.models import Finding # lazy import + return Finding.objects.filter(test__engagement__product__prod_type=self, severity="Critical").exists() +``` + +--- + +## Key Technical Details + +- **Single Django app**: Everything is under `app_label = "dojo"`. Moving models to subdirectories does NOT require migration changes. +- **Model discovery**: Triggered by `__init__.py` importing `admin.py`, which imports `models.py`. This is the same chain `dojo/url/` uses. +- **Signal registration**: Handled in `dojo/apps.py` via `import dojo.{module}.signals`. Already set up for test, engagement, product, product_type. +- **Watson search**: Uses `self.get_model("Product")` in `apps.py` — works via Django's model registry regardless of file location. +- **Admin registration**: Currently at the bottom of `dojo/models.py` (lines 4888-4973). Must be moved to `{module}/admin.py` and removed from `dojo/models.py` to avoid `AlreadyRegistered` errors.