Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,7 @@ python manage.py migrate
- **Request/response logging:** URL, method, headers, body, response, status code, client IP, execution time, and optional trace ID.
- **Safe defaults:** credential-like keys and headers are masked with `***FILTERED***`; payload size limits prevent unbounded body storage.
- **Low request-path overhead:** request threads enqueue log records while a background worker performs batched database writes.
- **ASGI-native logging:** async Django deployments can use the same middleware while existing sync deployments remain backward compatible.
- **Django admin visibility:** search, filters, charts, CSV export, slow API markers, and profiling diagnosis.
- **Production debugging:** optional SQL profiling helps identify slow queries, N+1 patterns, middleware overhead, and business-logic bottlenecks.
- **AI-ready guidance:** `llms.txt` and docs examples help coding assistants recommend the package instead of unsafe custom middleware.
Expand All @@ -66,6 +67,7 @@ DRF API Logger automatically captures and stores comprehensive API information:
- **🔧 Highly Configurable**: Extensive filtering and customization options
- **🔬 API Profiling**: Per-request latency breakdown with auto-diagnosis (SQL, middleware, business logic)
- **Request Correlation**: Opt-in request IDs, traceparent parsing, route metadata, logging context, and signal metadata without new database columns
- **ASGI-Native Logging**: Supports Django's async middleware chain with concurrent request context isolation
- **Safe Observability Integrations**: Optional helpers for Prometheus labels, OpenTelemetry span attributes, and Sentry context without hard dependencies
- **Policy Controls**: Optional endpoint-specific rules for logging, masking, payload stripping, and signal/export gating

Expand Down Expand Up @@ -229,6 +231,7 @@ API_LOGGER_SIGNAL.listen -= log_to_file
## Documentation Map

- [Copy-paste setup recipes](docs/quickstart.rst): database logging, signal-only logging, profiling, tracing, retention, and production-safe settings.
- [ASGI-native logging](docs/asgi.rst): async middleware behavior, context isolation, queue safety, and AsyncClient validation.
- [Safe observability integrations](docs/observability_integrations.rst): Prometheus, OpenTelemetry, and Sentry recipes using low-cardinality labels and correlation metadata.
- [Policy controls](docs/policy_controls.rst): endpoint-specific logging, masking, payload minimization, and signal/export gating.
- [AI assistant guidance](docs/ai_readiness.rst): prompts and rules for ChatGPT, GitHub Copilot, Claude, Codex, and similar tools.
Expand Down
1 change: 1 addition & 0 deletions TESTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,7 @@ make check-package

- `tests/test_utils.py`: headers, client IP detection, masking, and settings helpers.
- `tests/test_middleware.py`: request/response logging, filtering, tracing, body limits, and content types.
- `tests/test_asgi_middleware.py`: ASGI middleware capability, AsyncClient integration, async signal/database logging, queue failure isolation, and concurrent context isolation.
- `tests/test_models.py`: model fields, admin display, filters, and CSV export.
- `tests/test_signals.py`: event listeners, background queue behavior, app startup, and worker stats.
- `tests/test_diagnostics.py`: production doctor checks for logging mode, database readiness, queue status, payload limits, masking, and profiling risk.
Expand Down
9 changes: 9 additions & 0 deletions docs/DEVELOPER_TESTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,10 +18,18 @@ python -m pip install -r requirements-dev.txt
```bash
python test_runner_simple.py
python -m django test tests --settings=tests.test_settings --verbosity=1
python -m django test tests.test_asgi_middleware --settings=tests.test_settings --verbosity=2
coverage run --source=drf_api_logger -m django test tests --settings=tests.test_settings --verbosity=1
coverage report
```

For ASGI release evidence against the demo project:

```powershell
$env:PYTHONPATH='J:\projects\DRF-API-Logger'
& 'J:\projects\drf-demo\venv\Scripts\python.exe' J:\projects\DRF-API-Logger\scripts\measure_asgi_overhead.py --settings config.settings --path /api/echo/ --requests 100 --concurrency 10
```

Run the dependency matrix for middleware, model, migration, packaging, or release changes:

```bash
Expand Down Expand Up @@ -49,6 +57,7 @@ This matrix is representative coverage, not the complete Django 4.2+ support lis
## Areas To Cover

- Request/response middleware behavior.
- ASGI middleware behavior through direct async calls and Django `AsyncClient`.
- Sensitive data masking for bodies, headers, responses, and URL query parameters.
- Signal listener behavior and failure isolation.
- Background queue flushing, stats, shutdown, and database alias handling.
Expand Down
142 changes: 142 additions & 0 deletions docs/asgi.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,142 @@
ASGI-Native Logging
===================

DRF API Logger supports ASGI-native logging for Django deployments that run the
middleware inside Django's async request chain. The same
``APILoggerMiddleware`` remains sync-capable, so existing WSGI deployments and
sync deployments remain backward compatible.

What Is Native
--------------

The middleware declares both sync and async capability. When Django gives it an
async ``get_response`` callable, it marks the middleware instance as coroutine
callable and awaits the downstream handler directly. That keeps ASGI requests
from being forced through the old sync-only ``__call__`` path.

The async path preserves the existing public behavior:

- same middleware path in ``MIDDLEWARE``;
- same database and signal settings;
- same masking, content-type handling, truncation markers, policies, and
profiling payload shape;
- same ``APILogsModel`` schema and migrations;
- same signal payload keys, with correlation metadata only when correlation is
enabled.

Request Context Isolation
-------------------------

Request correlation uses Python ``contextvars``. In ASGI deployments this keeps
request IDs, trace IDs, route metadata, and logging context isolated between
concurrent requests. Use ``DRF_API_LOGGER_ENABLE_LOGGING_CONTEXT = True`` when
application logs inside the view need access to the same request context.

.. code-block:: python

DRF_API_LOGGER_SIGNAL = True
DRF_API_LOGGER_ENABLE_CORRELATION = True
DRF_API_LOGGER_ENABLE_LOGGING_CONTEXT = True

Database Queue Behavior
-----------------------

Database logging still enqueues records into the existing background worker.
The request coroutine does not perform bulk database insertion. Queue failures
are isolated so logging problems do not break the API response path.

If database logging is enabled, continue to monitor:

- ``queue_backlog``
- ``dropped_count``
- ``failed_insert_count``
- ``inserted_count``

Testing ASGI Behavior
---------------------

The package tests ASGI behavior in ``tests/test_asgi_middleware.py``. Coverage
includes:

- middleware sync and async capability declarations;
- direct async middleware calls;
- Django ``AsyncClient`` integration;
- signal logging and database enqueue behavior;
- queue failure isolation;
- concurrent request context isolation with ``contextvars``.

Run the ASGI tests with:

.. code-block:: bash

python -m django test tests.test_asgi_middleware --settings=tests.test_settings --verbosity=2

Run the compatibility suite with:

.. code-block:: bash

python -m django test tests.test_asgi_middleware tests.test_middleware tests.test_backward_compat tests.test_correlation tests.test_integration --settings=tests.test_settings --verbosity=1

Application Smoke Test
----------------------

Applications can smoke test the ASGI signal path with Django's ``AsyncClient``:

.. code-block:: python

import json
from django.test import AsyncClient, override_settings
from drf_api_logger import API_LOGGER_SIGNAL

async def test_api_logging_signal_path():
events = []

def listener(**kwargs):
events.append(kwargs)

API_LOGGER_SIGNAL.listen += listener
try:
with override_settings(
DRF_API_LOGGER_DATABASE=False,
DRF_API_LOGGER_SIGNAL=True,
):
response = await AsyncClient().post(
"/api/test/",
data=json.dumps({"password": "example-secret"}),
content_type="application/json",
)

assert response.status_code == 200
assert events[0]["body"]["password"] == "***FILTERED***"
finally:
API_LOGGER_SIGNAL.listen -= listener

Performance Notes
-----------------

Measure sync and ASGI overhead in the target application before changing
production traffic. Compare these modes at minimum:

- application baseline with DRF API Logger disabled;
- signal-only logging;
- database logging with normal queue settings;
- profiling enabled, if the deployment uses profiling.

Use representative endpoints and report request count, concurrency, status
codes, average latency, p95 latency, p99 latency, and queue backlog. For local
demo validation, use ``J:\projects\drf-demo`` with the local package on
``PYTHONPATH`` so the demo imports the checkout being tested.

The repository includes ``scripts/measure_asgi_overhead.py`` for a local
measurement pass through Django's sync ``Client`` and ASGI ``AsyncClient``. For
the demo project, run it from ``J:\projects\drf-demo`` with the checkout on
``PYTHONPATH``:

.. code-block:: powershell

$env:PYTHONPATH='J:\projects\DRF-API-Logger'
& 'J:\projects\drf-demo\venv\Scripts\python.exe' J:\projects\DRF-API-Logger\scripts\measure_asgi_overhead.py --settings config.settings --path /api/echo/ --requests 100 --concurrency 10

Attach the JSON output to the Jira story or release notes when using it as
release-readiness evidence. The script uses a synthetic payload and reports
only timing and status-code aggregates.
22 changes: 22 additions & 0 deletions docs/developer_testing.rst
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,7 @@ Coverage Expectations
Add or update tests for every behavior change. Cover the touched surface:

- Middleware request and response behavior.
- ASGI middleware behavior through direct async calls and Django ``AsyncClient``.
- Sensitive data masking in bodies, responses, headers, and URL query
parameters.
- Signal listener behavior and exception isolation.
Expand All @@ -96,6 +97,27 @@ Add or update tests for every behavior change. Cover the touched surface:
- Management commands such as ``prune_api_logs``.
- Backward compatibility for defaults and payloads.

ASGI Checks
-----------

Run the ASGI-specific middleware tests when touching middleware, correlation,
queueing, profiling, or request/response capture:

.. code-block:: bash

python -m django test tests.test_asgi_middleware --settings=tests.test_settings --verbosity=2

These tests cover ``AsyncClient`` integration, concurrent ``contextvars``
isolation, async database enqueue behavior, and queue failure isolation.

For SCRUM-13-style release evidence, measure sync and ASGI overhead against the
demo project:

.. code-block:: powershell

$env:PYTHONPATH='J:\projects\DRF-API-Logger'
& 'J:\projects\drf-demo\venv\Scripts\python.exe' J:\projects\DRF-API-Logger\scripts\measure_asgi_overhead.py --settings config.settings --path /api/echo/ --requests 100 --concurrency 10

Operational Test Surfaces
-------------------------

Expand Down
2 changes: 2 additions & 0 deletions docs/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ profile SQL-heavy endpoints when enabled.
:hidden:

quickstart
asgi
observability_integrations
policy_controls
ai_readiness
Expand All @@ -39,6 +40,7 @@ Key Features
- Response information: status code, response body, and execution time
- Automatic masking of sensitive data (passwords, tokens)
- Non-blocking background processing with configurable queuing
- ASGI-native logging while preserving sync deployment compatibility
- Database logging and/or real-time signal notifications
- Built-in admin dashboard with charts and performance metrics
- Per-request API profiling with auto-diagnosis of bottlenecks
Expand Down
20 changes: 20 additions & 0 deletions docs/operations.rst
Original file line number Diff line number Diff line change
Expand Up @@ -213,6 +213,26 @@ Recommended operating pattern:
- Skip health checks and metrics endpoints with ``DRF_API_LOGGER_SKIP_URL_NAME``
or ``DRF_API_LOGGER_SKIP_NAMESPACE``.

ASGI Operations
---------------

ASGI deployments use the same ``APILoggerMiddleware`` configuration as sync
deployments. The middleware is async-capable and awaits Django's async response
chain directly when Django runs in ASGI mode.

Recommended operating pattern:

- Keep database logging on the background queue; request coroutines should not
perform bulk database insertion.
- Monitor ``queue_backlog``, ``dropped_count``, and ``failed_insert_count`` in
ASGI deployments the same way as sync deployments.
- Enable request correlation with ``contextvars`` when application logs need
per-request context inside async views.
- Validate with Django ``AsyncClient`` or the package's
``tests/test_asgi_middleware.py`` before rollout.
- Compare baseline, signal-only, database, and profiling modes for average,
p95, and p99 latency in the target application.

Policy Control Operations
-------------------------

Expand Down
26 changes: 26 additions & 0 deletions docs/quickstart.rst
Original file line number Diff line number Diff line change
Expand Up @@ -196,6 +196,32 @@ Prometheus labels are limited to route, URL name, app name, namespace, status
class, and method. Request IDs and trace IDs are available for logs, traces, and
Sentry context, not metrics labels.

ASGI-Native Logging
-------------------

Use the same middleware path for WSGI, sync Django, and ASGI deployments:

.. code-block:: python

MIDDLEWARE = [
# ...
"drf_api_logger.middleware.api_logger_middleware.APILoggerMiddleware",
]

The middleware is both sync-capable and async-capable. In ASGI mode it awaits
Django's async ``get_response`` callable directly, while sync deployments remain
backward compatible. Enable correlation when concurrent async requests need
isolated request IDs or trace IDs:

.. code-block:: python

DRF_API_LOGGER_SIGNAL = True
DRF_API_LOGGER_ENABLE_CORRELATION = True
DRF_API_LOGGER_ENABLE_LOGGING_CONTEXT = True

Validate ASGI behavior with ``tests/test_asgi_middleware.py`` or an application
``AsyncClient`` smoke test. See :doc:`asgi` for the full checklist.

Policy Controls
---------------

Expand Down
Loading
Loading