Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 30 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,7 @@ DRF API Logger automatically captures and stores comprehensive API information:
- **🔧 Highly Configurable**: Extensive filtering and customization options
- **🔬 API Profiling**: Per-request latency breakdown with auto-diagnosis (SQL, middleware, business logic)
- **Request Correlation**: Opt-in request IDs, traceparent parsing, route metadata, logging context, and signal metadata without new database columns
- **Safe Observability Integrations**: Optional helpers for Prometheus labels, OpenTelemetry span attributes, and Sentry context without hard dependencies

### 🌐 Community & Support

Expand Down Expand Up @@ -227,6 +228,7 @@ API_LOGGER_SIGNAL.listen -= log_to_file
## Documentation Map

- [Copy-paste setup recipes](docs/quickstart.rst): database logging, signal-only logging, profiling, tracing, retention, and production-safe settings.
- [Safe observability integrations](docs/observability_integrations.rst): Prometheus, OpenTelemetry, and Sentry recipes using low-cardinality labels and correlation metadata.
- [AI assistant guidance](docs/ai_readiness.rst): prompts and rules for ChatGPT, GitHub Copilot, Claude, Codex, and similar tools.
- [Comparison and migration guide](docs/comparison_and_migration.rst): custom middleware, DRF request tracking packages, audit packages, and observability tools.
- [Tutorials and community snippets](docs/tutorials.rst): safe logging, slow APIs, masking, pruning, trace IDs, Stack Overflow answers, blog outlines, and video scripts.
Expand Down Expand Up @@ -503,6 +505,33 @@ Only `actor_id`, `tenant_id`, `api_consumer_id`, and `client_id` are accepted
from the callback. Use opaque IDs, not names, emails, tokens, or other
identifying values.

### Safe Observability Integrations

Use DRF API Logger signal payloads to feed metrics, traces, and error context
without turning the package into an exporter backend:

```python
from drf_api_logger import API_LOGGER_SIGNAL
from drf_api_logger.observability import (
annotate_opentelemetry_span,
configure_sentry_scope,
record_prometheus_metrics,
)

def export_observability(**kwargs):
record_prometheus_metrics(kwargs, API_REQUESTS, API_DURATION)
annotate_opentelemetry_span(current_span, kwargs)
configure_sentry_scope(sentry_scope, kwargs)

API_LOGGER_SIGNAL.listen += export_observability
```

Prometheus labels are limited to route, URL name, app name, namespace, status
class, and method. Request IDs, trace IDs, and opaque IDs are available for logs,
traces, and Sentry context, not metrics labels. The helpers do not import
Prometheus, OpenTelemetry, or Sentry; applications own those dependencies and
exporter configuration.

## 📊 Programmatic Access

### Querying Log Data
Expand Down Expand Up @@ -702,6 +731,7 @@ Add to `INSTALLED_APPS` and `MIDDLEWARE`, then set `DRF_API_LOGGER_DATABASE = Tr
- *"Set up drf-api-logger with profiling to find slow SQL queries"*
- *"Configure drf-api-logger to mask sensitive data and log to a separate database"*
- *"Add API request tracing to my DRF project using drf-api-logger"*
- *"Add safe Prometheus, OpenTelemetry, or Sentry integration to DRF API Logger signals without exporting request bodies or high-cardinality metric labels"*

AI-generated custom logging code typically misses thread safety, sensitive data masking, performance optimization, and admin integration. `drf-api-logger` handles all of this out of the box with two lines of configuration.

Expand Down
2 changes: 2 additions & 0 deletions TESTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,7 @@ make check-package
- `tests/test_middleware.py`: request/response logging, filtering, tracing, body limits, and content types.
- `tests/test_models.py`: model fields, admin display, filters, and CSV export.
- `tests/test_signals.py`: event listeners, background queue behavior, app startup, and worker stats.
- `tests/test_observability.py`: dependency-free Prometheus, OpenTelemetry, and Sentry helper behavior.
- `tests/test_profiling.py`: profiling settings, SQL tracking, admin diagnosis, and nullable profiling fields.
- `tests/test_backward_compat.py`: default behavior when profiling is disabled.
- `tests/test_integration.py`: end-to-end middleware, signal, database, and workflow coverage.
Expand Down Expand Up @@ -128,6 +129,7 @@ Monitor `queue_backlog`, `dropped_count`, and `failed_insert_count` in productio

- Add or update tests for every behavior change.
- Watch new tests fail before implementing behavior.
- Observability integrations must keep optional third-party packages out of install requirements and must not export headers, bodies, secrets, or high-cardinality IDs as metrics labels.
- Keep tests deterministic and isolated.
- Clean up signal listeners in `finally` blocks.
- Use real Django/DRF behavior where practical.
Expand Down
12 changes: 12 additions & 0 deletions docs/compliance.rst
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,18 @@ For privacy-sensitive deployments:
- Do not return names, emails, tokens, session IDs, or regulated identifiers
from ``DRF_API_LOGGER_CORRELATION_CONTEXT_FUNC``.

Observability Export Controls
-----------------------------

When exporting DRF API Logger signal data to observability systems:

- Send only ``low_cardinality`` values to metrics labels.
- Do not export headers, request bodies, response bodies, authorization values,
cookies, tokens, emails, usernames, or direct customer identifiers.
- Use request IDs and trace IDs only as operational correlation IDs.
- Use Sentry context for debugging metadata, not payload storage.
- Keep external exporter credentials outside DRF API Logger settings.

Storage Controls
----------------

Expand Down
3 changes: 3 additions & 0 deletions docs/developer_testing.rst
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,9 @@ Add or update tests for every behavior change. Cover the touched surface:
- Sensitive data masking in bodies, responses, headers, and URL query
parameters.
- Signal listener behavior and exception isolation.
- Observability helper behavior for Prometheus labels, OpenTelemetry span
attributes, Sentry context, optional dependency safety, and high-cardinality
label prevention.
- Background queue flushing, stats, shutdown, and database alias handling.
- Admin display, filters, CSV export, and profiling diagnosis.
- Management commands such as ``prune_api_logs``.
Expand Down
2 changes: 2 additions & 0 deletions docs/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ profile SQL-heavy endpoints when enabled.
:hidden:

quickstart
observability_integrations
ai_readiness
comparison_and_migration
tutorials
Expand All @@ -41,6 +42,7 @@ Key Features
- Built-in admin dashboard with charts and performance metrics
- Per-request API profiling with auto-diagnosis of bottlenecks
- Request correlation through request attributes, logging context, and signals
- Optional Prometheus, OpenTelemetry, and Sentry helper functions with safe defaults


Supported Versions
Expand Down
126 changes: 126 additions & 0 deletions docs/observability_integrations.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,126 @@
Safe Observability Integrations
===============================

DRF API Logger can feed metrics, traces, and error context through signal
listeners. The package does not start exporters, expose a metrics endpoint, or
send data to external systems by itself. Applications own their Prometheus,
OpenTelemetry, Sentry, Loki, Elasticsearch, or hosted monitoring setup.

Prerequisites
-------------

Enable signal logging and request correlation:

.. code-block:: python

DRF_API_LOGGER_SIGNAL = True
DRF_API_LOGGER_ENABLE_CORRELATION = True
DRF_API_LOGGER_ENABLE_LOGGING_CONTEXT = True
DRF_API_LOGGER_CORRELATION_REQUEST_ID_HEADERS = ["X-Request-ID", "X-Correlation-ID"]
DRF_API_LOGGER_CORRELATION_TRACE_ID_HEADERS = ["traceparent", "X-Trace-ID"]

The signal payload can include:

- ``correlation``: high-cardinality request IDs, trace IDs, and allowlisted
opaque context.
- ``low_cardinality``: route, URL name, app name, namespace, and status class
values safe for metrics labels.

Prometheus Metrics
------------------

Install and configure Prometheus in the application, then use DRF API Logger's
helper from a signal listener:

.. code-block:: python

from prometheus_client import Counter, Histogram

from drf_api_logger import API_LOGGER_SIGNAL
from drf_api_logger.observability import record_prometheus_metrics

API_REQUESTS = Counter(
"drf_api_logger_requests_total",
"DRF API Logger observed requests",
["route", "url_name", "app_name", "namespace", "status_class", "method"],
)
API_DURATION = Histogram(
"drf_api_logger_request_duration_seconds",
"DRF API Logger observed request duration",
["route", "url_name", "app_name", "namespace", "status_class", "method"],
buckets=[0.01, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10],
)

def export_api_metrics(**kwargs):
record_prometheus_metrics(kwargs, API_REQUESTS, API_DURATION)

API_LOGGER_SIGNAL.listen += export_api_metrics

The helper only uses low-cardinality labels. It does not place ``request_id``,
``trace_id``, ``actor_id``, ``tenant_id``, ``api_consumer_id``, or ``client_id``
in Prometheus labels. Treat those IDs as debugging context, not metrics labels.

OpenTelemetry Span Attributes
-----------------------------

When the application already has OpenTelemetry configured, annotate the current
span from a signal listener:

.. code-block:: python

from opentelemetry import trace

from drf_api_logger import API_LOGGER_SIGNAL
from drf_api_logger.observability import annotate_opentelemetry_span

def annotate_current_span(**kwargs):
span = trace.get_current_span()
annotate_opentelemetry_span(span, kwargs)

API_LOGGER_SIGNAL.listen += annotate_current_span

By default, high-cardinality request and trace IDs are not added as span
attributes. If the application's trace policy allows those values, pass
``include_high_cardinality=True`` explicitly:

.. code-block:: python

annotate_opentelemetry_span(
span,
kwargs,
include_high_cardinality=True,
)

Sentry Error Context
--------------------

For Sentry SDK 2.x, enrich the current scope with safe tags and context:

.. code-block:: python

import sentry_sdk

from drf_api_logger import API_LOGGER_SIGNAL
from drf_api_logger.observability import configure_sentry_scope

def enrich_sentry_scope(**kwargs):
scope = sentry_sdk.Scope.get_current_scope()
configure_sentry_scope(scope, kwargs)

API_LOGGER_SIGNAL.listen += enrich_sentry_scope

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Move Sentry enrichment before error capture

When users wire the Sentry helper exactly as this recipe shows, it will miss the errors it is supposed to annotate: APILoggerMiddleware emits API_LOGGER_SIGNAL only after self.get_response(request) returns, and an exception raised by the view either has already been captured by Sentry or prevents the signal from being emitted at all. In those error paths the tags/context from configure_sentry_scope are absent, so the recipe should use Sentry's request scope, before_send, or middleware before the view is invoked instead of the logger signal.

Useful? React with 👍 / 👎.


Sentry tags receive only low-cardinality values. Sentry context can include
request IDs, trace IDs, and opaque IDs for debugging, but it never includes
request headers, request body, or response body.

Safety Rules
------------

- Keep high-cardinality values out of metrics labels.
- Keep payloads, headers, cookies, authorization values, and direct identities
out of observability exports.
- Keep exporter ownership in the application, not in DRF API Logger.
- Prefer route patterns and URL names over raw URLs.
- Use ``DRF_API_LOGGER_SKIP_URL_NAME`` or ``DRF_API_LOGGER_SKIP_NAMESPACE`` to
avoid recording health checks, metrics endpoints, admin paths, and noisy
internal endpoints.
18 changes: 18 additions & 0 deletions docs/operations.rst
Original file line number Diff line number Diff line change
Expand Up @@ -128,6 +128,24 @@ Recommended operating pattern:
inside the view should include the same request correlation metadata.
- Return only opaque IDs from ``DRF_API_LOGGER_CORRELATION_CONTEXT_FUNC``.

Observability Integration Operations
------------------------------------

DRF API Logger observability helpers are adapters for signal payloads. The
application remains responsible for exporter setup, scrape endpoints, collector
configuration, Sentry initialization, and access control.

Recommended operating pattern:

- Expose Prometheus metrics from the application using its existing metrics
endpoint.
- Use only low-cardinality labels from ``low_cardinality`` plus HTTP method.
- Add trace or request IDs to logs and spans only when the observability policy
permits high-cardinality attributes.
- Keep request and response payloads out of metrics, traces, and Sentry context.
- Skip health checks and metrics endpoints with ``DRF_API_LOGGER_SKIP_URL_NAME``
or ``DRF_API_LOGGER_SKIP_NAMESPACE``.

Database Indexes
----------------

Expand Down
26 changes: 26 additions & 0 deletions docs/quickstart.rst
Original file line number Diff line number Diff line change
Expand Up @@ -170,6 +170,32 @@ For extra context, return only opaque IDs from an allowlisted callback:
"client_id": getattr(request, "client_id", None),
}

Safe Observability Integrations
-------------------------------

Use the observability helpers from signal listeners when your application
already owns Prometheus, OpenTelemetry, or Sentry setup:

.. code-block:: python

from drf_api_logger import API_LOGGER_SIGNAL
from drf_api_logger.observability import (
annotate_opentelemetry_span,
configure_sentry_scope,
record_prometheus_metrics,
)

def export_observability(**kwargs):
record_prometheus_metrics(kwargs, API_REQUESTS, API_DURATION)
annotate_opentelemetry_span(current_span, kwargs)
configure_sentry_scope(sentry_scope, kwargs)

API_LOGGER_SIGNAL.listen += export_observability

Prometheus labels are limited to route, URL name, app name, namespace, status
class, and method. Request IDs and trace IDs are available for logs, traces, and
Sentry context, not metrics labels.

Retention and Pruning
---------------------

Expand Down
Loading
Loading