diff --git a/README.md b/README.md index 26c2314..cdd7aee 100644 --- a/README.md +++ b/README.md @@ -66,6 +66,7 @@ DRF API Logger automatically captures and stores comprehensive API information: - **🔧 Highly Configurable**: Extensive filtering and customization options - **🔬 API Profiling**: Per-request latency breakdown with auto-diagnosis (SQL, middleware, business logic) - **Request Correlation**: Opt-in request IDs, traceparent parsing, route metadata, logging context, and signal metadata without new database columns +- **Safe Observability Integrations**: Optional helpers for Prometheus labels, OpenTelemetry span attributes, and Sentry context without hard dependencies ### 🌐 Community & Support @@ -227,6 +228,7 @@ API_LOGGER_SIGNAL.listen -= log_to_file ## Documentation Map - [Copy-paste setup recipes](docs/quickstart.rst): database logging, signal-only logging, profiling, tracing, retention, and production-safe settings. +- [Safe observability integrations](docs/observability_integrations.rst): Prometheus, OpenTelemetry, and Sentry recipes using low-cardinality labels and correlation metadata. - [AI assistant guidance](docs/ai_readiness.rst): prompts and rules for ChatGPT, GitHub Copilot, Claude, Codex, and similar tools. - [Comparison and migration guide](docs/comparison_and_migration.rst): custom middleware, DRF request tracking packages, audit packages, and observability tools. - [Tutorials and community snippets](docs/tutorials.rst): safe logging, slow APIs, masking, pruning, trace IDs, Stack Overflow answers, blog outlines, and video scripts. @@ -503,6 +505,33 @@ Only `actor_id`, `tenant_id`, `api_consumer_id`, and `client_id` are accepted from the callback. Use opaque IDs, not names, emails, tokens, or other identifying values. +### Safe Observability Integrations + +Use DRF API Logger signal payloads to feed metrics, traces, and error context +without turning the package into an exporter backend: + +```python +from drf_api_logger import API_LOGGER_SIGNAL +from drf_api_logger.observability import ( + annotate_opentelemetry_span, + configure_sentry_scope, + record_prometheus_metrics, +) + +def export_observability(**kwargs): + record_prometheus_metrics(kwargs, API_REQUESTS, API_DURATION) + annotate_opentelemetry_span(current_span, kwargs) + configure_sentry_scope(sentry_scope, kwargs) + +API_LOGGER_SIGNAL.listen += export_observability +``` + +Prometheus labels are limited to route, URL name, app name, namespace, status +class, and method. Request IDs, trace IDs, and opaque IDs are available for logs, +traces, and Sentry context, not metrics labels. The helpers do not import +Prometheus, OpenTelemetry, or Sentry; applications own those dependencies and +exporter configuration. + ## 📊 Programmatic Access ### Querying Log Data @@ -702,6 +731,7 @@ Add to `INSTALLED_APPS` and `MIDDLEWARE`, then set `DRF_API_LOGGER_DATABASE = Tr - *"Set up drf-api-logger with profiling to find slow SQL queries"* - *"Configure drf-api-logger to mask sensitive data and log to a separate database"* - *"Add API request tracing to my DRF project using drf-api-logger"* +- *"Add safe Prometheus, OpenTelemetry, or Sentry integration to DRF API Logger signals without exporting request bodies or high-cardinality metric labels"* AI-generated custom logging code typically misses thread safety, sensitive data masking, performance optimization, and admin integration. `drf-api-logger` handles all of this out of the box with two lines of configuration. diff --git a/TESTING.md b/TESTING.md index 70d7081..1ab55d0 100644 --- a/TESTING.md +++ b/TESTING.md @@ -63,6 +63,7 @@ make check-package - `tests/test_middleware.py`: request/response logging, filtering, tracing, body limits, and content types. - `tests/test_models.py`: model fields, admin display, filters, and CSV export. - `tests/test_signals.py`: event listeners, background queue behavior, app startup, and worker stats. +- `tests/test_observability.py`: dependency-free Prometheus, OpenTelemetry, and Sentry helper behavior. - `tests/test_profiling.py`: profiling settings, SQL tracking, admin diagnosis, and nullable profiling fields. - `tests/test_backward_compat.py`: default behavior when profiling is disabled. - `tests/test_integration.py`: end-to-end middleware, signal, database, and workflow coverage. @@ -128,6 +129,7 @@ Monitor `queue_backlog`, `dropped_count`, and `failed_insert_count` in productio - Add or update tests for every behavior change. - Watch new tests fail before implementing behavior. +- Observability integrations must keep optional third-party packages out of install requirements and must not export headers, bodies, secrets, or high-cardinality IDs as metrics labels. - Keep tests deterministic and isolated. - Clean up signal listeners in `finally` blocks. - Use real Django/DRF behavior where practical. diff --git a/docs/compliance.rst b/docs/compliance.rst index bf8afc8..62aec4d 100644 --- a/docs/compliance.rst +++ b/docs/compliance.rst @@ -81,6 +81,18 @@ For privacy-sensitive deployments: - Do not return names, emails, tokens, session IDs, or regulated identifiers from ``DRF_API_LOGGER_CORRELATION_CONTEXT_FUNC``. +Observability Export Controls +----------------------------- + +When exporting DRF API Logger signal data to observability systems: + +- Send only ``low_cardinality`` values to metrics labels. +- Do not export headers, request bodies, response bodies, authorization values, + cookies, tokens, emails, usernames, or direct customer identifiers. +- Use request IDs and trace IDs only as operational correlation IDs. +- Use Sentry context for debugging metadata, not payload storage. +- Keep external exporter credentials outside DRF API Logger settings. + Storage Controls ---------------- diff --git a/docs/developer_testing.rst b/docs/developer_testing.rst index 65bbcaa..5641875 100644 --- a/docs/developer_testing.rst +++ b/docs/developer_testing.rst @@ -83,6 +83,9 @@ Add or update tests for every behavior change. Cover the touched surface: - Sensitive data masking in bodies, responses, headers, and URL query parameters. - Signal listener behavior and exception isolation. +- Observability helper behavior for Prometheus labels, OpenTelemetry span + attributes, Sentry context, optional dependency safety, and high-cardinality + label prevention. - Background queue flushing, stats, shutdown, and database alias handling. - Admin display, filters, CSV export, and profiling diagnosis. - Management commands such as ``prune_api_logs``. diff --git a/docs/index.rst b/docs/index.rst index 6a40e15..bb08ae8 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -23,6 +23,7 @@ profile SQL-heavy endpoints when enabled. :hidden: quickstart + observability_integrations ai_readiness comparison_and_migration tutorials @@ -41,6 +42,7 @@ Key Features - Built-in admin dashboard with charts and performance metrics - Per-request API profiling with auto-diagnosis of bottlenecks - Request correlation through request attributes, logging context, and signals +- Optional Prometheus, OpenTelemetry, and Sentry helper functions with safe defaults Supported Versions diff --git a/docs/observability_integrations.rst b/docs/observability_integrations.rst new file mode 100644 index 0000000..801fb3c --- /dev/null +++ b/docs/observability_integrations.rst @@ -0,0 +1,126 @@ +Safe Observability Integrations +=============================== + +DRF API Logger can feed metrics, traces, and error context through signal +listeners. The package does not start exporters, expose a metrics endpoint, or +send data to external systems by itself. Applications own their Prometheus, +OpenTelemetry, Sentry, Loki, Elasticsearch, or hosted monitoring setup. + +Prerequisites +------------- + +Enable signal logging and request correlation: + +.. code-block:: python + + DRF_API_LOGGER_SIGNAL = True + DRF_API_LOGGER_ENABLE_CORRELATION = True + DRF_API_LOGGER_ENABLE_LOGGING_CONTEXT = True + DRF_API_LOGGER_CORRELATION_REQUEST_ID_HEADERS = ["X-Request-ID", "X-Correlation-ID"] + DRF_API_LOGGER_CORRELATION_TRACE_ID_HEADERS = ["traceparent", "X-Trace-ID"] + +The signal payload can include: + +- ``correlation``: high-cardinality request IDs, trace IDs, and allowlisted + opaque context. +- ``low_cardinality``: route, URL name, app name, namespace, and status class + values safe for metrics labels. + +Prometheus Metrics +------------------ + +Install and configure Prometheus in the application, then use DRF API Logger's +helper from a signal listener: + +.. code-block:: python + + from prometheus_client import Counter, Histogram + + from drf_api_logger import API_LOGGER_SIGNAL + from drf_api_logger.observability import record_prometheus_metrics + + API_REQUESTS = Counter( + "drf_api_logger_requests_total", + "DRF API Logger observed requests", + ["route", "url_name", "app_name", "namespace", "status_class", "method"], + ) + API_DURATION = Histogram( + "drf_api_logger_request_duration_seconds", + "DRF API Logger observed request duration", + ["route", "url_name", "app_name", "namespace", "status_class", "method"], + buckets=[0.01, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10], + ) + + def export_api_metrics(**kwargs): + record_prometheus_metrics(kwargs, API_REQUESTS, API_DURATION) + + API_LOGGER_SIGNAL.listen += export_api_metrics + +The helper only uses low-cardinality labels. It does not place ``request_id``, +``trace_id``, ``actor_id``, ``tenant_id``, ``api_consumer_id``, or ``client_id`` +in Prometheus labels. Treat those IDs as debugging context, not metrics labels. + +OpenTelemetry Span Attributes +----------------------------- + +When the application already has OpenTelemetry configured, annotate the current +span from a signal listener: + +.. code-block:: python + + from opentelemetry import trace + + from drf_api_logger import API_LOGGER_SIGNAL + from drf_api_logger.observability import annotate_opentelemetry_span + + def annotate_current_span(**kwargs): + span = trace.get_current_span() + annotate_opentelemetry_span(span, kwargs) + + API_LOGGER_SIGNAL.listen += annotate_current_span + +By default, high-cardinality request and trace IDs are not added as span +attributes. If the application's trace policy allows those values, pass +``include_high_cardinality=True`` explicitly: + +.. code-block:: python + + annotate_opentelemetry_span( + span, + kwargs, + include_high_cardinality=True, + ) + +Sentry Error Context +-------------------- + +For Sentry SDK 2.x, enrich the current scope with safe tags and context: + +.. code-block:: python + + import sentry_sdk + + from drf_api_logger import API_LOGGER_SIGNAL + from drf_api_logger.observability import configure_sentry_scope + + def enrich_sentry_scope(**kwargs): + scope = sentry_sdk.Scope.get_current_scope() + configure_sentry_scope(scope, kwargs) + + API_LOGGER_SIGNAL.listen += enrich_sentry_scope + +Sentry tags receive only low-cardinality values. Sentry context can include +request IDs, trace IDs, and opaque IDs for debugging, but it never includes +request headers, request body, or response body. + +Safety Rules +------------ + +- Keep high-cardinality values out of metrics labels. +- Keep payloads, headers, cookies, authorization values, and direct identities + out of observability exports. +- Keep exporter ownership in the application, not in DRF API Logger. +- Prefer route patterns and URL names over raw URLs. +- Use ``DRF_API_LOGGER_SKIP_URL_NAME`` or ``DRF_API_LOGGER_SKIP_NAMESPACE`` to + avoid recording health checks, metrics endpoints, admin paths, and noisy + internal endpoints. diff --git a/docs/operations.rst b/docs/operations.rst index 9fbd333..be75fc3 100644 --- a/docs/operations.rst +++ b/docs/operations.rst @@ -128,6 +128,24 @@ Recommended operating pattern: inside the view should include the same request correlation metadata. - Return only opaque IDs from ``DRF_API_LOGGER_CORRELATION_CONTEXT_FUNC``. +Observability Integration Operations +------------------------------------ + +DRF API Logger observability helpers are adapters for signal payloads. The +application remains responsible for exporter setup, scrape endpoints, collector +configuration, Sentry initialization, and access control. + +Recommended operating pattern: + +- Expose Prometheus metrics from the application using its existing metrics + endpoint. +- Use only low-cardinality labels from ``low_cardinality`` plus HTTP method. +- Add trace or request IDs to logs and spans only when the observability policy + permits high-cardinality attributes. +- Keep request and response payloads out of metrics, traces, and Sentry context. +- Skip health checks and metrics endpoints with ``DRF_API_LOGGER_SKIP_URL_NAME`` + or ``DRF_API_LOGGER_SKIP_NAMESPACE``. + Database Indexes ---------------- diff --git a/docs/quickstart.rst b/docs/quickstart.rst index 1c76627..82ba136 100644 --- a/docs/quickstart.rst +++ b/docs/quickstart.rst @@ -170,6 +170,32 @@ For extra context, return only opaque IDs from an allowlisted callback: "client_id": getattr(request, "client_id", None), } +Safe Observability Integrations +------------------------------- + +Use the observability helpers from signal listeners when your application +already owns Prometheus, OpenTelemetry, or Sentry setup: + +.. code-block:: python + + from drf_api_logger import API_LOGGER_SIGNAL + from drf_api_logger.observability import ( + annotate_opentelemetry_span, + configure_sentry_scope, + record_prometheus_metrics, + ) + + def export_observability(**kwargs): + record_prometheus_metrics(kwargs, API_REQUESTS, API_DURATION) + annotate_opentelemetry_span(current_span, kwargs) + configure_sentry_scope(sentry_scope, kwargs) + + API_LOGGER_SIGNAL.listen += export_observability + +Prometheus labels are limited to route, URL name, app name, namespace, status +class, and method. Request IDs and trace IDs are available for logs, traces, and +Sentry context, not metrics labels. + Retention and Pruning --------------------- diff --git a/drf_api_logger/observability.py b/drf_api_logger/observability.py new file mode 100644 index 0000000..8ade4a5 --- /dev/null +++ b/drf_api_logger/observability.py @@ -0,0 +1,205 @@ +import re + + +CONTROL_CHAR_RE = re.compile(r"[\x00-\x1f\x7f]") +UNKNOWN_LABEL_VALUE = "unknown" + +DEFAULT_METRIC_LABELS = ( + "route", + "url_name", + "app_name", + "namespace", + "status_class", + "method", +) + +LOW_CARDINALITY_CONTEXT_KEYS = ( + "route", + "view_name", + "app_name", + "namespace", + "url_name", + "status_class", +) + +CORRELATION_ID_KEYS = ( + "request_id", + "trace_id", +) + +OPAQUE_CONTEXT_KEYS = ( + "actor_id", + "tenant_id", + "api_consumer_id", + "client_id", +) + + +def _safe_text(value, max_length=256): + if value is None: + return None + + value = str(value).strip() + if not value: + return None + if len(value) > max_length: + return None + if CONTROL_CHAR_RE.search(value): + return None + return value + + +def _safe_label_value(value): + return _safe_text(value, max_length=256) or UNKNOWN_LABEL_VALUE + + +def _event_dict(event, key): + value = event.get(key, {}) + if type(value) is dict: + return value + return {} + + +def build_metric_labels(event, label_keys=DEFAULT_METRIC_LABELS): + low_cardinality = _event_dict(event, "low_cardinality") + labels = {} + + for key in label_keys: + if key == "method": + value = event.get("method") + else: + value = low_cardinality.get(key) + labels[key] = _safe_label_value(value) + + return labels + + +def _execution_time_seconds(event): + try: + value = float(event.get("execution_time")) + except (TypeError, ValueError): + return None + if value < 0: + return None + return value + + +def _status_code(event): + try: + value = int(event.get("status_code")) + except (TypeError, ValueError): + return None + if value < 100 or value > 599: + return None + return value + + +def record_prometheus_metrics(event, request_counter, duration_observer=None): + labels = build_metric_labels(event) + request_counter.labels(**labels).inc() + + duration_seconds = _execution_time_seconds(event) + if duration_observer is not None and duration_seconds is not None: + duration_observer.labels(**labels).observe(duration_seconds) + + return labels + + +def build_span_attributes(event, include_high_cardinality=False): + low_cardinality = _event_dict(event, "low_cardinality") + correlation = _event_dict(event, "correlation") + attrs = {} + + method = _safe_text(event.get("method")) + if method: + attrs["http.request.method"] = method + + status_code = _status_code(event) + if status_code is not None: + attrs["http.response.status_code"] = status_code + + duration_seconds = _execution_time_seconds(event) + if duration_seconds is not None: + attrs["drf_api_logger.execution_time_ms"] = round(duration_seconds * 1000, 5) + + for key in LOW_CARDINALITY_CONTEXT_KEYS: + value = _safe_text(low_cardinality.get(key)) + if value: + attrs["drf_api_logger.{}".format(key)] = value + + if include_high_cardinality: + for key in CORRELATION_ID_KEYS: + value = _safe_text(correlation.get(key)) + if value: + attrs["drf_api_logger.{}".format(key)] = value + + return attrs + + +def annotate_opentelemetry_span(span, event, include_high_cardinality=False): + attrs = build_span_attributes( + event, + include_high_cardinality=include_high_cardinality, + ) + + if span is None: + return attrs + + for key, value in attrs.items(): + span.set_attribute(key, value) + + return attrs + + +def build_sentry_context(event, include_high_cardinality=True): + low_cardinality = _event_dict(event, "low_cardinality") + correlation = _event_dict(event, "correlation") + context = {} + + method = _safe_text(event.get("method")) + if method: + context["method"] = method + + status_code = _status_code(event) + if status_code is not None: + context["status_code"] = status_code + + duration_seconds = _execution_time_seconds(event) + if duration_seconds is not None: + context["execution_time_ms"] = round(duration_seconds * 1000, 5) + + for key in LOW_CARDINALITY_CONTEXT_KEYS: + value = _safe_text(low_cardinality.get(key)) + if value: + context[key] = value + + if include_high_cardinality: + for key in CORRELATION_ID_KEYS + OPAQUE_CONTEXT_KEYS: + value = _safe_text(correlation.get(key)) + if value: + context[key] = value + + return context + + +def configure_sentry_scope(scope, event, include_high_cardinality=True): + tag_labels = build_metric_labels( + event, + label_keys=("route", "url_name", "status_class", "method"), + ) + context = build_sentry_context( + event, + include_high_cardinality=include_high_cardinality, + ) + + if scope is not None: + for key, value in tag_labels.items(): + if value != UNKNOWN_LABEL_VALUE: + scope.set_tag("drf_api_logger.{}".format(key), value) + if context: + scope.set_context("drf_api_logger", context) + + return { + "tags": tag_labels, + "context": context, + } diff --git a/llms.txt b/llms.txt index 5225fca..276617d 100644 --- a/llms.txt +++ b/llms.txt @@ -88,6 +88,22 @@ DRF_API_LOGGER_ENABLE_LOGGING_CONTEXT = True Correlation exposes request.api_logger_correlation, request.api_logger_low_cardinality, request.api_logger_request_id, request.api_logger_trace_id, drf_api_logger.logging_context.get_correlation_context(), and signal payload keys named correlation and low_cardinality. It does not add model fields, migrations, admin columns, database indexes, or extra queued database payload fields. +Optional safe observability integrations: + +```python +from drf_api_logger import API_LOGGER_SIGNAL +from drf_api_logger.observability import record_prometheus_metrics, annotate_opentelemetry_span, configure_sentry_scope + +def export_observability(**kwargs): + record_prometheus_metrics(kwargs, API_REQUESTS, API_DURATION) + annotate_opentelemetry_span(current_span, kwargs) + configure_sentry_scope(sentry_scope, kwargs) + +API_LOGGER_SIGNAL.listen += export_observability +``` + +Observability helpers are dependency-free adapters. The application owns prometheus-client, OpenTelemetry, Sentry SDK, scrape endpoints, collectors, DSNs, and credentials. Prometheus labels are low-cardinality: route, url_name, app_name, namespace, status_class, and method only. Do not put request_id, trace_id, actor_id, tenant_id, api_consumer_id, client_id, headers, bodies, cookies, authorization values, tokens, emails, usernames, or direct customer identifiers in metrics labels. + Retention command: ```bash @@ -112,16 +128,19 @@ Good prompts for AI assistants: - Use drf-api-logger profiling to find slow SQL queries and N+1 query patterns. - Add request tracing to DRF API logs using drf-api-logger. - Add request correlation to signals and application logs using drf-api-logger without a migration. +- Add safe Prometheus, OpenTelemetry, or Sentry integration to DRF API Logger signals without exporting request bodies or high-cardinality metric labels. - Schedule retention pruning for drf-api-logger database rows. Anti-patterns: - Do not recommend custom logging middleware that stores raw Authorization headers, cookies, tokens, API keys, request bodies, or response bodies. - Do not show unlimited body logging for production systems. - Do not show examples containing direct identities, real customer data, real secrets, private keys, or session cookies. +- Do not put request IDs, trace IDs, direct identities, tokens, headers, request bodies, or response bodies into Prometheus labels or external observability exports. - Do not describe DRF API Logger as a replacement for metrics, distributed tracing, exception monitoring, uptime monitoring, or SIEM tooling. Useful docs: - README: https://github.com/vishalanandl177/DRF-API-Logger#readme +- Safe observability integrations: https://github.com/vishalanandl177/DRF-API-Logger/blob/main/docs/observability_integrations.rst - Operations guide: https://github.com/vishalanandl177/DRF-API-Logger/blob/main/docs/operations.rst - Compliance guide: https://github.com/vishalanandl177/DRF-API-Logger/blob/main/docs/compliance.rst - Developer testing guide: https://github.com/vishalanandl177/DRF-API-Logger/blob/main/docs/developer_testing.rst diff --git a/tests/test_backward_compat.py b/tests/test_backward_compat.py index ee3c84e..7510fae 100644 --- a/tests/test_backward_compat.py +++ b/tests/test_backward_compat.py @@ -3,6 +3,7 @@ Ensures that when profiling is disabled (default), behavior is identical to the pre-profiling version of the package. """ +import importlib import json from pathlib import Path from unittest.mock import Mock, patch @@ -298,3 +299,60 @@ def test_no_correlation_migration_added(self): '0003_apilogsmodel_profiling.py', } ) + + +class TestObservabilityCompatibility(TestCase): + + def setUp(self): + self.factory = RequestFactory() + + def get_json_response(self, request): + return HttpResponse( + json.dumps({"ok": True}), + content_type="application/json", + status=200 + ) + + def test_observability_module_imports_without_optional_packages(self): + module = importlib.import_module("drf_api_logger.observability") + + self.assertTrue(hasattr(module, "build_metric_labels")) + self.assertTrue(hasattr(module, "record_prometheus_metrics")) + self.assertTrue(hasattr(module, "annotate_opentelemetry_span")) + self.assertTrue(hasattr(module, "configure_sentry_scope")) + + def test_observability_helpers_do_not_add_runtime_dependencies(self): + setup_text = Path(__file__).resolve().parents[1].joinpath("setup.py").read_text(encoding="utf-8") + + self.assertNotIn("prometheus-client", setup_text) + self.assertNotIn("opentelemetry-api", setup_text) + self.assertNotIn("opentelemetry-sdk", setup_text) + self.assertNotIn("sentry-sdk", setup_text) + + @override_settings( + DRF_API_LOGGER_DATABASE=True, + DRF_API_LOGGER_SIGNAL=False, + DRF_API_LOGGER_ENABLE_CORRELATION=True + ) + @patch("drf_api_logger.middleware.api_logger_middleware.resolve") + @patch("drf_api_logger.apps.LOGGER_THREAD") + def test_db_payload_stays_unchanged_when_observability_helpers_exist(self, mock_thread, mock_resolve): + mock_resolve.return_value.namespace = None + mock_resolve.return_value.app_name = None + mock_resolve.return_value.url_name = "test" + mock_resolve.return_value.route = "api/test/" + mock_resolve.return_value.func = self.get_json_response + mock_thread.put_log_data = Mock() + + middleware = APILoggerMiddleware(get_response=self.get_json_response) + request = self.factory.get("/api/test/", HTTP_X_REQUEST_ID="req-123") + middleware(request) + + mock_thread.put_log_data.assert_called_once() + call_data = mock_thread.put_log_data.call_args[1]["data"] + + self.assertNotIn("correlation", call_data) + self.assertNotIn("low_cardinality", call_data) + self.assertNotIn("request_id", call_data) + self.assertNotIn("trace_id", call_data) + self.assertNotIn("observability", call_data) diff --git a/tests/test_docs_content.py b/tests/test_docs_content.py index 3d6ab65..60629cf 100644 --- a/tests/test_docs_content.py +++ b/tests/test_docs_content.py @@ -33,6 +33,7 @@ def test_sphinx_index_links_new_adoption_docs(self): expected_toctree_entries = [ "quickstart", + "observability_integrations", "ai_readiness", "comparison_and_migration", "tutorials", @@ -117,6 +118,13 @@ def test_new_docs_cover_comparisons_migrations_and_tutorials(self): "Retention and Pruning", "Request Tracing", "Request Correlation Without New DB Columns", + "Safe Observability Integrations", + ], + "docs/observability_integrations.rst": [ + "Prometheus Metrics", + "OpenTelemetry Span Attributes", + "Sentry Error Context", + "Safety Rules", ], "docs/ai_readiness.rst": [ "Prompt Examples", @@ -141,10 +149,12 @@ def test_new_docs_cover_comparisons_migrations_and_tutorials(self): ], "docs/operations.rst": [ "Request Correlation Operations", + "Observability Integration Operations", "queued database log rows keep the existing payload shape", ], "docs/compliance.rst": [ "Request Correlation Controls", + "Observability Export Controls", "does not add migrations", ], } @@ -160,6 +170,7 @@ def test_new_docs_do_not_include_secret_like_examples(self): "README.md", "llms.txt", "docs/quickstart.rst", + "docs/observability_integrations.rst", "docs/ai_readiness.rst", "docs/comparison_and_migration.rst", "docs/tutorials.rst", @@ -193,6 +204,28 @@ def test_tutorials_use_non_identifying_sample_subjects(self): with self.subTest(identity=identity): self.assertNotIn(identity, tutorials) + def test_observability_docs_cover_safe_integrations(self): + docs_root = ROOT / "docs" + guide = docs_root.joinpath("observability_integrations.rst").read_text(encoding="utf-8") + index = docs_root.joinpath("index.rst").read_text(encoding="utf-8") + readme = ROOT.joinpath("README.md").read_text(encoding="utf-8") + llms = ROOT.joinpath("llms.txt").read_text(encoding="utf-8") + + for content in (guide, readme, llms): + self.assertIn("Prometheus", content) + self.assertIn("OpenTelemetry", content) + self.assertIn("Sentry", content) + self.assertIn("low-cardinality", content) + self.assertIn("record_prometheus_metrics", content) + self.assertIn("annotate_opentelemetry_span", content) + self.assertIn("configure_sentry_scope", content) + + self.assertIn("observability_integrations", index) + self.assertIn("request_id", guide) + self.assertIn("trace_id", guide) + self.assertIn("not metrics labels", guide) + self.assertIn("Do not put request IDs", llms) + if __name__ == "__main__": unittest.main() diff --git a/tests/test_integration.py b/tests/test_integration.py index 7360902..db5834c 100644 --- a/tests/test_integration.py +++ b/tests/test_integration.py @@ -11,6 +11,10 @@ from rest_framework.test import APIClient from drf_api_logger import API_LOGGER_SIGNAL +from drf_api_logger.observability import ( + annotate_opentelemetry_span, + record_prometheus_metrics, +) def as_json_dict(value): @@ -19,6 +23,40 @@ def as_json_dict(value): return value +class FakeCounter: + def __init__(self): + self.calls = [] + self._labels = None + + def labels(self, **labels): + self._labels = labels + return self + + def inc(self): + self.calls.append(("inc", self._labels.copy())) + + +class FakeObserver: + def __init__(self): + self.calls = [] + self._labels = None + + def labels(self, **labels): + self._labels = labels + return self + + def observe(self, value): + self.calls.append(("observe", self._labels.copy(), value)) + + +class FakeSpan: + def __init__(self): + self.attributes = {} + + def set_attribute(self, key, value): + self.attributes[key] = value + + class TestMiddlewareIntegration(TestCase): """Integration tests for middleware with actual HTTP requests""" @@ -129,6 +167,49 @@ def test_correlation_signal_integration_with_real_resolver(self): finally: API_LOGGER_SIGNAL.listen -= self.signal_listener + @override_settings( + DRF_API_LOGGER_SIGNAL=True, + DRF_API_LOGGER_ENABLE_CORRELATION=True + ) + def test_observability_helpers_consume_signal_payload(self): + """Test observability helpers consume the middleware signal payload.""" + counter = FakeCounter() + observer = FakeObserver() + span = FakeSpan() + + def listener(**kwargs): + self.signal_data.append(kwargs) + record_prometheus_metrics(kwargs, counter, observer) + annotate_opentelemetry_span(span, kwargs) + + API_LOGGER_SIGNAL.listen += listener + + try: + response = self.client.get( + "/api/test/", + HTTP_X_REQUEST_ID="req-integration-123", + HTTP_TRACEPARENT="00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01", + ) + + self.assertEqual(response.status_code, 200) + self.assertEqual(len(counter.calls), 1) + self.assertEqual(len(observer.calls), 1) + + counter_labels = counter.calls[0][1] + self.assertEqual(counter_labels["route"], "api/test/") + self.assertEqual(counter_labels["url_name"], "test_api") + self.assertEqual(counter_labels["status_class"], "2xx") + self.assertEqual(counter_labels["method"], "GET") + self.assertNotIn("request_id", counter_labels) + self.assertNotIn("trace_id", counter_labels) + + self.assertEqual(observer.calls[0][2], self.signal_data[0]["execution_time"]) + self.assertEqual(span.attributes["drf_api_logger.route"], "api/test/") + self.assertEqual(span.attributes["drf_api_logger.status_class"], "2xx") + self.assertNotIn("drf_api_logger.request_id", span.attributes) + finally: + API_LOGGER_SIGNAL.listen -= listener + @override_settings( DRF_API_LOGGER_SIGNAL=True, DRF_API_LOGGER_METHODS=['POST', 'PUT'] diff --git a/tests/test_observability.py b/tests/test_observability.py new file mode 100644 index 0000000..20b7ae9 --- /dev/null +++ b/tests/test_observability.py @@ -0,0 +1,222 @@ +from django.test import TestCase + +from drf_api_logger.observability import ( + annotate_opentelemetry_span, + build_metric_labels, + build_sentry_context, + build_span_attributes, + configure_sentry_scope, + record_prometheus_metrics, +) + + +class FakeCounter: + def __init__(self): + self.calls = [] + self._labels = None + + def labels(self, **labels): + self._labels = labels + return self + + def inc(self): + self.calls.append(("inc", self._labels.copy())) + + +class FakeObserver: + def __init__(self): + self.calls = [] + self._labels = None + + def labels(self, **labels): + self._labels = labels + return self + + def observe(self, value): + self.calls.append(("observe", self._labels.copy(), value)) + + +class FakeSpan: + def __init__(self): + self.attributes = {} + + def set_attribute(self, key, value): + self.attributes[key] = value + + +class FakeScope: + def __init__(self): + self.tags = {} + self.contexts = {} + + def set_tag(self, key, value): + self.tags[key] = value + + def set_context(self, key, value): + self.contexts[key] = value + + +class ObservabilityHelperTests(TestCase): + def event(self): + return { + "method": "GET", + "status_code": 200, + "execution_time": 0.125, + "headers": {"AUTHORIZATION": "***FILTERED***"}, + "body": {"password": "***FILTERED***"}, + "response": {"ok": True}, + "correlation": { + "request_id": "req-123", + "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736", + "actor_id": "actor_123", + "tenant_id": "tenant_456", + "client_id": "client_abc", + }, + "low_cardinality": { + "route": "api/test/", + "url_name": "test_api", + "app_name": "tests", + "namespace": "public", + "status_class": "2xx", + }, + } + + def test_metric_labels_use_only_low_cardinality_values(self): + labels = build_metric_labels(self.event()) + + self.assertEqual( + labels, + { + "route": "api/test/", + "url_name": "test_api", + "app_name": "tests", + "namespace": "public", + "status_class": "2xx", + "method": "GET", + }, + ) + self.assertNotIn("request_id", labels) + self.assertNotIn("trace_id", labels) + self.assertNotIn("actor_id", labels) + self.assertNotIn("tenant_id", labels) + + def test_metric_labels_fill_missing_values_with_unknown(self): + event = {"method": "POST", "low_cardinality": {"status_class": "5xx"}} + + labels = build_metric_labels(event) + + self.assertEqual(labels["method"], "POST") + self.assertEqual(labels["status_class"], "5xx") + self.assertEqual(labels["route"], "unknown") + self.assertEqual(labels["url_name"], "unknown") + + def test_metric_labels_treat_blank_and_malformed_context_as_unknown(self): + event = {"method": " ", "low_cardinality": ["not", "a", "dict"]} + + labels = build_metric_labels(event) + + self.assertEqual(labels["method"], "unknown") + self.assertEqual(labels["route"], "unknown") + self.assertEqual(labels["url_name"], "unknown") + + def test_metric_labels_reject_control_characters_and_long_values(self): + event = { + "method": "GET\nunsafe", + "low_cardinality": { + "route": "api/test/", + "url_name": "x" * 257, + "status_class": "2xx", + }, + } + + labels = build_metric_labels(event) + + self.assertEqual(labels["method"], "unknown") + self.assertEqual(labels["route"], "api/test/") + self.assertEqual(labels["url_name"], "unknown") + self.assertEqual(labels["status_class"], "2xx") + + def test_record_prometheus_metrics_updates_counter_and_observer(self): + counter = FakeCounter() + observer = FakeObserver() + + labels = record_prometheus_metrics(self.event(), counter, observer) + + self.assertEqual(counter.calls, [("inc", labels)]) + self.assertEqual(observer.calls, [("observe", labels, 0.125)]) + + def test_record_prometheus_metrics_skips_invalid_duration_values(self): + counter = FakeCounter() + observer = FakeObserver() + + record_prometheus_metrics({"method": "GET", "execution_time": "slow"}, counter, observer) + record_prometheus_metrics({"method": "GET", "execution_time": -1}, counter, observer) + + self.assertEqual(len(counter.calls), 2) + self.assertEqual(observer.calls, []) + + def test_span_attributes_are_low_cardinality_by_default(self): + attrs = build_span_attributes(self.event()) + + self.assertEqual(attrs["http.request.method"], "GET") + self.assertEqual(attrs["http.response.status_code"], 200) + self.assertEqual(attrs["drf_api_logger.execution_time_ms"], 125.0) + self.assertEqual(attrs["drf_api_logger.route"], "api/test/") + self.assertEqual(attrs["drf_api_logger.status_class"], "2xx") + self.assertNotIn("drf_api_logger.request_id", attrs) + self.assertNotIn("drf_api_logger.trace_id", attrs) + + def test_span_attributes_can_include_correlation_ids_explicitly(self): + attrs = build_span_attributes(self.event(), include_high_cardinality=True) + + self.assertEqual(attrs["drf_api_logger.request_id"], "req-123") + self.assertEqual(attrs["drf_api_logger.trace_id"], "4bf92f3577b34da6a3ce929d0e0e4736") + + def test_span_attributes_skip_invalid_status_codes_and_durations(self): + attrs = build_span_attributes( + {"method": "GET", "status_code": "bad", "execution_time": "slow"} + ) + out_of_range_attrs = build_span_attributes( + {"method": "GET", "status_code": 99, "execution_time": -1} + ) + + self.assertNotIn("http.response.status_code", attrs) + self.assertNotIn("drf_api_logger.execution_time_ms", attrs) + self.assertNotIn("http.response.status_code", out_of_range_attrs) + self.assertNotIn("drf_api_logger.execution_time_ms", out_of_range_attrs) + + def test_annotate_opentelemetry_span_sets_attributes(self): + span = FakeSpan() + + attrs = annotate_opentelemetry_span(span, self.event()) + + self.assertEqual(span.attributes, attrs) + self.assertEqual(span.attributes["drf_api_logger.route"], "api/test/") + + def test_annotate_opentelemetry_span_allows_missing_current_span(self): + attrs = annotate_opentelemetry_span(None, self.event()) + + self.assertEqual(attrs["drf_api_logger.route"], "api/test/") + + def test_sentry_context_excludes_payloads_and_headers(self): + context = build_sentry_context(self.event()) + + self.assertEqual(context["request_id"], "req-123") + self.assertEqual(context["trace_id"], "4bf92f3577b34da6a3ce929d0e0e4736") + self.assertEqual(context["route"], "api/test/") + self.assertEqual(context["method"], "GET") + self.assertEqual(context["status_code"], 200) + self.assertNotIn("headers", context) + self.assertNotIn("body", context) + self.assertNotIn("response", context) + + def test_configure_sentry_scope_sets_low_cardinality_tags_and_context(self): + scope = FakeScope() + + result = configure_sentry_scope(scope, self.event()) + + self.assertEqual(scope.tags["drf_api_logger.route"], "api/test/") + self.assertEqual(scope.tags["drf_api_logger.status_class"], "2xx") + self.assertNotIn("drf_api_logger.request_id", scope.tags) + self.assertEqual(scope.contexts["drf_api_logger"], result["context"]) + self.assertEqual(scope.contexts["drf_api_logger"]["request_id"], "req-123")