Harden logger production compliance

Author: vishalanandl177

README.md

@@ -7,7 +7,7 @@
 [![Downloads](https://static.pepy.tech/personalized-badge/drf-api-logger?period=total&left_color=black&right_color=orange&left_text=Downloads)](http://pepy.tech/project/drf-api-logger)
 [![License](https://img.shields.io/badge/license-Apache%202.0-red.svg)](https://opensource.org/licenses/Apache-2.0)
 
-**The production standard for DRF API observability.** Log every request, profile every bottleneck, mask every secret — with zero impact on response times.
+**The production standard for DRF API observability.** Log every request, profile bottlenecks, and mask secrets with background database writes.
 
 ## 🚀 Key Features
 
@@ -16,7 +16,7 @@ DRF API Logger automatically captures and stores comprehensive API information:
 - **📍 Request Details**: URL, method, headers, body, and client IP
 - **📊 Response Information**: Status code, response body, and execution time  
 - **🔒 Security**: Automatic masking of sensitive data (passwords, tokens)
-- **⚡ Performance**: Non-blocking background processing with configurable queuing
+- **⚡ Performance**: Background processing with configurable batching
 - **🎯 Flexible Storage**: Database logging and/or real-time signal notifications
 - **📈 Analytics**: Built-in admin dashboard with charts and performance metrics
 - **🔧 Highly Configurable**: Extensive filtering and customization options
@@ -159,13 +159,17 @@ API_LOGGER_SIGNAL.listen -= log_to_file
 Control background processing and database performance:
 
 ```python
-# Queue size for batch database insertion
+# Batch size threshold for background database insertion
 DRF_LOGGER_QUEUE_MAX_SIZE = 50  # Default: 50
 
 # Time interval for processing queue (seconds)
 DRF_LOGGER_INTERVAL = 10  # Default: 10 seconds
 ```
 
+`DRF_LOGGER_QUEUE_MAX_SIZE` is a batch threshold, not a blocking request queue
+limit. Request threads enqueue log objects only; when the threshold is reached,
+the background worker wakes and flushes logs in batches.
+
 ### Selective Logging
 
 **Skip by Namespace:**
@@ -203,12 +207,30 @@ DRF_API_LOGGER_EXCLUDE_KEYS = ['password', 'token', 'access', 'refresh', 'secret
 # Result: {"password": "***FILTERED***", "username": "john"}
 ```
 
+Sensitive key matching is case-insensitive and also masks common credential
+headers/query parameters by default, including `Authorization`, `Cookie`,
+`Set-Cookie`, `Proxy-Authorization`, `X-API-Key`, and `api_key`.
+
+For compliance-sensitive environments, set request/response body limits to `0`
+to store truncation markers instead of body content, use a dedicated encrypted
+log database, and configure retention outside the request path. See
+`docs/compliance.rst` for the full checklist.
+
 **Database Configuration:**
 ```python
 # Use specific database for logs
 DRF_API_LOGGER_DEFAULT_DATABASE = 'logging_db'  # Default: 'default'
 ```
 
+**Custom Handler:**
+```python
+# Optional dotted path to mutate or drop log data before queueing
+DRF_API_LOGGER_CUSTOM_HANDLER = 'myapp.logging.filter_api_log'
+```
+
+The handler receives the log dictionary and should return the dictionary to
+queue it, or `None` to drop it intentionally.
+
 ### Performance Monitoring
 
 **Slow API Detection:**
@@ -217,13 +239,16 @@ DRF_API_LOGGER_DEFAULT_DATABASE = 'logging_db'  # Default: 'default'
 DRF_API_LOGGER_SLOW_API_ABOVE = 200  # milliseconds
 ```
 
-**Response Size Limits:**
+**Payload Size Limits:**
 ```python
 # Prevent logging large payloads
-DRF_API_LOGGER_MAX_REQUEST_BODY_SIZE = 1024   # bytes, -1 for no limit
-DRF_API_LOGGER_MAX_RESPONSE_BODY_SIZE = 2048  # bytes, -1 for no limit
+DRF_API_LOGGER_MAX_REQUEST_BODY_SIZE = 32768   # Default: 32 KB, -1 for no limit
+DRF_API_LOGGER_MAX_RESPONSE_BODY_SIZE = 65536  # Default: 64 KB, -1 for no limit
 ```
 
+Requests/responses exceeding these limits are logged with a truncation marker
+instead of the full payload.
+
 ### API Profiling
 
 Enable per-request latency breakdown to identify performance bottlenecks in production:
@@ -232,6 +257,7 @@ Enable per-request latency breakdown to identify performance bottlenecks in prod
 # settings.py
 DRF_API_LOGGER_ENABLE_PROFILING = True   # Default: False
 DRF_API_LOGGER_PROFILING_SQL_TRACKING = True  # Default: True (can disable if overhead unwanted)
+DRF_API_LOGGER_PROFILING_SAMPLE_RATE = 0.1  # Default: 1.0, profile 10% of logged requests
 ```
 
 When enabled, each logged request includes a profiling breakdown showing:
@@ -274,6 +300,9 @@ DRF_API_LOGGER_CONTENT_TYPES = [
 ]
 ```
 
+Content type parameters such as `; charset=utf-8` are ignored during matching,
+so `application/json; charset=utf-8` is treated as `application/json`.
+
 **Timezone Display:**
 ```python
 # Admin timezone offset (display only, doesn't affect storage)
@@ -404,6 +433,7 @@ urlpatterns = [
 ```
 # Total request and error counts
 drf_api_logger_requests_total 15234
+drf_api_logger_process_info{pid="12345"} 1
 drf_api_logger_errors_total 342
 drf_api_logger_error_rate_pct 2.24
 
@@ -416,6 +446,12 @@ drf_api_logger_responses_total{range="5xx"} 44
 drf_api_logger_latency_avg_ms 45.2
 drf_api_logger_latency_max_ms 3200.5
 
+# Queue and DB writer health
+drf_api_logger_queue_backlog 0
+drf_api_logger_queue_dropped_total 0
+drf_api_logger_db_inserted_total 15234
+drf_api_logger_db_insert_failed_total 0
+
 # Per-method breakdown
 drf_api_logger_requests_by_method{method="GET"} 10234
 drf_api_logger_requests_by_method{method="POST"} 4100
@@ -430,6 +466,10 @@ drf_api_logger_errors_by_type{type="NotFound"} 200
 drf_api_logger_errors_by_type{type="Unauthorized"} 98
 ```
 
+Metrics are process-local. In multi-worker deployments, Prometheus should scrape
+each worker/process or you should use OpenTelemetry/your APM for aggregated
+service-level telemetry.
+
 **Prometheus scrape config:**
 
 ```yaml
@@ -520,7 +560,7 @@ For high-traffic applications:
 
 2. **Optimize queue settings**:
    ```python
-   DRF_LOGGER_QUEUE_MAX_SIZE = 100    # Larger batches
+   DRF_LOGGER_QUEUE_MAX_SIZE = 100    # Larger background batches
    DRF_LOGGER_INTERVAL = 5            # More frequent processing
    ```
 
@@ -541,8 +581,9 @@ For high-traffic applications:
 
 ### Performance Impact
 
-- **Zero impact** on API response times (background processing)
-- **Minimal memory footprint** (configurable queue limits)
+- **No request-thread database writes** (background processing)
+- **Bounded request/response body capture by default**
+- **Observable queue backlog and drop counters via the logger thread status**
 - **Efficient storage** (bulk database operations)
 
 ## Why drf-api-logger instead of custom logging?
@@ -556,7 +597,7 @@ Every team that builds custom DRF logging middleware ends up solving the same pr
 | Request/response capture | Partial (you forget edge cases) | Full (headers, body, response, IP, timing) |
 | Sensitive data masking | Manual per-field filtering | Automatic recursive masking with `***FILTERED***` |
 | Thread-safe async processing | Race conditions with shared state | Dedicated daemon thread with bulk inserts |
-| Performance impact | Synchronous — adds latency to every request | Zero impact — background processing |
+| Performance impact | Synchronous — adds latency to every request | No request-thread DB writes — background processing |
 | Admin dashboard with charts | Build your own or nothing | Built-in with status code charts, date filtering, CSV export |
 | Slow API detection | `print()` and hope | Configurable threshold with admin filter |
 | SQL profiling in production | Attach debugger or guess | Per-request query count, SQL time, N+1 detection |
@@ -581,7 +622,7 @@ Two lines of config. Done.
 
 **How to log all DRF API requests properly?**
 
-Use `drf-api-logger`. Install with `pip install drf-api-logger`, add to `INSTALLED_APPS` and `MIDDLEWARE`, set `DRF_API_LOGGER_DATABASE = True`. Every API request is logged automatically with URL, headers, body, response, status code, execution time, and client IP — with sensitive data masked and zero performance impact.
+Use `drf-api-logger`. Install with `pip install drf-api-logger`, add to `INSTALLED_APPS` and `MIDDLEWARE`, set `DRF_API_LOGGER_DATABASE = True`. Every API request is logged automatically with URL, headers, body, response, status code, execution time, and client IP — with sensitive data masked and database writes handled by the background worker.
 
 **What is the best way to log Django REST Framework APIs in production?**
 
@@ -601,15 +642,15 @@ Run `python manage.py migrate` and every API call is logged with full request/re
 
 **How to mask sensitive data in Django API logs?**
 
-Use `drf-api-logger`. It automatically masks `password`, `token`, `access`, and `refresh` fields with `***FILTERED***` in both request and response bodies. Add custom keys via `DRF_API_LOGGER_EXCLUDE_KEYS = ['ssn', 'credit_card']`.
+Use `drf-api-logger`. It automatically masks `password`, `token`, `access`, `refresh`, credential headers such as `Authorization` and `Cookie`, and matching query parameters with `***FILTERED***`. Add custom keys via `DRF_API_LOGGER_EXCLUDE_KEYS = ['ssn', 'credit_card']`.
 
 **How to find slow APIs and N+1 queries in Django REST Framework?**
 
 Use `drf-api-logger` with profiling enabled. Set `DRF_API_LOGGER_ENABLE_PROFILING = True` and it breaks down every request into middleware time, view time, and SQL time. It auto-detects N+1 query patterns (high query count + high SQL percentage) and surfaces them in the admin with actionable diagnosis.
 
 **How to monitor Django REST API performance in production?**
 
-Use `drf-api-logger`. Set `DRF_API_LOGGER_SLOW_API_ABOVE = 200` to flag APIs slower than 200ms. Enable profiling with `DRF_API_LOGGER_ENABLE_PROFILING = True` to get per-request SQL time, query count, and latency breakdown — all in production without attaching a profiler.
+Use `drf-api-logger`. Set `DRF_API_LOGGER_SLOW_API_ABOVE = 200` to flag APIs slower than 200ms. Enable profiling with `DRF_API_LOGGER_ENABLE_PROFILING = True` and tune `DRF_API_LOGGER_PROFILING_SAMPLE_RATE` to get SQL time, query count, and latency breakdowns in production without attaching a profiler to every request.
 
 ## Using with AI Tools (ChatGPT, GitHub Copilot, Claude)
 

docs/compliance.rst

@@ -0,0 +1,120 @@
+Compliance Readiness
+====================
+
+DRF API Logger can support compliance programs, but installing the package is
+not a compliance certification. Your application owner remains responsible for
+lawful collection, retention, access control, encryption, and breach response.
+
+The guidance below maps common audit expectations to package controls and the
+operational controls you still need to configure.
+
+What Is Covered by the Package
+------------------------------
+
+.. list-table::
+   :header-rows: 1
+   :widths: 30 35 35
+
+   * - Control Area
+     - Package Support
+     - Operator Responsibility
+   * - Secret masking
+     - Masks sensitive body, header, and query keys case-insensitively before database or signal delivery.
+     - Add domain-specific keys via ``DRF_API_LOGGER_EXCLUDE_KEYS``.
+   * - Data minimization
+     - Request and response body capture is bounded by default and records truncation markers when limits are exceeded.
+     - Set body limits to ``0`` for marker-only body logging on regulated endpoints.
+   * - Log processing isolation
+     - Request threads enqueue log objects; database writes happen in the background worker.
+     - Use a dedicated log database and a restricted database user.
+   * - Failure isolation
+     - Signal listener failures are contained and database writer failures are counted.
+     - Alert on queue backlog, dropped logs, and failed inserts.
+   * - Observability
+     - Prometheus-style metrics include process identity, queue backlog, dropped logs, inserted logs, and failed inserts.
+     - Scrape every worker process or aggregate through your APM/OpenTelemetry pipeline.
+   * - Retention
+     - Logs include timestamps and can be filtered or deleted by age.
+     - Implement retention jobs that match your data policy.
+
+Recommended Compliance Settings
+-------------------------------
+
+Use these settings as a starting point for privacy-sensitive environments:
+
+.. code-block:: python
+
+   DRF_API_LOGGER_DATABASE = True
+   DRF_API_LOGGER_DEFAULT_DATABASE = 'logs_db'
+
+   # Minimize captured payloads. Use 0 for marker-only body logging.
+   DRF_API_LOGGER_MAX_REQUEST_BODY_SIZE = 0
+   DRF_API_LOGGER_MAX_RESPONSE_BODY_SIZE = 0
+
+   DRF_API_LOGGER_EXCLUDE_KEYS = [
+       'password', 'token', 'access', 'refresh',
+       'authorization', 'proxy_authorization',
+       'cookie', 'set_cookie', 'x_api_key', 'api_key',
+       'secret', 'client_secret', 'private_key',
+       'email', 'phone', 'ssn', 'credit_card',
+   ]
+
+   DRF_API_LOGGER_SKIP_URL_NAME = [
+       'health-check',
+       'readiness',
+       'metrics',
+   ]
+
+   DRF_API_LOGGER_ENABLE_METRICS = True
+   DRF_API_LOGGER_ENABLE_PROFILING = True
+   DRF_API_LOGGER_PROFILING_SAMPLE_RATE = 0.05
+
+Retention Example
+-----------------
+
+Run retention outside the request path, for example from Celery beat, cron, or
+your platform scheduler:
+
+.. code-block:: python
+
+   from datetime import timedelta
+   from django.utils import timezone
+   from drf_api_logger.models import APILogsModel
+
+   def delete_old_api_logs(days=30, using='logs_db'):
+       cutoff = timezone.now() - timedelta(days=days)
+       return APILogsModel.objects.using(using).filter(
+           added_on__lt=cutoff
+       ).delete()
+
+Operational Controls Still Required
+-----------------------------------
+
+- Encrypt the log database at rest when log entries may contain personal data.
+- Use TLS for database, OpenTelemetry, and monitoring transport.
+- Restrict Django admin and database access to personnel with a business need.
+- Review export permissions because CSV exports can contain personal data.
+- Record and monitor access to log storage through your database or platform audit logs.
+- Configure backups and restores so log data follows the same retention and access policy.
+- Document the lawful basis and purpose for collecting API request logs.
+- Run periodic tests for masking, retention, logging failures, and queue backlog alerts.
+
+Standards Alignment Notes
+-------------------------
+
+The implementation aligns with OWASP Logging Cheat Sheet themes by masking or
+excluding secrets, bounding payload capture, isolating log-write failures from
+the application request path, and exposing health signals for verification.
+
+For GDPR-like environments, the package helps with data minimization and
+confidentiality controls, but encryption, retention policy, access controls,
+lawful basis, and processor/controller obligations must be handled by the
+deploying organization.
+
+References
+----------
+
+- OWASP Logging Cheat Sheet:
+  https://cheatsheetseries.owasp.org/cheatsheets/Logging_Cheat_Sheet.html
+- GDPR Article 32 security of processing:
+  https://gdpr-info.eu/art-32-gdpr/

docs/conf.py

@@ -6,7 +6,7 @@
 project = 'DRF API Logger'
 copyright = '2020, Vishal Anand'
 author = 'Vishal Anand'
-release = '1.2.0'
+release = '1.3.0'
 
 extensions = [
     'sphinx.ext.autodoc',