Skip to content

Home

Jump to bottom
Vishal Anand edited this page May 14, 2026 · 3 revisions

DRF API Logger Wiki

DRF API Logger is a Django REST Framework observability package for capturing API request and response data, storing it safely, and using it to investigate production behavior.

Use this wiki as the operational guide. Keep the README short for installation and quick start; keep deeper production, performance, compliance, and troubleshooting notes here.

What It Does

DRF API Logger can capture:

  • API URL and path
  • HTTP method
  • Request headers
  • Request body
  • Response body
  • Status code
  • Client IP address
  • Execution time
  • Request timestamp
  • Optional tracing ID
  • Optional profiling data such as SQL time and query count

It supports two output modes:

  • Database logging through APILogsModel
  • Signal-based logging through API_LOGGER_SIGNAL

Both modes can be enabled together when needed.

Quick Start

Install the package:

pip install drf-api-logger

Add the app:

INSTALLED_APPS = [
    # ...
    "drf_api_logger",
]

Add the middleware:

MIDDLEWARE = [
    # ...
    "drf_api_logger.middleware.api_logger_middleware.APILoggerMiddleware",
]

Enable database logging:

DRF_API_LOGGER_DATABASE = True

Run migrations:

python manage.py migrate

After this, API requests are logged and can be viewed in Django Admin.

Recommended Production Setup

For production, start with this conservative configuration:

DRF_API_LOGGER_DATABASE = True
DRF_API_LOGGER_SIGNAL = False

DRF_LOGGER_QUEUE_MAX_SIZE = 100
DRF_LOGGER_INTERVAL = 5

DRF_API_LOGGER_MAX_REQUEST_BODY_SIZE = 32768
DRF_API_LOGGER_MAX_RESPONSE_BODY_SIZE = 65536

DRF_API_LOGGER_EXCLUDE_KEYS = [
    "password",
    "token",
    "access",
    "refresh",
    "secret",
    "api_key",
    "client_secret",
]

DRF_API_LOGGER_SKIP_URL_NAME = ["health-check", "metrics"]
DRF_API_LOGGER_METHODS = ["GET", "POST", "PUT", "PATCH", "DELETE"]
DRF_API_LOGGER_ENABLE_PROFILING = False

For high-traffic systems, keep profiling disabled by default and enable it temporarily or with sampling:

DRF_API_LOGGER_ENABLE_PROFILING = True
DRF_API_LOGGER_PROFILING_SAMPLE_RATE = 0.05

Core Settings

Setting Default Purpose
DRF_API_LOGGER_DATABASE False Enables database-backed log storage.
DRF_API_LOGGER_SIGNAL False Emits log records to custom signal listeners.
DRF_LOGGER_QUEUE_MAX_SIZE 50 Bulk insert batch size and queue wake-up threshold. Must be greater than 0.
DRF_LOGGER_INTERVAL 10 Background flush interval in seconds. Must be greater than 0.
DRF_API_LOGGER_DEFAULT_DATABASE "default" Database alias used for storing log rows.
DRF_API_LOGGER_SKIP_URL_NAME [] Skips logging for matching Django URL names.
DRF_API_LOGGER_SKIP_NAMESPACE [] Skips logging for matching URL namespaces.
DRF_API_LOGGER_METHODS [] Logs only listed HTTP methods. Empty means all methods.
DRF_API_LOGGER_STATUS_CODES [] Logs only listed response codes. Empty means all status codes.
DRF_API_LOGGER_EXCLUDE_KEYS built-in sensitive keys plus custom keys Adds extra keys to mask recursively.
DRF_API_LOGGER_MAX_REQUEST_BODY_SIZE 32768 Maximum request body bytes to store. Use -1 for no limit.
DRF_API_LOGGER_MAX_RESPONSE_BODY_SIZE 65536 Maximum response body bytes to store. Use -1 for no limit.
DRF_API_LOGGER_ENABLE_PROFILING False Enables per-request profiling data.
DRF_API_LOGGER_PROFILING_SQL_TRACKING True Captures SQL query count and SQL time when profiling is enabled.
DRF_API_LOGGER_PROFILING_SAMPLE_RATE 1.0 Fraction of logged requests to profile. Clamped between 0.0 and 1.0.
DRF_API_LOGGER_CONTENT_TYPES default set plus custom values Adds loggable response/request content types.
DRF_API_LOGGER_TIMEDELTA 0 Admin display offset in minutes. Storage is unchanged.
DRF_API_LOGGER_PATH_TYPE "ABSOLUTE" Controls stored URL format. Options: ABSOLUTE, FULL_PATH, RAW_URI.
DRF_API_LOGGER_ENABLE_TRACING False Adds a tracing ID to each request.
DRF_API_LOGGER_TRACING_FUNC unset Dotted path to a custom trace ID generator.
DRF_API_LOGGER_TRACING_ID_HEADER_NAME unset Header name to read an existing trace ID from.
DRF_API_LOGGER_CUSTOM_HANDLER unset Dotted path to a function that transforms or drops log records before queueing.

Database Logging

Database logging is the normal production path:

DRF_API_LOGGER_DATABASE = True

Request threads do not write logs directly to the database. They enqueue log records. A background daemon thread flushes records with bulk_create.

This keeps database writes out of the request path, but the application still pays for:

  • Reading request body
  • Reading response body when content type is loggable
  • JSON decode/serialization for supported payloads
  • Recursive sensitive-data masking
  • Queue insertion

The database worker tracks queue and insert statistics through LOGGER_THREAD.get_status().

Example diagnostic use:

from drf_api_logger import apps as logger_apps

def api_logger_status():
    if not logger_apps.LOGGER_THREAD:
        return {"enabled": False}
    return logger_apps.LOGGER_THREAD.get_status()

Watch these fields in production:

  • queue_backlog
  • inserted_count
  • dropped_count
  • failed_insert_count
  • batch_size
  • interval

Signal Logging

Signal logging is useful when logs should go somewhere other than the package database table:

DRF_API_LOGGER_SIGNAL = True

Example:

from drf_api_logger import API_LOGGER_SIGNAL

def send_to_observability_platform(**data):
    # Send data to your logging or analytics system.
    pass

API_LOGGER_SIGNAL.listen += send_to_observability_platform

Important production note: signal listeners run synchronously in the request/response flow. Keep signal listeners fast. Do not do slow network calls directly inside a listener unless you dispatch them to a queue.

Selective Logging

Use selective logging to reduce cost, noise, and risk.

Skip internal endpoints:

DRF_API_LOGGER_SKIP_URL_NAME = ["health-check", "metrics"]
DRF_API_LOGGER_SKIP_NAMESPACE = ["admin", "internal"]

Log only selected methods:

DRF_API_LOGGER_METHODS = ["POST", "PUT", "PATCH", "DELETE"]

Log only selected status codes:

DRF_API_LOGGER_STATUS_CODES = [400, 401, 403, 404, 429, 500]

Admin requests are automatically skipped. Static and media requests are also skipped.

Payload Handling

By default, the middleware stores JSON and selected safe content types. Large payloads are replaced with a truncation marker instead of being stored.

Defaults:

DRF_API_LOGGER_MAX_REQUEST_BODY_SIZE = 32768
DRF_API_LOGGER_MAX_RESPONSE_BODY_SIZE = 65536

Special response handling:

  • Streaming responses are stored as ** Streaming **
  • GZIP responses are stored as ** GZIP Archive **
  • Binary responses are stored as ** Binary File **
  • Calendar responses are stored as ** Calendar **

Add custom content types only when there is a clear reason:

DRF_API_LOGGER_CONTENT_TYPES = [
    "application/json",
    "application/vnd.api+json",
    "application/xml",
    "text/csv",
]

Avoid unlimited payload capture in production unless the API is low-volume and the data is safe:

DRF_API_LOGGER_MAX_REQUEST_BODY_SIZE = -1
DRF_API_LOGGER_MAX_RESPONSE_BODY_SIZE = -1

Sensitive Data Masking

Sensitive fields are replaced with:

***FILTERED***

Built-in sensitive keys include:

  • password
  • token
  • access
  • refresh
  • authorization
  • proxy_authorization
  • cookie
  • set_cookie
  • x_api_key
  • api_key
  • secret
  • client_secret
  • private_key
  • sessionid
  • csrfmiddlewaretoken

Matching is case-insensitive and treats hyphens and underscores equivalently.

Add project-specific sensitive keys:

DRF_API_LOGGER_EXCLUDE_KEYS = [
    "ssn",
    "credit_card",
    "patient_id",
    "customer_secret",
]

Masking applies recursively to dictionaries and lists. Query parameters in stored API URLs are also masked when the parameter name matches a sensitive key.

Custom Handler

Use a custom handler to transform, enrich, or drop records before database queueing:

DRF_API_LOGGER_CUSTOM_HANDLER = "myapp.logging.clean_api_log"

Example:

def clean_api_log(data):
    if data["api"].endswith("/health/"):
        return None
    data["headers"].pop("AUTHORIZATION", None)
    return data

Return None to intentionally drop a log entry.

This is the best place for project-specific compliance filtering.

Profiling

Profiling adds timing breakdowns to logged requests:

DRF_API_LOGGER_ENABLE_PROFILING = True
DRF_API_LOGGER_PROFILING_SQL_TRACKING = True
DRF_API_LOGGER_PROFILING_SAMPLE_RATE = 1.0

Captured profiling areas:

  • Middleware work before the view
  • View and serialization time
  • Middleware work after the view
  • SQL total time
  • SQL query count

Use profiling to identify:

  • N+1 query patterns
  • Few but slow SQL queries
  • Business logic or external API slowness
  • Middleware overhead

Production guidance:

  • Keep profiling disabled by default on very high traffic systems.
  • Use DRF_API_LOGGER_PROFILING_SAMPLE_RATE to profile a small fraction of requests.
  • Disable SQL tracking if the query instrumentation overhead is not acceptable:
DRF_API_LOGGER_PROFILING_SQL_TRACKING = False

Request Tracing

Enable tracing:

DRF_API_LOGGER_ENABLE_TRACING = True

Use an existing inbound trace header:

DRF_API_LOGGER_TRACING_ID_HEADER_NAME = "X-Trace-ID"

Or provide a custom generator:

DRF_API_LOGGER_TRACING_FUNC = "myapp.tracing.generate_trace_id"

The middleware also attaches the trace ID to the request:

def view(request):
    trace_id = getattr(request, "tracing_id", None)

Admin Dashboard

With database logging enabled, Django Admin can be used to:

  • Search logs by API, body, response, and headers
  • Filter by method, status code, date, and performance
  • Inspect request and response data
  • Review slow APIs
  • Review profiling details when enabled
  • Export or query log data through normal Django model access

Restrict admin access carefully. API logs may still contain personal or business-sensitive data even when masking is enabled.

Performance And Production Risks

The current design keeps database writes out of the request path, but production systems should still watch these areas.

Large Payloads

Large request or response bodies increase CPU and memory pressure because the middleware may read, decode, serialize, and mask payloads.

Recommendation:

  • Keep payload limits enabled.
  • Avoid -1 limits in production.
  • Skip endpoints that return large files or large reports.

Queue Backlog

The background queue is intentionally unbounded so request threads do not block on logging. If the database is slow or unavailable, memory can grow as logs accumulate.

Recommendation:

  • Alert on queue_backlog.
  • Alert on failed_insert_count.
  • Use a dedicated log database if needed.
  • Keep retention cleanup in place.

Database Growth

API logs can grow quickly.

Recommendation:

  • Add retention policy.
  • Archive or delete old logs.
  • Index high-use query fields.
  • Keep log storage separate from transactional tables on high-volume systems.

Useful indexes:

CREATE INDEX idx_api_logs_added_on ON drf_api_logs(added_on);
CREATE INDEX idx_api_logs_api_method ON drf_api_logs(api, method);
CREATE INDEX idx_api_logs_status_code ON drf_api_logs(status_code);

Signal Listener Latency

Signal listeners run synchronously.

Recommendation:

  • Keep listeners lightweight.
  • Push external calls to Celery, RQ, Kafka, Redis, or another queue.
  • Avoid blocking network calls directly in listeners.

Profiling Overhead

SQL profiling uses Django query collection. It is valuable for diagnosis but should be sampled on high-traffic systems.

Recommendation:

  • Use profiling during investigations.
  • Use sampling in production.
  • Disable SQL tracking when the query-count detail is not needed.

Compliance Checklist

DRF API Logger helps with compliance readiness, but it does not make an application compliant by itself.

Before production use in regulated environments:

  • Identify whether request/response bodies contain PII, PHI, PCI, secrets, or customer confidential data.
  • Add domain-specific sensitive fields to DRF_API_LOGGER_EXCLUDE_KEYS.
  • Keep payload limits enabled.
  • Use DRF_API_LOGGER_CUSTOM_HANDLER to remove fields that should never be logged.
  • Restrict Django Admin access to log records.
  • Store logs in an encrypted database or encrypted volume.
  • Define retention and deletion rules.
  • Document who can access logs and why.
  • Avoid logging full card numbers, government IDs, medical records, or long-lived credentials.
  • Confirm log storage region/data residency requirements.
  • Include API logs in incident-response and breach-review processes.

Recommended compliance-oriented configuration:

DRF_API_LOGGER_DATABASE = True
DRF_API_LOGGER_SIGNAL = False
DRF_API_LOGGER_MAX_REQUEST_BODY_SIZE = 8192
DRF_API_LOGGER_MAX_RESPONSE_BODY_SIZE = 8192
DRF_API_LOGGER_EXCLUDE_KEYS = [
    "password",
    "token",
    "authorization",
    "cookie",
    "api_key",
    "secret",
    "ssn",
    "credit_card",
]
DRF_API_LOGGER_CUSTOM_HANDLER = "myapp.logging.compliance_filter"

Troubleshooting

No Logs Are Created

Check:

  • DRF_API_LOGGER_DATABASE = True or DRF_API_LOGGER_SIGNAL = True
  • Middleware is present in MIDDLEWARE
  • drf_api_logger is present in INSTALLED_APPS
  • Migrations were run
  • The request is not an admin/static/media request
  • The URL name or namespace is not skipped
  • The status code and method match configured filters

Migration Or Table Error

Run:

python manage.py migrate

If using a dedicated database, confirm the database router and DRF_API_LOGGER_DEFAULT_DATABASE point to the right alias.

Response Body Is Empty

Likely causes:

  • Response content type is not in DRF_API_LOGGER_CONTENT_TYPES
  • Response is streaming
  • Response is binary/gzip/calendar
  • Response body exceeded DRF_API_LOGGER_MAX_RESPONSE_BODY_SIZE

Request Body Is Empty

Likely causes:

  • Request body is not decodable as UTF-8
  • Request content type is unsupported
  • Request body exceeded DRF_API_LOGGER_MAX_REQUEST_BODY_SIZE
  • Request body was unavailable from Django request internals

Application Memory Grows During Outage

Likely cause: database logging queue is accumulating because inserts are failing or slow.

Check:

logger_apps.LOGGER_THREAD.get_status()

Watch queue_backlog and failed_insert_count.

API Latency Increased After Enabling Logger

Check:

  • Payload size limits
  • Large response bodies
  • Signal listeners doing slow work
  • Profiling enabled at 1.0
  • SQL tracking enabled under high traffic
  • Custom handler doing expensive processing

Operational Runbook

For a new production rollout:

  1. Enable database logging in staging.
  2. Add sensitive fields to DRF_API_LOGGER_EXCLUDE_KEYS.
  3. Keep payload limits conservative.
  4. Skip health, metrics, and high-volume internal endpoints.
  5. Run a load test with realistic payloads.
  6. Watch API latency, queue_backlog, insert failures, and database write volume.
  7. Enable profiling with sampling only if needed.
  8. Add retention cleanup before going live.
  9. Review admin permissions for log access.
  10. Document the final logging policy.

Useful Links

Clone this wiki locally