Merge pull request lightspeed-core#1582 from major/feat/sentry-support

RSPEED-2928: Add optional Sentry error tracking integration

Author: tisnik

docs/README.md

@@ -47,6 +47,8 @@ See the full documentation at [`../README.md`](../README.md) or browse sub-pages
 
 [Providers](https://lightspeed-core.github.io/lightspeed-stack/providers.html)
 
+[Sentry error tracking](https://lightspeed-core.github.io/lightspeed-stack/sentry.html)
+
 [User data collection](https://lightspeed-core.github.io/lightspeed-stack/user_data_collection.html)
 
 [Database structure](https://lightspeed-core.github.io/lightspeed-stack/DB/index.html)

docs/sentry.md

@@ -0,0 +1,162 @@
+# Sentry Error Tracking Integration
+
+Sentry integration is **completely optional** and **disabled by default**. Most deployments will not use it. The service starts and runs normally with no Sentry configuration present.
+
+When enabled, Sentry captures unhandled exceptions and performance traces from the FastAPI application and sends them to your Sentry project for monitoring and alerting.
+
+## Overview
+
+Enabling Sentry gives you:
+
+- **Error tracking** - unhandled exceptions are captured and reported with full stack traces
+- **Performance tracing** - a sample of POST requests are traced end-to-end
+- **Release tagging** - errors are tagged with the running service version for easier triage
+
+The integration is initialized at service startup and flushes any pending events on shutdown (2-second timeout).
+
+## Configuration
+
+Sentry is configured entirely through environment variables. No changes to `lightspeed-stack.yaml` are required.
+
+### Environment Variables
+
+| Variable | Required | Default | Description |
+|----------|----------|---------|-------------|
+| `SENTRY_DSN` | Yes (to enable) | - | Data Source Name from your Sentry project. Setting this variable enables Sentry. The DSN value is never written to logs to prevent credential exposure. |
+| `SENTRY_ENVIRONMENT` | No | `development` | Environment tag attached to all events. Set this explicitly in production deployments. Use values like `production`, `stage`, or `dev` to distinguish clusters or deployment stages. |
+| `SENTRY_CA_CERTS` | No | - | Path to a CA certificate bundle file. Only needed when your Sentry instance uses a private or internal CA. If the file is missing at startup, the SDK proceeds without custom certificates and logs a warning. |
+
+### Enabling Sentry
+
+Set `SENTRY_DSN` to the DSN string from your Sentry project settings:
+
+```bash
+export SENTRY_DSN="https://examplePublicKey@o0.ingest.sentry.io/0"
+```
+
+The service logs `Sentry initialized` on startup when the DSN is present, or `Sentry DSN not configured, skipping initialization` when it is not.
+
+## Behavior Details
+
+### FastAPI Integration
+
+The Sentry FastAPI integration is configured to capture only `POST` requests. `GET` requests (health checks, model listings, etc.) are not traced.
+
+### Trace Sampling
+
+Of the captured POST requests, the following routes are always excluded from tracing regardless of the sample rate:
+
+- `/readiness`
+- `/liveness`
+- `/metrics`
+- `/` (root)
+
+The remaining eligible requests are sampled at 25% for performance tracing. This keeps trace volume low and avoids noise from health check and metrics scrape traffic.
+
+### Privacy
+
+`send_default_pii` is set to `False`. Sentry will not attach user IP addresses, HTTP headers, or other personally identifiable information to events.
+
+### Release Tagging
+
+Every event is tagged with the running service version in the format `lightspeed-stack@{version}` (for example, `lightspeed-stack@0.5.0`). This makes it straightforward to correlate errors with specific releases in the Sentry UI.
+
+### Shutdown Behavior
+
+When the service shuts down, it flushes any buffered Sentry events before exiting. The flush has a 2-second timeout to avoid delaying shutdown.
+
+## OpenShift Deployment
+
+### Setting the DSN via a Secret
+
+Store the Sentry DSN in an OpenShift Secret rather than hardcoding it in a Deployment manifest:
+
+```bash
+oc create secret generic sentry-credentials \
+  --from-literal=dsn="https://examplePublicKey@o0.ingest.sentry.io/0"
+```
+
+Reference the Secret in your Deployment or Pod spec:
+
+```yaml
+env:
+  - name: SENTRY_DSN
+    valueFrom:
+      secretKeyRef:
+        name: sentry-credentials
+        key: dsn
+```
+
+### Setting the Environment Tag
+
+Use `SENTRY_ENVIRONMENT` to label events by cluster or deployment stage. This makes it easy to filter events in the Sentry UI:
+
+```yaml
+env:
+  - name: SENTRY_DSN
+    valueFrom:
+      secretKeyRef:
+        name: sentry-credentials
+        key: dsn
+  - name: SENTRY_ENVIRONMENT
+    value: "production"
+```
+
+Set this to `stage`, `dev`, or any label that matches your deployment topology.
+
+### Private CA Certificates (Enterprise Sentry Instances)
+
+Most deployers will not need this. If your Sentry instance is hosted internally and uses a certificate signed by a private CA, the SDK will fail to connect without the CA bundle.
+
+Mount the CA bundle into the pod using a ConfigMap or Secret, then point `SENTRY_CA_CERTS` at the mount path.
+
+**Using a ConfigMap:**
+
+```bash
+oc create configmap sentry-ca --from-file=ca-bundle.crt=/path/to/your/ca-bundle.crt
+```
+
+```yaml
+volumes:
+  - name: sentry-ca
+    configMap:
+      name: sentry-ca
+
+containers:
+  - name: lightspeed-stack
+    volumeMounts:
+      - name: sentry-ca
+        mountPath: /etc/sentry-ca
+        readOnly: true
+    env:
+      - name: SENTRY_DSN
+        valueFrom:
+          secretKeyRef:
+            name: sentry-credentials
+            key: dsn
+      - name: SENTRY_CA_CERTS
+        value: "/etc/sentry-ca/ca-bundle.crt"
+```
+
+If the file is not present at the path specified by `SENTRY_CA_CERTS`, the service logs a warning and continues without custom CA certificates. It will not fail to start.
+
+## Troubleshooting
+
+### Events Not Appearing in Sentry
+
+1. Confirm `SENTRY_DSN` is set and the value is correct. Check the service logs for `Sentry initialized` at startup.
+2. Verify network connectivity from the pod to the Sentry ingest endpoint. DNS resolution failures and firewall rules are common causes.
+3. If using a private Sentry instance, confirm `SENTRY_CA_CERTS` points to a valid CA bundle and the file is readable by the service process.
+4. Check that the DSN belongs to the correct Sentry project and organization.
+
+### Warning: CA Cert File Not Found
+
+```text
+CA cert file specified by SENTRY_CA_CERTS not found at /etc/sentry-ca/ca-bundle.crt; proceeding without custom CA certs
+```
+
+The path set in `SENTRY_CA_CERTS` does not exist. Verify the ConfigMap or Secret is mounted correctly and the mount path matches the environment variable value.
+
+### Events Appear in Wrong Environment
+
+Check the value of `SENTRY_ENVIRONMENT`. If it is not set, events are tagged as `development` by default. Set the variable explicitly to match your deployment stage.

pyproject.toml

@@ -73,6 +73,8 @@ dependencies = [
     # To be able to fix multiple CVEs, also LCORE-1117
     "requests>=2.33.0",
     "datasets>=4.7.0",
+    # Used for error tracking and monitoring
+    "sentry-sdk[fastapi]>=2.58.0",
 ]
 
 

src/app/main.py

@@ -4,6 +4,7 @@
 from collections.abc import AsyncIterator
 from contextlib import asynccontextmanager
 
+import sentry_sdk  # pyright: ignore[reportMissingImports]
 from fastapi import FastAPI, HTTPException
 from fastapi.middleware.cors import CORSMiddleware
 from fastapi.responses import JSONResponse
@@ -22,6 +23,7 @@
 from configuration import configuration
 from log import get_logger
 from models.responses import InternalServerErrorResponse
+from sentry import initialize_sentry
 from utils.common import register_mcp_servers_async
 from utils.llama_stack_version import check_llama_stack_version
 
@@ -44,6 +46,8 @@ async def lifespan(_app: FastAPI) -> AsyncIterator[None]:
     """
     configuration.load_configuration(os.environ["LIGHTSPEED_STACK_CONFIG_PATH"])
 
+    initialize_sentry()
+
     azure_config = configuration.configuration.azure_entra_id
     if azure_config is not None:
         AzureEntraIDManager().set_config(azure_config)
@@ -81,8 +85,13 @@ async def lifespan(_app: FastAPI) -> AsyncIterator[None]:
     yield
 
     # Cleanup resources on shutdown
-    await shutdown_background_topic_summary_tasks()
-    await A2AStorageFactory.cleanup()
+    try:
+        await shutdown_background_topic_summary_tasks()
+        await A2AStorageFactory.cleanup()
+    finally:
+        # Flush pending Sentry events after cleanup so any errors during
+        # shutdown are captured before the process exits.
+        sentry_sdk.flush(timeout=2)
     logger.info("App shutdown complete")
 
 

src/constants.py

@@ -249,3 +249,27 @@
 RLSAPI_V1_QUESTION_MAX_LENGTH: Final[int] = 32_768
 # Maximum character length for the serialized /v1/responses request body (64 KiB)
 RESPONSES_REQUEST_MAX_SIZE: Final[int] = 65_536
+
+# Sentry configuration constants
+# Environment variable name for the Sentry DSN (Data Source Name)
+SENTRY_DSN_ENV_VAR: Final[str] = "SENTRY_DSN"
+# Environment variable name for the Sentry environment tag
+SENTRY_ENVIRONMENT_ENV_VAR: Final[str] = "SENTRY_ENVIRONMENT"
+# Default Sentry environment when SENTRY_ENVIRONMENT is not set
+SENTRY_DEFAULT_ENVIRONMENT: Final[str] = "development"
+# Default trace sample rate (fraction of transactions to capture)
+SENTRY_DEFAULT_TRACES_SAMPLE_RATE: Final[float] = 0.25
+# Routes excluded from Sentry trace sampling (health checks, metrics, root).
+# Note: health and metrics routers are mounted WITHOUT a /v1 prefix
+# (see the setup_routers function in src/app/routers.py), so ASGI paths are
+# /readiness, /liveness, /metrics.
+SENTRY_EXCLUDED_ROUTES: Final[tuple[str, ...]] = (
+    "/readiness",
+    "/liveness",
+    "/metrics",
+    "/",
+)
+# Environment variable name for the Sentry CA certificate bundle path.
+# Set this to a file path (e.g. /etc/pki/tls/certs/ca-bundle.crt) when
+# connecting to a Sentry instance that uses a private or internal CA.
+SENTRY_CA_CERTS_ENV_VAR: Final[str] = "SENTRY_CA_CERTS"

src/sentry.py

@@ -0,0 +1,112 @@
+"""Sentry error tracking initialization and configuration."""
+
+import os
+
+import sentry_sdk  # pyright: ignore[reportMissingImports]
+from sentry_sdk.integrations.fastapi import (  # pyright: ignore[reportMissingImports]
+    FastApiIntegration,
+)
+
+import version
+from constants import (
+    SENTRY_CA_CERTS_ENV_VAR,
+    SENTRY_DEFAULT_ENVIRONMENT,
+    SENTRY_DEFAULT_TRACES_SAMPLE_RATE,
+    SENTRY_DSN_ENV_VAR,
+    SENTRY_ENVIRONMENT_ENV_VAR,
+    SENTRY_EXCLUDED_ROUTES,
+)
+from log import get_logger
+
+logger = get_logger(__name__)
+
+
+def sentry_traces_sampler(tracing_context: dict) -> float:
+    """
+    Determine the trace sample rate for a given request.
+
+    Excludes health check, metrics, and root routes from trace sampling to
+    reduce noise. All other routes use the default sample rate.
+
+    Parameters:
+    ----------
+        tracing_context (dict): The Sentry tracing context containing ASGI
+            scope information, including the request path.
+
+    Returns:
+    -------
+        float: 0.0 for excluded routes (no sampling), or
+            SENTRY_DEFAULT_TRACES_SAMPLE_RATE for all other routes.
+    """
+    asgi_scope = tracing_context.get("asgi_scope", {})
+    path = asgi_scope.get("path") if isinstance(asgi_scope, dict) else None
+
+    if path is not None:
+        if path == "/":
+            return 0.0
+        if any(
+            route != "/" and path.endswith(route) for route in SENTRY_EXCLUDED_ROUTES
+        ):
+            return 0.0
+
+    return SENTRY_DEFAULT_TRACES_SAMPLE_RATE
+
+
+def initialize_sentry() -> None:
+    """
+    Initialize Sentry error tracking if a DSN is configured.
+
+    Reads the SENTRY_DSN environment variable. If not set or empty, logs an
+    informational message and returns without initializing Sentry. When a DSN
+    is present, initializes the Sentry SDK with custom trace sampling, FastAPI
+    integration, and optional CA certificate configuration.
+
+    When SENTRY_CA_CERTS is set to a file path, that certificate bundle is
+    passed to the SDK for Sentry instances using private or internal CAs.
+
+    The DSN value is never logged to prevent accidental credential exposure.
+
+    Parameters:
+    ----------
+        None
+
+    Returns:
+    -------
+        None
+    """
+    dsn = os.environ.get(SENTRY_DSN_ENV_VAR)
+
+    if not dsn:
+        logger.info("Sentry DSN not configured, skipping initialization")
+        return
+
+    ca_certs = None
+    ca_certs_path = os.environ.get(SENTRY_CA_CERTS_ENV_VAR)
+    if ca_certs_path:
+        if os.path.exists(ca_certs_path):
+            ca_certs = ca_certs_path
+        else:
+            logger.warning(
+                "CA cert file specified by %s not found at %s; "
+                "proceeding without custom CA certs",
+                SENTRY_CA_CERTS_ENV_VAR,
+                ca_certs_path,
+            )
+
+    environment = os.environ.get(SENTRY_ENVIRONMENT_ENV_VAR, SENTRY_DEFAULT_ENVIRONMENT)
+
+    try:
+        sentry_sdk.init(
+            dsn=dsn,
+            environment=environment,
+            traces_sampler=sentry_traces_sampler,
+            send_default_pii=False,
+            ca_certs=ca_certs,
+            integrations=[FastApiIntegration(http_methods_to_capture=("POST",))],
+            release=f"lightspeed-stack@{version.__version__}",
+        )
+        logger.info("Sentry initialized")
+    except Exception:  # pylint: disable=broad-exception-caught
+        logger.exception(
+            "Failed to initialize Sentry, continuing without error tracking"
+        )