feat!: polishing project for v4.0.0 (#391)

* fix: started fixes before 4.0

* fix: updated README

* fix: last fix before release

Author: TeKrop

AGENTS.md

@@ -41,9 +41,9 @@ Four strict DDD layers; dependencies only flow inward.
 
 | Layer | May depend on | Must not depend on |
 |---|---|---|
-| `domain` | stdlib only | `adapters`, `api`, `infrastructure` |
-| `adapters` | `domain` | `api` |
-| `api` | `domain`, `adapters` (via DI) | — |
+| `domain` | stdlib, `infrastructure` (logger only) | `adapters`, `api` |
+| `adapters` | `domain`, `infrastructure` | `api` |
+| `api` | `domain`, `adapters` (via DI), `infrastructure` | — |
 | `infrastructure` | anything | — |
 
 ```
@@ -52,7 +52,7 @@ app/
 ├── config.py                      # Pydantic BaseSettings (settings singleton)
 ├── domain/
 │   ├── enums.py                   # All domain enums; HeroKey/MapKey built dynamically from CSV
-│   ├── exceptions.py              # Domain exceptions (ParserParsingError, RateLimitedError, …)
+│       ├── exceptions.py              # Domain exceptions (ParserParsingError, ParserInternalError, RateLimitedError, …)
 │   ├── models/player.py           # PlayerIdentity, PlayerRequest dataclasses
 │   ├── parsers/                   # HTML parsers (stateless functions, selectolax)
 │   ├── ports/                     # typing.Protocol interfaces (structural typing)
@@ -74,13 +74,14 @@ app/
 ├── api/
 │   ├── dependencies.py            # FastAPI Depends() providers + type aliases
 │   ├── exception_handlers.py
-│   ├── helpers.py                 # overfast_internal_error, Discord webhook
+│       ├── helpers.py                 # SWR headers and response helpers (routes_responses, apply_swr_headers, …)
 │   ├── lifespan.py
 │   ├── responses.py               # ASCIIJSONResponse (default response class)
 │   ├── models/                    # Pydantic response models
 │   └── routers/
 ├── infrastructure/
 │   ├── decorators.py              # @rate_limited
+│   ├── helpers.py                 # overfast_internal_error, send_discord_webhook_message
 │   ├── logger.py                  # loguru logger
 │   └── metaclasses.py             # Singleton with clear_all()
 └── monitoring/                    # Prometheus metrics + middleware

README.md

@@ -6,7 +6,7 @@
 [![Issues](https://img.shields.io/github/issues/TeKrop/overfast-api)](https://github.com/TeKrop/overfast-api/issues)
 [![Documentation](https://img.shields.io/badge/documentation-yes-brightgreen.svg)](https://overfast-api.tekrop.fr)
 [![License: MIT](https://img.shields.io/github/license/TeKrop/overfast-api)](https://github.com/TeKrop/overfast-api/blob/master/LICENSE)
-![Mockup OverFast API](https://files.tekrop.fr/overfast_api_logo_full_1000.png)
+![Mockup OverFast API](static/logo.png)
 
 > OverFast API provides comprehensive data on Overwatch heroes, game modes, maps, and player statistics by scraping Blizzard pages. Built with **FastAPI** and **Selectolax**, **PostgreSQL** for persistent storage, **Stale-While-Revalidate caching** via **Valkey** and **nginx (OpenResty)**, **taskiq** background workers, and **TCP Slow Start + AIMD throttling** for Blizzard requests.
 
@@ -16,6 +16,7 @@
 * [💽 Run as developer](#-run-as-developer)
 * [👨‍💻 Technical details](#-technical-details)
 * [🐍 Architecture](#-architecture)
+* [📊 Monitoring](#-monitoring)
 * [🤝 Contributing](#-contributing)
 * [🚀 Community projects](#-community-projects)
 * [🙏 Credits](#-credits)
@@ -74,7 +75,7 @@ just format    # Run ruff formatter
 ```
 
 ### Testing
-The code has been tested using unit testing, except some rare parts which are not relevant to test. There are tests on the parsers classes, the common classes, but also on the commands (run in CLI) and the API views (using FastAPI TestClient class).
+The code has been tested using unit testing, except some rare parts which are not relevant to test. There are tests on the parsers, the domain services, the adapters, and the API views (using FastAPI TestClient class). Tests are organized to mirror the DDD layer structure: `tests/domain/`, `tests/adapters/`, `tests/infrastructure/`, `tests/players/`, `tests/heroes/`, etc.
 
 Running tests with coverage (default)
 ```shell
@@ -83,17 +84,18 @@ just test
 
 Running tests with given args (without coverage)
 ```shell
-just test tests/common
-make test PYTEST_ARGS="tests/common"
+just test tests/domain/services
+make test PYTEST_ARGS="tests/domain/services"
 ```
 
 
 ### Pre-commit
 The project is using [pre-commit](https://pre-commit.com/) framework to ensure code quality before making any commit on the repository. After installing the project dependencies, you can install the pre-commit by using the `pre-commit install` command.
 
-The configuration can be found in the `.pre-commit-config.yaml` file. It consists in launching 2 processes on modified files before making any commit :
-- `ruff` for linting and code formatting (with `ruff format`)
-- `sourcery` for more code quality checks and a lot of simplifications
+The configuration can be found in the `.pre-commit-config.yaml` file. It runs the following checks on modified files before each commit:
+- `ruff` for linting and auto-fixing
+- `ruff format` for code formatting
+- `ty` for type checking
 
 ## 👨‍💻 Technical details
 
@@ -284,6 +286,22 @@ stateDiagram-v2
     AIMD --> AIMD : non-200 — reset streak
 ```
 
+## 📊 Monitoring
+
+OverFast API ships an optional observability stack built on **Prometheus** and **Grafana**. When enabled, metrics are collected from two sources: **Nginx/OpenResty** (all requests, including Valkey cache hits) via a Lua module, and **FastAPI middleware** (requests that reach the app — cache misses only). Both sources normalize dynamic path segments (player IDs, hero keys) to prevent cardinality explosion, using matching Python and Lua implementations.
+
+![Grafana dashboard screenshot](static/monitoring/grafana_dashboard.png)
+
+Key metrics tracked include request rates and status code distribution, cache hit/miss ratios, Blizzard upstream call rates, AIMD throttle state, and background task throughput. Auto-provisioned Grafana dashboards cover API usage, API health, Blizzard calls, tasks & rate limiting, and system metrics.
+
+Enable the full stack with a single command:
+
+```shell
+just up monitoring=true
+```
+
+For detailed metric definitions, normalization rules, dashboard descriptions, and troubleshooting, see [app/monitoring/README.md](app/monitoring/README.md).
+
 ## 🤝 Contributing
 
 Contributions, issues and feature requests are welcome ! Do you want to update the heroes data (health, armor, shields, etc.) or the maps list ? Don't hesitate to consult the dedicated [CONTRIBUTING file](https://github.com/TeKrop/overfast-api/blob/main/CONTRIBUTING.md).

app/adapters/blizzard/throttle.py

@@ -23,9 +23,9 @@
 from typing import TYPE_CHECKING
 
 from app.adapters.cache.valkey_cache import ValkeyCache
-from app.api.helpers import send_discord_webhook_message
 from app.config import settings
 from app.domain.exceptions import RateLimitedError
+from app.infrastructure.helpers import send_discord_webhook_message
 from app.infrastructure.logger import logger
 from app.infrastructure.metaclasses import Singleton
 from app.monitoring.metrics import (

app/adapters/tasks/worker.py

@@ -41,7 +41,6 @@
     get_storage,
     get_task_queue,
 )
-from app.api.helpers import send_discord_webhook_message
 from app.config import settings
 from app.domain.enums import HeroKey, Locale
 from app.domain.parsers.heroes import fetch_heroes_html, parse_heroes_html
@@ -53,6 +52,7 @@
     PlayerService,
     RoleService,
 )
+from app.infrastructure.helpers import send_discord_webhook_message
 from app.infrastructure.logger import logger
 from app.monitoring.metrics import (
     background_refresh_completed_total,

app/api/exception_handlers.py

@@ -7,9 +7,9 @@
 from fastapi.exceptions import ResponseValidationError
 from starlette.exceptions import HTTPException as StarletteHTTPException
 
-from app.api.helpers import overfast_internal_error
 from app.api.responses import ASCIIJSONResponse
-from app.domain.exceptions import OverfastError
+from app.domain.exceptions import OverfastError, ParserInternalError
+from app.infrastructure.helpers import overfast_internal_error
 
 if TYPE_CHECKING:
     from fastapi import FastAPI, Request
@@ -33,6 +33,10 @@ async def overfast_error_handler(_: Request, exc: OverfastError):
             status_code=exc.status_code,
         )
 
+    @app.exception_handler(ParserInternalError)
+    async def parser_internal_error_handler(_: Request, exc: ParserInternalError):
+        raise overfast_internal_error(exc.blizzard_url, exc.cause) from exc
+
     @app.exception_handler(ResponseValidationError)
     async def pydantic_validation_error_handler(
         request: Request, error: ResponseValidationError

app/api/helpers.py

@@ -1,12 +1,9 @@
 """API Helpers module"""
 
-import traceback
-from datetime import UTC, datetime
 from functools import cache
-from typing import TYPE_CHECKING, Any
+from typing import TYPE_CHECKING
 
-import httpx
-from fastapi import HTTPException, status
+from fastapi import status
 
 from app.api.models.errors import (
     BlizzardErrorMessage,
@@ -15,8 +12,6 @@
     RateLimitErrorMessage,
 )
 from app.config import settings
-from app.infrastructure.decorators import rate_limited
-from app.infrastructure.logger import logger
 
 if TYPE_CHECKING:
     from fastapi import Request, Response
@@ -112,145 +107,6 @@
 }
 
 
-def overfast_internal_error(url: str, error: Exception) -> HTTPException:
-    """Returns an Internal Server Error. Also log it and eventually send
-    a Discord notification via a webhook if configured.
-    """
-
-    # Get error details
-    error_str = str(error)
-    error_type = type(error).__name__
-
-    # Log the critical error with full traceback
-    logger.critical(
-        "Internal server error for URL {} : {}\n{}",
-        url,
-        error_str,
-        traceback.format_exc(),
-    )
-
-    # If we're using a profiler, it means we're debugging, raise the error
-    # directly in order to have proper backtrace in logs
-    if settings.profiler:
-        raise error  # pragma: no cover
-
-    # Truncate error message for Discord (keep first part which is most relevant)
-    max_error_length = 900  # Field value limit is 1024, leave room for formatting
-    if len(error_str) > max_error_length:
-        # For validation errors, try to show just the summary
-        if "validation error" in error_str.lower():
-            lines = error_str.split("\n")
-            error_str = "\n".join(
-                lines[:5]
-            )  # First 5 lines usually contain the key info
-            if len(error_str) > max_error_length:
-                error_str = error_str[:max_error_length]
-        else:
-            error_str = error_str[:max_error_length]
-
-    # Send a message to the given channel using Discord Webhook URL
-    send_discord_webhook_message(
-        title="🚨 Internal Server Error",
-        url=f"{settings.app_base_url}{url}" if not url.startswith("http") else url,
-        fields=[
-            {"name": "Error Type", "value": f"`{error_type}`", "inline": True},
-            {"name": "Endpoint", "value": f"`{url}`", "inline": True},
-            {
-                "name": "Error Message",
-                "value": f"```\n{error_str}\n```",
-                "inline": False,
-            },
-        ],
-        color=0xE74C3C,  # Red
-    )
-
-    return HTTPException(
-        status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
-        detail=settings.internal_server_error_message,
-    )
-
-
-def _truncate_text(text: str, max_length: int, suffix: str = "...") -> str:
-    """Truncate text to max length with suffix if needed."""
-    if len(text) <= max_length:
-        return text
-    return text[: max_length - len(suffix)] + suffix
-
-
-def _truncate_embed_fields(
-    fields: list[dict[str, Any]],
-) -> list[dict[str, Any]]:
-    """Truncate field names and values to Discord limits."""
-    max_field_name_length = 250  # Actual limit: 256
-    max_field_value_length = 1000  # Actual limit: 1024
-
-    for field in fields:
-        name = field.get("name", "")
-        value = field.get("value", "")
-
-        if isinstance(name, str) and len(name) > max_field_name_length:
-            field["name"] = _truncate_text(name, max_field_name_length)
-        if isinstance(value, str) and len(value) > max_field_value_length:
-            field["value"] = _truncate_text(
-                value, max_field_value_length, "\n*(truncated)*"
-            )
-
-    return fields
-
-
-@rate_limited(max_calls=1, interval=1800)
-def send_discord_webhook_message(
-    *,
-    title: str | None = None,
-    description: str | None = None,
-    url: str | None = None,
-    fields: list[dict[str, Any]] | None = None,
-    color: int | None = None,
-) -> httpx.Response | None:
-    """Helper method for sending a Discord webhook message using modern embed syntax.
-    It's limited to one call per 30 minutes with the same parameters.
-
-    Args:
-        title: Optional title for the embed (max 256 chars)
-        description: Optional description text (max 4096 chars)
-        url: Optional URL to make the title clickable
-        fields: Optional list of field dicts with 'name', 'value', and optional 'inline' keys
-        color: Optional color for the embed (decimal format, e.g., 0xFF0000 for red)
-    """
-    if not settings.discord_webhook_enabled:
-        logger.error("{}: {}", title, description)
-        return None
-
-    # Apply Discord embed length limits
-    if title:
-        title = _truncate_text(title, 250)
-    if description:
-        description = _truncate_text(description, 4000, "\n\n*(truncated)*")
-    if fields:
-        fields = _truncate_embed_fields(fields)
-
-    # Build the embed payload
-    embed = {
-        "color": color or 0xE74C3C,  # Default to red for errors/alerts
-        "timestamp": datetime.now(UTC).isoformat(),
-    }
-
-    if title:
-        embed["title"] = title
-    if description:
-        embed["description"] = description
-    if url:
-        embed["url"] = url
-    if fields:
-        embed["fields"] = fields
-
-    payload = {"username": "OverFast API", "embeds": [embed]}
-
-    return httpx.post(  # pragma: no cover
-        settings.discord_webhook_url, json=payload, timeout=10
-    )
-
-
 @cache
 def get_human_readable_duration(duration: int) -> str:
     # Define the time units

app/domain/exceptions.py

@@ -1,6 +1,6 @@
 """Set of custom exceptions used in the API"""
 
-from fastapi import status
+from http import HTTPStatus
 
 
 class RateLimitedError(Exception):
@@ -21,7 +21,7 @@ def __str__(self) -> str:
 class OverfastError(Exception):
     """Generic OverFast API Exception"""
 
-    status_code = status.HTTP_500_INTERNAL_SERVER_ERROR
+    status_code = HTTPStatus.INTERNAL_SERVER_ERROR.value
     message = "OverFast API Error"
 
     def __str__(self):
@@ -33,7 +33,7 @@ class ParserBlizzardError(OverfastError):
     initialization, usually when the data is not available
     """
 
-    status_code = status.HTTP_500_INTERNAL_SERVER_ERROR
+    status_code = HTTPStatus.INTERNAL_SERVER_ERROR.value
     message = "Parser Blizzard Error"
 
     def __init__(self, status_code: int, message: str):
@@ -45,14 +45,30 @@ def __init__(self, status_code: int, message: str):
 class ParserParsingError(OverfastError):
     """Exception raised when there was an error during data parsing"""
 
-    status_code = status.HTTP_500_INTERNAL_SERVER_ERROR
+    status_code = HTTPStatus.INTERNAL_SERVER_ERROR.value
     message = "Parser Parsing Error"
 
     def __init__(self, message: str):
         super().__init__()
         self.message = message
 
 
+class ParserInternalError(OverfastError):
+    """Raised by domain services when an unexpected parsing failure occurs.
+
+    Carries the ``blizzard_url`` that was being parsed and the underlying
+    ``cause`` so the API layer can call ``overfast_internal_error`` and send
+    an alert without the domain needing to know about FastAPI or Discord.
+    """
+
+    message = "Internal Server Error"
+
+    def __init__(self, blizzard_url: str, cause: Exception):
+        super().__init__()
+        self.blizzard_url = blizzard_url
+        self.cause = cause
+
+
 class SearchDataRetrievalError(OverfastError):
     """Generic search data retrieval Exception (namecards, titles, etc.)"""