feat(debug): global debug mode (correlation ids, SQL detail, JFR profiling) (#6378)

[laugiov]

Add a global debug mode (correlation ids, SQL detail, JFR profiling)

Closes #6378

What this adds

A global, instance-wide debug mode to diagnose bugs and performance issues on on-prem installations.
It is delivered in the product (application code, no infra) and is fully disabled by default. It has
three parts, all behind a single flag:

• Correlation id in every log line via Micrometer Tracing. All log lines of a request or job
  share the same traceId. No tracing backend is required; ids go to the normal log output.
• SQL detail via datasource-proxy: every statement (ORM and native) is logged with its execution
  time and parameters, on the dedicated io.openaev.debug.sql logger.
• JVM profiling via the JDK-built-in Java Flight Recorder: a bounded recording is started, dumped
  on a timer and flushed on shutdown to a known directory. No extra dependency, no agent.

The toggle scope is global. Per-request scoping is out of scope for this POC.

Known limitation

The trace id stays in-process and across Spring @Async, but does not cross the RabbitMQ or Quartz
boundaries, so a simulation is not traced end to end yet. That propagation is the follow-up tracked in
#6379.

⚠️ Needs reviewer input: should tracing run on OpenTelemetry?

Please look at this point and help decide. The owners' decision is to unify on OpenTelemetry, but
that change is blocked today and I did not force it. Here is the situation in plain terms.

Two different things share one toolbox. "Observability" has two parts:

• Metrics = numbers on dashboards (requests/min, DB load, ...). OpenAEV already sends these to the
  Filigran telemetry server. This is the existing, working setup.
• Tracing = following one request end to end. That is the traceId this PR adds.

Both can run on the same library, OpenTelemetry (OTel). The decision was: put both on OTel, one
stack.

Why it is blocked (verified, not theory). OpenAEV configures OTel manually for metrics only:
io.openaev.telemetry.OpenTelemetryConfig exposes one OpenTelemetry bean with a meter provider but
no tracer provider (and a noop fallback when the telemetry endpoint is unreachable). Spring
Boot's tracing auto-config creates its OpenTelemetry only @ConditionalOnMissingBean -- so it sees
the project's bean, backs off, and the tracer is never wired. Tested by swapping the dependency to
micrometer-tracing-bridge-otel: a real request then logged an empty trace id. So the PR keeps
the self-contained Brave bridge, which produces correct trace ids today.

What it would take to unblock (small, but in sensitive code). Give the metrics OpenTelemetry
bean a tracer provider too (a few dozen lines), and make sure tracing does not get switched off by
the metrics noop fallback (the trace id in logs must not depend on the telemetry server being up).
The change is small but lives in the metrics code that feeds production dashboards, so it should be
done/validated by the metrics owners, not forced from this debug feature. Once done, switching the
debug mode from Brave to OTel is a one-line dependency change.

Ask for the reviewer (metrics/Platform): is it safe to extend the metrics OpenTelemetry bean
with a tracer provider (decoupled from the noop fallback), so we can run on a single OTel stack? If
yes, that is a small follow-up; if not, we keep Brave. Full analysis in ADR-0002-tracing-stack.md
(Status: Accepted decision, implementation blocked).

How to enable / disable

Off by default. Enable (preferably via env var so it is easy to revert):

OPENAEV_DEBUG_ENABLED=true

or openaev.debug.enabled=true in application.properties. Disable by removing the flag and
restarting. While active, a loud warning banner is logged at startup and repeated on a fixed
interval. See docs/docs/administration/debug-mode.md for the full configuration reference and the
writable volume needed in containers.

Safety

• Off by default, near-zero overhead when off. No datasource proxy on the query path, no
  parameter capture, no tracer, no extra files. Proven by a structural test (the datasource is the
  plain pool, not a proxy, when the flag is off).
• Mandatory masking. Secrets and personal data in SQL parameters and fields are masked before
  they reach the logs (key based + value based, both configurable). Tests assert known secret/PII
  patterns never appear.
• No new attack surface. No new port, no new endpoint.
• Production barrier. Disabled by default; in production (no dev/test/ci profile) it refuses
  to start unless openaev.debug.allow-in-production=true is also set, with an optional
  auto-disable-after timer and a loud repeated banner.
• Deny-by-default masking option. masking.mask-all-parameters=true masks every value, keeping
  only column name and type.
• Bounded when on. The SQL log is a rotated file (off the console). JFR recordings are bounded in
  size/age/duration and their periodic dumps are bounded by a retention pass (jfr.max-dump-files,
  jfr.max-total-dump-size). On a read-only filesystem JFR fails loudly and the SQL log falls back to
  the console; the app keeps running.
• Single profiler. When the Pyroscope agent is enabled, JFR does not start (hard-enforced).

Changed after the review

• JFR dump retention added (the periodic dumps no longer accumulate without bound).
• Real production barrier (DebugEnabledCondition + allow-in-production) and auto-disable-after,
  replacing the previous advisory-only guardrail.
• Deny-by-default masking (mask-all-parameters).
• Bounded regex scan (8 KB) with mask-before-truncate, instead of truncate-before-regex (which would
  have leaked a prefix of a long secret).
• Single-profiler enforcement (JFR yields to Pyroscope).
• Doc/report accuracy fixes (the "files never grow unbounded" and "env-var as a security control"
  claims were corrected; the RabbitMQ/Quartz limit is now stated up front).

Owners' decisions applied

• Tracing stack (decided: OpenTelemetry) -- BLOCKED, kept Brave. See the "Needs reviewer input"
  section above for the full explanation and the question for the metrics owners. In short: the
  project's metrics-only OpenTelemetry bean prevents Boot from wiring the tracer, so the OTel bridge
  produced no trace id (verified); the PR keeps Brave until the metrics bean is extended.
• Profilers: keep both, by deployment. JFR for on-prem, Pyroscope for cloud; the single-profiler
  lock (JFR yields to Pyroscope) stays. Documented.
• Tenant tag compatible with both mechanisms. New DebugTenantSource reads the v2 request
  selector (path {tenantId} / X-Tenant-Ids header) and falls back to v1 TenantContext. There is
  no ambient accessor for the resolved v2 TxCtx (it is argument-passed by design), so the resolver
  reads the same request inputs the v2 resolver uses, isolated in one class for a trivial future
  switch.
• mask-all also masks statement-text literals. In mask-all-parameters, single-quoted literals in
  the statement text are blanked too (bounded), so an inline secret in a native query does not leak.
• Debug-on in the cloud: accepted, bounded risk. No new barrier; operator guide documents the
  incident settings (auto-disable-after, masking, multi-tenant aggregation note). Formal ISO/SOC
  validation deferred to security.

Dependencies added

• io.micrometer:micrometer-tracing-bridge-brave (Apache-2.0, version from the Spring Boot BOM)
• net.ttddyy:datasource-proxy:1.10 (Apache-2.0, pinned)

JFR uses the JDK; no dependency added. Details in docs/debug-mode-poc/DEPENDENCIES.md.

Notes for reviewers

• The project is on Spring Boot 3.5.15 / Java 21; all versions are aligned on that BOM.
• Operator reference (committed): docs/docs/administration/debug-mode.md.
• The ADR (p6spy vs datasource-proxy), the findings report (security coverage, container impact,
  RabbitMQ/Quartz async-propagation analysis) and the dependency/license list are kept as separate
  review artifacts and are not committed in this PR.

Example output

One request, GET /api/scenarios/sc-42. Console keeps the app logs and the ORM summary; the
verbose per-statement SQL goes to the rotated file openaev-debug-sql.log.

Console — every line carries the request's trace and tenant:

INFO [trace=6a3c4dea tenant=0e7c2f1a-tenant-acme] ...ScenarioApi : Loading scenario sc-42
INFO [trace=6a3c4dea tenant=0e7c2f1a-tenant-acme] ...ScenarioApi : Scenario sc-42 loaded
WARN [trace=6a3c4dea tenant=0e7c2f1a-tenant-acme] ...debug.orm : ORM GET /api/scenarios/sc-42: 13 queries (2 distinct), 7ms
N+1 SUSPECTED: 'select team_name from teams where team_id = ?' executed 12x (6ms total)

SQL file openaev-debug-sql.log — one line per statement, masked:

... INFO [trace=6a3c4dea... tenant=...acme] sql success=true time=1ms statement=insert into users (user_id, user_email, user_password) values (?, ?, ?) params=[{user_id=u-1, user_email=MASKED, user_password=MASKED}]
... INFO [trace=6a3c4dea... tenant=...acme] sql success=true time=5ms statement=select team_name from teams where team_id = ? params=[{team_id=team-0}]
... the same SELECT 11 more times, one per team -- the N+1 the summary flagged

Reading it:

• trace= correlation id shared by every line of the request (full 32-hex in the logs); filter on it.
• tenant= the tenant the request targets (default tenant when not tenant-scoped).
• time= JDBC execution time per statement.
• statement= the real SQL sent to PostgreSQL, with ? placeholders.
• params=[{column=value}] bound params by column; user_email (PII) and user_password (sensitive
  column) are ***MASKED***; user_id/team_id are shown.
• The ORM ... line: one summary per request, flagging the N+1 (same SELECT once per team).

[Copilot]

Pull request overview

Adds a global, instance-wide debug mode for OpenAEV (disabled by default) intended to help diagnose on-prem bugs/perf issues by enabling: trace correlation IDs in logs, SQL statement logging with parameter masking, per-request ORM/N+1 insight, and bounded JFR profiling, plus operator documentation and extensive test coverage.

Changes:

• Introduces io.openaev.debug runtime components (debug gating/guardrail, datasource proxy + SQL masking logger, ORM insight filter, tenant MDC tagging, JFR recording manager, log routing to rotated SQL file).
• Wires debug defaults into application.properties and adds an administration guide (docs/docs/administration/debug-mode.md).
• Adds unit + integration tests covering correlation, masking scenarios, production barrier behavior, JFR retention, read-only FS degradation, and async trace propagation.

Reviewed changes

Copilot reviewed 42 out of 42 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
openaev-api/src/main/resources/application.properties	Adds debug-mode configuration defaults and ties tracing enablement to the debug flag.
openaev-api/pom.xml	Adds debug-mode dependencies (Micrometer tracing bridge, datasource-proxy) and related parsing dependency.
docs/docs/administration/debug-mode.md	Operator documentation: enable/disable, production barrier, masking, rotation, container guidance.
openaev-api/src/main/java/io/openaev/debug/DebugProperties.java	Configuration properties model for debug mode (SQL/JFR/masking).
openaev-api/src/main/java/io/openaev/debug/DebugEnabledCondition.java	Production barrier condition gating debug-mode bean creation.
openaev-api/src/main/java/io/openaev/debug/DebugActivationGuard.java	Logs whether debug mode was refused/allowed in “production mode”.
openaev-api/src/main/java/io/openaev/debug/DebugConfiguration.java	Spring wiring for debug-mode beans (proxy, listeners, filters, interceptors, manager).
openaev-api/src/main/java/io/openaev/debug/DebugRuntimeState.java	Runtime “active” switch to support auto-disable without removing the proxy.
openaev-api/src/main/java/io/openaev/debug/DataSourceProxyBeanPostProcessor.java	Wraps the Spring datasource with datasource-proxy when debug SQL is enabled.
openaev-api/src/main/java/io/openaev/debug/MaskingSqlLoggingListener.java	Logs SQL with timing + masked parameters and feeds per-request ORM stats.
openaev-api/src/main/java/io/openaev/debug/SqlParameterColumnResolver.java	Best-effort placeholder→column mapping to support key-based masking.
openaev-api/src/main/java/io/openaev/debug/SensitiveDataMasker.java	Key/value-pattern masking + mask-all support (incl. statement literal blanking).
openaev-api/src/main/java/io/openaev/debug/OrmInsightContext.java	Thread-local per-request SQL aggregation context.
openaev-api/src/main/java/io/openaev/debug/RequestQueryStats.java	Aggregates per-request statement counts/timings and repeated-statement detection.
openaev-api/src/main/java/io/openaev/debug/OrmInsightFilter.java	Emits per-request ORM summary and flags N+1/chatty requests.
openaev-api/src/main/java/io/openaev/debug/JfrRecordingManager.java	Starts bounded JFR recording, periodic dumps, and retention pruning.
openaev-api/src/main/java/io/openaev/debug/DebugSqlLogFileConfigurer.java	Routes verbose SQL logs to a rotated file under the debug output dir.
openaev-api/src/main/java/io/openaev/debug/DebugTenantSource.java	Resolves tenant tag from v2 selectors with sanitization, falling back to TenantContext.
openaev-api/src/main/java/io/openaev/debug/DebugTenantMdcInterceptor.java	Adds/removes tenant tag in MDC per request.
openaev-api/src/main/java/io/openaev/debug/DebugWebMvcConfigurer.java	Registers debug-only MVC interceptors.
openaev-api/src/main/java/io/openaev/debug/DebugModeManager.java	Lifecycle: banner, repeating warning, JFR start/stop, and auto-disable.
openaev-api/src/test/resources/application.properties	Mirrors debug logger/tracing defaults for the test environment.
openaev-api/src/test/java/io/openaev/debug/TraceCorrelationTest.java	Verifies MDC trace correlation across app logs and SQL logs.
openaev-api/src/test/java/io/openaev/debug/SqlParameterColumnResolverTest.java	Unit tests for placeholder→column mapping.
openaev-api/src/test/java/io/openaev/debug/SqlLoggingMaskingTest.java	End-to-end SQL logging + masking over H2 via datasource-proxy.
openaev-api/src/test/java/io/openaev/debug/SqlLoggingScenariosTest.java	Scenario coverage: batch, update, mask-all, truncation, slow-query threshold, etc.
openaev-api/src/test/java/io/openaev/debug/SensitiveDataMaskerTest.java	Unit tests for key/value masking and scan behavior.
openaev-api/src/test/java/io/openaev/debug/RequestQueryStatsTest.java	Unit tests for request aggregation and offender ordering.
openaev-api/src/test/java/io/openaev/debug/OrmInsightContextTest.java	Unit tests for thread-local lifecycle and record no-op behavior.
openaev-api/src/test/java/io/openaev/debug/OrmInsightFilterTest.java	End-to-end filter behavior (N+1, chatty, masking in summary).
openaev-api/src/test/java/io/openaev/debug/JfrRecordingManagerTest.java	Tests for JFR start/stop, periodic dumps, read-only failure, retention, disabled mode.
openaev-api/src/test/java/io/openaev/debug/DebugTenantSourceTest.java	Tests v2 selector precedence + sanitization fallback behavior.
openaev-api/src/test/java/io/openaev/debug/DebugTenantMdcInterceptorTest.java	Tests MDC tagging and cleanup for tenant.
openaev-api/src/test/java/io/openaev/debug/DebugSqlLogFileConfigurerTest.java	Tests rotated file routing, restore-on-stop, and console fallback.
openaev-api/src/test/java/io/openaev/debug/DebugPropertiesBindingTest.java	Tests safe defaults and Spring binding for nested types/lists.
openaev-api/src/test/java/io/openaev/debug/DebugModeToggleTest.java	Verifies beans/proxy presence when enabled and absence when disabled/refused.
openaev-api/src/test/java/io/openaev/debug/DebugEnabledConditionTest.java	Unit tests for the production barrier matching logic.
openaev-api/src/test/java/io/openaev/debug/DebugModeManagerTest.java	Tests banner/repeating warning, auto-disable, and Pyroscope coexistence behavior.
openaev-api/src/test/java/io/openaev/debug/DebugModeOffE2ETest.java	Real-app proof that debug mode off has no proxy/beans and no debug logs.
openaev-api/src/test/java/io/openaev/debug/DebugModeE2ETest.java	Real-context test: proxy unwraps, request correlation, masking on real DB.
openaev-api/src/test/java/io/openaev/debug/DebugModeReadOnlyFsE2ETest.java	Real-app degradation test: JFR fails with non-writable dir but app still serves.
openaev-api/src/test/java/io/openaev/debug/DebugAsyncCorrelationE2ETest.java	Real-app test: trace correlation preserved across Spring async executor boundary.

[Copilot]

Pull request overview

Copilot reviewed 46 out of 46 changed files in this pull request and generated 4 comments.

[codecov]

Codecov Report

❌ Patch coverage is 86.73835% with 74 lines in your changes missing coverage. Please review.
✅ Project coverage is 44.14%. Comparing base (67f4102) to head (e62e2e2).

Files with missing lines	Patch %	Lines
...ain/java/io/openaev/debug/JfrRecordingManager.java	82.14%	13 Missing and 7 partials ⚠️
...c/main/java/io/openaev/debug/OrmInsightFilter.java	78.84%	7 Missing and 4 partials ⚠️
...va/io/openaev/debug/MaskingSqlLoggingListener.java	81.48%	3 Missing and 7 partials ⚠️
...a/io/openaev/debug/SqlParameterColumnResolver.java	83.33%	3 Missing and 4 partials ⚠️
...c/main/java/io/openaev/debug/DebugModeManager.java	91.37%	1 Missing and 4 partials ⚠️
.../main/java/io/openaev/debug/DebugTenantSource.java	76.19%	2 Missing and 3 partials ⚠️
...in/java/io/openaev/debug/DebugActivationGuard.java	66.66%	2 Missing and 2 partials ⚠️
...va/io/openaev/debug/DebugSqlLogFileConfigurer.java	92.85%	2 Missing and 2 partials ⚠️
...ain/java/io/openaev/debug/SensitiveDataMasker.java	89.47%	1 Missing and 3 partials ⚠️
.../io/openaev/debug/DebugLogCorrelationListener.java	80.00%	2 Missing ⚠️
... and 2 more

❌ Your project check has failed because the head coverage (2.88%) is below the target coverage (80.00%). You can increase the head coverage or adjust the target coverage.

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #6380      +/-   ##
============================================
+ Coverage     43.75%   44.14%   +0.39%     
- Complexity     7208     7394     +186     
============================================
  Files          2289     2308      +19     
  Lines         63185    63743     +558     
  Branches       8421     8480      +59     
============================================
+ Hits          27647    28140     +493     
- Misses        33776    33784       +8     
- Partials       1762     1819      +57     

Flag	Coverage Δ
backend	66.69% <86.73%> (+0.30%)	⬆️
e2e	18.37% <ø> (ø)
frontend	2.88% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[antoinemzs]

Some nitpicks but also a logging format change that cannot be disabled (should only exist in debug=true mode)

Trace when everything off (large empty brackets)

2026-06-29T09:38:47.286+02:00  INFO 106352 --- [OpenAEV API] [  restartedMain] [                                                 ] io.openaev.App                           : Starting App using Java 21.0.11 with PID 106352 (/home/antoinemzs/filigran/openbas/openaev-api/target/classes started by antoinemzs in /home/antoinemzs/filigran/openbas)

When enabled, the brackets hold a correlation id

2026-06-29T09:47:21.539+02:00  INFO 116789 --- [OpenAEV API] [0.0-8080-exec-1] [6a42230910f449788694b1176553beca-09a186ba46edc482] io.openaev.debug.orm                     : ORM GET /api/xtm-composer/d9491550-1ffc-4070-8291-334f2f327919/connector-instances: 8 queries (8 distinct), 6ms

[antoinemzs]

Define MDC?

[laugiov]

Mapped Diagnostic Context, i will define it in the documentation ;)

[laugiov]

Done. MDC is now spelled out on first use: "the MDC (Mapped Diagnostic Context, the per-thread key/value bag the logging library attaches to every line)".

[antoinemzs]

Excellent doc thanks

[antoinemzs]

Maybe worth liaising with platform if we want to make this persistent, or it risks being deleted every time we recycle the container.

Similarly, let's think about recommending making this persistent in the demo compose file (repo docker)

[laugiov]

Good point. The output dir is configurable (openaev.debug.output-dir) but defaults to ./logs/debug, which sits on the container's ephemeral layer and is wiped on recycle. I added an explicit note in the docs: point it at a persistent volume so the SQL logs and JFR dumps survive a restart (the container example already uses a named volume).
For the two follow-ups: I'll liaise with platform on whether to make it persistent by default.