feat(jobs): add controller diagnostics and per-module E2E config pooling (#440)

Add a `diagnostics` module to the jobs controller that collects a structured
snapshot of job/step/task state for debugging stuck or failing jobs. Log
snapshots at DEBUG level in the scheduler and reconciler, with a
`include_job_logs_in_diagnostics` config flag (default off) to gate raw log
inclusion since logs may contain secrets or PII.

Refactor the E2E harness to support per-module config selection via an
`e2e_config()` pytest mark. Config layers (YAML files + inline overlays) are
resolved to a canonical hash and pooled, so modules sharing the same effective
config reuse one running services instance and it is torn down as soon as the
last consumer finishes. Extract all service lifecycle logic from conftest.py
into a new `services_pool.py` module.

Add `e2e/configs/local-subprocess.yaml` and a new `test_jobs_auth.py` suite
that enables auth and exercises job principal propagation end-to-end using
unsigned JWTs. Add unit tests for diagnostics, scheduler priority ordering,
subprocess backend, and timestamp contracts.

Signed-off-by: Ryan S <267728323+ironcommit@users.noreply.github.com>

Author: ironcommit

docs/set-up/config-reference.mdx

@@ -422,6 +422,8 @@ jobs:
   schedule_interval_seconds: 5
   # Register the subprocess/default execution profile. When unset, defaults to true for docker/none runtimes and false for kubernetes.
   enable_subprocess_executor:
+  # Include raw job log lines in controller diagnostics snapshots. Disabled by default because job logs may contain secrets or PII. Enable only for local debugging or test environments. | default: False
+  include_job_logs_in_diagnostics: false
 ```
 
 ### `models`

e2e/configs/local-subprocess.yaml

@@ -0,0 +1,68 @@
+# Local E2E config for hosts without Docker.
+#
+# This keeps the explicit subprocess/default jobs profile required by
+# translate_cpu_container_steps_to_subprocess(), while avoiding the default
+# docker job backends that are derived from platform.runtime: "docker".
+
+platform:
+  runtime: "none"
+  base_url: "http://0.0.0.0:8080"
+
+service: {}
+
+auth:
+  enabled: false
+  allow_unsigned_jwt: true
+  policy_decision_point_provider: embedded
+  policy_decision_point_base_url: "http://localhost:8080"
+  policy_data_refresh_interval: 2
+  bundle_cache_seconds: 15
+  admin_email: "admin@example.com"
+
+entities: {}
+
+jobs:
+  # Local E2E-only debugging aid. This may leak secrets or PII from job output,
+  # so it must remain disabled in non-test configs.
+  include_job_logs_in_diagnostics: true
+  executors:
+    - provider: subprocess
+      profile: default
+      backend: subprocess
+      config:
+        working_directory: .tmp/e2e/subprocess-jobs
+        cleanup_completed_jobs_immediately: false
+        ttl_seconds_before_active: 60
+        ttl_seconds_active: 3600
+        ttl_seconds_after_finished: 300
+  executor_defaults:
+    subprocess:
+      working_directory: .tmp/e2e/subprocess-jobs
+      cleanup_completed_jobs_immediately: false
+      ttl_seconds_before_active: 60
+      ttl_seconds_active: 3600
+      ttl_seconds_after_finished: 300
+
+evaluator:
+  recreate_existing_system_entities: true
+
+safe_synthesizer: {}
+
+models:
+  controller:
+    interval_seconds: 5
+    model_deployment_garbage_collection_ttl_seconds: 30
+
+inference_gateway: {}
+
+secrets:
+  allow_key_creation: true
+
+files:
+  default_storage_config:
+    type: local
+    path: .tmp/e2e/files
+
+studio:
+  static_files_path: web/packages/studio/dist
+  sandbox_enabled: true