feat: structured JSONL logging, CLI query tools, and root cause analysis#110
Open
ElNiak wants to merge 9 commits into
Open
feat: structured JSONL logging, CLI query tools, and root cause analysis#110ElNiak wants to merge 9 commits into
ElNiak wants to merge 9 commits into
Conversation
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
|
✅ PR Validation Passed Your changes look good! The quick validation checks have passed:
The full CI pipeline will run when this PR is merged or when targeting the main branches. |
Contributor
There was a problem hiding this comment.
Pull request overview
Introduces structured JSONL logging and new CLI/reporting utilities to query logs, render timelines/artifacts, and perform root-cause analysis, alongside refactors to event construction and environment/service mixins.
Changes:
- Replace plain text file logging with
structured.jsonlusing a contextvars-backedLogContext, plus anEventStreamRecorderthat writes events into the same stream. - Add reporting/query utilities (
LogQueryEngine, timeline renderer, artifacts browser, root-cause analyzer + builtin failure patterns) and expose them via new CLI commands (panther logs,panther report,panther debug). - Refactor protocol managers to use topology base classes and refactor/inline various mixins (execution/environment and event factory methods).
Reviewed changes
Copilot reviewed 99 out of 120 changed files in this pull request and generated 11 comments.
Show a summary per file
| File | Description |
|---|---|
| panther/tools/docs_gen/mkdocs/gen_ref_pages.py | Removes unused commented docs-gen logic. |
| panther/plugins/services/service_event_mixin.py | Switches test event emission to factory-based TestEvent. |
| panther/plugins/services/iut/quic/quinn/quinn.py | Updates IUT service manager to use ServiceManagerEventMixin. |
| panther/plugins/services/iut/quic/quiche/quiche.py | Updates IUT service manager to use ServiceManagerEventMixin. |
| panther/plugins/services/iut/quic/quic_go/quic_go.py | Updates IUT service manager to use ServiceManagerEventMixin. |
| panther/plugins/services/iut/quic/quant/quant.py | Updates IUT service manager to use ServiceManagerEventMixin. |
| panther/plugins/services/iut/quic/picoquic_shadow/picoquic_shadow.py | Updates IUT service manager to use ServiceManagerEventMixin. |
| panther/plugins/services/iut/quic/picoquic/picoquic.py | Updates IUT service manager to use ServiceManagerEventMixin. |
| panther/plugins/services/iut/quic/mvfst/mvfst.py | Updates IUT service manager to use ServiceManagerEventMixin. |
| panther/plugins/services/iut/quic/lsquic/lsquic.py | Updates IUT service manager to use ServiceManagerEventMixin. |
| panther/plugins/services/iut/quic/aioquic/aioquic.py | Updates IUT service manager to use ServiceManagerEventMixin. |
| panther/plugins/services/iut/minip/ping_pong/ping_pong.py | Updates IUT service manager to use ServiceManagerEventMixin. |
| panther/plugins/services/iut/iut_event_mixin.py | Removes the IUT event mixin facade module. |
| panther/plugins/protocols/peer_to_peer/peer_to_peer.py | Adds peer-to-peer protocol base with role validation semantics. |
| panther/plugins/protocols/peer_to_peer/init.py | Exposes PeerToPeerProtocolBase as package API. |
| panther/plugins/protocols/client_server/quic/quic.py | Switches QUIC protocol manager to ClientServerProtocolBase. |
| panther/plugins/protocols/client_server/minip/minip.py | Switches MiniP protocol manager to ClientServerProtocolBase. |
| panther/plugins/protocols/client_server/client_server.py | Adds client-server protocol base with shared role/port semantics. |
| panther/plugins/protocols/client_server/init.py | Exposes ClientServerProtocolBase as package API. |
| panther/plugins/environments/network_environment_mixin.py | Removes legacy network environment mixin. |
| panther/plugins/environments/network_environment/shadow_ns/shadow_ns.py | Cleans up unused experiment-finished-early import. |
| panther/plugins/environments/network_environment/localhost_single_container/localhost_single_container.py | Cleans up unused experiment-finished-early import. |
| panther/plugins/environments/execution_environment_mixin.py | Removes legacy execution environment mixin. |
| panther/plugins/environments/execution_environment/base_execution_environment.py | Inlines environment/execution initialization + command modification logic. |
| panther/plugins/environments/environment_plugin_mixin.py | Removes legacy environment plugin mixin. |
| panther/core/utils/structured_formatter.py | Adds JSONL formatter for structured logging output. |
| panther/core/utils/logging_mixin.py | Refactors logger mixin to be feature-aware without FeatureLoggerMixin. |
| panther/core/utils/logger_factory.py | Configures structured JSONL file handler + feature injection filter. |
| panther/core/utils/log_context.py | Adds contextvars-based LogContext + log_context() manager. |
| panther/core/utils/feature_logger_mixin.py | Replaces mixin with get_feature_logger() helper. |
| panther/core/utils/init.py | Updates utils public API exports (removes FeatureLoggerMixin). |
| panther/core/test_cases/test_case_impl.py | Pushes LogContext for the duration of a test run. |
| panther/core/test_cases/mixins/service_management.py | Adds per-service/per-phase LogContext during setup/prepare/teardown. |
| panther/core/reporting/timeline_renderer.py | Adds timeline rendering over structured logs. |
| panther/core/reporting/root_cause_analyzer.py | Adds root-cause analyzer over structured logs + patterns. |
| panther/core/reporting/log_query_engine.py | Adds streaming JSONL query engine with composable filters. |
| panther/core/reporting/failure_patterns.py | Adds built-in failure pattern catalog for RCA. |
| panther/core/reporting/experiment_reporter.py | Integrates RCA + artifacts into JSON/Markdown experiment reports. |
| panther/core/reporting/artifact_browser.py | Adds artifact inventory browser (index-first, walk fallback). |
| panther/core/reporting/init.py | Exports new reporting utilities in package API. |
| panther/core/outputs/output_index.py | Adds output index builder (output_index.json) for artifact manifests. |
| panther/core/outputs/output_aggregator.py | Registers collected outputs into the output index (if provided). |
| panther/core/outputs/init.py | Exposes OutputIndexBuilder in outputs package API. |
| panther/core/observer/workflow/init.py | Docstring cleanup (no functional changes). |
| panther/core/observer/impl/storage_observer.py | Updates typed handlers, docstring notes structured.jsonl path. |
| panther/core/observer/impl/state_observer.py | Simplifies event typing to BaseEvent in handlers. |
| panther/core/observer/impl/metrics_observer.py | Simplifies event typing to BaseEvent in handlers. |
| panther/core/observer/impl/logger_observer.py | Simplifies event typing to BaseEvent in handlers. |
| panther/core/observer/impl/event_stream_recorder.py | Adds observer that appends events into structured.jsonl. |
| panther/core/observer/impl/command_audit_observer.py | Refactors audit observer to accept BaseEvent (currently unsubscribed). |
| panther/core/observer/impl/init.py | Exports EventStreamRecorder. |
| panther/core/observer/factory.py | Registers event_stream observer type by default. |
| panther/core/metrics/resource_monitor.py | Docstring cleanup (no functional changes). |
| panther/core/metrics/metric_types.py | Adds metric dataclasses (Metric, TimingContext). |
| panther/core/metrics/enums.py | Docstring cleanup (no functional changes). |
| panther/core/experiment_observer.py | Registers EventStreamRecorder during observer setup. |
| panther/core/events/test/emitter.py | Switches test emitter to TestEvent factory methods and explicit imports. |
| panther/core/events/test/init.py | Narrows exported test event surface to factory-based API. |
| panther/core/events/step/events.py | Replaces many subclasses with factory classmethods on StepEvent. |
| panther/core/events/step/emitter.py | Switches step emitter to StepEvent factory methods. |
| panther/core/events/step/init.py | Narrows exported step event surface to factory-based API. |
| panther/core/events/service/emitter.py | Switches service emitter to ServiceEvent factory methods. |
| panther/core/events/service/init.py | Narrows exported service event surface to factory-based API. |
| panther/core/events/plugin/emitter.py | Switches plugin emitter to PluginEvent factory methods. |
| panther/core/events/plugin/init.py | Narrows exported plugin event surface to factory-based API. |
| panther/core/events/experiment/events.py | Introduces factory methods on ExperimentEvent and keeps some compat subclasses. |
| panther/core/events/experiment/emitter.py | Switches experiment emitter to ExperimentEvent factory methods. |
| panther/core/events/experiment/init.py | Narrows exported experiment event surface to factory-based API. |
| panther/core/events/environment/init.py | Narrows exported environment event surface to factory-based API. |
| panther/core/events/assertion/events.py | Replaces many subclasses with factory classmethods on AssertionEvent. |
| panther/core/events/assertion/emitter.py | Switches assertion emitter to AssertionEvent factory methods. |
| panther/core/events/assertion/init.py | Narrows exported assertion event surface to factory-based API. |
| panther/core/docker_builder/platform_detection.py | Adds Docker platform/buildx detection mixin. |
| panther/core/command_processor/mixins/modification_mixin.py | Removes command modification mixin (logic inlined elsewhere). |
| panther/core/command_processor/mixins/init.py | Updates mixins exports to remove modification mixin. |
| panther/core/command_processor/init.py | Updates command processor docs/exports to remove modification mixin. |
| panther/config/core/validators/init.py | Switches from lazy __getattr__ imports to eager exports. |
| panther/config/core/models/global_config.py | Replaces debug_file_logging with structured_log_file config option. |
| panther/cli/core/main.py | Registers new CLI command groups (logs/report/debug/build-metrics). |
| panther/cli/commands/logs.py | Adds structured log query/errors CLI commands. |
| panther/cli/commands/debug.py | Adds developer introspection CLI for events/observers. |
| panther/cli/commands/build_metrics.py | Adds CLI to view/export build/test metrics. |
| experiment-config/base/experiment_config_example_minimal_docker_no_buildx.yaml | Removes debug_file_logging from example config. |
| experiment-config/base/experiment_config_example_minimal_docker.yaml | Removes debug_file_logging from example config. |
| experiment-config/base/experiment_config_example_minimal.yaml | Removes debug_file_logging from example config. |
You can also share your feedback on Copilot code review. Take the survey.
Comment on lines
+11
to
+13
| Thread safety: writes are protected by a threading.Lock to prevent | ||
| interleaving with concurrent log handler writes. | ||
| """ |
Comment on lines
+81
to
+83
| with self._lock: | ||
| with open(self.output_path, "a", encoding="utf-8") as fh: | ||
| fh.write(line + "\n") |
Comment on lines
+72
to
+76
| # Track processed events for deduplication | ||
| if hasattr(event, "id"): | ||
| if event.id in self.processed_events_uuids: | ||
| return True | ||
| self.processed_events_uuids.append(event.id) |
Comment on lines
+54
to
+56
| def get_supported_event_types(self) -> list: | ||
| """Return the list of event types this observer handles.""" | ||
| return [ | ||
| CommandGenerationStartedEvent, | ||
| CommandGeneratedEvent, | ||
| CommandModifiedEvent, | ||
| ConfigGeneratedEvent, | ||
| ] | ||
|
|
||
| def handle_command_generation_started( | ||
| self, event: CommandGenerationStartedEvent | ||
| ) -> None: | ||
| return [] |
| @@ -6,7 +6,7 @@ | |||
| from panther.plugins.core.plugin_decorators import register_plugin | |||
| from panther.plugins.core.structures.plugin_type import PluginType | |||
| from panther.plugins.services.base.rust_quic_base import RustQUICServiceManager | |||
| from panther.plugins.services.iut.iut_event_mixin import IUTManagerEventMixin | |||
| from panther.plugins.services.service_event_mixin import ServiceManagerEventMixin | |||
Comment on lines
+196
to
+198
| # Push log context for this test; will be popped in the finally block | ||
| _ctx_token = log_context(test_id=self.test_name) | ||
| _ctx_token.__enter__() |
Comment on lines
+443
to
+444
| # Pop log context for this test | ||
| _ctx_token.__exit__(None, None, None) |
Comment on lines
+69
to
+73
| # Collect all matching records (no limit yet — we sort first) | ||
| records = list(self._engine.query(log_filter)) | ||
| records = self._sort_by_timestamp(records) | ||
| if limit > 0: | ||
| records = records[:limit] |
|
|
||
| import json | ||
| import re | ||
| from dataclasses import dataclass, field |
Comment on lines
+173
to
+184
| event_filter = LogFilter(sources={"event"}) | ||
| for record in self._engine.query(event_filter): | ||
| message = record.get("message", "") | ||
| event_type = record.get("event_type", "") | ||
| # Include events that look error-related | ||
| if any( | ||
| kw in (message + " " + event_type).lower() | ||
| for kw in ("error", "fail", "crash", "timeout", "killed") | ||
| ): | ||
| # Avoid duplicates | ||
| if record not in error_records: | ||
| error_records.append(record) |
This was referenced Mar 16, 2026
Open
…d classes Track 1 - God class splits: - Split DockerBuilder into platform_detection, buildx_operations, image_operations - Split MetricsCollector into metric_types, metric_analysis - Split ExperimentManager into experiment_phases, experiment_cleanup Track 2 - Event/observer simplification: - Replace ~130 event subclasses with factory classmethods on 8 base classes - Simplify ITypedObserver from 881 to ~80 lines with dynamic dispatch - Remove all backward-compat alias functions from events - Clean up events/__init__.py from 404 to ~100 lines - Update all emitters to use factory methods directly Track 3 - Validator cleanup: - Remove dead validator exports from pydantic_factories.py - Simplify validators/__init__.py (remove __getattr__ machinery) Net reduction: ~5,600 lines across 40 files. 870 tests passing.
…, and mixin inlining Wire hidden core modules to CLI (report, build-metrics, debug commands), delete orphaned tools and unused mixins, inline single-use mixins into their consumers, replace empty protocol placeholders with intermediate base classes (ClientServerProtocolBase, PeerToPeerProtocolBase). Net: -1,746 LOC, 13 files deleted, 5 created, 10 new CLI commands.
Add LogContext (contextvars-based) and StructuredJsonFormatter that replaces flat text file logs with structured JSONL. Every log record now carries experiment_id, test_id, service_id, phase context automatically via contextvars — zero changes to call sites needed. - New: log_context.py (LogContext dataclass + context manager) - New: structured_formatter.py (JSONL logging.Formatter) - Modified: logger_factory.py (structured file handler replaces text) - Modified: global_config.py (removed debug_file_logging, added structured_log_file) - Removed: debug_file_logging references from configs and experiment_manager
Wire LogContext into experiment lifecycle boundaries so that every log record and event emitted during a phase/test/service carries the relevant identifiers (experiment_id, test_id, service_id, phase). Context propagation locations: - ExperimentManager.initialize_experiments: experiment_id + phase - ExperimentManager.run_tests: experiment_id + test_execution phase - ExperimentManager.run_tests per-test loop: test_id - ExperimentManager.cleanup: experiment_id + cleanup phase - TestCase.run: test_id (via manual __enter__/__exit__) - ServiceManagementMixin: service_id + phase for setup_testers, setup_implementations, prepare_services, teardown_services New EventStreamRecorder observer: - Appends every event to the same structured.jsonl used by StructuredJsonFormatter, interleaving events and log records chronologically in a single timeline file - Uses "source": "event" and "level": "EVENT" to distinguish from logging records - Reads LogContext for experiment/test/service/phase fields - Thread-safe file writes via threading.Lock - Registered as a global observer in ExperimentObserverMixin with low priority (10) so business observers run first Also: - Register EventStreamRecorder in observer factory as "event_stream" - Add deprecation note to StorageObserver about separate JSONL files now being redundant with structured.jsonl (kept for backward compat) - 12 new unit tests covering schema, context propagation, deduplication, thread safety, error handling
Streaming JSONL log query engine (LogQueryEngine, LogFilter) that reads structured.jsonl files without loading them fully into memory. Supports filtering by level, service, test, phase, time range, correlation ID, message regex, and source. CLI commands: panther logs query <dir> --level/--service/--test/--phase/... panther logs errors <dir> (shorthand for --level ERROR,CRITICAL) Both commands support --json (default JSONL) and --human (table) output. 35 unit tests cover all filter combinations, file discovery, malformed input handling, and the count() aggregation.
Create an incremental, thread-safe output index that tracks every file produced during an experiment and serializes it as output_index.json. - New: panther/core/outputs/output_index.py with OutputIndexBuilder, FileEntry, and detect_type_and_format helper - Integrate into ExperimentManager: init at startup, register known files (config, reports, metrics, structured log), flush at cleanup - Extend OutputAggregator with optional output_index_builder param to register collected environment outputs in the manifest - 35 unit tests covering serialization, auto-detection, dedup, thread safety, and aggregator integration
…mands Add TimelineRenderer for chronological swimlane display of structured log entries, and ArtifactBrowser for listing/filtering experiment output files from output_index.json (with directory walk fallback). Register both as `panther report timeline` and `panther report artifacts` CLI subcommands with filtering by test, service, time range, phase, and artifact type.
…matching Add declarative FailurePattern library (8 built-in patterns derived from ErrorCategory) and RootCauseAnalyzer that reads structured.jsonl via LogQueryEngine to produce ranked root-cause findings with log excerpts and actionable suggestions. Extend ExperimentReporter with optional RCA and artifact inventory sections in both JSON and Markdown reports (gracefully skipped when structured.jsonl is absent). Add `panther report diagnose` CLI subcommand with --json/--human output.
ElNiak
force-pushed
the
feature/structured-logging-overhaul
branch
from
2c38f50 to
7e32dc3
Compare
|
✅ PR Validation Passed Your changes look good! The quick validation checks have passed:
The full CI pipeline will run when this PR is merged or when targeting the main branches. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
structured.jsonl) usingcontextvars-based context propagation — every log entry automatically carries experiment_id, test_id, service_id, and phase without any call-site changesstructured.jsonl, giving a single chronological file of logs + eventsoutput_index.jsonmanifest tracking every output file with type, format, and provenance metadatapanther logs queryandpanther logs errorscommands with filtering by level, service, test, phase, time range, correlation ID, and regex patternspanther report timeline(correlated multi-service timeline) andpanther report artifacts(artifact browser with filtering)panther report diagnosewith 8 built-in failure patterns (docker build, cert errors, Ivy compilation, segfault, OOM, timeouts, etc.) that detect root causes and suggest fixesNew CLI commands
Test plan