Add sealed ETL denial analysis (replacement for #699) by richiemsft · Pull Request #709 · microsoft/mxc

richiemsft · 2026-07-30T05:36:04Z

📖 Description

Replacement for #699, which was mistakenly marked merged when the stacked parent branch temporarily received child-stack history during a stack-handling mistake. This PR restores the intended review boundary on top of repaired Integration PR #696.

Adds the denial-analysis layer on top of #696:

Introduces the cross-platform learning_mode_core denial model, summary, analysis, framing, and emission primitives.
Adds the Windows sealed-ETL decoder using OpenTrace/ProcessTrace and TDH metadata.
Decodes and normalizes file, registry, UI, and capability-related Learning Mode events.
Streams raw diagnostics, handles producer pointer widths, and enforces canonical user-visible file paths.
Adds lm_analyze as a developer-only trace diagnostic.

This PR is intentionally stacked on #696 and should merge after it.

🔗 References

🔍 Validation

cargo test -p learning_mode_core
cargo test -p learning_mode_windows --all-targets
cargo clippy -p learning_mode_core -p learning_mode_windows --all-targets -- -D warnings
Combined stack validation was also run with the AppContainer and executor crates.

✅ Checklist

Signed the Contributor License Agreement
Linked to an issue
Updated documentation (if applicable)
Updated Copilot instructions (if build, architecture, or conventions changed)
If this PR changes Cargo.lock, the dependency-feed-check check passes (see docs/pull-requests.md)

📋 Issue Type

Bug fix
Feature
Task

GitHub Actions runs the PR validation build automatically. The ADO pipeline
(MXC-PR-Build) is the Azure version of the PR pipeline, kept in parity with the GitHub
Actions build; it runs on merge to main, and Microsoft reviewers with write access can trigger it
on a PR with /azp run. See docs/pull-requests.md.

Microsoft Reviewers: Open in CodeFlow

Introduce the learning_mode_core crate as the cross-platform hinge of the captureDenials pipeline (PR4/PR5 foundation): - model: DeniedResource / ResourceType (file|ui|network|capability|other) / AccessType, camelCase wire contract with round-trip + key-stability tests. - summary: DenialSummary terminator (exitCode, totalDenials, deniedResourcesTruncated). - frame: self-describing type-tagged DenialFrame (denial|summary). - emit: RFC 7464 (0x1E-framed) NDJSON stream writer over any io::Write. - analyze: DenialAnalyzer decode trait + AnalyzeError, implemented per-OS by the backend ETL decoders (Windows first). No OS-specific code; 16 unit tests + 1 doc-test, clippy/fmt clean. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Decode the learning-mode `.etl` that CaptureSession::finish seals into cross-platform learning_mode_core::DeniedResource values, implementing the DenialAnalyzer trait. - path_norm: kernel-form -> drive-letter, incl. the `\??\C:\...` / `\??\UNC\...` DOS-devices prefix the trace actually emits. - tdh_decode: TDH EVENT_RECORD -> (event_id, props). - extractors: route event 14/4907 (access check) by ObjectType (File -> file, Key -> registry/other, empty -> capability) and derive accessType from AccessMask with a File/Registry right vocabulary (Write > Execute > Read); event 27 -> Ui; event 28 -> capability denial (Denied gate, payload ProcessId). Missing/unparseable mask degrades to Unknown rather than dropping the denial. - etl_decode: file-mode OpenTraceW/ProcessTrace loop + dedup by (user-visible path, accessType). - examples/lm_analyze: NDJSON emit + `--raw` event dump for validation. Event vocabulary and AccessMask classification confirmed on the GE_CURRENT x64 VM against real block-and-log (Mode="Normal") and allow-and-log (Mode="Permissive") captures: File/registry denials decode identically in both modes; capability denials arrive as empty-ObjectType event 14 (permissive) or event 28 (block). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Factor the pure decode composition (extract_denial routing -> path normalisation -> dedup) out of EtlDenialAnalyzer::analyze into resources_from_events, and cover it with tests built from the event shapes captured on hardware for both learning modes: - block-and-log (Mode="Normal"): file/registry event 14 + capability event 28 - allow-and-log (Mode="Permissive"): capability folded into empty-ObjectType event 14 (no event 28) - empty-path capabilities from event 14 and event 28 collapse to one resource - non-actionable object types / not-denied records / unknown event IDs dropped A live ETW/TDH fixture read is intentionally not used: it needs the provider manifests registered on the machine, and a raw capture embeds token SIDs that don't belong in a public repo. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: dfea4d42-e8a5-42ef-978a-6434e8bf89ec

Allocate TRACE_EVENT_INFO storage with its required alignment and decode UTF-16 payloads from byte pairs so unaligned ETW data never forms invalid Rust references. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: dfea4d42-e8a5-42ef-978a-6434e8bf89ec

Stream raw ETL diagnostics, honor producer pointer width, normalize Windows path aliases, correct access masks and empty identifiers, and serialize FILETIME losslessly for JSON consumers. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: dfea4d42-e8a5-42ef-978a-6434e8bf89ec

Describe the record-separator-framed output as an RFC 7464 JSON text sequence rather than NDJSON across the core API and diagnostic example. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: dfea4d42-e8a5-42ef-978a-6434e8bf89ec

Normalize DOS devices, mapped drives, MUP redirectors, and DFS paths to absolute user-visible forms while omitting pipes, relative paths, and unknown device namespaces. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: dfea4d42-e8a5-42ef-978a-6434e8bf89ec

Document lm_analyze as a developer-only tool and explain why the sealed, policy-oriented learning-mode decoder does not directly reuse the diagnostic console's real-time display consumer. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: dfea4d42-e8a5-42ef-978a-6434e8bf89ec

azure-pipelines · 2026-07-30T05:36:13Z

Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run.

Copilot

Pull request overview

Adds cross-platform denial primitives and Windows sealed-ETL analysis for Learning Mode captures.

Changes:

Adds denial models, summaries, framing, and emission.
Implements bounded ETL decoding, extraction, normalization, and deduplication.
Adds the Windows-only lm_analyze diagnostic.

Reviewed changes

Copilot reviewed 15 out of 16 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
`src/core/learning_mode_core/src/summary.rs`	Defines capture summaries.
`src/core/learning_mode_core/src/model.rs`	Defines denial wire models.
`src/core/learning_mode_core/src/lib.rs`	Exposes the core API.
`src/core/learning_mode_core/src/frame.rs`	Defines framed records.
`src/core/learning_mode_core/src/emit.rs`	Emits RFC 7464 streams.
`src/core/learning_mode_core/src/analyze.rs`	Defines analyzer abstractions.
`src/core/learning_mode_core/Cargo.toml`	Configures the new crate.
`src/Cargo.toml`	Registers the workspace crate.
`src/Cargo.lock`	Records dependency changes.
`src/backends/learning_mode/windows/src/tdh_decode.rs`	Decodes TDH properties.
`src/backends/learning_mode/windows/src/path_norm.rs`	Normalizes Windows paths.
`src/backends/learning_mode/windows/src/lib.rs`	Exports analyzer APIs.
`src/backends/learning_mode/windows/src/extractors.rs`	Extracts typed denials.
`src/backends/learning_mode/windows/src/etl_decode.rs`	Processes sealed ETL files.
`src/backends/learning_mode/windows/examples/lm_analyze.rs`	Adds the diagnostic CLI.
`src/backends/learning_mode/windows/Cargo.toml`	Adds the core dependency.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3a8860a9-bb20-48ef-91a1-3da8e34b92fb

richiemsft · 2026-07-30T18:07:18Z

/azp run

azure-pipelines · 2026-07-30T18:07:36Z

Azure Pipelines: Successfully started running 1 pipeline(s).

richiemsft and others added 11 commits July 29, 2026 22:26

Update decoder terminology for capture modes

52b9691

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: dfea4d42-e8a5-42ef-978a-6434e8bf89ec

Update PR4 workspace lockfile

9f9df31

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: dfea4d42-e8a5-42ef-978a-6434e8bf89ec

Harden Learning Mode ETL analysis

2ec77aa

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: dfea4d42-e8a5-42ef-978a-6434e8bf89ec

richiemsft requested a review from a team as a code owner July 30, 2026 05:36

This was referenced Jul 30, 2026

Emit captureDenials JSON output (replacement for #701) #710

Open

Wire captureDenials to Windows Learning Mode runtime #696

Open

richiemsft requested a review from Copilot July 30, 2026 05:42

Copilot started reviewing on behalf of richiemsft July 30, 2026 05:42 View session

Copilot AI reviewed Jul 30, 2026

View reviewed changes

Comment thread src/backends/learning_mode/windows/src/extractors.rs Outdated

Comment thread src/backends/learning_mode/windows/src/tdh_decode.rs Outdated

Comment thread src/backends/learning_mode/windows/examples/lm_analyze.rs Outdated

Comment thread src/backends/learning_mode/windows/Cargo.toml

Address PR 709 review feedback

af3d5bc

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3a8860a9-bb20-48ef-91a1-3da8e34b92fb

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add sealed ETL denial analysis (replacement for #699) - #709

Add sealed ETL denial analysis (replacement for #699)#709
richiemsft wants to merge 12 commits into
user/saulg/learning-mode-endtoendfrom
user/saulg/learning-mode-analyze-replacement

richiemsft commented Jul 30, 2026 •

edited by microsoft-github-policy-service Bot

Loading

Uh oh!

azure-pipelines Bot commented Jul 30, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

richiemsft commented Jul 30, 2026

Uh oh!

azure-pipelines Bot commented Jul 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

richiemsft commented Jul 30, 2026 • edited by microsoft-github-policy-service Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📖 Description

🔗 References

🔍 Validation

✅ Checklist

📋 Issue Type

Microsoft Reviewers: Open in CodeFlow

Uh oh!

azure-pipelines Bot commented Jul 30, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

richiemsft commented Jul 30, 2026

Uh oh!

azure-pipelines Bot commented Jul 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

richiemsft commented Jul 30, 2026 •

edited by microsoft-github-policy-service Bot

Loading