Trace Proposal

docs/Experimental-Features.md

@@ -138,3 +138,176 @@ A layer is converted to a Zephyr module with the standard Zephyr entry points:
 The exported module can then be consumed by a Zephyr application by adding the module path to `ZEPHYR_EXTRA_MODULES` (or through a west manifest) and enabling the desired `CONFIG_...` symbols in `prj.conf`.
 
 Example: the [cmsis-to-zephyr-concept `ml_inference` example](https://github.com/brondani/cmsis-to-zephyr-concept/tree/main/ml_inference) shows a minimal end-to-end flow: exporting a layer as a Zephyr module and consuming it from a Zephyr application.
+
+## Trace
+
+Trace captures target execution data via SWO (or trace port), ETB, MTB, and Event Recorder.
+
+> Note
+> 
+> - `<target-set>` is the base name of the `*.ctrace.yml` file. This name is derived from the selected target-set, i.e. `SDS+AppKit-E8@HIL.ctrace.yml` is `<target-set>` = `SDS+AppKit-E8@HIL`.
+
+The file `*.ctrace.yml` specifies the trace information that should be captured from the target. This file is created using the CMSIS-Debugger with dialog support (UX potentially similar to SDS extension).
+
+pyOCD stores raw trace streams from various sources in binary files, for example `.trace/<target-set>/ETM.raw`, `.trace/<target-set>/ER.raw`.
+
+CMSIS Debugger post-processes raw streams into CSV files (on demand for raw views) and [CTF format v1.8.3](https://diamon.org/ctf/v1.8.3/) for viewers and analysis tools. CTF output is written to `.trace/<target-set>/ctf`.
+
+CMSIS Debugger uses the [CDT Trace](https://github.com/eclipse-cdt-cloud/vscode-trace-extension) extension as the VS Code viewer. This viewer requires a [Trace Compass XML Analysis file](https://archive.eclipse.org/tracecompass/doc/stable/org.eclipse.tracecompass.doc.user/Data-driven-analysis.html) that defines analysis views. The XML Analysis file should be generated by a pre-processing tool from the trace configuration files listed below.
+
+The trace solution supports both interactive debug workflows and CI workflows.
+
+### Tools and Extensions
+
+The following responsibilities are split across tools and VS Code extensions.
+
+**[Arm CMSIS Solution](https://marketplace.visualstudio.com/items?itemName=Arm.cmsis-csolution):**
+
+- Input: `*.csolution.yml` and optional `.cmsis/*.dbgconf`.
+- Output: `*.cbuild-run.yml` consumed by pyOCD.
+
+**[Arm CMSIS Debugger](https://marketplace.visualstudio.com/items?itemName=Arm.vscode-cmsis-debugger) - Trace View:**
+
+- Input: `.cmsis/*.ctrace.yml`, raw trace files in `.trace`, ELF/DWARF symbols.
+- Output:
+  - updated `.cmsis/*.ctrace.yml` with resolved symbolic locations,
+  - CTF output in `.trace/ctf` for [VSCode Trace Server](https://marketplace.visualstudio.com/items?itemName=eclipse-cdt.vscode-trace-server),
+  - trace analysis views and CSV export for raw trace data files.
+
+**[pyOCD](pyOCD-Debugger.md):**
+
+- Input: `*.cbuild-run.yml` and `.cmsis/*.ctrace.yml` (`register-values`).
+- Output: raw trace data files in `.trace`.
+
+**CLI symbol mapper for CI:**
+
+- Purpose: resolve ELF symbol values and update `.cmsis/*.ctrace.yml` in CI workflows.
+
+### Interactive Debug Workflow
+
+Phase                 | Where          | Action                                                                 | Output artifact
+:---------------------|:--------------:|:-----------------------------------------------------------------------|:-------------------------------
+Configure             | CMSIS Debugger | Load existing `.cmsis/*.ctrace.yml` at IDE start or target-set change. | n/a
+Build update          | CMSIS Debugger | On update of `*.cbuild-run.yml`, resolve symbols from ELF and validate. | Updated `register-values` in `*.ctrace.yml`
+Start debug           | pyOCD          | Clear previous raw trace files and load `register-values` from `*.ctrace.yml`   | n/a
+Edit trace settings   | CMSIS Debugger | Update `.cmsis/*.ctrace.yml`, re-resolve symbols, persist config.  | Saved `.cmsis/*.ctrace.yml` with update `register-values`.
+Capture               | pyOCD          | Capture trace during execution; A monitor command triggers `*.ctrace.yml` reload, changes delete raw trace data files.  | Raw trace data files in `.trace`
+Analyze               | CMSIS Debugger | Open raw trace data files generate views, CSV, or CTF output (utilizing `*.ctrace.yml` values).     | `.trace/ctf` files and analysis views
+
+### CI Workflow
+
+CI requires a prepared `.cmsis/*.ctrace.yml` (created interactively or maintained manually in source control).
+
+CI phase              | Where          | Action                                                                 | Output artifact
+:---------------------|:--------------:|:-----------------------------------------------------------------------|:-------------------------------
+Build                 | CMSIS-Toolbox  | Build the application.                                                  | ELF file and `*.cbuild-run.yml`
+Resolve symbols       | CLI tool       | After build, resolve symbols in `.cmsis/*.ctrace.yml` using ELF/DWARF and validate. | Updated `register-values` in `*.ctrace.yml`
+Run and capture       | pyOCD          | Load image and register values, delete previous raw trace data files, then capture trace data.               | Raw trace data files in `.trace`
+Post-process/analyze  | CMSIS Debugger | Convert raw data to CTF and open analysis views (desktop step).        | `.trace/ctf` and reports
+
+### Configuration files
+
+- Trace infrastructure (trace clock, ETB setup, trace pins, and related hardware setup) is configured in `*.csolution.yml` and optional target-set specific `.cmsis/*.dbgconf`.
+- Trace information (captured data configuration) is configured in Trace View of **[Arm CMSIS Debugger](https://marketplace.visualstudio.com/items?itemName=Arm.vscode-cmsis-debugger)** and stored in target-set specific `.cmsis/*.ctrace.yml` (for example `SDS+AppKit-E8@HIL.ctrace.yml`).
+
+#### Structure of `*.ctrace.yml`
+
+```yml
+ctrace:   # start of file (configure trace)
+# User-authored fields (created by dialog in CMSIS-Debugger):
+  created-by: CMSIS-Debugger v1.4.0
+  instructions:   # enable instruction trace (ETB, MTB only)
+    start:        # with cross trigger
+      - location: mySymbol    # can be code or data symbol
+        value: 0x10           # optional with data symbol
+
+  timestamp:      # enable timestamps
+
+  data:           # enable data trace
+    - location:   mysymbol  # required, symbol or address
+      access:     rw        # optional, default write
+      size:       8         # optional, default sizeof(symbol)
+      pc:         no        # optional, default PC value enabled
+    - location:
+            :
+
+  exceptions:       # enable exception trace
+    - pname: Core0  # only for Core0 (default all cores) 
+
+
+  events:           # enable all events or a specific list
+    - event: ///core1/CPI    # Cycles per instruction
+    - event: EXE    # Exception overhead
+        ...
+ # todo how to specify Cortex-M85 PMU Events
+
+  itm:            # enable ITM
+    - pname: Core0
+      enable: 0xFFFFFFFF   
+      privilege: 0x8  
+
+# Generated fields (managed by CMSIS Debugger and CLI tooling, do not edit manually):
+  ELF-files:  # use for mapping locations
+    - file: program1.axf
+      pname: Core0
+    - file: program2.axf
+      pname: Core1
+
+  register-values:
+    - pname: Core0
+      ITM:
+        TER: 0xFFFFFFFF
+        TPR: 0x8
+      DTW:
+        xxx:
+        symbol: mysymbol
+
+
+    - pname: Core1
+      ITM:
+      DTW:
+```
+
+Validation rules:
+
+- `location` entries accept symbol names or numeric addresses.
+- `pname` defaults to all cores if omitted; set it to scope trace by core.
+- `register-values` must be regenerated after each build that changes symbols.
+- Fields under `ELF-files` and `register-values` are generated artifacts.
+
+### Target trace information setup
+
+The trace information configuration is written to `DWT` and `ITM` registers of the Cortex-M system. The setup is injected through generated `register-values` and loaded by pyOCD to the target.
+
+In a later step, a pre-processing tool may generate `.trace/README.md` with setup instructions and code snippets that can be inserted in the application code. This file can also be consumed by automation and AI-based tooling.
+
+### Overall directory and file structure
+
+Directory                  | Description
+:--------------------------|:--------------------
+`.` (root)                 | Contains *.csolution.yml project file.
+`.cmsis`                   | Contains `<target-set>.ctrace.yml` that configures which trace information is captured (effectively the DWT, ITM register setup).
+`.trace/<target-set>`      | Contains raw trace data files.
+`.trace/<target-set>/ctf`  | Contains CTF files (`metadata`, `stream_0`, `stream_1`, etc.)
+
+### Initial Implementation
+
+Initial release focus is pyOCD with SWO UART and interactive operation in CMSIS-Debugger.
+
+Subsequent releases extend this initial solution to:
+
+- Instruction Trace (ETB, MTB), EventRecorder
+- CI Workflow
+
+### Open design questions
+
+- How should cross-trigger functionality be represented in `*.ctrace.yml`?
+- Should core selection (`pname`) support richer selectors beyond current core names?\
+- UX: how to expose configuration and captured trace information.
+- Rename "TRACE AND LIVE VIEW" to "TARGET MONITOR" with sub-sections LIVE WATCH and TRACE.
+
+### Out-of-Scope
+
+- Trace **does not** include RTT (SystemView) or STDIO UART output, which use separate data output paths.
+- ITM0 is often used for printf output; this output should not be routed to trace components and should instead go to an output window or debug console.
+