Merge pull request #150 from cubewise-code/feat/issue-146-detailed-results

Add --detailed-results for per-execution cube rows (closes #146)

Author: nicolasbisurgi

CHANGELOG.md

@@ -0,0 +1,16 @@
+# Changelog
+
+All notable changes to RushTI are documented in this file.
+
+## Unreleased — `feat/issue-146-detailed-results`
+
+- Add `--detailed-results` for per-execution cube rows (closes #146).
+- Log migration hint at run start when `--detailed-results` is enabled.
+- Fix: `--tm1-instance` CLI argument now overrides `default_tm1_instance` for
+  results push and auto-load, not just for taskfile read.
+- Fix: parameter parser no longer treats backslash as an escape character
+  inside quoted values; Windows-style paths (e.g. `F:\Cons\Go_Files\`) parse
+  correctly in both TM1-sourced and text-based taskfiles.
+- Change: parameters in pushed result rows are now rendered as inline
+  `key="value"` pairs (matching the input cube format) instead of a JSON
+  dictionary.

docs/advanced/cli-reference.md

@@ -60,6 +60,7 @@ rushti --tasks FILE [options]          # 'run' is the default command
 | `--force` | `-f` | FLAG | `false` | Bypass exclusive mode checks and run immediately. |
 | `--optimize` | | CHOICE | *(none)* | Enable task optimization with a scheduling algorithm: `longest_first` or `shortest_first`. |
 | `--no-checkpoint` | | FLAG | `false` | Disable checkpoint saving for this run. |
+| `--detailed-results` | | FLAG | `false` | When pushing results to TM1, emit one row per executed TI instead of summarizing expanded tasks. Each row gets a fresh sequential `task_id`; the original `task_id` is preserved in the new `original_task_id` measure. See the [TM1 integration docs](../features/tm1-integration.md#detailed-results) for the join contract. |
 | `--tm1-instance` | | STR | *(none)* | Read task file from TM1 instead of disk. Requires `--workflow`. |
 | `--workflow` | `-W` | STR | *(none)* | Workflow identifier. Defaults to JSON metadata `workflow` field or the taskfile filename stem if omitted. Required when using `--tm1-instance`. |
 | `--log-level` | `-L` | CHOICE | `INFO` | Override log level for this run. |
@@ -284,7 +285,7 @@ rushti stats export --workflow daily-etl --run-id 20260115_103000 --output run.c
 
 ### rushti stats visualize
 
-Generate an interactive HTML dashboard with Gantt charts, success rates, and execution trends. Automatically opens the dashboard in the default browser.
+Generate an interactive HTML dashboard with Gantt charts, success rates, and execution trends, plus a companion DAG visualization rebuilt from the latest run's recorded predecessors. Automatically opens the dashboard in the default browser.
 
 ```bash
 rushti stats visualize --workflow daily-etl
@@ -298,6 +299,11 @@ rushti stats visualize --workflow daily-etl --runs 10 --output dashboard.html
 | `--output` | `-o` | PATH | Output HTML file path (default: `visualizations/rushti_dashboard_<id>.html`) |
 | `--settings` | `-s` | PATH | Path to `settings.ini` |
 
+!!! info "DAG reflects the latest executed run"
+    Both the dashboard and the DAG are sourced from the stats database — not from the live taskfile or TM1 cube definition. Editing a workflow definition without re-running it will not update either visualization. Re-run the workflow to refresh.
+
+    When a task expanded via MDX wildcards (one parent task fanned out to N executions), the DAG now renders **one node per executed TI**, suffixing shared `task_id` values (`2.1`, `2.2`, `2.3`). Predecessors fan out to all expansions of the parent. Earlier versions deduped by `task_id`, hiding fan-out and making the DAG appear unchanged across edits to expandable parameters.
+
 ---
 
 ### rushti stats analyze

docs/advanced/migration-guide.md

@@ -105,6 +105,71 @@ id=4 instance=tm1srv01 process=Load.Cube pYear=2026 predecessors=3
 
 ---
 
+## Numeric `task_id` Required {#numeric-task_id-required}
+
+Starting with the version that introduces `--detailed-results`, the JSON taskfile schema rejects non-integer `task_id` values. RushTI was always intended to use numeric IDs — the `rushti_task_id` cube dimension is pre-populated with integer-named elements `"1".."5000"` and result rows can only land in the cube if their `task_id` matches an element. Earlier releases were lax about this; the validator now enforces it explicitly.
+
+**Accepted forms:**
+
+- JSON integer: `"id": 5`
+- Integer-shaped string: `"id": "5"`
+
+**Rejected (now raises a `TaskfileValidationError` at parse time):**
+
+| Bad value | Reason |
+|-----------|--------|
+| `"id": 0` | Zero is not a positive integer; cube dimension starts at 1 |
+| `"id": -3` | Negatives have no cube element |
+| `"id": "05"` | Leading-zero strings don't match the integer-named cube elements (`"5"` not `"05"`) |
+| `"id": 5.0` | Floats are not integers |
+| `"id": "task-1"` | Non-integer string |
+| `"id": "abc"` | Non-integer string |
+
+The same rule applies to entries in `predecessors`.
+
+### How to Migrate Legacy Taskfiles
+
+If your taskfile uses string IDs like `"task-1"`, `"extract-load"`, or `"e1"`, rewrite them as integers (`1`, `2`, `3`, ...) and update every `predecessors` reference accordingly. The order doesn't matter — pick whatever sequence is convenient.
+
+```json
+// Before
+{
+  "tasks": [
+    {"id": "extract-1", "instance": "tm1srv01", "process": "..." },
+    {"id": "transform-1", "predecessors": ["extract-1"], "instance": "tm1srv01", "process": "..." }
+  ]
+}
+
+// After
+{
+  "tasks": [
+    {"id": 1, "instance": "tm1srv01", "process": "..." },
+    {"id": 2, "predecessors": [1], "instance": "tm1srv01", "process": "..." }
+  ]
+}
+```
+
+The error message you'll see if validation fails:
+
+```
+Task file validation failed:
+  - Task 0: 'id' must be a positive integer. Got: 'extract-1'. Task IDs must be positive integers (the rushti_task_id cube dimension uses integer member names).
+```
+
+---
+
+## TM1 Model Auto-Upgrade
+
+`rushti build` is now non-destructive by default. When you upgrade RushTI to a release that adds new measure elements (e.g. `original_task_id`), the next `rushti build --tm1-instance <inst>` patches the existing model in place:
+
+- **Dimensions.** Missing measure elements are added; existing elements and their attribute values are left alone. Cube data is preserved.
+- **`}rushti.load.results`.** The TI process is application-owned and is always replaced with the latest body — TI processes are stateless, so this is safe.
+- **`--force`.** Still wipes and rebuilds dimensions and the cube (data loss). Use only on dev/test environments.
+
+You don't need a separate command — bare `rushti build` does the right thing. CI/CD pipelines that run `rushti build` on every release continue to work across version upgrades without manual intervention.
+
+---
+
 ## Step-by-Step Migration
 
 ### Phase 1: Install and Verify (Day 1)

docs/advanced/settings-reference.md

@@ -119,6 +119,7 @@ Controls TM1-based read/write integration: reading task files from a TM1 cube an
 |---------|------|---------|-------------|
 | `push_results` | bool | `false` | Upload the results CSV to the TM1 Applications folder after each run. The file is named `rushti_{workflow}_{run_id}.csv` (with `.blb` extension for TM1 < v12). |
 | `auto_load_results` | bool | `false` | After uploading results (requires `push_results = true`), call the `}rushti.load.results` TI process on the target TM1 instance to load the CSV into the rushti cube. Passes `pSourceFile` and `pTargetCube` parameters. The process must exist on the target instance. |
+| `detailed_results` | bool | `false` | Emit one cube row per executed TI when pushing results, instead of summarizing expanded tasks (only meaningful with `push_results = true`). Each row gets a fresh sequential `task_id`; the original IDs are preserved in the new `original_task_id` measure. See [TM1 integration: detailed results](../features/tm1-integration.md#detailed-results). |
 | `default_tm1_instance` | str | *(none)* | Default TM1 instance name (from `config.ini`) used for reading task files and writing results. Required when `push_results` is enabled. |
 | `default_rushti_cube` | str | `rushti` | Name of the TM1 cube for task definitions and execution results. Created by the `rushti build` command. |
 | `default_workflow_dim` | str | `rushti_workflow` | Dimension name for workflow identifiers. |
@@ -128,7 +129,9 @@ Controls TM1-based read/write integration: reading task files from a TM1 cube an
 
 **Overridable via CLI:** `--tm1-instance`, `--workflow` / `-W`
 
-**Overridable via JSON task file:** `push_results`, `auto_load_results`
+**Overridable via CLI:** `--detailed-results`
+
+**Overridable via JSON task file:** `push_results`, `auto_load_results`, `detailed_results`
 
 !!! info "Setting Up TM1 Integration"
     Run `rushti build --tm1-instance <instance>` to create the required dimensions and cube automatically before enabling `push_results`.

docs/features/dag-execution.md

@@ -139,6 +139,8 @@ Open the HTML file in any browser to see an interactive graph with:
 - Click a node to see task details (instance, process, parameters)
 - Search and filter by task ID or process name
 
+`rushti tasks visualize` shows the **definition** (one node per task in the file). For a post-execution view that includes MDX expansions as separate nodes, use `rushti stats visualize --workflow <id>` instead — that one rebuilds the DAG from the latest recorded run, with one node per executed TI.
+
 ### Validate Before Running
 
 Check for structural problems without connecting to TM1:
@@ -184,3 +186,4 @@ Settings are resolved in priority order: CLI arguments > JSON task file settings
 - **[Advanced Task Files](../advanced/advanced-task-files.md)** — Stages, timeouts, expandable parameters
 - **[Task File Basics](../getting-started/task-files.md)** — Complete format reference (JSON and TXT)
 - **[Self-Optimization](optimization.md)** — Let RushTI learn from past runs and optimize task ordering
+- **[TM1 Integration: Detailed Results](tm1-integration.md#detailed-results)** — Per-execution rows in the cube for expanded tasks, joined via `original_task_id`

docs/features/tm1-integration.md

@@ -320,6 +320,42 @@ ExecuteCommand(cmd, 1);
 
 ---
 
+## Detailed Results
+
+By default, when a workflow uses [expandable tasks](../advanced/advanced-task-files.md) (parameters with the `*` suffix), all expansions of one parent task share the same `task_id` and the cube load step collapses them into a single summary row per original `task_id`. This is the only viable behaviour with the cube's `task_id` dimension, but it hides per-execution detail (individual durations, which specific parameter combo failed, retry counts per expansion).
+
+The `--detailed-results` flag (or `detailed_results = true` in `settings.ini` / the taskfile `settings` block) flips this: every executed TI gets its own row in the cube.
+
+### What Changes in the Cube
+
+- **One row per executed TI.** A workflow `[1, 2*(60), 3]` produces 62 rows instead of 3.
+- **Sequential gap-free `task_id`.** All rows are renumbered starting at 1 (`1..62`).
+- **`original_task_id` measure preserves identity.** Every row carries the pre-renumber ID, so the 60 expansions of `2*` all show `original_task_id = "2"`.
+- **`predecessors` references `original_task_id`.** Downstream tasks reconcile fan-in by joining `predecessors` against `original_task_id` in the same view — not against the new `task_id`.
+- **The stats DB is unchanged.** Results in the local SQLite/DynamoDB store remain keyed by the original `task_id`. The renumbering is purely a presentation step at upload time.
+
+The new `original_task_id` measure is added to the `rushti_measure` dimension automatically the next time you run `rushti build` (the upgrade is non-destructive — existing cube data is preserved). The `}rushti.load.results` TI is also rewritten to populate the new column.
+
+### Cross-Run Stability
+
+`task_id` values under detailed-results are **not stable across runs** — if the MDX in an expandable task returns a different number of members between runs, every renumbered ID downstream shifts. This is by design. The stable identity that survives across runs is the workflow definition stored under the `Input` element of the `rushti_run_id` dimension. For cross-run analysis, join through that input snapshot, the `task_signature` field in the stats DB, or the `original_task_id` measure.
+
+### Stage Grouping Recommendation
+
+Assign a unique `stage` value per expandable task so the fan-out groups visually in the cube. Every expansion inherits the parent's stage and lands together in views.
+
+### Dimension Sizing
+
+The `rushti_task_id` dimension defaults to 5,000 members. A workflow whose detailed-results renumbering exceeds 5,000 will fail to load into the cube — extend the dimension before enabling detailed-results on very wide workflows. RushTI does not auto-grow the dimension at upload time and does not perform extra TM1 round-trips to detect this case.
+
+### DAG Visualization Mirrors the Same Behavior
+
+`rushti stats visualize` rebuilds the DAG from the latest run's recorded predecessors. When that run involved expansions, the DAG now renders **one node per executed TI**, suffixing shared `task_id` values (`2.1`, `2.2`, `2.3`) and fanning predecessor edges out to every expansion of the parent. This is the visual analog of `--detailed-results` in the cube: per-execution rows in the cube ↔ per-execution nodes in the DAG.
+
+The stats database is the single source of truth for both the dashboard and the DAG. Editing the workflow definition (in the cube or in a JSON file) without re-running it will **not** refresh either visualization — re-run the workflow to update.
+
+---
+
 ## Customize Further
 
 - **[Statistics & Dashboards](statistics.md)** — The local SQLite database that feeds TM1 integration

docs/getting-started/task-files.md

@@ -29,8 +29,10 @@ TXT files are the simplest way to get started. Each line is a task, and you have
     id="4" predecessors="1,3" instance="tm1srv01" process="}bedrock.server.wait" pWaitSec=3
     ```
 
-!!! tip "Use integer IDs"
-    Always use integers (`"1"`, `"2"`, `"3"`, ...) for task IDs. This keeps things simple and is required if you plan to use [expandable parameters](../advanced/advanced-task-files.md) with MDX expressions.
+!!! warning "Task IDs must be positive integers"
+    `id` must be a positive integer (`1`, `2`, `3`, ...) — either a JSON integer (`"id": 1`) or an integer-shaped string (`"id": "1"`). Zero, negatives, leading-zero strings (`"05"`), floats, and non-numeric strings (`"task-1"`, `"abc"`) are rejected at parse time. Predecessors must follow the same rule.
+
+    The reason is concrete: the `rushti_task_id` cube dimension is pre-populated with integer-named elements `"1".."5000"`. Result rows can only land in the cube if their `task_id` matches an existing dimension element. See the [migration guide](../advanced/migration-guide.md) if you have legacy taskfiles using string IDs.
 
 ---
 

src/rushti/_shlex_utils.py

@@ -0,0 +1,34 @@
+"""Shared shlex helpers for rushti parameter parsing.
+
+TM1 process parameters never use ``\\`` as an escape character — it is always
+a literal path separator (e.g. ``F:\\Cons\\Go_Files\\``). The default
+``shlex.split(..., posix=True)`` treats backslash as an escape, which trips
+on Windows-style paths and drops the whole parameter set.
+
+Implementation note: ``shlex.split(..., posix=False)`` preserves backslashes
+as literals but does *not* group whitespace-containing quoted strings
+(``"Working Forecast"`` would split on the inner space). To get both
+behaviors — literal backslashes AND quote-grouping — we pre-escape every
+``\\`` in the input so POSIX mode preserves it, then tokenize with POSIX
+semantics and strip the (already-consumed) outer quotes via the shlex
+parser itself.
+"""
+
+from __future__ import annotations
+
+import shlex
+from typing import List
+
+
+def shlex_split_literal_backslashes(text: str) -> List[str]:
+    """Tokenize ``text`` like :func:`shlex.split` but treat backslash literally.
+
+    Quote-grouping still applies: ``param="value with spaces"`` yields one
+    token ``param=value with spaces``.
+
+    :param text: The raw parameter string.
+    :return: List of tokens with backslashes preserved.
+    :raises ValueError: If quotes are unbalanced.
+    """
+    pre_escaped = text.replace("\\", "\\\\")
+    return shlex.split(pre_escaped, posix=True)

src/rushti/cli.py

@@ -337,6 +337,18 @@ def create_argument_parser() -> argparse.ArgumentParser:
         help="Disable checkpoint saving for this run",
     )
 
+    parser.add_argument(
+        "--detailed-results",
+        dest="detailed_results",
+        action="store_true",
+        default=None,
+        help=(
+            "Emit one row per executed TI in the cube results, instead of "
+            "summarizing expansions. Each row gets a fresh sequential task_id "
+            "and the original_task_id measure preserves the pre-renumber identity."
+        ),
+    )
+
     parser.add_argument(
         "--optimize",
         dest="optimize",
@@ -410,6 +422,7 @@ def parse_named_arguments(argv: list):
         "tm1_instance": tm1_instance,
         "workflow": workflow,
         "log_level": args.log_level,
+        "detailed_results": args.detailed_results,
     }
 
     return tasks_file_path, cli_args
@@ -619,6 +632,15 @@ def main() -> int:
     # Apply CLI overrides to settings
     settings = get_effective_settings(settings, cli_args=cli_args)
 
+    if settings.tm1_integration.detailed_results:
+        logging.info(
+            "Detailed results enabled: one cube row per executed TI. "
+            "If you upgraded RushTI from an earlier release, run "
+            "'rushti build --tm1-instance <instance>' once to add the "
+            "'original_task_id' measure to the rushti_measure dimension. "
+            "The build is non-destructive — existing cube data is preserved."
+        )
+
     # Extract final values (CLI overrides settings.ini)
     max_workers = (
         cli_args["max_workers"]
@@ -1092,12 +1114,17 @@ def main() -> int:
                         connect_to_tm1_instance,
                         build_results_dataframe,
                         summarize_expanded_tasks,
+                        assign_unique_task_ids,
                     )
 
-                    tm1_instance = settings.tm1_integration.default_tm1_instance
-                    if tm1_instance:
+                    # CLI --tm1-instance wins over default_tm1_instance for every
+                    # TM1 operation in this run (taskfile read, results push,
+                    # auto_load_results). Silent precedence — no extra warning
+                    # when CLI overrides default.
+                    upload_instance = tm1_instance or settings.tm1_integration.default_tm1_instance
+                    if upload_instance:
                         tm1_upload = connect_to_tm1_instance(
-                            tm1_instance,
+                            upload_instance,
                             CONFIG,
                         )
                         try:
@@ -1106,7 +1133,10 @@ def main() -> int:
                                 workflow,
                                 ctx.execution_logger.run_id if ctx.execution_logger else "",
                             )
-                            results_df = summarize_expanded_tasks(results_df)
+                            if settings.tm1_integration.detailed_results:
+                                results_df = assign_unique_task_ids(results_df)
+                            else:
+                                results_df = summarize_expanded_tasks(results_df)
                             if not results_df.empty:
                                 file_name = upload_results_to_tm1(
                                     tm1_upload,
@@ -1130,13 +1160,13 @@ def main() -> int:
                                     )
                                     logger.info(
                                         "Executed }rushti.load.results on %s",
-                                        tm1_instance,
+                                        upload_instance,
                                     )
                                     if not success:
                                         logger.warning(
                                             "auto_load_results: Failed to execute "
                                             "}rushti.load.results on %s: %s",
-                                            tm1_instance,
+                                            upload_instance,
                                             status,
                                         )
                         finally: