Merge pull request #153 from cubewise-code/feat/per-workflow-tm1-instance

feat: per-workflow tm1_instance result-push target

Author: nicolasbisurgi

CHANGELOG.md

@@ -2,6 +2,29 @@
 
 All notable changes to RushTI are documented in this file.
 
+## Unreleased
+
+- **Per-workflow `tm1_instance` setting** for results push and auto-load.
+  Set it inside a JSON taskfile's `settings` block to override the
+  `settings.ini` default per workflow. Resolution chain (highest wins):
+  CLI `--tm1-instance` > taskfile `settings.tm1_instance` >
+  `settings.ini [tm1_integration].tm1_instance` >
+  `settings.ini [tm1_integration].default_tm1_instance` (deprecated).
+  RushTI now logs which tier supplied the target at push time.
+- **Behavioural change:** `push_results` and `auto_load_results` set
+  inside taskfile JSON now take effect. They were silently ignored in
+  the `run` path before this release. If you have divergent values
+  between taskfile JSON and `settings.ini`, reconcile them before
+  upgrading — the taskfile value will now win.
+- **Soft deprecation:** `settings.ini [tm1_integration].default_tm1_instance`.
+  Rename to `tm1_instance`. The old key is honoured indefinitely as the
+  final fallback in the resolution chain; a one-shot `DEPRECATION:`
+  warning fires at settings load only when it's the value actually being
+  used.
+- **Soft deprecation:** `rushti tasks push --target-tm1-instance`. Use
+  `--tm1-instance` instead. The legacy flag is aliased and continues to
+  work; a `DEPRECATION:` warning fires on use.
+
 ## Unreleased — `feat/issue-146-detailed-results`
 
 - Add `--detailed-results` for per-execution cube rows (closes #146).

CONTEXT.md

@@ -0,0 +1,106 @@
+# RushTI — Domain Context
+
+This file captures the canonical vocabulary used in RushTI. Terms here are
+load-bearing: they appear in code, docs, settings, and user-facing messages.
+When a term feels ambiguous in conversation, anchor it to a definition here.
+
+> Scope: single context. No `CONTEXT-MAP.md` — the whole repository shares
+> this vocabulary.
+
+---
+
+## Core terms
+
+### Taskfile
+The structured input that drives a `rushti run`. Materialised as either:
+- a **JSON task file** (`.json`) parsed by `parse_json_taskfile`, OR
+- a **TXT task file** (`.txt`) converted on read by `convert_txt_to_json`, OR
+- a **TM1 cube taskfile** read by `read_taskfile_from_tm1` (an MDX query
+  against the `rushti` cube using the `Input` element of `rushti_run_id`).
+
+The in-memory shape is the `Taskfile` dataclass (`taskfile.py`) — a container
+with `version`, `metadata`, `settings`, and `tasks`. Always prefer
+**"taskfile"** (one word, lowercase) in docs; **"JSON task file"** is also
+acceptable when the format matters.
+
+Do **not** call it `workflow.json` — that name implies the file *is* a
+workflow, which conflates the container with one of its metadata fields.
+
+### Workflow
+The logical identifier for a run, stored in `metadata.workflow` of a taskfile
+and as an element of the `rushti_workflow` dimension. A taskfile contains
+tasks *for* one workflow.
+
+Use **"workflow"** as an adjective for scope ("workflow-level setting",
+"per-workflow override") rather than as a synonym for "taskfile".
+
+### Task
+A single TI process execution inside a taskfile. Each task has:
+- `id` — a positive integer string, used as an element name in
+  `rushti_task_id`.
+- `instance` — **the TM1 instance where this task executes** (task-level).
+- `process` — the TI process name on that instance.
+- `parameters`, `predecessors`, `stage`, `timeout`, etc.
+
+### TM1 instance
+A named TM1 server defined in `config.ini`. Three roles can apply
+contextually — context disambiguates, not the name:
+
+| Role | Meaning | Where it shows up |
+|---|---|---|
+| **Source** | Where a taskfile is read *from* (cube source) | `rushti run --tm1-instance X` |
+| **Execution target** | Where a task executes | task-level `instance` field |
+| **Results target** | Where execution results are pushed | settings.ini or taskfile `tm1_instance`; also CLI `--tm1-instance` as fallback |
+
+The canonical setting key for the results target is `tm1_instance` (in both
+`settings.ini [tm1_integration]` and the taskfile `settings` block). The
+deprecated alias is `default_tm1_instance`.
+
+There is no separate "source TM1 instance" override — `--tm1-instance` is
+both the source and (in absence of higher-precedence overrides) the results
+target for a `run` invocation.
+
+### Source vs target — naming convention
+RushTI does not prefix instance names with `source_` or `target_`. The
+*command* or the *config section* makes the role obvious:
+- `rushti run --tm1-instance X` — X is the source for cube-read, and the
+  fallback results target if nothing else is set.
+- `rushti tasks push --tm1-instance X` — X is the target of the push
+  (canonical form). The legacy alias `--target-tm1-instance` is deprecated.
+- `[tm1_integration].tm1_instance` — the section header makes "target" clear.
+
+This convention is intentional: prefixes don't scale across surfaces and
+fight against contextual disambiguation. See
+[[adr/0001-tm1-instance-resolution]] for the full rationale.
+
+---
+
+## Settings precedence
+
+The effective value for a settings-driven knob is resolved in this order
+(highest wins):
+
+1. **CLI arguments** — e.g. `--max-workers`, `--tm1-instance`.
+2. **Taskfile settings block** — e.g. `{"settings": {"max_workers": 8}}`
+   in JSON. Applied via `_apply_json_settings` inside
+   `get_effective_settings` *after* the taskfile is parsed.
+3. **`settings.ini`** — the canonical defaults file at
+   `config/settings.ini`.
+4. **Built-in defaults** — the dataclass field defaults in
+   `settings.py`.
+
+Three knobs are *not* settings-driven and don't follow this chain:
+- Per-task `instance` and `process` — taskfile-only, no fallback.
+- TM1 connection parameters — `config.ini` only.
+
+---
+
+## Files of record
+
+| File | Purpose |
+|---|---|
+| `config/settings.ini` | Execution defaults (user-editable). |
+| `config/settings.ini.template` | Documented example, shipped in repo. |
+| `config/config.ini` | TM1 connection parameters per instance. |
+| `archive/{workflow}/{run_id}.json` | Snapshot of the taskfile actually executed, for audit + DAG reconstruction. |
+| `data/rushti_stats.db` | Local SQLite stats database (when `[stats] enabled = true`). |

config/settings.ini.template

@@ -113,7 +113,7 @@
 # To set up TM1 integration:
 # 1. Run: python rushti.py build --tm1-instance tm1srv01
 #    This creates the required dimensions and cube automatically.
-# 2. Set push_results = true and configure default_tm1_instance below
+# 2. Set push_results = true and configure tm1_instance below
 #
 # Default: false
 # push_results = false
@@ -126,9 +126,18 @@
 # Default: false
 # auto_load_results = false
 
-# Default TM1 instance for reading taskfiles and writing results
-# Used when --tm1-instance is specified without an instance, and for auto-upload
-# Must be defined in config.ini
+# TM1 instance used as the results-push target when no higher-precedence
+# override is set. Resolved against a 4-tier chain (highest wins):
+#   1. CLI --tm1-instance
+#   2. Taskfile JSON settings.tm1_instance
+#   3. This key
+#   4. default_tm1_instance below (deprecated)
+# Must be defined in config.ini.
+# tm1_instance = tm1srv01
+
+# DEPRECATED: use tm1_instance above. Still honoured indefinitely as the
+# final fallback; a one-shot DEPRECATION warning fires at settings load
+# *only* when tm1_instance is unset and this key is the value being used.
 # default_tm1_instance = tm1srv01
 
 # Default cube name for task definitions and results

docs/advanced/cli-reference.md

@@ -235,8 +235,8 @@ rushti tasks push --tasks daily-etl.json --tm1-instance tm1srv01
 | Option | Short | Type | Description |
 |--------|-------|------|-------------|
 | `--tasks` | `-t` | PATH | Local JSON task file to push (*required*) |
-| `--tm1-instance` | | STR | Source TM1 instance (if loading from TM1) |
-| `--target-tm1-instance` | | STR | Target TM1 instance for the push |
+| `--tm1-instance` | | STR | TM1 instance to push the taskfile to. Context disambiguates the role — on `tasks push` this is the destination. |
+| `--target-tm1-instance` | | STR | **Deprecated alias** for `--tm1-instance`. Still works; emits a `DEPRECATION:` warning when used. |
 | `--settings` | `-s` | PATH | Path to `settings.ini` |
 
 ---

docs/advanced/settings-reference.md

@@ -120,7 +120,8 @@ Controls TM1-based read/write integration: reading task files from a TM1 cube an
 | `push_results` | bool | `false` | Upload the results CSV to the TM1 Applications folder after each run. The file is named `rushti_{workflow}_{run_id}.csv` (with `.blb` extension for TM1 < v12). |
 | `auto_load_results` | bool | `false` | After uploading results (requires `push_results = true`), call the `}rushti.load.results` TI process on the target TM1 instance to load the CSV into the rushti cube. Passes `pSourceFile` and `pTargetCube` parameters. The process must exist on the target instance. |
 | `detailed_results` | bool | `false` | Emit one cube row per executed TI when pushing results, instead of summarizing expanded tasks (only meaningful with `push_results = true`). Each row gets a fresh sequential `task_id`; the original IDs are preserved in the new `original_task_id` measure. See [TM1 integration: detailed results](../features/tm1-integration.md#detailed-results). |
-| `default_tm1_instance` | str | *(none)* | Default TM1 instance name (from `config.ini`) used for reading task files and writing results. Required when `push_results` is enabled. |
+| `tm1_instance` | str | *(none)* | TM1 instance (from `config.ini`) used as the **results-push target** when no higher-precedence override is set. Resolved per [the 4-tier chain](../features/tm1-integration.md#per-workflow-target-instance): CLI `--tm1-instance` > taskfile `settings.tm1_instance` > this key > `default_tm1_instance` (deprecated). Required when `push_results` is enabled and no higher tier is set. |
+| `default_tm1_instance` | str | *(none)* | **Deprecated** — use `tm1_instance`. Honoured indefinitely as the final fallback in the resolution chain; a one-shot `DEPRECATION:` warning fires at settings load *only* when it's actually being used (i.e. `tm1_instance` is unset). |
 | `default_rushti_cube` | str | `rushti` | Name of the TM1 cube for task definitions and execution results. Created by the `rushti build` command. |
 | `default_workflow_dim` | str | `rushti_workflow` | Dimension name for workflow identifiers. |
 | `default_task_id_dim` | str | `rushti_task_id` | Dimension name for task sequence elements (1--5000 default elements). |
@@ -131,7 +132,7 @@ Controls TM1-based read/write integration: reading task files from a TM1 cube an
 
 **Overridable via CLI:** `--detailed-results`
 
-**Overridable via JSON task file:** `push_results`, `auto_load_results`, `detailed_results`
+**Overridable via JSON task file:** `push_results`, `auto_load_results`, `detailed_results`, `tm1_instance`
 
 !!! info "Setting Up TM1 Integration"
     Run `rushti build --tm1-instance <instance>` to create the required dimensions and cube automatically before enabling `push_results`.
@@ -293,7 +294,7 @@ Copy this template to `config/settings.ini` and uncomment the settings you want
 # To set up TM1 integration:
 # 1. Run: rushti build --tm1-instance tm1srv01
 #    This creates the required dimensions and cube automatically.
-# 2. Set push_results = true and configure default_tm1_instance below
+# 2. Set push_results = true and configure tm1_instance below
 #
 # Default: false
 # push_results = false
@@ -308,6 +309,11 @@ Copy this template to `config/settings.ini` and uncomment the settings you want
 
 # Default TM1 instance for reading taskfiles and writing results
 # Must be defined in config.ini
+# tm1_instance = tm1srv01
+
+# DEPRECATED: use tm1_instance above. Honoured indefinitely as the final
+# fallback in the resolution chain; a one-shot DEPRECATION warning fires
+# only when this key is the value actually being used.
 # default_tm1_instance = tm1srv01
 
 # Default cube name for task definitions and results

docs/features/tm1-integration.md

@@ -148,7 +148,7 @@ retention_days = 90
 
 [tm1_integration]
 push_results = true
-default_tm1_instance = tm1srv01
+tm1_instance = tm1srv01
 default_rushti_cube = rushti
 ```
 
@@ -163,7 +163,7 @@ To automatically load the CSV into the cube after each push, enable `auto_load_r
 [tm1_integration]
 push_results = true
 auto_load_results = true
-default_tm1_instance = tm1srv01
+tm1_instance = tm1srv01
 default_rushti_cube = rushti
 ```
 
@@ -198,14 +198,66 @@ All TM1 integration settings live in the `[tm1_integration]` section of `config/
 |---------|---------|-------------|
 | `push_results` | `false` | Push a results CSV to TM1 after each run |
 | `auto_load_results` | `false` | Automatically call `}rushti.load.results` to load the CSV into the cube |
-| `default_tm1_instance` | *(none)* | TM1 instance to use for reading/writing cube data |
+| `tm1_instance` | *(none)* | Default results-push target. Resolved against the [4-tier chain](#per-workflow-target-instance) — CLI `--tm1-instance` > taskfile `settings.tm1_instance` > this key > `default_tm1_instance` (deprecated). |
+| `default_tm1_instance` | *(none)* | **Deprecated** alias for `tm1_instance`. Honoured as the final fallback; warns at load only when actually used. |
 | `default_rushti_cube` | `rushti` | Name of the cube to use |
 | `dim_workflow` | `rushti_workflow` | Workflow dimension name |
 | `dim_task_id` | `rushti_task_id` | Task ID dimension name |
 | `dim_run_id` | `rushti_run_id` | Run ID dimension name |
 
 ---
 
+### Per-workflow target instance
+
+The `tm1_instance` value that receives the results push is resolved against a four-tier precedence chain. The first non-empty value wins:
+
+| Tier | Source | Where it's set |
+|------|--------|----------------|
+| 1 | CLI | `rushti run --tm1-instance <name>` |
+| 2 | Taskfile JSON | `settings.tm1_instance` inside the JSON task file |
+| 3 | settings.ini (canonical) | `[tm1_integration].tm1_instance` |
+| 4 | settings.ini (deprecated) | `[tm1_integration].default_tm1_instance` |
+
+Tier 2 is the new workflow-level override — drop `tm1_instance` into any JSON task file's `settings` block to push that workflow's results to a specific instance without touching `settings.ini`:
+
+```json
+{
+  "version": "2.0",
+  "metadata": { "workflow": "daily-finance-close" },
+  "settings": {
+    "push_results": true,
+    "auto_load_results": true,
+    "tm1_instance": "tm1prod"
+  },
+  "tasks": [ /* ... */ ]
+}
+```
+
+When the upload runs, RushTI logs which tier supplied the value:
+
+```
+INFO  Result upload target: tm1prod (source: taskfile)
+```
+
+If all four tiers are empty and `push_results` is on, the run still succeeds — RushTI logs a warning and skips the upload:
+
+```
+WARNING  push_results enabled but no TM1 instance configured (CLI --tm1-instance, taskfile settings.tm1_instance, or settings.ini tm1_instance).
+```
+
+!!! info "Task-level `instance` vs workflow-level `tm1_instance`"
+    Two `instance`-flavoured fields live at different nesting levels inside a JSON task file:
+
+    - **`tasks[*].instance`** — *task-level execution target.* Where the individual TI process runs. Required on every task.
+    - **`settings.tm1_instance`** — *workflow-level results target.* Where the run's results are pushed at the end. Optional; overrides settings.ini.
+
+    They are independent. A workflow can execute its tasks across several instances (one per task) and push the consolidated results back to a single target instance.
+
+!!! warning "Cube-sourced taskfiles have no tier 2"
+    When a taskfile is loaded from the TM1 cube via `rushti run --tm1-instance X --workflow Y`, the cube schema does not carry a `settings` block — only task definitions. Resolution skips tier 2 and falls through directly to settings.ini (tiers 3 and 4). The CLI value (tier 1) still acts as both the source for the cube read and, in absence of other settings, the fallback results target.
+
+---
+
 ## Build Command Options
 
 ```bash