Skip to content

Commit 6097bd6

Browse files
feat: per-workflow tm1_instance with 4-tier resolution
Introduce a canonical `tm1_instance` setting for the results-push target, resolvable per workflow via the taskfile JSON `settings` block. The deprecated `default_tm1_instance` is honoured as a final fallback and warns only when it is the value actually being used. Precedence (highest wins): 1. CLI `run --tm1-instance` 2. Taskfile JSON `settings.tm1_instance` 3. `settings.ini [tm1_integration].tm1_instance` 4. `settings.ini [tm1_integration].default_tm1_instance` (deprecated) Wiring fix: `get_effective_settings(..., json_settings=...)` now actually runs after the taskfile is parsed, so `push_results` and `auto_load_results` set inside taskfile JSON take effect. This is a behavioural change called out in CHANGELOG — taskfile-JSON values that were previously silently ignored will now win over `settings.ini`. `rushti tasks push --target-tm1-instance` is aliased to `--tm1-instance`; the legacy flag continues to work but logs a DEPRECATION warning on use. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent 9504ea6 commit 6097bd6

14 files changed

Lines changed: 756 additions & 120 deletions

File tree

CHANGELOG.md

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,29 @@
22

33
All notable changes to RushTI are documented in this file.
44

5+
## Unreleased
6+
7+
- **Per-workflow `tm1_instance` setting** for results push and auto-load.
8+
Set it inside a JSON taskfile's `settings` block to override the
9+
`settings.ini` default per workflow. Resolution chain (highest wins):
10+
CLI `--tm1-instance` > taskfile `settings.tm1_instance` >
11+
`settings.ini [tm1_integration].tm1_instance` >
12+
`settings.ini [tm1_integration].default_tm1_instance` (deprecated).
13+
RushTI now logs which tier supplied the target at push time.
14+
- **Behavioural change:** `push_results` and `auto_load_results` set
15+
inside taskfile JSON now take effect. They were silently ignored in
16+
the `run` path before this release. If you have divergent values
17+
between taskfile JSON and `settings.ini`, reconcile them before
18+
upgrading — the taskfile value will now win.
19+
- **Soft deprecation:** `settings.ini [tm1_integration].default_tm1_instance`.
20+
Rename to `tm1_instance`. The old key is honoured indefinitely as the
21+
final fallback in the resolution chain; a one-shot `DEPRECATION:`
22+
warning fires at settings load only when it's the value actually being
23+
used.
24+
- **Soft deprecation:** `rushti tasks push --target-tm1-instance`. Use
25+
`--tm1-instance` instead. The legacy flag is aliased and continues to
26+
work; a `DEPRECATION:` warning fires on use.
27+
528
## Unreleased — `feat/issue-146-detailed-results`
629

730
- Add `--detailed-results` for per-execution cube rows (closes #146).

CONTEXT.md

Lines changed: 106 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,106 @@
1+
# RushTI — Domain Context
2+
3+
This file captures the canonical vocabulary used in RushTI. Terms here are
4+
load-bearing: they appear in code, docs, settings, and user-facing messages.
5+
When a term feels ambiguous in conversation, anchor it to a definition here.
6+
7+
> Scope: single context. No `CONTEXT-MAP.md` — the whole repository shares
8+
> this vocabulary.
9+
10+
---
11+
12+
## Core terms
13+
14+
### Taskfile
15+
The structured input that drives a `rushti run`. Materialised as either:
16+
- a **JSON task file** (`.json`) parsed by `parse_json_taskfile`, OR
17+
- a **TXT task file** (`.txt`) converted on read by `convert_txt_to_json`, OR
18+
- a **TM1 cube taskfile** read by `read_taskfile_from_tm1` (an MDX query
19+
against the `rushti` cube using the `Input` element of `rushti_run_id`).
20+
21+
The in-memory shape is the `Taskfile` dataclass (`taskfile.py`) — a container
22+
with `version`, `metadata`, `settings`, and `tasks`. Always prefer
23+
**"taskfile"** (one word, lowercase) in docs; **"JSON task file"** is also
24+
acceptable when the format matters.
25+
26+
Do **not** call it `workflow.json` — that name implies the file *is* a
27+
workflow, which conflates the container with one of its metadata fields.
28+
29+
### Workflow
30+
The logical identifier for a run, stored in `metadata.workflow` of a taskfile
31+
and as an element of the `rushti_workflow` dimension. A taskfile contains
32+
tasks *for* one workflow.
33+
34+
Use **"workflow"** as an adjective for scope ("workflow-level setting",
35+
"per-workflow override") rather than as a synonym for "taskfile".
36+
37+
### Task
38+
A single TI process execution inside a taskfile. Each task has:
39+
- `id` — a positive integer string, used as an element name in
40+
`rushti_task_id`.
41+
- `instance`**the TM1 instance where this task executes** (task-level).
42+
- `process` — the TI process name on that instance.
43+
- `parameters`, `predecessors`, `stage`, `timeout`, etc.
44+
45+
### TM1 instance
46+
A named TM1 server defined in `config.ini`. Three roles can apply
47+
contextually — context disambiguates, not the name:
48+
49+
| Role | Meaning | Where it shows up |
50+
|---|---|---|
51+
| **Source** | Where a taskfile is read *from* (cube source) | `rushti run --tm1-instance X` |
52+
| **Execution target** | Where a task executes | task-level `instance` field |
53+
| **Results target** | Where execution results are pushed | settings.ini or taskfile `tm1_instance`; also CLI `--tm1-instance` as fallback |
54+
55+
The canonical setting key for the results target is `tm1_instance` (in both
56+
`settings.ini [tm1_integration]` and the taskfile `settings` block). The
57+
deprecated alias is `default_tm1_instance`.
58+
59+
There is no separate "source TM1 instance" override — `--tm1-instance` is
60+
both the source and (in absence of higher-precedence overrides) the results
61+
target for a `run` invocation.
62+
63+
### Source vs target — naming convention
64+
RushTI does not prefix instance names with `source_` or `target_`. The
65+
*command* or the *config section* makes the role obvious:
66+
- `rushti run --tm1-instance X` — X is the source for cube-read, and the
67+
fallback results target if nothing else is set.
68+
- `rushti tasks push --tm1-instance X` — X is the target of the push
69+
(canonical form). The legacy alias `--target-tm1-instance` is deprecated.
70+
- `[tm1_integration].tm1_instance` — the section header makes "target" clear.
71+
72+
This convention is intentional: prefixes don't scale across surfaces and
73+
fight against contextual disambiguation. See
74+
[[adr/0001-tm1-instance-resolution]] for the full rationale.
75+
76+
---
77+
78+
## Settings precedence
79+
80+
The effective value for a settings-driven knob is resolved in this order
81+
(highest wins):
82+
83+
1. **CLI arguments** — e.g. `--max-workers`, `--tm1-instance`.
84+
2. **Taskfile settings block** — e.g. `{"settings": {"max_workers": 8}}`
85+
in JSON. Applied via `_apply_json_settings` inside
86+
`get_effective_settings` *after* the taskfile is parsed.
87+
3. **`settings.ini`** — the canonical defaults file at
88+
`config/settings.ini`.
89+
4. **Built-in defaults** — the dataclass field defaults in
90+
`settings.py`.
91+
92+
Three knobs are *not* settings-driven and don't follow this chain:
93+
- Per-task `instance` and `process` — taskfile-only, no fallback.
94+
- TM1 connection parameters — `config.ini` only.
95+
96+
---
97+
98+
## Files of record
99+
100+
| File | Purpose |
101+
|---|---|
102+
| `config/settings.ini` | Execution defaults (user-editable). |
103+
| `config/settings.ini.template` | Documented example, shipped in repo. |
104+
| `config/config.ini` | TM1 connection parameters per instance. |
105+
| `archive/{workflow}/{run_id}.json` | Snapshot of the taskfile actually executed, for audit + DAG reconstruction. |
106+
| `data/rushti_stats.db` | Local SQLite stats database (when `[stats] enabled = true`). |

config/settings.ini.template

Lines changed: 13 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -113,7 +113,7 @@
113113
# To set up TM1 integration:
114114
# 1. Run: python rushti.py build --tm1-instance tm1srv01
115115
# This creates the required dimensions and cube automatically.
116-
# 2. Set push_results = true and configure default_tm1_instance below
116+
# 2. Set push_results = true and configure tm1_instance below
117117
#
118118
# Default: false
119119
# push_results = false
@@ -126,9 +126,18 @@
126126
# Default: false
127127
# auto_load_results = false
128128

129-
# Default TM1 instance for reading taskfiles and writing results
130-
# Used when --tm1-instance is specified without an instance, and for auto-upload
131-
# Must be defined in config.ini
129+
# TM1 instance used as the results-push target when no higher-precedence
130+
# override is set. Resolved against a 4-tier chain (highest wins):
131+
# 1. CLI --tm1-instance
132+
# 2. Taskfile JSON settings.tm1_instance
133+
# 3. This key
134+
# 4. default_tm1_instance below (deprecated)
135+
# Must be defined in config.ini.
136+
# tm1_instance = tm1srv01
137+
138+
# DEPRECATED: use tm1_instance above. Still honoured indefinitely as the
139+
# final fallback; a one-shot DEPRECATION warning fires at settings load
140+
# *only* when tm1_instance is unset and this key is the value being used.
132141
# default_tm1_instance = tm1srv01
133142

134143
# Default cube name for task definitions and results

docs/advanced/cli-reference.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -235,8 +235,8 @@ rushti tasks push --tasks daily-etl.json --tm1-instance tm1srv01
235235
| Option | Short | Type | Description |
236236
|--------|-------|------|-------------|
237237
| `--tasks` | `-t` | PATH | Local JSON task file to push (*required*) |
238-
| `--tm1-instance` | | STR | Source TM1 instance (if loading from TM1) |
239-
| `--target-tm1-instance` | | STR | Target TM1 instance for the push |
238+
| `--tm1-instance` | | STR | TM1 instance to push the taskfile to. Context disambiguates the role — on `tasks push` this is the destination. |
239+
| `--target-tm1-instance` | | STR | **Deprecated alias** for `--tm1-instance`. Still works; emits a `DEPRECATION:` warning when used. |
240240
| `--settings` | `-s` | PATH | Path to `settings.ini` |
241241

242242
---

docs/advanced/settings-reference.md

Lines changed: 9 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -120,7 +120,8 @@ Controls TM1-based read/write integration: reading task files from a TM1 cube an
120120
| `push_results` | bool | `false` | Upload the results CSV to the TM1 Applications folder after each run. The file is named `rushti_{workflow}_{run_id}.csv` (with `.blb` extension for TM1 < v12). |
121121
| `auto_load_results` | bool | `false` | After uploading results (requires `push_results = true`), call the `}rushti.load.results` TI process on the target TM1 instance to load the CSV into the rushti cube. Passes `pSourceFile` and `pTargetCube` parameters. The process must exist on the target instance. |
122122
| `detailed_results` | bool | `false` | Emit one cube row per executed TI when pushing results, instead of summarizing expanded tasks (only meaningful with `push_results = true`). Each row gets a fresh sequential `task_id`; the original IDs are preserved in the new `original_task_id` measure. See [TM1 integration: detailed results](../features/tm1-integration.md#detailed-results). |
123-
| `default_tm1_instance` | str | *(none)* | Default TM1 instance name (from `config.ini`) used for reading task files and writing results. Required when `push_results` is enabled. |
123+
| `tm1_instance` | str | *(none)* | TM1 instance (from `config.ini`) used as the **results-push target** when no higher-precedence override is set. Resolved per [the 4-tier chain](../features/tm1-integration.md#per-workflow-target-instance): CLI `--tm1-instance` > taskfile `settings.tm1_instance` > this key > `default_tm1_instance` (deprecated). Required when `push_results` is enabled and no higher tier is set. |
124+
| `default_tm1_instance` | str | *(none)* | **Deprecated** — use `tm1_instance`. Honoured indefinitely as the final fallback in the resolution chain; a one-shot `DEPRECATION:` warning fires at settings load *only* when it's actually being used (i.e. `tm1_instance` is unset). |
124125
| `default_rushti_cube` | str | `rushti` | Name of the TM1 cube for task definitions and execution results. Created by the `rushti build` command. |
125126
| `default_workflow_dim` | str | `rushti_workflow` | Dimension name for workflow identifiers. |
126127
| `default_task_id_dim` | str | `rushti_task_id` | Dimension name for task sequence elements (1--5000 default elements). |
@@ -131,7 +132,7 @@ Controls TM1-based read/write integration: reading task files from a TM1 cube an
131132

132133
**Overridable via CLI:** `--detailed-results`
133134

134-
**Overridable via JSON task file:** `push_results`, `auto_load_results`, `detailed_results`
135+
**Overridable via JSON task file:** `push_results`, `auto_load_results`, `detailed_results`, `tm1_instance`
135136

136137
!!! info "Setting Up TM1 Integration"
137138
Run `rushti build --tm1-instance <instance>` to create the required dimensions and cube automatically before enabling `push_results`.
@@ -293,7 +294,7 @@ Copy this template to `config/settings.ini` and uncomment the settings you want
293294
# To set up TM1 integration:
294295
# 1. Run: rushti build --tm1-instance tm1srv01
295296
# This creates the required dimensions and cube automatically.
296-
# 2. Set push_results = true and configure default_tm1_instance below
297+
# 2. Set push_results = true and configure tm1_instance below
297298
#
298299
# Default: false
299300
# push_results = false
@@ -308,6 +309,11 @@ Copy this template to `config/settings.ini` and uncomment the settings you want
308309

309310
# Default TM1 instance for reading taskfiles and writing results
310311
# Must be defined in config.ini
312+
# tm1_instance = tm1srv01
313+
314+
# DEPRECATED: use tm1_instance above. Honoured indefinitely as the final
315+
# fallback in the resolution chain; a one-shot DEPRECATION warning fires
316+
# only when this key is the value actually being used.
311317
# default_tm1_instance = tm1srv01
312318

313319
# Default cube name for task definitions and results

docs/features/tm1-integration.md

Lines changed: 55 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -148,7 +148,7 @@ retention_days = 90
148148

149149
[tm1_integration]
150150
push_results = true
151-
default_tm1_instance = tm1srv01
151+
tm1_instance = tm1srv01
152152
default_rushti_cube = rushti
153153
```
154154

@@ -163,7 +163,7 @@ To automatically load the CSV into the cube after each push, enable `auto_load_r
163163
[tm1_integration]
164164
push_results = true
165165
auto_load_results = true
166-
default_tm1_instance = tm1srv01
166+
tm1_instance = tm1srv01
167167
default_rushti_cube = rushti
168168
```
169169

@@ -198,14 +198,66 @@ All TM1 integration settings live in the `[tm1_integration]` section of `config/
198198
|---------|---------|-------------|
199199
| `push_results` | `false` | Push a results CSV to TM1 after each run |
200200
| `auto_load_results` | `false` | Automatically call `}rushti.load.results` to load the CSV into the cube |
201-
| `default_tm1_instance` | *(none)* | TM1 instance to use for reading/writing cube data |
201+
| `tm1_instance` | *(none)* | Default results-push target. Resolved against the [4-tier chain](#per-workflow-target-instance) — CLI `--tm1-instance` > taskfile `settings.tm1_instance` > this key > `default_tm1_instance` (deprecated). |
202+
| `default_tm1_instance` | *(none)* | **Deprecated** alias for `tm1_instance`. Honoured as the final fallback; warns at load only when actually used. |
202203
| `default_rushti_cube` | `rushti` | Name of the cube to use |
203204
| `dim_workflow` | `rushti_workflow` | Workflow dimension name |
204205
| `dim_task_id` | `rushti_task_id` | Task ID dimension name |
205206
| `dim_run_id` | `rushti_run_id` | Run ID dimension name |
206207

207208
---
208209

210+
### Per-workflow target instance
211+
212+
The `tm1_instance` value that receives the results push is resolved against a four-tier precedence chain. The first non-empty value wins:
213+
214+
| Tier | Source | Where it's set |
215+
|------|--------|----------------|
216+
| 1 | CLI | `rushti run --tm1-instance <name>` |
217+
| 2 | Taskfile JSON | `settings.tm1_instance` inside the JSON task file |
218+
| 3 | settings.ini (canonical) | `[tm1_integration].tm1_instance` |
219+
| 4 | settings.ini (deprecated) | `[tm1_integration].default_tm1_instance` |
220+
221+
Tier 2 is the new workflow-level override — drop `tm1_instance` into any JSON task file's `settings` block to push that workflow's results to a specific instance without touching `settings.ini`:
222+
223+
```json
224+
{
225+
"version": "2.0",
226+
"metadata": { "workflow": "daily-finance-close" },
227+
"settings": {
228+
"push_results": true,
229+
"auto_load_results": true,
230+
"tm1_instance": "tm1prod"
231+
},
232+
"tasks": [ /* ... */ ]
233+
}
234+
```
235+
236+
When the upload runs, RushTI logs which tier supplied the value:
237+
238+
```
239+
INFO Result upload target: tm1prod (source: taskfile)
240+
```
241+
242+
If all four tiers are empty and `push_results` is on, the run still succeeds — RushTI logs a warning and skips the upload:
243+
244+
```
245+
WARNING push_results enabled but no TM1 instance configured (CLI --tm1-instance, taskfile settings.tm1_instance, or settings.ini tm1_instance).
246+
```
247+
248+
!!! info "Task-level `instance` vs workflow-level `tm1_instance`"
249+
Two `instance`-flavoured fields live at different nesting levels inside a JSON task file:
250+
251+
- **`tasks[*].instance`** — *task-level execution target.* Where the individual TI process runs. Required on every task.
252+
- **`settings.tm1_instance`** — *workflow-level results target.* Where the run's results are pushed at the end. Optional; overrides settings.ini.
253+
254+
They are independent. A workflow can execute its tasks across several instances (one per task) and push the consolidated results back to a single target instance.
255+
256+
!!! warning "Cube-sourced taskfiles have no tier 2"
257+
When a taskfile is loaded from the TM1 cube via `rushti run --tm1-instance X --workflow Y`, the cube schema does not carry a `settings` block — only task definitions. Resolution skips tier 2 and falls through directly to settings.ini (tiers 3 and 4). The CLI value (tier 1) still acts as both the source for the cube read and, in absence of other settings, the fallback results target.
258+
259+
---
260+
209261
## Build Command Options
210262

211263
```bash

0 commit comments

Comments
 (0)