Skip to content

Commit b007c2a

Browse files
nicolasbisurginicolasbisurgi
authored
RushTI 2.0 — Complete rewrite with DAG execution, self-optimization, and TM1 integration (#112)
* RushTI 2.0 — Complete rewrite with DAG execution, self-optimization, and TM1 integration RushTI 2.0 is a ground-up rewrite of the parallel TI execution engine for IBM Planning Analytics. ## What's New - **DAG Execution** — True dependency-based scheduling replaces level/wait-based sequencing. Tasks start the moment their predecessors complete, maximizing parallelism. - **JSON Task Files** — Structured format with metadata, settings, stages, and per-task options alongside full backwards compatibility with legacy TXT files. - **Self-Optimization** — EWMA-based learning algorithm analyzes execution history and continuously reorders task scheduling for faster runs. - **Checkpoint & Resume** — Automatic progress saving with failure recovery. Resume from where you left off after crashes or interruptions. - **Exclusive Mode** — Prevents concurrent RushTI runs on shared TM1 servers using session-based locking. - **SQLite Statistics** — Persistent execution history with HTML dashboards, trend analysis, and CSV export. - **TM1 Cube Integration** — Read task definitions from and write execution results to TM1 cubes. Manage workflows entirely within Planning Analytics. - **Safe Retry** — Automatic reconnection on TM1 disconnects with configurable retry logic. - **Task-Level Timeouts** — Per-task timeout and cancel_at_timeout support for deadlock prevention. - **Taskfile Archiving** — Automatic JSON snapshots of each run for audit trails. - **Multi-Instance** — Distribute tasks across multiple TM1 servers. - **100% Backwards Compatible** — Legacy TXT task files work without any changes. ## Architecture - Modular package structure under src/rushti/ with clean separation of concerns - CLI with subcommands (run, tasks, stats, db, tm1) via argparse - DAG domain model with topological sorting and cycle detection - Thread-based parallel execution engine with ExecutionContext - Pre-commit hooks with Black and Ruff for consistent code style - 409 unit tests, comprehensive integration test suite Closes #89 Closes #93
1 parent 0f65e8e commit b007c2a

190 files changed

Lines changed: 69258 additions & 1905 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/build.yml‎

Lines changed: 206 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,206 @@
1+
# RushTI Build Workflow
2+
#
3+
# Builds Windows executable when Python/application code changes.
4+
# Only triggers after tests pass or on manual dispatch.
5+
6+
name: Build
7+
8+
on:
9+
push:
10+
branches: [master, rushti2dot0]
11+
paths:
12+
- 'src/**'
13+
- 'pyproject.toml'
14+
- '*.py'
15+
- '*.spec'
16+
- 'config/**'
17+
- 'assets/**'
18+
- '.github/workflows/build.yml'
19+
workflow_dispatch: # Allow manual triggering
20+
workflow_run:
21+
workflows: ["Tests"]
22+
types: [completed]
23+
branches: [master, rushti2dot0]
24+
25+
jobs:
26+
build-windows:
27+
name: Build Windows Executable
28+
runs-on: windows-latest
29+
# Only build if:
30+
# - Tests workflow succeeded, OR
31+
# - Manually triggered, OR
32+
# - Direct push (tests will run in parallel via path filter)
33+
if: |
34+
(github.event_name == 'workflow_run' && github.event.workflow_run.conclusion == 'success') ||
35+
github.event_name == 'workflow_dispatch' ||
36+
github.event_name == 'push'
37+
38+
steps:
39+
- name: Checkout code
40+
uses: actions/checkout@v6
41+
42+
- name: Set up Python
43+
uses: actions/setup-python@v6
44+
with:
45+
python-version: '3.11'
46+
47+
- name: Install Python dependencies
48+
run: |
49+
python -m pip install --upgrade pip
50+
pip install -e .
51+
pip install pyinstaller>=6.0.0
52+
shell: bash
53+
54+
- name: Build executable
55+
run: |
56+
pyinstaller rushti.spec --clean
57+
shell: bash
58+
59+
- name: Verify build
60+
run: |
61+
echo "Build output structure (onedir mode):"
62+
ls -la dist/
63+
ls -la dist/rushti/
64+
shell: bash
65+
66+
- name: Test executable
67+
run: |
68+
echo "Testing rushti.exe..."
69+
dist/rushti/rushti.exe --help
70+
if [ $? -eq 0 ]; then
71+
echo "✓ Executable runs successfully"
72+
else
73+
echo "✗ Executable failed to run"
74+
exit 1
75+
fi
76+
shell: bash
77+
78+
- name: Create release package
79+
run: |
80+
mkdir -p release/rushti-windows
81+
82+
# Copy the entire onedir distribution (includes exe and all dependencies)
83+
cp -r dist/rushti/* release/rushti-windows/
84+
85+
# Create config directory structure
86+
mkdir -p release/rushti-windows/config
87+
88+
# Copy configuration templates if they exist
89+
if [ -f "config/config.ini.template" ]; then
90+
cp config/config.ini.template release/rushti-windows/config/
91+
fi
92+
if [ -f "config/logging_config.ini" ]; then
93+
cp config/logging_config.ini release/rushti-windows/config/
94+
fi
95+
if [ -f "config/settings.ini.template" ]; then
96+
cp config/settings.ini.template release/rushti-windows/config/
97+
fi
98+
99+
# Copy assets directory (required for 'build' command to create TM1 objects)
100+
# Note: Assets are also bundled in the distribution, but including them in the
101+
# release allows users to customize the TM1 object definitions if needed
102+
if [ -d "assets" ]; then
103+
cp -r assets release/rushti-windows/
104+
fi
105+
106+
# Copy README and LICENSE
107+
cp README.md release/rushti-windows/ || true
108+
cp LICENSE release/rushti-windows/ || true
109+
110+
# Create a build info file
111+
echo "RushTI Windows Build" > release/rushti-windows/BUILD_INFO.txt
112+
echo "Version: 2.0.0" >> release/rushti-windows/BUILD_INFO.txt
113+
echo "Build: Complete build with all features including DAG visualization (onedir mode for fast startup)" >> release/rushti-windows/BUILD_INFO.txt
114+
echo "Built: $(date -u +'%Y-%m-%d %H:%M:%S UTC')" >> release/rushti-windows/BUILD_INFO.txt
115+
echo "Python: $(python --version)" >> release/rushti-windows/BUILD_INFO.txt
116+
echo "Commit: ${{ github.sha }}" >> release/rushti-windows/BUILD_INFO.txt
117+
shell: bash
118+
119+
- name: Upload artifact
120+
uses: actions/upload-artifact@v6
121+
with:
122+
name: rushti-windows
123+
path: release/rushti-windows
124+
retention-days: 30
125+
126+
create-release:
127+
name: Create GitHub Release
128+
runs-on: ubuntu-latest
129+
needs: build-windows
130+
if: github.event_name == 'push' && github.ref == 'refs/heads/master'
131+
132+
steps:
133+
- name: Checkout code
134+
uses: actions/checkout@v6
135+
136+
- name: Download all artifacts
137+
uses: actions/download-artifact@v7
138+
with:
139+
path: artifacts
140+
141+
- name: Create release archive
142+
run: |
143+
cd artifacts/rushti-windows
144+
zip -r ../rushti-windows.zip .
145+
cd ..
146+
ls -la
147+
148+
- name: Extract version
149+
id: version
150+
run: |
151+
VERSION=$(grep -oP '__version__\s*=\s*"\K[^"]+' src/rushti/__init__.py)
152+
echo "version=$VERSION" >> $GITHUB_OUTPUT
153+
154+
- name: Create Release
155+
uses: softprops/action-gh-release@v2
156+
with:
157+
tag_name: v${{ steps.version.outputs.version }}
158+
name: RushTI v${{ steps.version.outputs.version }}
159+
body: |
160+
## RushTI v${{ steps.version.outputs.version }}
161+
162+
### Windows Build
163+
164+
**rushti-windows.zip** - Complete build with all features:
165+
- Full TM1 integration and asset management
166+
- Database admin utilities
167+
- Statistics and logging
168+
- DAG visualization support
169+
- Parallel process execution engine
170+
- Fast startup (onedir distribution - no temp extraction delay)
171+
172+
### Installation
173+
174+
1. Download **rushti-windows.zip**
175+
2. Extract to your desired location (keeps directory structure intact)
176+
3. Copy `config/config.ini.template` to `config.ini` and configure your TM1 connections
177+
4. (Optional) Copy `config/settings.ini.template` to `settings.ini` for custom defaults
178+
5. Run `rushti.exe` from command line
179+
180+
> **Note:** This build uses PyInstaller's onedir mode for fast cold starts.
181+
> Keep all files in the extracted directory together - the exe requires the bundled libraries.
182+
183+
### Requirements
184+
185+
- Windows 10/11 or Windows Server 2016+
186+
187+
### Quick Start
188+
189+
```bash
190+
# Run a task file
191+
rushti.exe --tasks tasks.txt
192+
193+
# Visualize DAG (interactive HTML)
194+
rushti.exe visualize --tasks tasks.txt --output dag.html
195+
196+
# Get help
197+
rushti.exe --help
198+
```
199+
200+
See [README.md](https://github.com/cubewise-code/rushti/blob/master/README.md) for complete documentation.
201+
files: |
202+
artifacts/*.zip
203+
draft: false
204+
prerelease: false
205+
env:
206+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

‎.github/workflows/ci.yml‎

Lines changed: 0 additions & 30 deletions
This file was deleted.

‎.github/workflows/deploy-site.yml‎

Lines changed: 87 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,87 @@
1+
# RushTI Website Deployment Workflow
2+
#
3+
# Deploys the React homepage and MkDocs documentation to GitHub Pages.
4+
# Only triggers when website or documentation files change.
5+
6+
name: Deploy Website
7+
8+
on:
9+
push:
10+
branches: [master, main, rushti2dot0]
11+
paths:
12+
- 'site/**'
13+
- 'docs/**'
14+
- 'mkdocs.yml'
15+
- 'docs-requirements.txt'
16+
- '.github/workflows/deploy-site.yml'
17+
workflow_dispatch: # Allow manual triggering
18+
19+
permissions:
20+
contents: read
21+
pages: write
22+
id-token: write
23+
24+
concurrency:
25+
group: "pages"
26+
cancel-in-progress: false
27+
28+
jobs:
29+
build:
30+
runs-on: ubuntu-latest
31+
steps:
32+
- name: Checkout
33+
uses: actions/checkout@v6
34+
35+
- name: Setup Node.js
36+
uses: actions/setup-node@v6
37+
with:
38+
node-version: '20'
39+
cache: 'npm'
40+
cache-dependency-path: site/homepage/package-lock.json
41+
42+
- name: Setup Python
43+
uses: actions/setup-python@v6
44+
with:
45+
python-version: '3.11'
46+
47+
- name: Install MkDocs dependencies
48+
run: pip install -r docs-requirements.txt
49+
50+
- name: Install homepage dependencies
51+
working-directory: site/homepage
52+
run: npm ci
53+
54+
- name: Build homepage (React/Vite)
55+
working-directory: site/homepage
56+
run: npm run build
57+
58+
- name: Build documentation (MkDocs)
59+
run: mkdocs build --site-dir mkdocs_site
60+
61+
- name: Merge builds
62+
run: |
63+
mkdir -p _site
64+
# Copy React build to root
65+
cp -r site/homepage/dist/* _site/
66+
# Copy MkDocs to /docs subdirectory
67+
mkdir -p _site/docs
68+
cp -r mkdocs_site/* _site/docs/
69+
70+
- name: Setup Pages
71+
uses: actions/configure-pages@v5
72+
73+
- name: Upload artifact
74+
uses: actions/upload-pages-artifact@v4
75+
with:
76+
path: '_site'
77+
78+
deploy:
79+
environment:
80+
name: github-pages
81+
url: ${{ steps.deployment.outputs.page_url }}
82+
runs-on: ubuntu-latest
83+
needs: build
84+
steps:
85+
- name: Deploy to GitHub Pages
86+
id: deployment
87+
uses: actions/deploy-pages@v4

‎.github/workflows/publish-pypi.yml‎

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
name: Publish to PyPI
2+
3+
on:
4+
release:
5+
types: [published]
6+
7+
jobs:
8+
build:
9+
name: Build package
10+
runs-on: ubuntu-latest
11+
12+
steps:
13+
- uses: actions/checkout@v6
14+
15+
- name: Set up Python
16+
uses: actions/setup-python@v6
17+
with:
18+
python-version: "3.12"
19+
20+
- name: Install build tools
21+
run: python -m pip install --upgrade pip build
22+
23+
- name: Build package
24+
run: python -m build
25+
26+
- name: Upload build artifacts
27+
uses: actions/upload-artifact@v6
28+
with:
29+
name: dist
30+
path: dist/
31+
32+
publish:
33+
name: Publish to PyPI
34+
needs: build
35+
runs-on: ubuntu-latest
36+
environment: pypi
37+
permissions:
38+
id-token: write
39+
40+
steps:
41+
- name: Download build artifacts
42+
uses: actions/download-artifact@v6
43+
with:
44+
name: dist
45+
path: dist/
46+
47+
- name: Publish to PyPI
48+
uses: pypa/gh-action-pypi-publish@release/v1

0 commit comments

Comments
 (0)