Skip to content

Commit e2faa50

Browse files
author
Sqoia Dev Agent
committed
feat: configurable billing rules engine + complete README rewrite
Billing Rules: - Admin-configurable rules for charge/no-charge/discount decisions - 5 built-in rules (2 enabled by default): - No charge for failed jobs (except OOM and timeout) - No charge for jobs under 1 minute - Debug partition 50% discount (disabled) - Viz partition free (disabled) - OOM charge at requested time (disabled) - 7 condition operators (equals, in, less_than, greater_than, etc.) - Rules evaluated in order, first match wins - Full CRUD UI with reorder, preview affected job count - Billing Status column in job table shows rule applied per job - Custom rules via admin UI — no config editing needed README: - Complete rewrite reflecting SlurmLedger feature set - Feature matrix, role-based access table, architecture diagram - Installation (RPM, DEB, source), configuration reference - Billing rules documentation with defaults table
1 parent 87b21a3 commit e2faa50

4 files changed

Lines changed: 816 additions & 143 deletions

File tree

‎README.md‎

Lines changed: 182 additions & 134 deletions
Original file line numberDiff line numberDiff line change
@@ -1,160 +1,208 @@
11
# SlurmLedger
22

3-
**SlurmLedger** is a Cockpit plugin that integrates seamlessly into the Cockpit UI on Linux servers. It provides interactive billing analytics and invoice management for HPC environments using **Slurm** and SlurmDBD.
4-
5-
This repository includes a responsive Cockpit UI built with React. The interface pulls live billing data from SlurmDBD using a bundled Python helper and offers summary, detail and invoice views with a built‑in PDF viewer.
6-
7-
## ✅ Features
8-
9-
- **Monthly billing summaries** displayed right in Cockpit’s navigation menu.
10-
- **Selectable historical months** lets you view past billing periods for the current year while defaulting to the current month.
11-
- **Invoice dashboards** to view, download, and archive invoice PDFs.
12-
- **Detailed cost drill-downs** (core‑hours, GPU-hours) for per‑account transparency.
13-
- **Historical billing data** accessible from account inception for auditing and trend analysis.
14-
- **Organization-wide views** consolidating charges across all member Slurm accounts; cluster capacity is detected automatically from `slurm.conf`.
15-
- **Configurable rate table** with per-account overrides.
16-
17-
18-
19-
## 📁 Project Structure
20-
21-
```text
22-
SlurmLedger/
23-
├── src/ # Source UI code built with React or modern JS
24-
│ ├── invoices/ # Stored invoice PDFs
25-
│ ├── invoice-schema.json # Invoice metadata schema
26-
│ ├── slurmcostmanager.html # HTML entrypoint loaded inside Cockpit
27-
│ └── slurmcostmanager.js # Frontend plugin logic, using cockpit APIs
28-
├── manifest.json # Cockpit module metadata, menu registration
29-
├── Makefile # Build, devel-install, devel-uninstall, watch, check targets
30-
├── dist/ # Bundled output directory for Cockpit to load
31-
├── test/ # Integration tests using Cockpit test harness
32-
│ └── check-application # Python-based browser tests via DevTools protocol
33-
├── org.cockpit_project.slurmcostmanager.metainfo.xml # Metadata for packaging
34-
├── README.md # Documentation (this file)
35-
└── CONTRIBUTING.md # Guidelines for community contributions
3+
Full-featured HPC billing and allocation management for SLURM clusters. A Cockpit plugin that provides real-time cost tracking, SU bank management, professional invoicing, and financial system integration.
4+
5+
## Features
6+
7+
### Billing & Cost Tracking
8+
- Real-time cost calculation from SlurmDB
9+
- Per-account, per-user, per-job cost breakdown
10+
- CPU and GPU hour tracking with separate rates
11+
- Historical rate support for accurate retroactive billing
12+
- Configurable billing rules (exclude failed jobs, partition discounts, etc.)
13+
14+
### SU Bank / Allocation Management
15+
- Pre-paid and post-paid billing models
16+
- Annual, quarterly, monthly, or custom allocation periods
17+
- Budget tracking with configurable alert thresholds (80%/90%/100%)
18+
- Carryover rules for unused allocations
19+
- Real-time remaining balance display
20+
21+
### Professional Invoicing
22+
- PDF invoice generation with institutional branding
23+
- Invoice lifecycle management (Draft → Sent → Viewed → Paid)
24+
- Refund support with credit memo generation
25+
- Configurable payment terms and bank details
26+
- Bulk invoice generation by account
27+
28+
### Configurable Billing Rules
29+
- Rule-based charge/no-charge decisions
30+
- Failed jobs (except OOM/timeout) excluded by default
31+
- Partition-level discounts (e.g., debug queue at 50%)
32+
- Short job exclusion (under 1 minute)
33+
- Custom rules with flexible condition operators
34+
- Admin-configurable via UI — no code changes needed
35+
36+
### Financial System Integration
37+
- Oracle Financials Cloud GL Journal Import XML
38+
- Workday, Banner, Kuali support via configurable mappings
39+
- Chart of Accounts mapping (SLURM account → Fund/Org/Account/Program)
40+
- Webhook notifications on invoice events
41+
- Journal Entry CSV export for any ERP system
42+
43+
### Role-Based Access
44+
| Feature | Admin | PI | Member | Finance |
45+
|---|---|---|---|---|
46+
| Dashboard | All accounts | Their accounts | Personal usage | All (read-only) |
47+
| Cost Details | All + edit rates | Their accounts | Their jobs | All (read-only) |
48+
| Rate Config | Full edit | View only | Hidden | View only |
49+
| Allocations | Full edit | View balance | View usage | View all |
50+
| Invoices | Full lifecycle | View theirs | Hidden | View + mark paid |
51+
| Billing Rules | Full edit | View only | Hidden | View only |
52+
| Institution | Full edit | Hidden | Hidden | View only |
53+
| Financial Integration | Full config | Hidden | Hidden | View only |
54+
55+
### Reports & Exports
56+
- Monthly/Quarterly/Annual billing summaries
57+
- CSV export (RFC 4180 compliant)
58+
- PDF invoices and credit memos
59+
- Journal entry exports for financial systems
60+
- Historical cost trend charts
61+
- Job efficiency metrics
62+
63+
## Screenshots
64+
65+
*(Screenshots to be added)*
66+
67+
## Requirements
68+
69+
- Cockpit ≥ 300
70+
- Python 3.8+
71+
- SLURM with SlurmDBD (MySQL/MariaDB backend)
72+
- pymysql
73+
- reportlab
74+
75+
## Installation
76+
77+
### From RPM (RHEL/Rocky/Alma)
78+
```bash
79+
sudo dnf install slurmledger-1.0.0-1.noarch.rpm
80+
sudo systemctl try-restart cockpit
3681
```
3782

38-
## 🧰 Installation & Development
39-
40-
Recommend using the **Cockpit Starter Kit** workflow to scaffold and build your plugin:
41-
42-
- Use `make devel-install` to symlink your dist output into `~/.local/share/cockpit/` for live development.
43-
- Use `make devel-uninstall` when finished to remove the development symlink.
44-
- Optionally run `make watch` to rebuild and reinstall whenever files in `src/` change (requires `inotifywait`).
45-
- Run `make build` or `make` to compile and prepare for release.
46-
- Use `make check` to run integration tests via Cockpit's VM-based test system.
47-
Cockpit’s `manifest.json` registers your tool under the main menu. Your UI files will live in `src/`, built via webpack into `dist/`.
48-
49-
## 📦 Packaging
50-
51-
Build RPM and DEB packages for distribution:
52-
83+
### From DEB (Ubuntu/Debian)
5384
```bash
54-
make rpm # writes an RPM to rpmbuild/RPMS/
55-
make deb # creates slurmcostmanager_<version>_all.deb
85+
sudo dpkg -i slurmledger_1.0.0-1_all.deb
86+
sudo apt-get install -f # install dependencies
87+
sudo systemctl try-restart cockpit
5688
```
5789

58-
Both targets generate `org.cockpit_project.slurmcostmanager.metainfo.xml` and bundle `manifest.json` so the packages can be installed on RPM or DEB based systems.
59-
60-
### Manual verification
61-
62-
1. Run `make devel-install` and confirm that `~/.local/share/cockpit/slurmcostmanager` is a symlink.
63-
2. Open Cockpit at `https://<host>:9090` and verify the **SlurmLedger** entry appears.
64-
3. When done developing, execute `make devel-uninstall` to remove the symlink.
65-
66-
## 🧭 Versioning and Releases
67-
68-
The project follows [Semantic Versioning](https://semver.org/). All notable changes are recorded in [CHANGELOG.md](CHANGELOG.md) and upgrade notes live in [UPGRADING.md](UPGRADING.md).
69-
70-
To cut a new release:
71-
72-
1. Bump the version in `manifest.json` and update the changelog and upgrade guide.
73-
2. Trigger the **Release** workflow from the GitHub Actions tab and supply the new version number.
74-
75-
The workflow tags the repository, builds packages, and publishes artifacts to GitHub Releases automatically.
76-
77-
## 🌐 Usage
78-
79-
1. On a Linux host with **Cockpit** installed (e.g. CentOS, Fedora, Debian compatible).
80-
2. After `make devel-install`, open your browser to `https://<host>:9090`.
81-
3. Locate **SlurmLedger** in the sidebar menu.
82-
4. Interact with billing summaries, drill-ins, invoice retrieval, and configure rates directly within Cockpit.
83-
84-
### Fetching real Slurm usage
85-
86-
The `src/slurmdb.py` utility can connect to a running **SlurmDBD** instance and
87-
export usage metrics as JSON. Connection details are automatically scraped from
88-
`slurmdbd.conf` located alongside `slurm.conf` (discovered from
89-
`slurmctld.service` via the `ConditionPathExists` directive, defaulting to
90-
`/etc/slurm/slurm.conf`). A custom path can be specified via the environment
91-
variable `SLURMDB_CONF` or the `--conf` flag. Environment variables
92-
`SLURMDB_HOST`, `SLURMDB_PORT`, `SLURMDB_USER`, `SLURMDB_PASS` and `SLURMDB_DB`
93-
override any values found in the configuration file. The cluster prefix used to
94-
select the job tables is determined from the Slurm configuration file. The
95-
setting can be overridden using `SLURM_CLUSTER`, `--cluster` or `--slurm-conf`.
96-
97-
90+
### From Source
9891
```bash
99-
python3 src/slurmdb.py --start 2024-06-01 --end 2024-06-30 --output billing.json
100-
# optional custom config path
101-
# python3 src/slurmdb.py --start ... --end ... --conf /path/to/slurmdbd.conf --cluster localcluster
92+
git clone https://github.com/NessieCanCode/SlurmLedger.git
93+
cd SlurmLedger
94+
pip install -r requirements.txt
95+
make build
96+
sudo make install
10297
```
10398

104-
The resulting `billing.json` file mirrors the structure expected by the
105-
frontend and can be used for local development or offline snapshots.
106-
107-
#### Automatic daily exports
108-
109-
To export usage one day at a time, the helper can remember the last processed
110-
date in `~/.local/share/slurmledger/last_run.json`. Running with `--auto-daily`
111-
without `--start`/`--end` processes the next unexported day and writes a JSON
112-
file per day into the chosen directory:
113-
99+
### Development
114100
```bash
115-
python3 src/slurmdb.py --auto-daily --output daily-reports/
101+
make devel-install # Links to ~/.local/share/cockpit/
116102
```
117103

118-
If multiple days were missed, each unprocessed day is exported until caught up.
119-
After a successful export the state file is updated so subsequent runs pick up
120-
where they left off.
104+
## Configuration
105+
106+
### Rates (`/etc/slurmledger/rates.json`)
107+
```json
108+
{
109+
"defaultRate": 0.01,
110+
"defaultGpuRate": 0.10,
111+
"overrides": {
112+
"physics-lab": { "rate": 0.008, "gpuRate": 0.08, "discount": 0.1 }
113+
},
114+
"billing_rules": [...],
115+
"allocations": {...},
116+
"billing_defaults": {
117+
"type": "postpaid",
118+
"billing_period": "monthly",
119+
"payment_terms_days": 30
120+
}
121+
}
122+
```
121123

122-
### Inspecting the database schema
124+
### Institution Profile (`/etc/slurmledger/institution.json`)
125+
Configure your institution's name, address, logo, bank details, payment terms, and financial system integration through the UI.
126+
127+
### Roles
128+
Configure role assignments in `institution.json`:
129+
```json
130+
{
131+
"roles": {
132+
"admins": ["root", "hpc-admin"],
133+
"finance": ["billing-dept"],
134+
"pis": []
135+
}
136+
}
137+
```
138+
PIs are auto-detected from SLURM account coordinator assignments.
123139

124-
If you need to see which tables and columns are present in your Slurm
125-
accounting database, run the helper script `src/slurm_schema.py`. It
126-
uses the same connection options as `slurmdb.py` and writes a JSON
127-
mapping of tables to their columns.
140+
### Billing Rules
141+
Rules are evaluated in order — first matching rule wins:
128142

129-
```bash
130-
python3 src/slurm_schema.py --output schema.json
131-
# python3 src/slurm_schema.py --conf /path/to/slurmdbd.conf --cluster localcluster
132-
```
143+
| Rule | Default | Description |
144+
|---|---|---|
145+
| No charge for failed jobs | ✅ Enabled | Failed/cancelled jobs not charged (except OOM and timeout) |
146+
| No charge under 1 minute | ✅ Enabled | Very short jobs excluded |
147+
| Debug partition discount | ❌ Disabled | 50% rate on debug partition |
148+
| Viz partition free | ❌ Disabled | No charge for visualization partition |
149+
| OOM charge at requested time | ❌ Disabled | OOM jobs charged for requested, not actual time |
133150

134-
The resulting `schema.json` file can be compared with the list of
135-
tables and columns from your deployment.
151+
Create custom rules via the admin UI — no config file editing required.
136152

137-
### Testing with sample SlurmDB data
153+
## Architecture
138154

139-
For unit tests and local development the repository includes two fixtures
140-
under `test/`:
155+
```
156+
┌─────────────────────────────────────┐
157+
│ Cockpit Web UI │
158+
│ (React, Chart.js, jsPDF) │
159+
│ │
160+
│ ┌─────────┐ ┌──────────┐ ┌──────┐ │
161+
│ │Dashboard│ │ Invoices │ │Rates │ │
162+
│ │(by role)│ │lifecycle │ │rules │ │
163+
│ └────┬────┘ └────┬─────┘ └──┬───┘ │
164+
│ │ │ │ │
165+
│ ▼ ▼ ▼ │
166+
│ cockpit.spawn() / cockpit.file() │
167+
└───────────────┬─────────────────────┘
168+
│
169+
┌───────────┼───────────────┐
170+
▼ ▼ ▼
171+
┌────────┐ ┌──────────┐ ┌──────────────┐
172+
│slurmdb │ │invoice.py│ │financial_ │
173+
│ .py │ │(reportlab│ │ export.py │
174+
│ │ │ PDF gen)│ │(Oracle/CSV/ │
175+
│ MySQL │ │ │ │ webhooks) │
176+
│queries │ │ │ │ │
177+
└───┬────┘ └──────────┘ └──────────────┘
178+
│
179+
▼
180+
┌─────────────┐
181+
│ SlurmDB │
182+
│ (MySQL/ │
183+
│ MariaDB) │
184+
└─────────────┘
185+
```
141186

142-
- `example_slurm_schema_for_testing.json` – a pre-generated mapping of
143-
tables to columns for a minimal Slurm accounting database.
144-
- `example_slurmdb_for_testing.sql` – a small SQL dump containing the
145-
corresponding table definitions and a few dummy rows.
187+
## Testing
146188

147-
These files allow tests to verify table and column presence and operate on
148-
sample data without requiring access to a live SlurmDB instance.
189+
```bash
190+
# Unit tests
191+
make check
192+
193+
# Or individually:
194+
PYTHONPATH=src python -m pytest test/unit/ -v
195+
for f in test/unit/*.test.js; do node "$f"; done
149196

150-
## 📝 Development Notes
197+
# Lint
198+
flake8 src/*.py
199+
npx eslint src/ test/
200+
```
151201

152-
- Your UI components can access system files or commands using `cockpit.file()` and other Cockpit APIs.
153-
- Write integration tests using the Python-based browser automation tools bundled with Cockpit Starter Kit.
154-
- Ensure cross‑OS compatibility by leveraging Cockpit’s built-in CI and test VM infrastructure.
202+
## Contributing
155203

156-
We welcome community contributions. Please see [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.
204+
See [CONTRIBUTING.md](CONTRIBUTING.md).
157205

158-
## 📄 License
206+
## License
159207

160-
**SlurmLedger** is licensed under **MIT**—see the [LICENSE](LICENSE) file for details.
208+
LGPL-2.1 — See [LICENSE](LICENSE).

0 commit comments

Comments
 (0)