|
| 1 | +# SlurmLedger Production Setup Guide |
| 2 | + |
| 3 | +## Prerequisites |
| 4 | + |
| 5 | +- SLURM cluster with SlurmDBD configured (MySQL/MariaDB backend) |
| 6 | +- Cockpit installed on the head/admin node |
| 7 | +- Root access (for initial setup only) |
| 8 | +- Python 3.9+ |
| 9 | + |
| 10 | +## Step 1: Install Dependencies |
| 11 | + |
| 12 | +### RHEL/Rocky Linux 9 |
| 13 | +```bash |
| 14 | +sudo dnf install cockpit python3 python3-pymysql |
| 15 | +sudo pip3 install reportlab |
| 16 | +sudo systemctl enable --now cockpit.socket |
| 17 | +``` |
| 18 | + |
| 19 | +### Ubuntu 22.04+ |
| 20 | +```bash |
| 21 | +sudo apt install cockpit python3 python3-pymysql python3-reportlab |
| 22 | +sudo systemctl enable --now cockpit.socket |
| 23 | +``` |
| 24 | + |
| 25 | +## Step 2: Install SlurmLedger |
| 26 | + |
| 27 | +### From Release Package |
| 28 | +```bash |
| 29 | +# Download latest release |
| 30 | +curl -LO https://github.com/NessieCanCode/SlurmLedger/releases/latest/download/slurmledger-1.0.0-1.noarch.rpm |
| 31 | +sudo dnf install slurmledger-1.0.0-1.noarch.rpm |
| 32 | +``` |
| 33 | + |
| 34 | +### From Source |
| 35 | +```bash |
| 36 | +git clone https://github.com/NessieCanCode/SlurmLedger.git |
| 37 | +cd SlurmLedger |
| 38 | +pip3 install -r requirements.txt |
| 39 | +make build |
| 40 | +sudo make install |
| 41 | +``` |
| 42 | + |
| 43 | +## Step 3: Configure File Permissions |
| 44 | + |
| 45 | +```bash |
| 46 | +# Create config directory |
| 47 | +sudo mkdir -p /etc/slurmledger |
| 48 | +sudo mkdir -p /etc/slurmledger/invoices |
| 49 | +sudo mkdir -p /var/log/slurmledger |
| 50 | + |
| 51 | +# Copy default configs (only if they don't exist) |
| 52 | +sudo cp -n /usr/share/cockpit/slurmledger/rates.json /etc/slurmledger/ |
| 53 | +sudo cp -n /usr/share/cockpit/slurmledger/institution.json /etc/slurmledger/ |
| 54 | + |
| 55 | +# Set permissions |
| 56 | +sudo chown -R root:cockpit-ws /etc/slurmledger |
| 57 | +sudo chmod 750 /etc/slurmledger |
| 58 | +sudo chmod 640 /etc/slurmledger/*.json |
| 59 | +sudo chmod 750 /etc/slurmledger/invoices |
| 60 | + |
| 61 | +# Log directory |
| 62 | +sudo chown root:root /var/log/slurmledger |
| 63 | +sudo chmod 750 /var/log/slurmledger |
| 64 | +``` |
| 65 | + |
| 66 | +## Step 4: Verify SlurmDB Access |
| 67 | + |
| 68 | +SlurmLedger reads database credentials from `slurmdbd.conf`. Verify the Cockpit user can read it: |
| 69 | + |
| 70 | +```bash |
| 71 | +# Check slurmdbd.conf is readable |
| 72 | +sudo cat /etc/slurm/slurmdbd.conf | grep StoragePass |
| 73 | +# Should show the database password |
| 74 | + |
| 75 | +# Test database connectivity |
| 76 | +python3 /usr/share/cockpit/slurmledger/slurmdb.py \ |
| 77 | + --start $(date -d '1 month ago' +%Y-%m-%d) \ |
| 78 | + --end $(date +%Y-%m-%d) \ |
| 79 | + --output - | python3 -m json.tool | head -20 |
| 80 | +``` |
| 81 | + |
| 82 | +If this fails, check: |
| 83 | +- `slurmdbd.conf` has correct `StorageHost`, `StorageUser`, `StoragePass` |
| 84 | +- MySQL/MariaDB is running and accessible |
| 85 | +- The storage user has SELECT access to the `slurm_acct_db` database |
| 86 | + |
| 87 | +## Step 5: Access the Plugin |
| 88 | + |
| 89 | +1. Open a browser to `https://your-server:9090` |
| 90 | +2. Log in with an admin account |
| 91 | +3. Click "SlurmLedger" in the left navigation |
| 92 | +4. The **Setup Wizard** will guide you through initial configuration: |
| 93 | + - **Step 1**: Enter institution details (name, address, contacts) |
| 94 | + - **Step 2**: Set billing rates (CPU rate per core-hour, GPU rate) |
| 95 | + - **Step 3**: Test database connection |
| 96 | + |
| 97 | +## Step 6: Configure Billing Rates |
| 98 | + |
| 99 | +Navigate to **Administration → Rate Configuration**: |
| 100 | + |
| 101 | +1. Set the default CPU rate ($/core-hour). Common range: $0.005 – $0.05 |
| 102 | +2. Set the default GPU rate ($/GPU-hour). Common range: $0.05 – $0.50 |
| 103 | +3. Add per-account overrides if needed (e.g., discounted rates for funded groups) |
| 104 | +4. Click **Save** |
| 105 | + |
| 106 | +### Historical Rates |
| 107 | +If rates changed in the past, add entries in the **Historical Rates** section so retroactive billing is accurate. |
| 108 | + |
| 109 | +## Step 7: Set Up Allocations (Optional) |
| 110 | + |
| 111 | +Navigate to **Administration → Allocations**: |
| 112 | + |
| 113 | +1. For each account, choose: |
| 114 | + - **Pre-paid**: Account has a fixed SU budget. Jobs are blocked when exhausted. |
| 115 | + - **Post-paid**: Account is billed after the fact. No enforcement. |
| 116 | +2. Set budget (in Service Units / core-hours) |
| 117 | +3. Set allocation period (annual, quarterly, monthly) |
| 118 | +4. Configure alert thresholds (default: 80%, 90%, 100%) |
| 119 | + |
| 120 | +## Step 8: Configure Billing Rules |
| 121 | + |
| 122 | +Navigate to **Administration → Billing Rules**: |
| 123 | + |
| 124 | +Default rules (adjust as needed): |
| 125 | +- ✅ **No charge for failed jobs** (except OOM and timeout) |
| 126 | +- ✅ **No charge for jobs under 1 minute** |
| 127 | +- ❌ Debug partition 50% discount (enable if applicable) |
| 128 | +- ❌ Visualization partition free (enable if applicable) |
| 129 | + |
| 130 | +## Step 9: Configure Institution Profile |
| 131 | + |
| 132 | +Navigate to **Administration → Institution Profile**: |
| 133 | + |
| 134 | +1. Fill in all required fields (name, address, contacts) |
| 135 | +2. Upload institution logo (for invoices) |
| 136 | +3. Enter bank/payment details (for invoice footer) |
| 137 | +4. Set payment terms (e.g., "Net 30") |
| 138 | + |
| 139 | +## Step 10: Set Up Balance Enforcement (Optional) |
| 140 | + |
| 141 | +For pre-paid allocations, install the cron job: |
| 142 | + |
| 143 | +```bash |
| 144 | +# Create cron job for hourly balance checks |
| 145 | +sudo tee /etc/cron.d/slurmledger-enforcer << 'EOF' |
| 146 | +# SlurmLedger Balance Enforcer — check allocations hourly |
| 147 | +0 * * * * root /usr/bin/python3 /usr/share/cockpit/slurmledger/balance_enforcer.py --enforce --log /var/log/slurmledger/enforcer.log |
| 148 | +EOF |
| 149 | +sudo chmod 644 /etc/cron.d/slurmledger-enforcer |
| 150 | + |
| 151 | +# Test it first (dry run): |
| 152 | +sudo python3 /usr/share/cockpit/slurmledger/balance_enforcer.py --check |
| 153 | +``` |
| 154 | + |
| 155 | +The enforcer uses SLURM's native `GrpTRESMins` limit to cap accounts at their allocation. Jobs submitted after the limit is reached will be held in PENDING state with reason `AssocGrpCPUMinutesLimit`. |
| 156 | + |
| 157 | +## Step 11: Configure Financial Integration (Optional) |
| 158 | + |
| 159 | +Navigate to **Administration → Financial Integration**: |
| 160 | + |
| 161 | +1. Select your ERP system (Oracle Financials, Workday, Banner, Kuali, or Generic Webhook) |
| 162 | +2. Enter webhook URL and API key |
| 163 | +3. Map SLURM accounts to Chart of Accounts codes |
| 164 | +4. Click **Test Connection** |
| 165 | + |
| 166 | +## Step 12: Set Up Roles |
| 167 | + |
| 168 | +Navigate to **Administration → Institution Profile** (roles section in institution.json): |
| 169 | + |
| 170 | +```json |
| 171 | +{ |
| 172 | + "roles": { |
| 173 | + "admins": ["root", "hpc-admin"], |
| 174 | + "finance": ["billing-dept"], |
| 175 | + "pis": [] |
| 176 | + } |
| 177 | +} |
| 178 | +``` |
| 179 | + |
| 180 | +- **Admins**: Full access to all features |
| 181 | +- **Finance**: Read-only access to invoices, can mark invoices as paid |
| 182 | +- **PIs**: Auto-detected from SLURM account coordinators |
| 183 | +- **Members**: Anyone else — sees only their own usage |
| 184 | + |
| 185 | +## Step 13: Generate Your First Invoice |
| 186 | + |
| 187 | +1. Navigate to **Detailed Transactions** |
| 188 | +2. Select a month and account |
| 189 | +3. Click **Export Invoice** |
| 190 | +4. Review the PDF — verify rates, line items, and totals |
| 191 | +5. Navigate to **Invoices** to see the invoice in the ledger |
| 192 | +6. Click **Mark as Sent** when you send it to the PI |
| 193 | + |
| 194 | +## Step 14: Set Up Backups |
| 195 | + |
| 196 | +```bash |
| 197 | +# Daily backup of all SlurmLedger config and data |
| 198 | +sudo tee /etc/cron.daily/slurmledger-backup << 'EOF' |
| 199 | +#!/bin/bash |
| 200 | +BACKUP_DIR=/backup/slurmledger/$(date +%Y%m%d) |
| 201 | +mkdir -p "$BACKUP_DIR" |
| 202 | +cp -a /etc/slurmledger/ "$BACKUP_DIR/" |
| 203 | +find /backup/slurmledger/ -maxdepth 1 -mtime +90 -exec rm -rf {} \; |
| 204 | +EOF |
| 205 | +sudo chmod 755 /etc/cron.daily/slurmledger-backup |
| 206 | +``` |
| 207 | + |
| 208 | +## Verification Checklist |
| 209 | + |
| 210 | +- [ ] Cockpit accessible at https://server:9090 |
| 211 | +- [ ] SlurmLedger appears in Cockpit navigation |
| 212 | +- [ ] Setup wizard completes successfully |
| 213 | +- [ ] Billing data loads for current month |
| 214 | +- [ ] Rates are configured (not demo values) |
| 215 | +- [ ] Institution profile is complete |
| 216 | +- [ ] Test invoice generates with correct branding |
| 217 | +- [ ] Invoice numbers are sequential |
| 218 | +- [ ] Balance enforcer runs without errors (dry run) |
| 219 | +- [ ] Backup cron is active |
| 220 | +- [ ] File permissions are correct on /etc/slurmledger/ |
| 221 | + |
| 222 | +## Troubleshooting |
| 223 | + |
| 224 | +### "Failed to load data" on first visit |
| 225 | +- Check `slurmdbd.conf` is readable by the Cockpit user |
| 226 | +- Verify MySQL is running: `systemctl status mariadb` |
| 227 | +- Test manually: `python3 slurmdb.py --start 2026-01-01 --end 2026-12-31 --output -` |
| 228 | + |
| 229 | +### Invoice PDF has no logo or bank details |
| 230 | +- Complete the Institution Profile in Administration |
| 231 | +- Upload a logo (PNG/JPG, under 256KB) |
| 232 | +- Fill in the bank/payment information |
| 233 | + |
| 234 | +### Balance enforcer says "No allocations configured" |
| 235 | +- Set up allocations in Administration → Allocations |
| 236 | +- Only "prepaid" allocations are enforced |
| 237 | + |
| 238 | +### Permission denied on config save |
| 239 | +- Check `/etc/slurmledger/` ownership: `ls -la /etc/slurmledger/` |
| 240 | +- Should be `root:cockpit-ws 750` for directory, `640` for files |
0 commit comments