Skip to content

Commit 876caaf

Browse files
author
Sqoia Dev Agent
committed
test: Docker Compose test environment with MariaDB + Cockpit
Test Environment: - docker-compose.yml: MariaDB 10.11 + Cockpit container - Dockerfile.test: Debian Bookworm with all dependencies - slurmdbd-test.conf: points at Docker MariaDB - rates-test.json: uses accounts from test fixture (oit, ucmerced) - institution-test.json: fully populated test profile - run-local-tests.sh: automated test suite (DB healthcheck, pytest, slurmdb.py integration, invoice generation, balance enforcer) - .dockerignore: keeps builds clean - Updated test/vm/Dockerfile: added reportlab + pytest - Updated README: Local Development & Testing section Usage: cd test && ./run-local-tests.sh # Cockpit at http://localhost:9090
1 parent 79d89d7 commit 876caaf

9 files changed

Lines changed: 312 additions & 5 deletions

‎.dockerignore‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
.git
2+
node_modules
3+
dist
4+
*.rpm
5+
*.deb
6+
test/vm

‎README.md‎

Lines changed: 38 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -201,21 +201,55 @@ python3 /usr/share/cockpit/slurmledger/balance_enforcer.py --check
201201

202202
The **Check Balances** button in the Admin Dashboard runs the same check interactively and displays results in the UI.
203203

204-
## Testing
204+
## Local Development & Testing
205205

206+
### Docker Compose (recommended)
207+
208+
Spins up MariaDB loaded with the test fixture, builds a Cockpit container with all dependencies, and runs the full test suite against the real database.
209+
210+
```bash
211+
cd test
212+
./run-local-tests.sh
213+
```
214+
215+
This will:
216+
1. Start MariaDB 10.11 and populate `slurm_acct_db` from the test fixture
217+
2. Build and start the Cockpit container with the plugin mounted read-only
218+
3. Run Python unit tests with `pytest`
219+
4. Run `slurmdb.py` against the live database and print a sample of the JSON output
220+
5. Smoke-test PDF invoice generation
221+
6. Smoke-test the balance enforcer in dry-run mode
222+
223+
Cockpit is available at `http://localhost:9090` once the environment is ready.
224+
225+
To stop the environment:
206226
```bash
207-
# Unit tests
227+
docker compose -f test/docker-compose.yml down
228+
```
229+
230+
### Manual unit tests
231+
232+
```bash
233+
# Unit tests only (no database required)
208234
make check
209235

210-
# Or individually:
211-
PYTHONPATH=src python -m pytest test/unit/ -v
236+
# Or directly:
237+
PYTHONPATH=src python3 -m pytest test/unit/ -v
212238
for f in test/unit/*.test.js; do node "$f"; done
213239

214240
# Lint
215241
flake8 src/*.py
216242
npx eslint src/ test/
217243
```
218244

245+
### Development install (live reload from source)
246+
247+
```bash
248+
make devel-install
249+
# Plugin is now linked at ~/.local/share/cockpit/slurmledger
250+
# Open Cockpit at https://localhost:9090
251+
```
252+
219253
## Contributing
220254

221255
See [CONTRIBUTING.md](CONTRIBUTING.md).

‎test/Dockerfile.test‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
FROM debian:bookworm
2+
3+
RUN apt-get update && apt-get install -y \
4+
cockpit \
5+
python3 python3-pymysql python3-pip \
6+
nodejs npm \
7+
&& pip3 install --break-system-packages reportlab pytest \
8+
&& apt-get clean \
9+
&& rm -rf /var/lib/apt/lists/*
10+
11+
# Create config and log directories with appropriate permissions
12+
RUN mkdir -p /etc/slurmledger /etc/slurmledger/invoices /var/log/slurmledger \
13+
&& chmod 750 /etc/slurmledger
14+
15+
# Copy the plugin source
16+
COPY src/ /usr/share/cockpit/slurmledger/
17+
18+
EXPOSE 9090
19+
20+
CMD ["cockpit-ws", "--no-tls", "--port", "9090"]

‎test/docker-compose.yml‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
version: '3.8'
2+
services:
3+
db:
4+
image: mariadb:10.11
5+
environment:
6+
MYSQL_ROOT_PASSWORD: testpass
7+
MYSQL_DATABASE: slurm_acct_db
8+
ports:
9+
- "3306:3306"
10+
volumes:
11+
- ./example_slurmdb_for_testing.sql:/docker-entrypoint-initdb.d/init.sql
12+
healthcheck:
13+
test: ["CMD", "mysqladmin", "ping", "-h", "localhost"]
14+
interval: 5s
15+
timeout: 3s
16+
retries: 10
17+
18+
slurmledger:
19+
build:
20+
context: ..
21+
dockerfile: test/Dockerfile.test
22+
depends_on:
23+
db:
24+
condition: service_healthy
25+
ports:
26+
- "9090:9090"
27+
volumes:
28+
- ../src:/usr/share/cockpit/slurmledger:ro
29+
- ./unit:/opt/slurmledger-tests/unit:ro
30+
- ./slurmdbd-test.conf:/etc/slurm/slurmdbd.conf:ro
31+
- ./rates-test.json:/etc/slurmledger/rates.json
32+
- ./institution-test.json:/etc/slurmledger/institution.json
33+
environment:
34+
- SLURM_CONF=/etc/slurm/slurm.conf

‎test/institution-test.json‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
{
2+
"institutionName": "Test University",
3+
"institutionAbbreviation": "TU",
4+
"campusDivision": "Research Computing",
5+
"institutionType": "R1 University",
6+
"primaryContact": {
7+
"fullName": "Jane Admin",
8+
"title": "HPC Systems Administrator",
9+
"email": "hpc-admin@test.example",
10+
"phone": "555-555-0100"
11+
},
12+
"secondaryContact": {
13+
"fullName": "John Finance",
14+
"email": "finance@test.example",
15+
"phone": "555-555-0101"
16+
},
17+
"streetAddress": "123 Research Drive",
18+
"city": "Testville",
19+
"state": "CA",
20+
"postalCode": "99999",
21+
"country": "US",
22+
"fiscalYearStartMonth": "7",
23+
"defaultCurrency": "USD",
24+
"departmentName": "Office of Information Technology",
25+
"costCenter": "CC-HPC-001",
26+
"notes": "Test environment — not for production use",
27+
"logo": ""
28+
}

‎test/rates-test.json‎

Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,71 @@
1+
{
2+
"defaultRate": 0.02,
3+
"defaultGpuRate": 0.2,
4+
"historicalRates": {
5+
"2024-01": 0.015
6+
},
7+
"historicalGpuRates": {
8+
"2024-01": 0.15
9+
},
10+
"overrides": {
11+
"research": {
12+
"rate": 0.01,
13+
"gpuRate": 0.1
14+
},
15+
"education": {
16+
"discount": 0.5
17+
},
18+
"oit": {
19+
"rate": 0.02,
20+
"gpuRate": 0.2
21+
}
22+
},
23+
"allocations": {
24+
"oit": {
25+
"type": "postpaid",
26+
"billing_period": "monthly",
27+
"budget_su": null
28+
},
29+
"ucmerced": {
30+
"type": "prepaid",
31+
"budget_su": 1000000,
32+
"period": "annual",
33+
"start_date": "2025-01-01",
34+
"end_date": "2025-12-31",
35+
"carryover": false,
36+
"alerts": [80, 90, 100]
37+
}
38+
},
39+
"billing_defaults": {
40+
"type": "postpaid",
41+
"billing_period": "monthly",
42+
"payment_terms_days": 30
43+
},
44+
"billing_rules": [
45+
{
46+
"id": "no-charge-failed",
47+
"name": "Don't charge failed jobs (except OOM and timeout)",
48+
"enabled": true,
49+
"condition": {
50+
"field": "state",
51+
"operator": "in",
52+
"values": ["FAILED", "CANCELLED", "NODE_FAIL"]
53+
},
54+
"exclude_states": ["OUT_OF_MEMORY", "TIMEOUT"],
55+
"action": "no_charge",
56+
"description": "Failed jobs are not billed unless they failed due to OOM or timeout"
57+
},
58+
{
59+
"id": "no-charge-short",
60+
"name": "Don't charge jobs under 1 minute",
61+
"enabled": true,
62+
"condition": {
63+
"field": "elapsed_seconds",
64+
"operator": "less_than",
65+
"value": 60
66+
},
67+
"action": "no_charge",
68+
"description": "Very short jobs are likely setup failures and not charged"
69+
}
70+
]
71+
}

‎test/run-local-tests.sh‎

Lines changed: 108 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,108 @@
1+
#!/bin/bash
2+
set -euo pipefail
3+
4+
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
5+
cd "$SCRIPT_DIR"
6+
7+
echo "=== Starting SlurmLedger Test Environment ==="
8+
9+
# Start services and rebuild if source has changed
10+
docker compose up -d --build
11+
12+
echo "Waiting for services to become healthy..."
13+
# Poll the DB healthcheck rather than sleeping a fixed interval
14+
for i in $(seq 1 30); do
15+
if docker compose exec -T db mysqladmin ping -h localhost --silent 2>/dev/null; then
16+
echo "Database is ready."
17+
break
18+
fi
19+
if [ "$i" -eq 30 ]; then
20+
echo "ERROR: Database did not become ready in time." >&2
21+
docker compose logs db
22+
exit 1
23+
fi
24+
sleep 2
25+
done
26+
27+
# Give cockpit-ws a moment to initialize after the DB is up
28+
sleep 3
29+
30+
# ---------------------------------------------------------------------------
31+
# Unit tests — run against the src/ Python modules directly.
32+
# src/ is bind-mounted at /usr/share/cockpit/slurmledger inside the container.
33+
# The unit tests live in test/unit/ on the host; copy them in at runtime so
34+
# pytest can discover them alongside the source they import.
35+
# ---------------------------------------------------------------------------
36+
echo ""
37+
echo "=== Running Python unit tests ==="
38+
docker compose exec -T slurmledger bash -c "
39+
PYTHONPATH=/usr/share/cockpit/slurmledger \
40+
python3 -m pytest /opt/slurmledger-tests/unit -v --tb=short
41+
" || echo "WARN: Some unit tests failed (see output above)"
42+
43+
# ---------------------------------------------------------------------------
44+
# Integration test — exercise slurmdb.py against the real MariaDB container.
45+
# Uses --conf to read connection details from the mounted slurmdbd.conf, and
46+
# --cluster localcluster to match the table prefix in the test fixture.
47+
# ---------------------------------------------------------------------------
48+
echo ""
49+
echo "=== Testing slurmdb.py against real database ==="
50+
docker compose exec -T slurmledger python3 /usr/share/cockpit/slurmledger/slurmdb.py \
51+
--conf /etc/slurm/slurmdbd.conf \
52+
--cluster localcluster \
53+
--start 2024-01-01 --end 2025-12-31 \
54+
--output - \
55+
| python3 -m json.tool | head -40
56+
57+
# ---------------------------------------------------------------------------
58+
# Invoice generation smoke test
59+
# ---------------------------------------------------------------------------
60+
echo ""
61+
echo "=== Testing invoice generation ==="
62+
echo '{
63+
"invoice_number": "TEST-001",
64+
"date": "2026-03-29",
65+
"items": [
66+
{"description": "CPU Hours (March 2026)", "qty": 1000, "rate": 0.02, "amount": 20.00}
67+
],
68+
"institution": {
69+
"institutionName": "Test University",
70+
"streetAddress": "123 Research Drive",
71+
"city": "Testville",
72+
"state": "CA",
73+
"postalCode": "99999"
74+
},
75+
"bank_info": ["Test Bank — ACH routing 000000000"],
76+
"notes": "Net 30. Test invoice — not for production use.",
77+
"subtotal": 20.00,
78+
"total_due": 20.00
79+
}' | docker compose exec -T slurmledger \
80+
python3 /usr/share/cockpit/slurmledger/invoice.py > /dev/null \
81+
&& echo "Invoice generation: OK" \
82+
|| echo "Invoice generation: FAILED"
83+
84+
# ---------------------------------------------------------------------------
85+
# Balance enforcer smoke test — no allocations will be overdrawn in the test
86+
# fixture, so we expect a clean exit or a "no allocations" message.
87+
# ---------------------------------------------------------------------------
88+
echo ""
89+
echo "=== Testing balance enforcer (dry-run) ==="
90+
docker compose exec -T slurmledger \
91+
python3 /usr/share/cockpit/slurmledger/balance_enforcer.py \
92+
--check --json 2>&1 \
93+
|| echo "(Expected: no allocations configured or scontrol unavailable in test env)"
94+
95+
# ---------------------------------------------------------------------------
96+
# Summary
97+
# ---------------------------------------------------------------------------
98+
echo ""
99+
echo "=== Test Environment Ready ==="
100+
echo "Cockpit UI: http://localhost:9090"
101+
echo " Log in with any system user account created inside the container, or"
102+
echo " create one with: docker compose exec slurmledger useradd -m -p \$(openssl passwd -1 testpass) testuser"
103+
echo ""
104+
echo "To stop and remove containers:"
105+
echo " docker compose -f test/docker-compose.yml down"
106+
echo ""
107+
echo "To tail application logs:"
108+
echo " docker compose -f test/docker-compose.yml logs -f slurmledger"

‎test/slurmdbd-test.conf‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
StorageType=accounting_storage/mysql
2+
StorageHost=db
3+
StoragePort=3306
4+
StorageUser=root
5+
StoragePass=testpass
6+
StorageLoc=slurm_acct_db

‎test/vm/Dockerfile‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,6 @@ RUN apt-get update && \
66
apt-get install -y python3 python3-pip && \
77
rm -rf /var/lib/apt/lists/*
88

9-
RUN pip3 install pymysql
9+
RUN pip3 install pymysql reportlab pytest
1010

1111
WORKDIR /src

0 commit comments

Comments
 (0)