Skip to content

Commit 68ecb10

Browse files
alanpeixinhoalanpeixinho
committed
fixup! feat: add prune_db command to prune old checkouts, builds and tests. * Delete rows older than a given age with manual cascade (checkout -> builds -> tests) * --origins scoping, --dry-run and a --yes confirmation. Extract parse_interval into a shared helper and add integration tests.
 * Add tables parameter. * Protect issue related rows by default, and add parameter to bypass. Signed-off-by: Alan Peixinho <alan.peixinho@profusion.mobi>
1 parent 552bcc4 commit 68ecb10

3 files changed

Lines changed: 323 additions & 47 deletions

File tree

‎backend/docs/prune_db command.md‎

Lines changed: 82 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,82 @@
1+
# prune_db Command Documentation
2+
3+
The `prune_db` command deletes old rows from `checkouts`, `builds`, and `tests` in the dashboard database. It is intended for routine retention: run with `--dry-run` first, review the counts, then execute without `--dry-run`.
4+
5+
Models use `DO_NOTHING` foreign keys, so the command applies manual cascade rules instead of relying on the database:
6+
7+
- An old checkout also removes its builds and tests, even when those children are newer than the cutoff.
8+
- An old build also removes its tests, even when they are newer than the cutoff.
9+
10+
## Parameters
11+
12+
### Required Parameters
13+
14+
- `--older-than`: Delete rows older than this age. Format: `'x days'`, `'x hours'`, or `'x minutes'` (for example, `'30 days'`).
15+
16+
### Optional Parameters
17+
18+
- `--tables`: Limit pruning to specific tables (comma-separated). Valid options: `checkouts`, `builds`, `tests`. Default: all three.
19+
- Cascade still applies inside the selected tables: a build matched because its checkout is old is deleted when `builds` is selected, and a test matched because its build is doomed is deleted when `tests` is selected.
20+
- Tables not listed are not deleted. For example, `--tables builds` removes old builds but leaves their tests in place. Selecting a parent without its children (e.g. only `checkouts`) can therefore leave orphaned rows.
21+
- `--origins`: Limit age-based pruning to specific origins (comma-separated). If omitted, any origin is considered.
22+
- Cascade ignores origin: once a parent row is doomed, its children are removed even if they belong to a different origin.
23+
- `--batch-size`: Number of rows deleted per batch (default: `10000`). Must be at least `1`.
24+
- `--skip-issue-protection`: Prune builds and tests linked to issues. By default, rows with an associated incident are kept.
25+
- `--dry-run`: Print counts without deleting anything.
26+
- `--yes`: Skip the confirmation prompt and delete immediately.
27+
28+
## Examples
29+
30+
### Preview what would be deleted
31+
32+
```bash
33+
python manage.py prune_db --older-than "30 days" --dry-run
34+
```
35+
36+
### Delete rows older than 90 days
37+
38+
```bash
39+
python manage.py prune_db --older-than "90 days" --yes
40+
```
41+
42+
### Prune only one origin
43+
44+
```bash
45+
python manage.py prune_db --older-than "30 days" --origins maestro,0dayci --dry-run
46+
```
47+
48+
### Prune only tests
49+
50+
```bash
51+
python manage.py prune_db --older-than "30 days" --tables tests --yes
52+
```
53+
54+
### Prune rows linked to issues (override default protection)
55+
56+
```bash
57+
python manage.py prune_db --older-than "30 days" --skip-issue-protection --yes
58+
```
59+
60+
## What Is Not Deleted
61+
62+
The command only touches `checkouts`, `builds`, and `tests`. Related tables are left as-is, including:
63+
64+
- `incidents` rows themselves (only used to decide which builds/tests/checkouts to keep)
65+
- `hardware_status`, `latest_checkout`, `tree_tests_rollup` (reference checkouts)
66+
- `pending_build`, `pending_test` (reference builds)
67+
68+
If those tables must stay consistent, plan separate cleanup or accept stale references until another process removes them.
69+
70+
## Recommended Workflow
71+
72+
1. Run with `--dry-run` and review per-table counts.
73+
2. Add `--origins` when retention should apply to specific origins only.
74+
3. Use `--tables` only when you intentionally want a partial prune.
75+
4. Run without `--dry-run`; confirm at the prompt unless `--yes` is set.
76+
77+
## Notes
78+
79+
- Deletion is batched and child-first (`tests`, then `builds`, then `checkouts`) to avoid orphans within the pruned set.
80+
- Each batch commits separately to keep locks short.
81+
- `--origins` scopes the age filter only. Cascade deletions do not re-check the child's origin.
82+
- By default, builds and tests referenced by `incidents` are not pruned, and their parent checkouts are kept too. Use `--skip-issue-protection` to delete them anyway (incident rows are not removed by this command).

‎backend/kernelCI_app/management/commands/prune_db.py‎

Lines changed: 109 additions & 41 deletions
Original file line numberDiff line numberDiff line change
@@ -5,13 +5,18 @@
55
integrity, deletion cascades manually (models use DO_NOTHING): a removed
66
checkout drags its builds, and a removed build drags its tests, even when those
77
children are newer than the cutoff.
8+
9+
Rows linked to an incident (an issue) are kept by default, together with their
10+
ancestors so nothing is orphaned; pass --skip-issue-protection to prune them too.
811
"""
912

1013
from django.core.management.base import BaseCommand, CommandError
1114
from django.db import connections
1215

1316
from kernelCI_app.management.commands.helpers.intervals import parse_interval
1417

18+
PRUNABLE_TABLES = ("checkouts", "builds", "tests")
19+
1520

1621
class Command(BaseCommand):
1722
help = "Prune checkouts, builds and tests older than a given age"
@@ -48,50 +53,69 @@ def add_arguments(self, parser):
4853
default=10000,
4954
help="Number of rows to delete per batch (default: 10000)",
5055
)
56+
parser.add_argument(
57+
"--tables",
58+
type=lambda s: [t.strip() for t in s.split(",")],
59+
default=list(PRUNABLE_TABLES),
60+
help="Limit pruning to specific tables (comma-separated: "
61+
f"{', '.join(PRUNABLE_TABLES)}). Only the listed tables are deleted; "
62+
"unlisted child tables are left untouched, so selecting a parent without "
63+
"its children (e.g. only 'checkouts') can leave orphans. Default: all.",
64+
)
65+
parser.add_argument(
66+
"--skip-issue-protection",
67+
action="store_true",
68+
help="Prune builds and tests linked to issues (default: keep rows with "
69+
"an associated incident)",
70+
)
5171

5272
def handle(self, *args, **options):
5373
try:
5474
cutoff = parse_interval(options["older_than"])
5575
except ValueError as e:
5676
raise CommandError(str(e)) from e
5777

78+
if options["batch_size"] < 1:
79+
raise CommandError(
80+
f"--batch-size must be at least 1 (got {options['batch_size']}). "
81+
"It sets how many rows are deleted per batch, so it needs to be a "
82+
"positive number."
83+
)
84+
85+
unknown_tables = [t for t in options["tables"] if t not in PRUNABLE_TABLES]
86+
if unknown_tables:
87+
raise CommandError(
88+
f"Unknown table(s): {', '.join(unknown_tables)}. "
89+
f"Valid options are: {', '.join(PRUNABLE_TABLES)}."
90+
)
91+
selected_tables = [t for t in PRUNABLE_TABLES if t in options["tables"]]
92+
protect_incidents = not options["skip_issue_protection"]
93+
5894
dry_run = options["dry_run"]
5995
origins = options["origins"]
6096

61-
# Values are always passed as query parameters; only static SQL fragments are
62-
# interpolated. Named parameters let us reuse the same dict across every query.
6397
params = {"cutoff": cutoff, "batch_size": options["batch_size"]}
64-
origin_cond = ""
98+
origins_condition = ""
6599
if origins:
66100
params["origins"] = list(origins)
67-
origin_cond = "AND origin = ANY(%(origins)s)"
101+
origins_condition = "AND origin = ANY(%(origins)s)"
68102

69-
# Age-based prune is origin-scoped; children of a doomed parent are removed
70-
# regardless of origin (and even if newer than the cutoff) to avoid orphans.
71-
checkout_where = f"_timestamp < %(cutoff)s {origin_cond}"
72-
build_where = (
73-
f"(_timestamp < %(cutoff)s {origin_cond}) "
74-
f"OR checkout_id IN (SELECT id FROM checkouts WHERE {checkout_where})"
75-
)
76-
test_where = (
77-
f"(_timestamp < %(cutoff)s {origin_cond}) "
78-
f"OR build_id IN (SELECT id FROM builds WHERE {build_where})"
79-
)
103+
where_clauses = self._build_where_clauses(origins_condition, protect_incidents)
80104

81105
with connections["default"].cursor() as cursor:
82-
checkout_count = self._count(cursor, "checkouts", checkout_where, params)
83-
build_count = self._count(cursor, "builds", build_where, params)
84-
test_count = self._count(cursor, "tests", test_where, params)
85-
total = checkout_count + build_count + test_count
86-
87-
self.stdout.write(
88-
f"Rows older than {cutoff.isoformat()} (including cascaded children):\n"
89-
f"* checkouts:\t{checkout_count:>8}\n"
90-
f"* builds:\t{build_count:>8}\n"
91-
f"* tests:\t{test_count:>8}\n"
92-
"----------------------\n"
93-
f"* total:\t{total:>8}"
94-
)
106+
counts = {
107+
t: self._count(cursor, t, where_clauses[t], params)
108+
for t in selected_tables
109+
}
110+
total = sum(counts.values())
111+
112+
lines = [f"Rows older than {cutoff.isoformat()}:"]
113+
lines += [f"* {t}:\t{counts[t]:>8}" for t in selected_tables]
114+
lines += ["----------------------", f"* total:\t{total:>8}"]
115+
lines.append("Note: counts include children cascaded from pruned parents.")
116+
if protect_incidents:
117+
lines.append("Note: rows linked to an incident are kept.")
118+
self.stdout.write("\n".join(lines))
95119

96120
if total == 0:
97121
self.stdout.write(self.style.SUCCESS("Nothing to prune."))
@@ -106,28 +130,72 @@ def handle(self, *args, **options):
106130
return
107131

108132
if not options["yes"]:
109-
answer = (
110-
input(
111-
f"Delete {checkout_count} checkouts, {build_count} builds and "
112-
f"{test_count} tests ({total} rows total)? [y/N] "
133+
summary = ", ".join(f"{counts[t]} {t}" for t in selected_tables)
134+
try:
135+
answer = (
136+
input(f"Delete {summary} ({total} rows total)? [y/N] ")
137+
.strip()
138+
.lower()
113139
)
114-
.strip()
115-
.lower()
116-
)
140+
except EOFError:
141+
answer = ""
117142
if answer not in ("y", "yes"):
118143
self.stdout.write("Aborted.")
119144
return
120145

121-
# Delete children before parents so the parent subqueries stay valid.
122-
# Each batch commits on its own (autocommit), keeping locks short; the
123-
# child-first order means no orphans exist between commits.
146+
# Delete child-first (reverse of PRUNABLE_TABLES order): each batch commits
147+
# on its own, so a crash mid-run leaves children already gone before their
148+
# parents, never the reverse. Reordering this would risk orphans.
124149
deleted = 0
125-
deleted += self._batch_delete(cursor, "tests", test_where, params)
126-
deleted += self._batch_delete(cursor, "builds", build_where, params)
127-
deleted += self._batch_delete(cursor, "checkouts", checkout_where, params)
150+
for table in reversed(selected_tables):
151+
deleted += self._batch_delete(
152+
cursor, table, where_clauses[table], params
153+
)
128154

129155
self.stdout.write(self.style.SUCCESS(f"Successfully pruned {deleted} rows."))
130156

157+
def _build_where_clauses(self, origins_condition, protect_incidents):
158+
"""Build the per-table WHERE clauses, chaining the cascade so each child
159+
matches when its parent is doomed, and appending incident protection when
160+
enabled."""
161+
age = f"_timestamp < %(cutoff)s {origins_condition}"
162+
163+
# A row tied to an incident is protected, along with the ancestors that would
164+
# otherwise be orphaned: a build is protected when it (or one of its tests) has
165+
# an incident, and a checkout when one of its builds is protected.
166+
incident_tests = "SELECT test_id FROM incidents WHERE test_id IS NOT NULL"
167+
incident_builds = (
168+
"SELECT build_id FROM incidents WHERE build_id IS NOT NULL "
169+
f"UNION SELECT build_id FROM tests WHERE id IN ({incident_tests})"
170+
)
171+
exclusion = (
172+
{
173+
"tests": f" AND id NOT IN ({incident_tests})",
174+
"builds": f" AND id NOT IN ({incident_builds})",
175+
"checkouts": (
176+
" AND id NOT IN "
177+
f"(SELECT checkout_id FROM builds WHERE id IN ({incident_builds}))"
178+
),
179+
}
180+
if protect_incidents
181+
else {}
182+
)
183+
184+
checkout_where = age + exclusion.get("checkouts", "")
185+
build_where = (
186+
f"(({age}) OR checkout_id IN (SELECT id FROM checkouts WHERE {checkout_where}))"
187+
+ exclusion.get("builds", "")
188+
)
189+
test_where = (
190+
f"(({age}) OR build_id IN (SELECT id FROM builds WHERE {build_where}))"
191+
+ exclusion.get("tests", "")
192+
)
193+
return {
194+
"checkouts": checkout_where,
195+
"builds": build_where,
196+
"tests": test_where,
197+
}
198+
131199
def _count(self, cursor, table, where, params):
132200
cursor.execute(f"SELECT COUNT(*) FROM {table} WHERE {where}", params)
133201
return cursor.fetchone()[0]

0 commit comments

Comments
 (0)