feat/prune age by alanpeixinho · Pull Request #1970 · kernelci/dashboard

alanpeixinho · 2026-07-03T17:21:19Z

Summary

Add prune_db, a manual management command to delete old data and reclaim space.

Deletes checkouts, builds and tests older than a given age (e.g. --older-than "30 days").
Manual cascade (models use DO_NOTHING): pruning a checkout removes its builds, pruning a build removes its tests — even when children are newer.
--origins scopes the age filter; cascade stays origin-agnostic so no orphans are left.
Batched deletes (--batch-size) that each commit on their own to keep locks short.
--dry-run and a per-table confirmation prompt showing exactly how many rows will be affected (skip with --yes).
Extracts parse_interval from update_db into a shared helper.

Usage

python manage.py prune_db --older-than "30 days" --dry-run                                                                                                                                                         
python manage.py prune_db --older-than "30 days" --origins maestro

How to Test

In a local database create newly seeded data poetry run python3 manage.py seed_data --clear --yes
Run command poetry run python3 manage.py prune_db --dry-run with different arguments for older-than parameter.
Verify the number of selected rows matches the database.
Run command poetry run python3 manage.py prune_db with different arguments for older-than parameter.
Verify the rows are properly deleted.

felipebergamin · 2026-07-06T11:46:27Z

+
+        # Values are always passed as query parameters; only static SQL fragments are
+        # interpolated. Named parameters let us reuse the same dict across every query.
+        params = {"cutoff": cutoff, "batch_size": options["batch_size"]}


a minor but I think that would be nice to validate the batch_size. Currently we are able to pass a negative value to the database.

django.db.utils.DataError: LIMIT must not be negative

felipebergamin · 2026-07-06T11:50:46Z

+"""
+Management command to prune old builds, tests and checkouts.
+
+Removes checkouts, builds and tests older than a given age. To keep referential
+integrity, deletion cascades manually (models use DO_NOTHING): a removed
+checkout drags its builds, and a removed build drags its tests, even when those
+children are newer than the cutoff.
+"""


Worth noting in docs that aggregate tables are not touched by this command.

tales-aparecida · 2026-07-06T12:16:51Z

@nuclearcat how long/far-back do we keep the raw KCIDB json payloads?

mentonin · 2026-07-03T21:07:29Z

@@ -0,0 +1,25 @@
+from datetime import datetime, timedelta


Django has a built-in parser django.utils.dateparse.parse_duration that could be used here

I took a look on the built-in parser. Unfortunately it has different standards from what we are following.

standard: "30 00:00:00" (%d %H:%M:%S)

ISO 8601: "P30D"

postgres: "3 days 04:05:06" (only works for days)
I believe we might benefit more of keeping our custom parser, we could even extend it to include weeks or months, especially in a scenario where we are working with longer ranges.
What you think?

nuclearcat · 2026-07-07T06:52:41Z

Each batch re-evaluates the full nested predicate. Add on top batch might be too small (we need to check how many tests per day we have, how long take deletion of 10k).
Consider materializing the ids once into a CREATE TEMP TABLE tobedeleted_... AS SELECT id .., and then batch-deleting by joining against it. Temp tables survive across the per-batch commits within the same session, so this fits the existing design.

alanpeixinho · 2026-07-10T15:14:11Z

Each batch re-evaluates the full nested predicate. Add on top batch might be too small (we need to check how many tests per day we have, how long take deletion of 10k). Consider materializing the ids once into a CREATE TEMP TABLE tobedeleted_... AS SELECT id .., and then batch-deleting by joining against it. Temp tables survive across the per-batch commits within the same session, so this fits the existing design.

Nice catch @nuclearcat . I have included a temp table to store ids before batch deleting.

* Delete rows older than a given age with manual cascade (checkout -> builds -> tests) * --origins scoping, --dry-run and a --yes confirmation. Extract parse_interval into a shared helper and add integration tests. Closes kernelci#1969 Signed-off-by: Alan Peixinho <alan.peixinho@profusion.mobi>

Signed-off-by: Alan Peixinho <alan.peixinho@profusion.mobi>

…tests. * Delete rows older than a given age with manual cascade (checkout -> builds -> tests) * --origins scoping, --dry-run and a --yes confirmation. Extract parse_interval into a shared helper and add integration tests. * Add tables parameter. * Protect issue related rows by default, and add parameter to bypass. Signed-off-by: Alan Peixinho <alan.peixinho@profusion.mobi>

Signed-off-by: Alan Peixinho <alan.peixinho@profusion.mobi>

mentonin · 2026-07-19T23:13:34Z

+                    return
+
+                if dry_run:
+                    self.stdout.write(


Should the warning go to stderr instead?

mentonin · 2026-07-19T23:16:43Z

+                    summary = ", ".join(f"{counts[t]} {t}" for t in selected_tables)
+                    try:
+                        answer = (
+                            input(f"Delete {summary} ({total} rows total)? [y/N] ")


We already printed counts + total, can we make this shorter?

Good point. Changed to simple message to confirm.

mentonin · 2026-07-19T23:20:00Z

+                        self.stdout.write("Aborted.")
+                        return
+
+                # Delete child-first (reverse of PRUNABLE_TABLES order): each batch


Document the strict parent-before-child order of PRUNABLE_TABLES at its definition

felipebergamin · 2026-07-21T22:16:54Z

+            f"DELETE FROM {temp_table} WHERE id IN "
+            f"(SELECT id FROM {temp_table} LIMIT %(batch_size)s) RETURNING id"
+            f") DELETE FROM {table} WHERE id IN (SELECT id FROM batch)"


even we are filtering unknown tables from command args with PRUNABLE_TABLES I think we should go safer here and quote the identifiers. DELETE FROM "{table}"

* Send dry-run warning to stderr * Shorten confirmation prompt now that counts are already printed * Document PRUNABLE_TABLES parent-before-child order * Quote table identifiers in prune SQL Signed-off-by: Alan Peixinho <alan.peixinho@profusion.mobi>

alanpeixinho force-pushed the feat/prune-age branch 2 times, most recently from e5c35c3 to 552bcc4 Compare July 3, 2026 20:04

felipebergamin reviewed Jul 6, 2026

View reviewed changes

alanpeixinho requested a review from felipebergamin July 6, 2026 21:20

mentonin reviewed Jul 6, 2026

View reviewed changes

alanpeixinho requested a review from mentonin July 7, 2026 12:45

alanpeixinho force-pushed the feat/prune-age branch 2 times, most recently from 9a454fa to 6995fc0 Compare July 8, 2026 21:16

alanpeixinho added 7 commits July 15, 2026 15:51

fix: Add DB env to integration

7ee4385

Signed-off-by: Alan Peixinho <alan.peixinho@profusion.mobi>

fixup! fix: Add DB env to integration

06c69f2

Signed-off-by: Alan Peixinho <alan.peixinho@profusion.mobi>

feat: Use temp table for deletion

e275881

Signed-off-by: Alan Peixinho <alan.peixinho@profusion.mobi>

alanpeixinho force-pushed the feat/prune-age branch from 6995fc0 to e275881 Compare July 15, 2026 18:51

mentonin reviewed Jul 19, 2026

View reviewed changes

felipebergamin reviewed Jul 21, 2026

View reviewed changes

alanpeixinho force-pushed the feat/prune-age branch from 8063d33 to 87a607e Compare July 22, 2026 13:51

Uh oh!

Conversation

alanpeixinho commented Jul 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Usage

How to Test

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tales-aparecida commented Jul 6, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nuclearcat commented Jul 7, 2026

Uh oh!

alanpeixinho commented Jul 10, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

alanpeixinho commented Jul 3, 2026 •

edited

Loading