Skip to content

fix(worker): Optimize ProcessFlakesTask N+1 queries for testruns and flakes#1213

Open
sentry[bot] wants to merge 1 commit into
mainfrom
seer/fix/worker-process-flakes-n1-ZOWj7Q
Open

fix(worker): Optimize ProcessFlakesTask N+1 queries for testruns and flakes#1213
sentry[bot] wants to merge 1 commit into
mainfrom
seer/fix/worker-process-flakes-n1-ZOWj7Q

Conversation

@sentry

@sentry sentry Bot commented Jun 20, 2026

Copy link
Copy Markdown
Contributor

This PR addresses an N+1 query performance issue in the app.tasks.flakes.ProcessFlakesTask by optimizing database interactions related to Testrun and Flake objects.

Problem:
Previously, process_flakes_for_commit would iterate over each ReportSession (upload) and:

  1. Call get_testruns(upload) for each upload, resulting in a separate database query for Testrun objects per upload.
  2. Call Testrun.objects.bulk_update for each upload after processing, leading to multiple update operations.
  3. In handle_pass, individual flake.save() calls were made when a flake expired, causing additional single-row updates.

Solution:

  1. Batch Testrun Fetching: A new function get_testruns_for_uploads was introduced to fetch all relevant Testrun objects for all uploads in a single batched query. These are then grouped by upload_id in Python.
  2. Consolidated Testrun Updates: process_single_upload was refactored to accept pre-fetched testruns. All modified Testrun objects are now accumulated and Testrun.objects.bulk_update is called only once after processing all uploads.
  3. Optimized Flake Updates: The individual flake.save() call in handle_pass for expiring flakes has been removed. Expiring flakes are now marked with end_date and handled by the existing Flake.objects.bulk_create(update_conflicts=True) operation at the end of process_flakes_for_commit, which efficiently updates or creates flakes in bulk.

Legal Boilerplate

Look, I get it. The entity doing business as "Sentry" was incorporated in the State of Delaware in 2015 as Functional Software, Inc. In 2022 this entity acquired Codecov and as result Sentry is going to need some rights from me in order to utilize my contributions in this PR. So here's the deal: I retain all rights, title and interest in and to my contributions, and by keeping this boilerplate intact I confirm that Sentry can use, modify, copy, and redistribute my contributions, under Sentry's choice of terms.

Fixes WORKER-YTH

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants