Reduce resource footprint of project metrics updates#6655
Merged
nscuro merged 1 commit intoJul 12, 2026
Conversation
✅ Snyk checks have passed. No issues have been found so far.
💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse. |
Up to standards ✅🟢 Issues
|
| Metric | Results |
|---|---|
| Coverage variation | ✅ +0.11% coverage variation (-1.00%) |
| Diff coverage | ✅ ∅ diff coverage (70.00%) |
Coverage variation details
Coverable lines Covered lines Coverage Common ancestor commit (341c917) 42816 37138 86.74% Head commit (d4c7216) 42816 (+0) 37187 (+49) 86.85% (+0.11%) Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch:
<coverage of head commit> - <coverage of common ancestor commit>
Diff coverage details
Coverable lines Covered lines Diff coverage Pull request (#6655) 0 0 ∅ (not applicable) Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified:
<covered lines added or modified>/<coverable lines added or modified> * 100%
NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.
2 tasks
Improves the efficiency of project metrics updates by calculating metrics of all components of a project in bulk, rather than calling the `UPDATE_COMPONENT_METRICS` stored proc in a loop. Adds a new `COMPUTE_COMPONENT_METRICS` function so the functionality can be shared between metrics updates of full projects, and of individual components. Also significantly reduces I/O overhead by being smarter about when to write new metrics rows. Most notably, this change skips row creation if another row with identical metrics values already exists *for the same day*. Note that this means that the `LAST_OCCURRENCE` column can no longer be used to judge metrics freshness *within a day*. Since as of v5, metrics aggregations have day-resolution, intra-day data points with identical values provide no value there. `LAST_OCCURRENCE` still advances at least once per day, since the first update of a day always inserts. Only intra-day resolution is lost, and nothing (we know of) relies on it. The I/O savings outweigh keeping the legacy behavior in any case. For a large project with 9k components in a test instance, runtime of `UPDATE_PROJECT_METRICS` dropped from ~3.5s to ~300ms. In the ideal case (successive runs on the same day without metrics changes), the previous implementation caused 3MB of WAL writes, where the new implementation causes single-digit kB writes. Signed-off-by: nscuro <nscuro@protonmail.com>
nscuro
force-pushed
the
reduce-project-metrics-update-io-cost
branch
from
ae8294c to
d4c7216
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Improves the efficiency of project metrics updates by calculating metrics of all components of a project in bulk, rather than calling the
UPDATE_COMPONENT_METRICSstored proc in a loop. Adds a newCOMPUTE_COMPONENT_METRICSfunction so the functionality can be shared between metrics updates of full projects, and of individual components.Also significantly reduces I/O overhead by being smarter about when to write new metrics rows. Most notably, this change skips row creation if another row with identical metrics values already exists for the same day.
Addressed Issue
Relates to #6642
Metrics updates showed up as main contributor to DB / system load there.
Additional Details
Note that this means that the
LAST_OCCURRENCEcolumn can no longer be used to judge metrics freshness within a day. Since as of v5, metrics aggregations have day-resolution, intra-day data points with identical values provide no value there.LAST_OCCURRENCEstill advances at least once per day, since the first update of a day always inserts. Only intra-day resolution is lost, and nothing (we know of) relies on it. The I/O savings outweigh keeping the legacy behavior in any case.For a large project with 9k components in a test instance, runtime of
UPDATE_PROJECT_METRICSdropped from ~3.5s to ~300ms. In the ideal case (successive runs on the same day without metrics changes), the previous implementation caused 3MB of WAL writes, where the new implementation causes single-digit kB writes.Checklist
This PR fixes a defect, and I have provided tests to verify that the fix is effectiveThis PR introduces changes to the database model, and I have updated the migration changelog accordinglyThis PR introduces new or alters existing behavior, and I have updated the documentation accordinglyThis PR is a substantial change (per the ADR criteria), and I have added an ADR underdocs/adr/