Python version of phase_stats.py calculation by ArneTR · Pull Request #1787 · green-coding-solutions/green-metrics-tool

ArneTR · 2026-07-22T11:01:48Z

This PR speeds up the processing for the phase stats by moving calculation out of the DB hotpath.

Analysis

We first assumed the bottle neck was the SQL queries, which became quite complex, or the big database with the big index (In our live DB of ~ 100 GB of raw measurement values)

However the slow down was the network connects. We make a separate SQL query per metric and phase. This can sum up to 6 x 30 calls easily. With setup and teardown each network call takes about 1 second.

Now we only do one big query to fetch all the values resulting in a 100x speedup

Summary

Replaced repeated per-phase, per-metric SQL aggregation with one full-run fetch and in-memory Python calculations.
Added percentile, derivative, sampling-rate, and phase-boundary handling to mirror the previous PostgreSQL behavior.
Moved carbon-intensity provider resolution into metric aggregation, including warnings for multiple providers.
Added MAX_POSTGRES_BIGINT and centralized Decimal() handling.
Expected to significantly reduce database connection overhead and improve processing speed.

ArneTR · 2026-07-22T11:01:58Z

@coderabbitai review

coderabbitai · 2026-07-22T11:02:05Z

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

coderabbitai · 2026-07-22T11:07:52Z

Warning

Review limit reached

@ArneTR, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 39 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 9fabd493-34aa-436a-9042-3889287c22b2

📥 Commits

Reviewing files that changed from the base of the PR and between 2f37f9f and 8d41a9b.

📒 Files selected for processing (1)

lib/phase_stats.py

📝 Walkthrough

Walkthrough

Phase-stat aggregation now fetches all run measurements once, groups them into per-metric time series, and computes phase aggregates in memory. New helpers reproduce percentile, derivative, sampling-rate, and phase-boundary calculations, including the first post-phase sample where applicable. The phase loop uses these calculations for each metric and resolves carbon-intensity providers while recording warnings for multiple providers.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly matches the main change: replacing phase_stats.py calculations with a Python implementation.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

coderabbitai

Actionable comments posted: 2

🧹 Nitpick comments (1)

tools/phase_stats.py (1)
28-28: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Optional: trailing space in status message / prefer logging.

print('Fetched data for ', args.run_id) emits "Fetched data for <id>" (double space from the comma separator). Minor cosmetic nit; consider an f-string or the project's logger for consistency.
♻️ Optional tweak
-    print('Fetched data for ', args.run_id)
+    print(f'Fetched data for {args.run_id}')

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: faa34ad5-df0f-4d1c-848d-42f244669fae

📥 Commits

Reviewing files that changed from the base of the PR and between b728830 and cd966e7.

📒 Files selected for processing (2)

lib/phase_stats.py
tools/phase_stats.py

ArneTR · 2026-07-22T12:12:53Z

@coderabbitai review

coderabbitai · 2026-07-22T12:12:59Z

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[skip ci]

ArneTR · 2026-07-23T12:07:54Z

@ribalba PR now ready for review. Since it is in the crucial calculation part please give it an extra pair of eyes

ArneTR · 2026-07-23T12:08:04Z

@coderabbitai review

coderabbitai · 2026-07-23T12:13:44Z

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

coderabbitai · 2026-07-23T12:14:17Z

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

github-actions · 2026-07-23T12:34:03Z

Eco CI Output [RUN-ID: 30005732884]:

Label	🖥 avg. CPU utilization [%]	🔋 Total Energy [Wh]	🔌 avg. Power [Watts]	Duration [Seconds]

Measurement #1	30.1481	1.8680	4.23	1590.40

Total Run	30.15	1.8680	4.23	1590.40

Additional overhead from Eco CI	N/A	0.0049	4.32	4.10

🌳 CO2 Data:
City: CONSTANT, Lat: , Lon:
IP:
CO₂ from energy is: 0.431519550 g
CO₂ from manufacturing (embodied carbon) is: 0.453763080 g
Carbon Intensity for this location: 231 gCO₂eq/kWh
SCI: 0.885283 gCO₂eq / pipeline run emitted

Total cost of whole PR so far:

[skip ci] * main: (fix): Custom metrics now allows valid floats and even negative numbers also (ci): Added Eco-CI measurement after setup (test-fix): Downgrading python version as 3.15 did not exist yet Upgraded Python version in CI/CD (feat): Making GitHub Action two step to debug setup vs. run better Bump actions/setup-python from 6 to 7 in /.github/actions/gmt-pytest (#1786) Bump fastapi from 0.139.0 to 0.139.2 (#1780) (feat): All config files are now in a config folder instead of clobbering the root dir and docker dir (#1793) Updated Cloud Energy Bump pandas from 3.0.3 to 3.0.5 (#1790) Bump aiohttp from 3.14.2 to 3.14.3 (#1792) Bump cachetools from 7.1.4 to 7.1.5 (#1791) (fix): 0 Values in CPU Utilization Provider for linux do not lead to Floating Point Exception anymore

ribalba

Looks ok. Few minor things

ribalba · 2026-07-29T18:02:20Z

+                if carbon_intensity is not None:
+                    phase_warnings.add(f"More than one carbon intensity provider is configured. Now using {metric}")
+                carbon_intensity = value_avg
+                chosen_carbon_metric_name = metric


Must't this be set before all the calculations are done?

ribalba · 2026-07-29T18:04:44Z

+        derivative_max = weighted_derivative_max # pylint: disable=possibly-used-before-assignment
+        derivative_min = weighted_derivative_min # pylint: disable=possibly-used-before-assignment
+
+    return (


I would probably make this an object as I found this quite confusing to map to vars

ribalba · 2026-07-29T18:05:41Z

+        WHERE mm.run_id = %s
+        ORDER BY mv.measurement_metric_id ASC, mv.time ASC
+    """
+    metric_time_series = {}


Will this not become huge on a very long run? What happens when you run this on a 45 min run?

ArneTR added 2 commits July 22, 2026 12:46

Python version

556cb7f

More clear acces instead of implied reference access

cd966e7

coderabbitai Bot reviewed Jul 22, 2026

View reviewed changes

Comment thread lib/phase_stats.py Outdated

Comment thread lib/phase_stats.py Outdated

ArneTR added 4 commits July 22, 2026 13:22

(style)

dd91d48

(style): Clarified some code implementations with comments

af848ed

Removed wrong commit part [skip ci]

7fc573c

Doing Decimal() in helper function

2f37f9f

ArneTR added 2 commits July 23, 2026 13:44

Clarifying comments

c9d37d7

[skip ci]

Rework

8d41a9b

ArneTR requested a review from ribalba July 23, 2026 12:07

ribalba approved these changes Jul 29, 2026

View reviewed changes

Uh oh!

Conversation

ArneTR commented Jul 22, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Analysis

Summary

Uh oh!

ArneTR commented Jul 22, 2026

Uh oh!

coderabbitai Bot commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coderabbitai Bot commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review limit reached

Walkthrough

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ArneTR commented Jul 22, 2026

Uh oh!

coderabbitai Bot commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ArneTR commented Jul 23, 2026

Uh oh!

ArneTR commented Jul 23, 2026

Uh oh!

coderabbitai Bot commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coderabbitai Bot commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 23, 2026

Uh oh!

ribalba left a comment

Choose a reason for hiding this comment

Uh oh!

ribalba Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

ribalba Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

ribalba Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ArneTR commented Jul 22, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Jul 22, 2026 •

edited

Loading

coderabbitai Bot commented Jul 22, 2026 •

edited

Loading

coderabbitai Bot commented Jul 22, 2026 •

edited

Loading

coderabbitai Bot commented Jul 23, 2026 •

edited

Loading

coderabbitai Bot commented Jul 23, 2026 •

edited

Loading