Skip to content

Commit 6589e77

Browse files
authored
fix: semantic memory leaks and stabilize IaC orchestration
2 parents 87e0e31 + 15f95c9 commit 6589e77

34 files changed

+807
-422
lines changed

.gcp/terraforms/iam_bindings.tf

Lines changed: 12 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -12,18 +12,21 @@ resource "google_service_account_iam_member" "github_deployer_sa" {
1212
# Roles for github-actions-deployer
1313
locals {
1414
deployer_roles = [
15-
"roles/run.developer",
16-
"roles/workflows.editor",
17-
"roles/cloudscheduler.admin",
18-
"roles/iam.serviceAccountUser"
15+
"roles/run.developer", # Manage Cloud Run jobs
16+
"roles/workflows.editor", # Manage Workflows
17+
"roles/cloudscheduler.admin", # Manage Scheduler
18+
"roles/iam.serviceAccountUser", # Act as SAs for jobs
19+
"roles/artifactregistry.admin", # Manage Artifact Registry
20+
"roles/eventarc.admin", # Manage Eventarc triggers
21+
"roles/storage.admin", # Manage buckets and state locking
22+
"roles/resourcemanager.projectIamAdmin", # Manage the IAM bindings in this code
23+
"roles/iam.workloadIdentityPoolAdmin", # Manage WIF in wif.tf
24+
"roles/monitoring.admin", # Manage Monitoring in monitoring.tf
25+
"roles/iam.serviceAccountAdmin", # Manage Alert policies in monitoring.tf
26+
"roles/iam.admin" # Manage Iam roles
1927
]
2028
}
2129

22-
# Enables to:
23-
# Deploy Cloud Run containers
24-
# Deploy pipeline-dispatcher.yml
25-
# Update the cron job
26-
# Attach SAs to Cloud Run
2730
resource "google_project_iam_member" "github_deployer_permissions" {
2831
for_each = toset(local.deployer_roles)
2932
project = var.project_id

.github/workflows/ci-infra.yml

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,11 @@ on:
1212
workflow_dispatch:
1313
workflow_call:
1414

15+
# Add concurrency control to prevent state lock issues
16+
concurrency:
17+
group: terraform-${{ env.ENV }}
18+
cancel-in-progress: false
19+
1520
permissions:
1621
contents: 'read'
1722
id-token: 'write'
@@ -50,4 +55,4 @@ jobs:
5055
TF_VAR_region: ${{ env.REGION }}
5156
TF_VAR_github_repo: ${{ env.GITHUB_REPO }}
5257
TF_VAR_alert_email_map: ${{ secrets.ALERT_EMAIL_MAP }}
53-
run: terraform apply -auto-approve -lock=false
58+
run: terraform apply -auto-approve

README.md

Lines changed: 48 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@ Small to mid-sized organizations are trapped in a cycle where they outgrow the c
1212
This project solves that challenge by delivering a highly resilient, event-driven data pipeline on Google Cloud Platform for reliable operational analytics. It guarantees data integrity through a strict Medallion architecture (Bronze, Silver, Gold) that relies on rigid data contracts and validation gates to catch and isolate bad data early in the lifecycle.
1313

1414
### Defensive Pipeline Architecture
15-
![WIP_pipeline_diagram_picture](https://still-working-on-it.need-to-finish-readme.first)
15+
![pipeline-orchestration-diagram](/assets/diagrams/01-pipeline-orchestration-diagram.png)
1616

1717
To eliminate the risk of cross-run data contamination and memory bloat, the pipeline employs a defensive state-management strategy where local compute environments are strictly temporary:
1818
* **Stateless Orchestration:** Every execution operates within an isolated, deterministic `run_id` workspace that is aggressively purged post-run.
@@ -67,29 +67,60 @@ The pipeline does not just move data; it actively defends the analytical layer f
6767

6868
The pipeline is explicitly engineered to process massive datasets within the rigid memory constraints of serverless compute (Cloud Run). By leveraging the Polars Rust engine (Lazy API & Streaming), the system achieves near-perfect memory density, operating consistently at the physical hardware ceiling.
6969

70-
**GCP Stress-Test Metrics (18 Million Row Snapshot)**
70+
### GCP Stress-Test Metrics (Scaling Efficiency)
7171

72-
![engine-performance-8gb](/assets/screenshots/engine-performance-8gb-2cpu.png)
72+
| 18M Snapshot (8GiB / 2 vCPU) | 36M Snapshot (16GiB / 4 vCPU) |
73+
| :---: | :---: |
74+
| ![engine-performance-8gb](/assets/screenshots/engine-performance-8gb-2cpu.png) | ![engine-performance-16gb](/assets/screenshots/engine-performance-16gb-4cpu.png) |
7375

74-
> The data used for this chart [`benchmarks/`](/assets/benchmarks/polars/18mrows_dataset_stats_log.csv) and the 18m rows dataset can be found her [`data/`](/data/)
76+
> Benchmark data: [`18m_stats_log.csv`](/assets/benchmarks/polars/18mrows_dataset_stats_log.csv) and [`36m_stats_log.csv`](/assets/benchmarks/polars/36mrows_dataset_stats_log.csv)
7577
78+
| Metric | 18M Rows (8GB / 2 vCPU) | 36M Rows (16GB / 4 vCPU) |
79+
| :--- | :--- | :--- |
80+
| **Throughput (Processing)** | ~116,000 Rows / Second | ~220,000 Rows / Second |
81+
| **Total Runtime (Wall-Clock)** | 02m 34s | 02m 43s |
82+
| **Memory Tax (Fixed)** | ~1.5 GiB | ~1.5 GiB |
83+
| **Effective Data Headroom** | ~6.5 GiB | ~14.5 GiB |
7684

77-
| Metric | Value (18M Row Peak Load) |
78-
| :--- | :--- |
79-
| **Throughput (Processing)** | ~116,000 Rows / Second |
80-
| **Total Runtime (Wall-Clock)** | 02m 34s |
81-
| **Compute Provision** | 2 vCPU / 8 GiB |
82-
| **Memory Tax (Fixed)** | ~1.5 GiB (OS / Sandbox / IO Buffers) |
83-
| **Effective Data Headroom** | ~6.5 GiB (Active Transformation) |
84-
85-
* **Linear Vertical Scaling:** Bumping the Cloud Run provision to 32GiB allows the same architecture to process ~72 Million rows without code changes.
86-
* **Predictable Capacity:** Identifying the 1.5GB "Memory Tax" allows for precise resource governance, ensuring jobs never fail due to unpredictable Signal 9 (OOM) events.
85+
* **Near-Linear Performance Scaling:** Doubling the compute and dataset size results in only a 9-second increase in wall-clock time, effectively doubling the throughput as the Polars engine saturates the additional vCPUs.
86+
* **Predictable Capacity:** Identifying the "Memory Tax" (OS/IO overhead) allows for precise resource governance, ensuring jobs never fail due to unpredictable Signal 9 (OOM) events.
8787
* **Zero-Idle Economics:** 100% serverless execution ensures zero billable time during idle periods, significantly reducing the Total Cost of Ownership (TCO) compared to dedicated cluster solutions.
8888

89-
**Measurement Methodology**
89+
### Cost Efficiency & Free-Tier
90+
91+
The pipeline's processing speed allows for a full analytical rebuild of 36M rows while remaining comfortably within the **GCP Cloud Run Free Tier** (180k vCPU-sec, 360k GiB-sec). This means a small-to-mid-sized organization can run this production-grade pipeline multiple times a day with **zero compute costs.**
92+
93+
| Compute Provision | Dataset | vCPU-Seconds / Run | GiB-Seconds / Run | Monthly Free-Tier Runs |
94+
| :--- | :--- | :--- | :--- | :--- |
95+
| **8 GiB / 2 vCPU** | ~18m rows | 308 | 1,232 | **~292 Runs / Month** |
96+
| **16 GiB / 4 vCPU** | ~36m rows | 652 | 2,608 | **~138 Runs / Month** |
97+
| **32 GiB / 8 vCPU** | ~72m rows | 1,304 | 5,216 | **~69 Runs / Month** |
98+
99+
> *Calculations based on verified benchmarks. Even at the highest 32GiB tier, the pipeline can execute a full state rebuild twice daily for $0*
100+
101+
### Measurement Methodology
90102
* **Performance Profiling:** Captured from production telemetry via the pipeline's native `run_duration` metadata, calculating the precise delta between `started_at` and `completed_at` timestamps.
91-
* **Memory Utilization:** Monitored via an integrated [`psutil.virtual_memory().used`](/assets/benchmarks/polars/README.md) profiling implementation to verify the actual resource footprint and confirm the physical ceiling for an 8GiB provision.
92-
* **Throughput Efficiency:** Leverages Polars' streaming evaluation to maintain high throughput and minimize CPU idle time during GCS I/O, providing a significant performance advantage over traditional eager-loading engines.
103+
* **Memory Utilization:** Monitored via an integrated [`psutil.virtual_memory().used`](/assets/benchmarks/polars/README.md) profiling implementation to verify the actual resource footprint and confirm the physical ceiling for 8GiB/16GiB provision.
104+
* **Throughput Efficiency:** Leverages Polars streaming evaluation to maintain high throughput and minimize CPU idle time during GCS I/O, providing a significant performance advantage over traditional eager-loading engines.
105+
106+
### **Scaling Roadmap: From Serverless to Enterprise Lakehouse**
107+
108+
To ensure the architecture survives the transition from millions to billions of rows, the pipeline is designed to evolve across three validated scaling paths. This roadmap prioritizes cost-efficiency at low volumes while providing a clear architectural pivot for enterprise-scale workloads.
109+
110+
#### **Stage 1: Temporal Sharding (Vertical Efficiency)**
111+
* **Strategy:** Refactor the `Assemble` stage to iterate through **yearly batch partitions** while `Semantic` stage to **streams output directly** to a GCS staging location.
112+
* **Publish Evolution:** Moves to a **Partitioned Atomic Swap**. Yearly shards are streamed directly to a staged GCS version prefix. The `Integrity Gate` validates cloud-side completeness before the `latest_version.json` pointer is updated.
113+
* **Trade-off:** **Latency vs. Memory.** Significantly increases total wall-clock time due to repeated I/O cycles, but allows 32GiB instances to process 100M+ rows by isolating join-intensity to specific temporal shards.
114+
115+
#### **Stage 2: Incremental Delta Architecture (Event-Driven)**
116+
* **Strategy:** Transition from a "Full Rebuild" batch model to a **Stateless Delta Propagation** model, processing only active deltas.
117+
* **Publish Evolution:** Moves to a **Checkpoint-based Commit**. Folder-based versioning is replaced by an atomic merge into the Gold layer. The "Pointer" evolves into a metadata watermark signifying data freshness to downstream consumers.
118+
* **Trade-off:** **Simplicity vs. Scale.** Eliminates memory constraints and reduces runtime costs, but sacrifices easy "point-in-time" folder recovery. Requires "Last-Mile" deduplication logic (e.g., SQL Views) for downstream consumers.
119+
120+
#### **Stage 3: BigQuery "Engine-as-a-Service" (The Enterprise Pivot)**
121+
* **Strategy:** Offload the `Assemble` and `Semantic` compute layers entirely to **BigQuery (ELT Pattern)**.
122+
* **Publish Evolution:** Moves to a **Atomic View Redirection**. The Python "Gatekeeper" builds semantics in a staging dataset and runs SQL-driven integrity checks. Publication is achieved by an atomic swap of a BigQuery Authorized View, replacing the file-based pointer system.
123+
* **Trade-off:** **Cost vs. Capability.** Provides an infinite scaling ceiling and removes all local infrastructure bounds, but introduces higher cost-per-query overhead and requires transitioning from local Parquet files to managed cloud storage.
93124

94125

95126
## Observability & Alerting
Lines changed: 165 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,165 @@
1+
view,timestamp,logger,memory,unit
2+
DEFAULT,2026-04-09T21:43:40.178678Z,METRIC_MEM:,2812.83,MB
3+
DEFAULT,2026-04-09T21:43:41.178984Z,METRIC_MEM:,2868.2,MB
4+
DEFAULT,2026-04-09T21:43:42.179341Z,METRIC_MEM:,2931.05,MB
5+
DEFAULT,2026-04-09T21:43:43.179687Z,METRIC_MEM:,2994.48,MB
6+
DEFAULT,2026-04-09T21:43:44.179962Z,METRIC_MEM:,3049.48,MB
7+
DEFAULT,2026-04-09T21:43:45.180339Z,METRIC_MEM:,3111.88,MB
8+
DEFAULT,2026-04-09T21:43:46.180721Z,METRIC_MEM:,3175.25,MB
9+
DEFAULT,2026-04-09T21:43:47.181107Z,METRIC_MEM:,3246.77,MB
10+
DEFAULT,2026-04-09T21:43:48.181558Z,METRIC_MEM:,3309.72,MB
11+
DEFAULT,2026-04-09T21:43:49.181470Z,METRIC_MEM:,3364.92,MB
12+
DEFAULT,2026-04-09T21:43:50.181366Z,METRIC_MEM:,3427.7,MB
13+
DEFAULT,2026-04-09T21:43:51.181351Z,METRIC_MEM:,3483.11,MB
14+
DEFAULT,2026-04-09T21:43:52.181694Z,METRIC_MEM:,3554.01,MB
15+
DEFAULT,2026-04-09T21:43:53.182108Z,METRIC_MEM:,3609.08,MB
16+
DEFAULT,2026-04-09T21:43:54.182588Z,METRIC_MEM:,3672.5,MB
17+
DEFAULT,2026-04-09T21:43:55.182979Z,METRIC_MEM:,3726.8,MB
18+
DEFAULT,2026-04-09T21:43:56.183486Z,METRIC_MEM:,3790.45,MB
19+
DEFAULT,2026-04-09T21:43:57.183846Z,METRIC_MEM:,3854.01,MB
20+
DEFAULT,2026-04-09T21:43:58.184338Z,METRIC_MEM:,3916.96,MB
21+
DEFAULT,2026-04-09T21:43:59.184674Z,METRIC_MEM:,3980.06,MB
22+
DEFAULT,2026-04-09T21:44:00.185069Z,METRIC_MEM:,4043.16,MB
23+
DEFAULT,2026-04-09T21:44:01.185349Z,METRIC_MEM:,4098.34,MB
24+
DEFAULT,2026-04-09T21:44:02.185794Z,METRIC_MEM:,4161.24,MB
25+
DEFAULT,2026-04-09T21:44:03.190091Z,METRIC_MEM:,4224.66,MB
26+
DEFAULT,2026-04-09T21:44:04.186657Z,METRIC_MEM:,4287.29,MB
27+
DEFAULT,2026-04-09T21:44:05.187081Z,METRIC_MEM:,4342.34,MB
28+
DEFAULT,2026-04-09T21:44:06.187557Z,METRIC_MEM:,4405.66,MB
29+
DEFAULT,2026-04-09T21:44:07.187831Z,METRIC_MEM:,4467.94,MB
30+
DEFAULT,2026-04-09T21:44:08.188119Z,METRIC_MEM:,4523.98,MB
31+
DEFAULT,2026-04-09T21:44:09.188007Z,METRIC_MEM:,4586.83,MB
32+
DEFAULT,2026-04-09T21:44:10.187799Z,METRIC_MEM:,4649.84,MB
33+
DEFAULT,2026-04-09T21:44:11.187840Z,METRIC_MEM:,4705.04,MB
34+
DEFAULT,2026-04-09T21:44:12.187947Z,METRIC_MEM:,4768.01,MB
35+
DEFAULT,2026-04-09T21:44:13.188380Z,METRIC_MEM:,4831.27,MB
36+
DEFAULT,2026-04-09T21:44:14.189078Z,METRIC_MEM:,4894.65,MB
37+
DEFAULT,2026-04-09T21:44:15.189546Z,METRIC_MEM:,4958.21,MB
38+
DEFAULT,2026-04-09T21:44:16.189898Z,METRIC_MEM:,5020.09,MB
39+
DEFAULT,2026-04-09T21:44:17.190829Z,METRIC_MEM:,5075.54,MB
40+
DEFAULT,2026-04-09T21:44:18.191150Z,METRIC_MEM:,5138.73,MB
41+
DEFAULT,2026-04-09T21:44:19.191518Z,METRIC_MEM:,5182.56,MB
42+
DEFAULT,2026-04-09T21:44:20.197169Z,METRIC_MEM:,5623.91,MB
43+
DEFAULT,2026-04-09T21:44:21.197587Z,METRIC_MEM:,6033.96,MB
44+
DEFAULT,2026-04-09T21:44:22.201299Z,METRIC_MEM:,6488.75,MB
45+
DEFAULT,2026-04-09T21:44:23.198486Z,METRIC_MEM:,6785.55,MB
46+
DEFAULT,2026-04-09T21:44:24.198818Z,METRIC_MEM:,6895.43,MB
47+
DEFAULT,2026-04-09T21:44:25.199286Z,METRIC_MEM:,7071.41,MB
48+
DEFAULT,2026-04-09T21:44:26.199620Z,METRIC_MEM:,7220.95,MB
49+
DEFAULT,2026-04-09T21:44:27.200130Z,METRIC_MEM:,7458.04,MB
50+
DEFAULT,2026-04-09T21:44:28.200477Z,METRIC_MEM:,7530.45,MB
51+
DEFAULT,2026-04-09T21:44:29.200378Z,METRIC_MEM:,7652.43,MB
52+
DEFAULT,2026-04-09T21:44:30.200421Z,METRIC_MEM:,7838.43,MB
53+
DEFAULT,2026-04-09T21:44:31.200275Z,METRIC_MEM:,8050.17,MB
54+
DEFAULT,2026-04-09T21:44:32.200504Z,METRIC_MEM:,8270.6,MB
55+
DEFAULT,2026-04-09T21:44:33.200943Z,METRIC_MEM:,8439.16,MB
56+
DEFAULT,2026-04-09T21:44:34.201362Z,METRIC_MEM:,8777.46,MB
57+
DEFAULT,2026-04-09T21:44:35.201918Z,METRIC_MEM:,8934.88,MB
58+
DEFAULT,2026-04-09T21:44:36.202212Z,METRIC_MEM:,8910.29,MB
59+
DEFAULT,2026-04-09T21:44:37.202609Z,METRIC_MEM:,7358.61,MB
60+
DEFAULT,2026-04-09T21:44:38.204970Z,METRIC_MEM:,7661.91,MB
61+
DEFAULT,2026-04-09T21:44:39.209060Z,METRIC_MEM:,7835.91,MB
62+
DEFAULT,2026-04-09T21:44:40.209130Z,METRIC_MEM:,7907.15,MB
63+
DEFAULT,2026-04-09T21:44:41.204203Z,METRIC_MEM:,8154.41,MB
64+
DEFAULT,2026-04-09T21:44:42.209313Z,METRIC_MEM:,8405.19,MB
65+
DEFAULT,2026-04-09T21:44:43.209838Z,METRIC_MEM:,7903.66,MB
66+
DEFAULT,2026-04-09T21:44:44.210360Z,METRIC_MEM:,8222.37,MB
67+
DEFAULT,2026-04-09T21:44:45.210692Z,METRIC_MEM:,8588.28,MB
68+
DEFAULT,2026-04-09T21:44:46.213686Z,METRIC_MEM:,9032.81,MB
69+
DEFAULT,2026-04-09T21:44:47.211559Z,METRIC_MEM:,9398.2,MB
70+
DEFAULT,2026-04-09T21:44:48.211871Z,METRIC_MEM:,9777.2,MB
71+
DEFAULT,2026-04-09T21:44:49.211810Z,METRIC_MEM:,10187.28,MB
72+
DEFAULT,2026-04-09T21:44:50.211790Z,METRIC_MEM:,10544.79,MB
73+
DEFAULT,2026-04-09T21:44:51.211702Z,METRIC_MEM:,10918.25,MB
74+
DEFAULT,2026-04-09T21:44:52.226469Z,METRIC_MEM:,11645.3,MB
75+
DEFAULT,2026-04-09T21:44:53.226946Z,METRIC_MEM:,12692.11,MB
76+
DEFAULT,2026-04-09T21:44:54.227541Z,METRIC_MEM:,13956.78,MB
77+
DEFAULT,2026-04-09T21:44:55.228051Z,METRIC_MEM:,13874.34,MB
78+
DEFAULT,2026-04-09T21:44:56.228511Z,METRIC_MEM:,12523.92,MB
79+
DEFAULT,2026-04-09T21:44:57.228953Z,METRIC_MEM:,12711.82,MB
80+
DEFAULT,2026-04-09T21:44:58.229409Z,METRIC_MEM:,12737.88,MB
81+
DEFAULT,2026-04-09T21:44:59.229860Z,METRIC_MEM:,12783.1,MB
82+
DEFAULT,2026-04-09T21:45:00.230271Z,METRIC_MEM:,12872.66,MB
83+
DEFAULT,2026-04-09T21:45:01.230679Z,METRIC_MEM:,12877.08,MB
84+
DEFAULT,2026-04-09T21:45:02.231040Z,METRIC_MEM:,12811.36,MB
85+
DEFAULT,2026-04-09T21:45:03.231455Z,METRIC_MEM:,12722.5,MB
86+
DEFAULT,2026-04-09T21:45:04.237488Z,METRIC_MEM:,12728.71,MB
87+
DEFAULT,2026-04-09T21:45:05.232293Z,METRIC_MEM:,12771.59,MB
88+
DEFAULT,2026-04-09T21:45:06.232823Z,METRIC_MEM:,12869.63,MB
89+
DEFAULT,2026-04-09T21:45:07.237757Z,METRIC_MEM:,12988.25,MB
90+
DEFAULT,2026-04-09T21:45:08.233418Z,METRIC_MEM:,13095.33,MB
91+
DEFAULT,2026-04-09T21:45:09.233396Z,METRIC_MEM:,13201.29,MB
92+
DEFAULT,2026-04-09T21:45:10.236817Z,METRIC_MEM:,13288.67,MB
93+
DEFAULT,2026-04-09T21:45:11.237004Z,METRIC_MEM:,13399.59,MB
94+
DEFAULT,2026-04-09T21:45:12.237337Z,METRIC_MEM:,13502.8,MB
95+
DEFAULT,2026-04-09T21:45:13.237766Z,METRIC_MEM:,13616.84,MB
96+
DEFAULT,2026-04-09T21:45:14.240824Z,METRIC_MEM:,13711.62,MB
97+
DEFAULT,2026-04-09T21:45:15.241266Z,METRIC_MEM:,13820.56,MB
98+
DEFAULT,2026-04-09T21:45:16.241645Z,METRIC_MEM:,13937.32,MB
99+
DEFAULT,2026-04-09T21:45:17.242003Z,METRIC_MEM:,14028.76,MB
100+
DEFAULT,2026-04-09T21:45:18.242372Z,METRIC_MEM:,14126.38,MB
101+
DEFAULT,2026-04-09T21:45:19.245015Z,METRIC_MEM:,14220.46,MB
102+
DEFAULT,2026-04-09T21:45:20.243002Z,METRIC_MEM:,14323.86,MB
103+
DEFAULT,2026-04-09T21:45:21.245997Z,METRIC_MEM:,14424.54,MB
104+
DEFAULT,2026-04-09T21:45:22.247559Z,METRIC_MEM:,7915.41,MB
105+
DEFAULT,2026-04-09T21:45:23.246817Z,METRIC_MEM:,8489.96,MB
106+
DEFAULT,2026-04-09T21:45:24.247168Z,METRIC_MEM:,8911.7,MB
107+
DEFAULT,2026-04-09T21:45:25.249494Z,METRIC_MEM:,9354.19,MB
108+
DEFAULT,2026-04-09T21:45:26.249975Z,METRIC_MEM:,9780.51,MB
109+
DEFAULT,2026-04-09T21:45:27.250390Z,METRIC_MEM:,10217.01,MB
110+
DEFAULT,2026-04-09T21:45:28.250624Z,METRIC_MEM:,10676.55,MB
111+
DEFAULT,2026-04-09T21:45:29.250558Z,METRIC_MEM:,11116.09,MB
112+
DEFAULT,2026-04-09T21:45:30.250662Z,METRIC_MEM:,11445.64,MB
113+
DEFAULT,2026-04-09T21:45:31.250691Z,METRIC_MEM:,11473.34,MB
114+
DEFAULT,2026-04-09T21:45:32.256485Z,METRIC_MEM:,10635.77,MB
115+
DEFAULT,2026-04-09T21:45:33.256709Z,METRIC_MEM:,9477.71,MB
116+
DEFAULT,2026-04-09T21:45:34.256519Z,METRIC_MEM:,8338.32,MB
117+
DEFAULT,2026-04-09T21:45:35.260632Z,METRIC_MEM:,8521.45,MB
118+
DEFAULT,2026-04-09T21:45:36.260800Z,METRIC_MEM:,8241.01,MB
119+
DEFAULT,2026-04-09T21:45:37.260872Z,METRIC_MEM:,8706.99,MB
120+
DEFAULT,2026-04-09T21:45:38.258259Z,METRIC_MEM:,8889.8,MB
121+
DEFAULT,2026-04-09T21:45:39.258646Z,METRIC_MEM:,9144.98,MB
122+
DEFAULT,2026-04-09T21:45:40.261191Z,METRIC_MEM:,9366.04,MB
123+
DEFAULT,2026-04-09T21:45:41.259351Z,METRIC_MEM:,9586.33,MB
124+
DEFAULT,2026-04-09T21:45:42.259706Z,METRIC_MEM:,9807.19,MB
125+
DEFAULT,2026-04-09T21:45:43.260068Z,METRIC_MEM:,10050.3,MB
126+
DEFAULT,2026-04-09T21:45:44.260485Z,METRIC_MEM:,10235.79,MB
127+
DEFAULT,2026-04-09T21:45:45.260779Z,METRIC_MEM:,10445.2,MB
128+
DEFAULT,2026-04-09T21:45:46.261136Z,METRIC_MEM:,10634.31,MB
129+
DEFAULT,2026-04-09T21:45:47.261490Z,METRIC_MEM:,10886.51,MB
130+
DEFAULT,2026-04-09T21:45:48.265978Z,METRIC_MEM:,11324.49,MB
131+
DEFAULT,2026-04-09T21:45:49.266059Z,METRIC_MEM:,12017.47,MB
132+
DEFAULT,2026-04-09T21:45:50.266025Z,METRIC_MEM:,11914.96,MB
133+
DEFAULT,2026-04-09T21:45:51.266082Z,METRIC_MEM:,11647.12,MB
134+
DEFAULT,2026-04-09T21:45:52.272392Z,METRIC_MEM:,11648.13,MB
135+
DEFAULT,2026-04-09T21:45:53.269008Z,METRIC_MEM:,9406.49,MB
136+
DEFAULT,2026-04-09T21:45:54.269512Z,METRIC_MEM:,9458.97,MB
137+
DEFAULT,2026-04-09T21:45:55.269918Z,METRIC_MEM:,9560.43,MB
138+
DEFAULT,2026-04-09T21:45:56.270231Z,METRIC_MEM:,9660.94,MB
139+
DEFAULT,2026-04-09T21:45:57.270653Z,METRIC_MEM:,9766.71,MB
140+
DEFAULT,2026-04-09T21:45:58.271050Z,METRIC_MEM:,9849.17,MB
141+
DEFAULT,2026-04-09T21:45:59.303917Z,METRIC_MEM:,9507.84,MB
142+
DEFAULT,2026-04-09T21:46:00.313037Z,METRIC_MEM:,9122.2,MB
143+
DEFAULT,2026-04-09T21:46:01.309538Z,METRIC_MEM:,9528.65,MB
144+
DEFAULT,2026-04-09T21:46:02.310012Z,METRIC_MEM:,9809.38,MB
145+
DEFAULT,2026-04-09T21:46:03.310421Z,METRIC_MEM:,10074.81,MB
146+
DEFAULT,2026-04-09T21:46:04.310900Z,METRIC_MEM:,10459.12,MB
147+
DEFAULT,2026-04-09T21:46:05.311218Z,METRIC_MEM:,10760.92,MB
148+
DEFAULT,2026-04-09T21:46:06.311655Z,METRIC_MEM:,11069.43,MB
149+
DEFAULT,2026-04-09T21:46:07.312043Z,METRIC_MEM:,11441.79,MB
150+
DEFAULT,2026-04-09T21:46:08.312403Z,METRIC_MEM:,11540.78,MB
151+
DEFAULT,2026-04-09T21:46:09.312344Z,METRIC_MEM:,11546.04,MB
152+
DEFAULT,2026-04-09T21:46:10.312541Z,METRIC_MEM:,10711.32,MB
153+
DEFAULT,2026-04-09T21:46:11.312428Z,METRIC_MEM:,9515.68,MB
154+
DEFAULT,2026-04-09T21:46:12.312786Z,METRIC_MEM:,8679.4,MB
155+
DEFAULT,2026-04-09T21:46:13.313152Z,METRIC_MEM:,8724.61,MB
156+
DEFAULT,2026-04-09T21:46:14.313501Z,METRIC_MEM:,8833.48,MB
157+
DEFAULT,2026-04-09T21:46:15.313877Z,METRIC_MEM:,8843.11,MB
158+
DEFAULT,2026-04-09T21:46:16.314200Z,METRIC_MEM:,8844.81,MB
159+
DEFAULT,2026-04-09T21:46:17.315061Z,METRIC_MEM:,8844.75,MB
160+
DEFAULT,2026-04-09T21:46:18.315452Z,METRIC_MEM:,8845.94,MB
161+
DEFAULT,2026-04-09T21:46:19.315830Z,METRIC_MEM:,8850.8,MB
162+
DEFAULT,2026-04-09T21:46:20.316259Z,METRIC_MEM:,8860.11,MB
163+
DEFAULT,2026-04-09T21:46:21.316618Z,METRIC_MEM:,8866.31,MB
164+
DEFAULT,2026-04-09T21:46:22.316957Z,METRIC_MEM:,8858.84,MB
165+
DEFAULT,2026-04-09T21:46:23.317325Z,METRIC_MEM:,8746.22,MB

0 commit comments

Comments
 (0)