fix(guard): add metrics by MasterPtato · Pull Request #2480 · rivet-dev/rivet

MasterPtato · 2025-05-27T20:45:33Z

Changes

MasterPtato · 2025-05-27T20:45:49Z

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

How to use the Graphite Merge Queue

Add the label merge-queue to this PR to add it to the merge queue.

You must have a Graphite account in order to use the merge queue. Sign up using this link.

_{An organization admin has enabled the Graphite Merge Queue in this repository.} _{Please do not merge from GitHub as this will restart CI on PRs being processed by the merge queue.}

This stack of pull requests is managed by Graphite. Learn more about stacking.

greptile-apps

PR Summary

This PR adds comprehensive metrics tracking to the Rivet Guard service, focusing on monitoring system performance and request handling.

Added Prometheus metrics in metrics.rs for tracking TCP connections, active requests, route cache size, rate limiters, and in-flight counters
Modified server.rs to increment/decrement connection and request metrics at key lifecycle points
Changed rate limiting in proxy_service.rs to use IP addresses instead of actor IDs for better tracking
Added metrics server configuration in main.rs with a dedicated task for metrics collection
Configured Prometheus target for guard service on port 8091 with 15-second scrape interval

_{7 file(s) reviewed, 6 comment(s)}
_{Edit PR Review Bot Settings | Greptile}

greptile-apps · 2025-05-27T20:46:21Z

+														metrics::ACTIVE_REQUESTS_TOTAL.inc();
+
+														let result = service_clone.process(req).await.map_err(|err| GlobalErrorWrapper{err});
+
+														metrics::ACTIVE_REQUESTS_TOTAL.dec();
+
+														result


style: Consider using a try-finally pattern to ensure metrics are always decremented, even if process() panics.

greptile-apps · 2025-05-27T20:46:22Z


 									// Accept TLS connection in a separate task to avoid ownership issues
 									tokio::spawn(async move {
+										metrics::TCP_CONNECTIONS_TOTAL.inc();


style: TCP connection metric incremented before TLS handshake. Consider moving after successful handshake to avoid counting failed connections.

greptile-apps · 2025-05-27T20:46:23Z

+					endpoint: "http://127.0.0.1:8091".into(),
+					scrape_interval: 15,


logic: Port 8091 is already used by the worker service (see line 215). Consider using a different port for guard metrics to avoid conflicts.

greptile-apps · 2025-05-27T20:46:24Z

+				components::vector::PrometheusTarget {
+					endpoint: "http://127.0.0.1:8091".into(),
+					scrape_interval: 15,
+				},


logic: Missing /metrics path in endpoint URL which is required for Prometheus scraping

Suggested change

components::vector::PrometheusTarget {

endpoint: "http://127.0.0.1:8091".into(),

scrape_interval: 15,

},

components::vector::PrometheusTarget {

endpoint: "http://127.0.0.1:8091/metrics".into(),

scrape_interval: 15,

},

greptile-apps · 2025-05-27T20:46:51Z

+	// Start metrics server
+	let metrics_task = tokio::spawn(rivet_metrics::run_standalone(config.clone()));


logic: No cleanup/shutdown handling for metrics task if main server exits or Ctrl+C is received. Should implement graceful shutdown.

greptile-apps · 2025-05-27T20:46:55Z

+			Err(e) => bail!(
+				"Failed to build dynamic hostname actor routing regex: {}",
+				e
+			),
 		};


style: Error handling style is inconsistent with the static regex case below (lines 181-184). Consider using the same style for both cases.

cloudflare-workers-and-pages · 2025-05-27T20:49:34Z

Deploying rivet with Cloudflare Pages

Latest commit:	`bd93030`
Status:	✅ Deploy successful!
Preview URL:	https://73a1cb0c.rivet.pages.dev
Branch Preview URL:	https://05-27-fix-guard-add-metrics.rivet.pages.dev

View logs

NathanFlurry

@greptileai

cloudflare-workers-and-pages · 2025-05-31T00:55:02Z

Deploying rivet-hub with Cloudflare Pages

Latest commit:	`bd93030`
Status:	✅ Deploy successful!
Preview URL:	https://a72cdbb1.rivet-hub-7jb.pages.dev
Branch Preview URL:	https://05-27-fix-guard-add-metrics.rivet-hub-7jb.pages.dev

View logs

cloudflare-workers-and-pages · 2025-05-31T00:55:20Z

Deploying rivet-studio with Cloudflare Pages

Latest commit:	`bd93030`
Status:	✅ Deploy successful!
Preview URL:	https://2c56ae00.rivet-studio.pages.dev
Branch Preview URL:	https://05-27-fix-guard-add-metrics.rivet-studio.pages.dev

View logs

MasterPtato requested a review from NathanFlurry May 27, 2025 20:45

greptile-apps Bot reviewed May 27, 2025

View reviewed changes

MasterPtato force-pushed the 05-27-fix_guard_add_metrics branch 2 times, most recently from 6db518e to 240796d Compare May 27, 2025 23:17

NathanFlurry reviewed May 27, 2025

View reviewed changes

MasterPtato force-pushed the 05-27-fix_guard_add_metrics branch 2 times, most recently from 6c38ea5 to 37074b9 Compare May 28, 2025 01:02

This was referenced May 28, 2025

fix(guard): replace internal caches with moka #2481

Closed

fix(guard): add more tokio runtime metrics, remove labels from metrics #2485

Closed

fix(cache): add traces #2489

Closed

feat: add pb usage metrics, server state #2503

Closed

NathanFlurry changed the base branch from main to graphite-base/2480 May 31, 2025 00:39

NathanFlurry force-pushed the 05-27-fix_guard_add_metrics branch from 37074b9 to 4d3b056 Compare May 31, 2025 00:39

NathanFlurry changed the base branch from graphite-base/2480 to 05-30-chore_expose_namespace_in_otel_attributes May 31, 2025 00:39

This was referenced May 31, 2025

chore: expose namespace in otel attributes #2501

Closed

chore(cluster): remove tcp & udp ports on gg #2502

Closed

MasterPtato mentioned this pull request May 31, 2025

chore: fix guard metrics #2504

Closed

This was referenced May 31, 2025

chore: update workspace #2505

Closed

feat(guard): add http ping endpoint to edge api hosts #2506

Closed

fix(guard): add metrics

bd93030

MasterPtato force-pushed the 05-27-fix_guard_add_metrics branch from 4d3b056 to bd93030 Compare June 2, 2025 18:13

MasterPtato force-pushed the 05-30-chore_expose_namespace_in_otel_attributes branch from 8f8d1e6 to 7b38fd6 Compare June 2, 2025 18:13

MasterPtato mentioned this pull request Jun 2, 2025

fix: fix pb metrics standalone #2518

Closed

NathanFlurry force-pushed the 05-30-chore_expose_namespace_in_otel_attributes branch from 7b38fd6 to 3115384 Compare June 2, 2025 20:12

NathanFlurry force-pushed the 05-27-fix_guard_add_metrics branch from bd93030 to 617a367 Compare June 2, 2025 20:12

NathanFlurry mentioned this pull request Jun 2, 2025

fix(pegboard): fix collecting metrics for actors without workflows #2519

Closed

graphite-app Bot closed this Jun 3, 2025

graphite-app Bot deleted the 05-27-fix_guard_add_metrics branch June 3, 2025 07:03

github-actions Bot mentioned this pull request Jun 3, 2025

chore(main): release 25.5.1 #2423

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(guard): add metrics#2480

fix(guard): add metrics#2480
MasterPtato wants to merge 1 commit into05-30-chore_expose_namespace_in_otel_attributesfrom
05-27-fix_guard_add_metrics

MasterPtato commented May 27, 2025

Uh oh!

MasterPtato commented May 27, 2025 •

edited

Loading

Uh oh!

greptile-apps Bot left a comment

Uh oh!

greptile-apps Bot May 27, 2025

Uh oh!

greptile-apps Bot May 27, 2025

Uh oh!

greptile-apps Bot May 27, 2025

Uh oh!

greptile-apps Bot May 27, 2025

Uh oh!

greptile-apps Bot May 27, 2025

Uh oh!

greptile-apps Bot May 27, 2025

Uh oh!

cloudflare-workers-and-pages Bot commented May 27, 2025 •

edited

Loading

Uh oh!

NathanFlurry left a comment

Uh oh!

cloudflare-workers-and-pages Bot commented May 31, 2025 •

edited

Loading

Uh oh!

cloudflare-workers-and-pages Bot commented May 31, 2025 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		endpoint: "http://127.0.0.1:8091".into(),
		scrape_interval: 15,

		// Start metrics server
		let metrics_task = tokio::spawn(rivet_metrics::run_standalone(config.clone()));

Conversation

MasterPtato commented May 27, 2025

Changes

Uh oh!

MasterPtato commented May 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

How to use the Graphite Merge Queue

Uh oh!

greptile-apps Bot left a comment

Choose a reason for hiding this comment

PR Summary

Uh oh!

greptile-apps Bot May 27, 2025

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot May 27, 2025

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot May 27, 2025

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot May 27, 2025

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot May 27, 2025

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot May 27, 2025

Choose a reason for hiding this comment

Uh oh!

cloudflare-workers-and-pages Bot commented May 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Deploying rivet with Cloudflare Pages

Uh oh!

NathanFlurry left a comment

Choose a reason for hiding this comment

Uh oh!

cloudflare-workers-and-pages Bot commented May 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Deploying rivet-hub with Cloudflare Pages

Uh oh!

cloudflare-workers-and-pages Bot commented May 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Deploying rivet-studio with Cloudflare Pages

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

MasterPtato commented May 27, 2025 •

edited

Loading

cloudflare-workers-and-pages Bot commented May 27, 2025 •

edited

Loading

cloudflare-workers-and-pages Bot commented May 31, 2025 •

edited

Loading

cloudflare-workers-and-pages Bot commented May 31, 2025 •

edited

Loading