Skip to content

[ISSUE #8090]♻️Report telemetry shutdown health and dropped log lines#8091

Merged
mxsm merged 1 commit into
mainfrom
mxsm/telemetry-shutdown-report
Jul 6, 2026
Merged

[ISSUE #8090]♻️Report telemetry shutdown health and dropped log lines#8091
mxsm merged 1 commit into
mainfrom
mxsm/telemetry-shutdown-report

Conversation

@mxsm

@mxsm mxsm commented Jul 6, 2026

Copy link
Copy Markdown
Owner

Which Issue(s) This PR Fixes(Closes)

Brief Description

This PR makes telemetry shutdown produce structured evidence instead of only returning success or failure.

Changes include:

  • Add TelemetryShutdownReport with subscriber, file sink, dropped log line, and provider shutdown fields.
  • Drop local file logging guards before shutting down OpenTelemetry logs, traces, metrics, and Prometheus handles.
  • Preserve logs, traces, and metrics shutdown errors in the report while still exposing into_result for fail-fast callers.
  • Record telemetry shutdown detail inside the broker shutdown report and mark the observability component unhealthy when provider shutdown fails.
  • Update Broker, NameServer, Proxy, and Controller entrypoints to consume the structured shutdown report.
  • Extend observability and broker tests for file sink reporting, no-op shutdown, and telemetry provider failure reporting.

How Did You Test This Change?

  • cargo fmt --all passed.
  • cargo test -p rocketmq-observability logging --all-features passed.
  • cargo test -p rocketmq-broker shutdown --all-features passed.
  • cargo test -p rocketmq-observability --test logging_bootstrap_guards --all-features passed.
  • cargo clippy --workspace --no-deps --all-targets --all-features -- -D warnings passed with CARGO_TARGET_DIR set to a fresh C drive temporary target directory.

Summary by CodeRabbit

  • New Features

    • Telemetry shutdown now produces richer health reports, including per-provider status and detailed error information.
    • Broker shutdown summaries now include observability details for easier troubleshooting.
  • Bug Fixes

    • Improved shutdown handling so telemetry cleanup continues across components even if one part fails.
    • Added clearer warnings when dropped log lines or unhealthy telemetry shutdowns are detected.

@mxsm mxsm merged commit 673d1a9 into main Jul 6, 2026
5 of 7 checks passed
@mxsm mxsm deleted the mxsm/telemetry-shutdown-report branch July 6, 2026 17:30
@coderabbitai

coderabbitai Bot commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 272dc614-14db-4f9f-80ef-c6b959042b03

📥 Commits

Reviewing files that changed from the base of the PR and between 85640dc and adccc07.

📒 Files selected for processing (9)
  • rocketmq-broker/src/bin/broker_bootstrap_server.rs
  • rocketmq-broker/src/broker_runtime.rs
  • rocketmq-controller/src/bin/controller_bootstrap.rs
  • rocketmq-namesrv/src/bin/namesrv_bootstrap_server.rs
  • rocketmq-observability/src/init.rs
  • rocketmq-observability/src/lib.rs
  • rocketmq-observability/src/logging.rs
  • rocketmq-observability/tests/logging_bootstrap_guards.rs
  • rocketmq-proxy/src/bin/rocketmq-proxy-rust.rs

Walkthrough

Introduces structured telemetry shutdown reporting: TelemetryProviderShutdownReport and TelemetryShutdownReport capture per-provider health, errors, and dropped log line details instead of a plain Result. Shutdown no longer short-circuits on provider failure. Broker shutdown reporting, and namesrv/broker/proxy/controller bootstrap entrypoints, are updated to consume these reports via into_result().

Changes

Telemetry shutdown reporting

Layer / File(s) Summary
Provider shutdown report and guard refactor
rocketmq-observability/src/init.rs
Adds TelemetryProviderShutdownReport with per-provider ok/error fields, is_healthy(), into_result(); TelemetryGuard::shutdown_inner now collects failures instead of early-returning and shutdown() delegates to shutdown_with_report().
TelemetryShutdownReport and TelemetryRuntimeGuard::shutdown
rocketmq-observability/src/logging.rs, rocketmq-observability/src/lib.rs
Adds TelemetryShutdownReport (health, JSON, into_result()); TelemetryRuntimeGuard::shutdown now computes subscriber/file-log/dropped-line state, drops logging guard before provider shutdown, and returns the report; type re-exported and tests updated.
Broker shutdown component integration
rocketmq-broker/src/broker_runtime.rs
Adds completed_with_detail and from_telemetry_shutdown_report to BrokerShutdownComponentReport; observability shutdown now built from the telemetry report with unhealthy-state warning logging; adds unit tests.
Bootstrap entrypoint updates
rocketmq-broker/src/bin/broker_bootstrap_server.rs, rocketmq-namesrv/src/bin/namesrv_bootstrap_server.rs, rocketmq-controller/src/bin/controller_bootstrap.rs, rocketmq-proxy/src/bin/rocketmq-proxy-rust.rs
Shutdown calls now chain .into_result() and match Ok(_report) instead of Ok(()), preserving prior return/context behavior.
Observability bootstrap guard tests
rocketmq-observability/tests/logging_bootstrap_guards.rs
Tests capture the shutdown report, assert health/file-log fields, and validate via report.into_result().expect(...).

Estimated code review effort: 3 (Moderate) | ~25 minutes

Possibly related issues

Possibly related PRs

  • mxsm/rocketmq-rust#7545: Both PRs modify shutdown_basic_service_with_report in broker_runtime.rs, extending the shutdown-report aggregation mechanism.

Suggested labels: enhancement✨

Suggested reviewers: SpaceXCN, TeslaRustor, rocketmq-rust-bot

Poem

A rabbit checks the telemetry gate,
shutdown reports now seal its fate 🐇
logs and traces, metrics too,
all reported, healthy and true.
No more silent early stop—
every provider gets its drop! 🥕

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch mxsm/telemetry-shutdown-report

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@rocketmq-rust-bot

Copy link
Copy Markdown
Collaborator

🔊@mxsm 🚀Thanks for your contribution🎉!

💡CodeRabbit(AI) will review your code first🔥!

Note

🚨The code review suggestions from CodeRabbit are to be used as a reference only, and the PR submitter can decide whether to make changes based on their own judgment. Ultimately, the project management personnel will conduct the final code review💥.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

AI review first Ai review pr first approved PR has approved auto merge refactor♻️ refactor code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Refactor♻️] Report telemetry shutdown health and dropped log lines

3 participants