Panic on backward system-clock adjustment (sleep/wake NTP correction) wedges IDS auth and downstream CloudKit sync

[richardpowellus]

Summary

CachedKeys::get_stale_time() in src/ids/identity_manager.rs#L45-L49 panics with Time went backwards whenever the system clock is corrected backward (e.g. NTP resync after sleep/wake). Because the function is called from is_valid() (L51-L58) and is_dirty() (L61-L65), which are invoked on every cached-identity check (L283, L305), the panic kills the IDS auth path and cascades into every downstream subsystem — most visibly CloudKit sync wedges into a retry loop until the app is force-killed.

Reproduction

Confirmed on Windows 11 (ARM64), OpenBubbles 1.18.800.0 (Microsoft Store) with rustpush as bundled.

1. Run OpenBubbles with CloudKit sync enabled (cloudSyncingEnabled: true).
2. Put the machine into Modern Standby for several hours/days.
3. Wake the machine. Windows NTP corrects the system clock backward (typically a few seconds; in my case ~6.5s after a 3.5-day sleep).
4. OpenBubbles thread immediately panics with PanicException(Time went backwards: SystemTimeError(<N>s)).
5. The CloudKit sync supervisor retries on a ~4.6s interval, producing a steady stream of failures (in my log: exactly 13 errors/min for ~9 minutes straight) and pegging one CPU core at ~96% indefinitely. The Flutter UI freezes; only force-killing the process recovers.

Evidence

Panic from the app log

2026-06-02T15:38:18.658213Z [ERROR] [BlueBubblesApp] PanicException(Time went backwards: SystemTimeError(4.9310864s))
2026-06-02T15:38:18.731222Z [ERROR] [BlueBubblesApp] PanicException(Time went backwards: SystemTimeError(4.7560443s))
2026-06-02T15:38:19.006534Z [ERROR] [BlueBubblesApp] PanicException(Time went backwards: SystemTimeError(5.7651135s))
2026-06-02T15:38:19.604706Z [ERROR] [BlueBubblesApp] PanicException(Time went backwards: SystemTimeError(3.8677244s))
2026-06-02T15:38:19.633064Z [ERROR] [BlueBubblesApp] PanicException(Time went backwards: SystemTimeError(3.8665786s))
... (panic stack into SimpleDecoder.decode at flutter_rust_bridge/src/codec/base.dart:35)

Correlated Windows Kernel-General clock-adjust events (same machine, same minute)

2026-06-02 08:37:32 (Event 1) — system time set to 2026-06-02T15:37:32Z from 2026-05-29T20:28:02Z (wake from 3.5-day sleep)
2026-06-02 08:38:18 (Event 1) — system time set to 2026-06-02T15:38:18.489Z from 2026-06-02T15:38:25.010Z   ← clock jumped 6.5s BACKWARD
2026-06-02 08:37:38 (Power 507) — exiting Modern Standby

The 6.5s backward NTP correction at 08:38:18 PDT (= 15:38:18 UTC) is exactly the trigger for the panic burst that started <200 ms later. The 3-5s panic magnitudes match the NTP correction window. After the initial burst, the retry loop maintained a steady 13 errors/min for 9+ minutes before I killed the process.

Steady cadence proves a wedged retry loop, not just a one-shot panic

Errors per minute on 2026-06-02:
  17:15  13
  17:14  13
  17:13  13
  17:12  13
  17:11  13
  17:10  13
  17:09  13
  17:08  13
  17:07  13

Root cause

src/ids/identity_manager.rs#L45-L49:

fn get_stale_time(&self) -> Duration {
    SystemTime::now()
        .duration_since(UNIX_EPOCH + Duration::from_millis(self.at_ms))
        .expect("Time went backwards")
}

When SystemTime::now() is earlier than self.at_ms (which happens whenever the wall clock is adjusted backward), duration_since returns Err(SystemTimeError) and .expect() panics. self.at_ms was captured before the NTP correction; SystemTime::now() is read after. The cache thinks its keys are from "the future" relative to the wall clock — which is physically impossible but trivially produced by the OS.

The panic propagates up through the Tokio task, the IDS lookup task is gone, but the supervising CloudKit code keeps retrying — and each retry hits the same panicking code path because the cached at_ms values are still in the (now-relative) future. The loop only breaks when the wall clock catches back up (potentially many seconds, or never, if drift cancels out).

Impact

• Severity: High. Single-event trigger → unrecoverable application hang requiring force-kill. Reproducible on every wake from a non-trivial sleep on Windows (where backward NTP corrections after Modern Standby are routine).
• Blast radius: Any code path that gates on CachedKeys::is_valid or is_dirty — i.e. every IDS-authenticated request, including CloudKit (gateway.icloud.com/ckdatabase/...), iMessage delivery checks, etc.
• User-visible behaviour: App freezes; CPU pegs at one full core; iCloud sync, contact sync, and message sending all silently break until the user notices and force-kills.

Proposed fix

Replace the panic with a safe fallback. Three options, ordered by minimal-change → robust:

1. Minimal — saturate to zero on backward skew:

fn get_stale_time(&self) -> Duration {
    SystemTime::now()
        .duration_since(UNIX_EPOCH + Duration::from_millis(self.at_ms))
        .unwrap_or(Duration::ZERO)
}

This treats a clock-backward situation as "the key was just refreshed," which is conservative (keys appear fresh until the next genuine staleness check). Safe because is_valid() returning true on fresh keys is the no-op path.

2. Better — return Duration::MAX (force re-auth):

.unwrap_or(Duration::MAX)

Forces an immediate refresh, which is the right behaviour if the wall clock genuinely moved by an unexpected amount.

3. Best — use a monotonic clock for staleness:
Store Instant::now() alongside at_ms at cache-insertion time and compute staleness against Instant. Monotonic clocks are immune to wall-clock skew. (Drawback: Instant is not persistable, so this only works for in-memory caches — but CachedKeys already looks in-memory based on the surrounding code.)

Related class of bugs

There are 15+ other SystemTime::now().duration_since(...).unwrap() / .expect() call sites in the codebase that have the same fundamental problem. Found via gh search code 'duration_since repo:OpenBubbles/rustpush':

• src/statuskit.rs — 2 sites
• src/passwords.rs — 1 site
• src/findmy.rs — 1 site
• src/ids/user.rs — 1 site
• src/util.rs — 2 sites (duration_since(SystemTime::UNIX_EPOCH).unwrap())
• src/auth.rs — mme_refreshed weekly check
• src/facetime.rs, src/imessage/aps_client.rs, src/imessage/messages.rs, src/icloud/keychain.rs, src/ids/identity_manager.rs, cloudkit-proto/src/lib.rs — various
• src/sharedstreams.rs — round_seconds()

identity_manager.rs#L45 is the one I caught panicking, but any of these can panic the same way under the right clock-skew conditions. Worth fixing as a class — perhaps a small helper duration_since_safe() in util.rs that returns Duration::ZERO (or whatever the call site needs) on Err.

Workaround for users until fixed

Disable CloudKit sync via flutter.cloudSyncingEnabled = false and flutter.attachmentSyncEnabled = false in shared_preferences.json. iMessages still deliver via APS reflection; only cross-device iCloud history sync is lost.

---

Happy to send a PR for option 1 or 3 if you have a preferred approach.

[richardpowellus]

Update — CloudKit sync wedges in a silent hot loop even without any clock skew

After filing this issue, I implemented a Windows Scheduled Task to auto-restart OpenBubbles on Power-Troubleshooter wake events as a workaround. I then re-enabled cloudSyncingEnabled and attachmentSyncEnabled to test.

Result: A fresh process launch with CloudKit sync enabled — on a machine that has been continuously awake for hours, with no clock-skew event in the Windows Kernel-General log — still produces the same hot-loop symptom:

• ~3 minutes after launch, CPU pegs at ~92% with one thread consuming 96% of process CPU (TID 41412: 121.4s of CPU out of 125.6s process total, ThreadState=Running).
• App remains Responding=True and the UI is usable, but a single rustpush worker thread is wedged.
• Zero Time went backwards panics this time.
• Zero [ERROR] lines in the log during the loop (other than the unrelated benign Android-billing and FaceTime-button startup exceptions that appear on every cold start).
• Last log line before the loop is the same as before: [INFO] [BlueBubblesApp] Doing cloudkit sync!

So the clock-skew panic at identity_manager.rs#L48 reported above is one trigger for the wedge, but there is at least one additional code path in the CloudKit sync that wedges silently without any clock event. The silent hot loop after Doing cloudkit sync! does not log anything that I can correlate from outside the binary.

Hypothesis: the CloudKit sync supervisor task is busy-spinning on a retry/poll loop that doesn't yield or log. Possibilities:

• Tight loop { match res { Err(_) => continue, ... } } on a CKDatabase request that immediately resolves with an error the code treats as retriable.
• A select! with a timer firing on a zero-duration Duration (which Duration::ZERO from a saturating clock subtraction could plausibly cause).
• A pending future that always returns Poll::Ready but the calling code never makes progress on the actual sync.

I'm flagging it here rather than opening a second issue because it manifests with the same user-visible symptom (CloudKit sync wedges, force-kill required) and is likely fixable in the same code area. Happy to file separately if you'd prefer.

Workaround that worked for me: Keep flutter.cloudSyncingEnabled = false and flutter.attachmentSyncEnabled = false. iMessages still deliver via APS reflection; the Scheduled Task is in place for when CloudKit sync is safe to re-enable.

Repro environment: same as original report (Windows 11 ARM64, OpenBubbles 1.18.800.0 from MS Store).

[richardpowellus]

Update 2 — Actual trigger identified: stale sync_continuation_token + unbounded loop

Followup to my previous comment. I tracked down the silent hot loop, and the trigger is more specific than I described:

Trigger: A messageSyncToken / chatSyncToken / attachmentSyncToken that has been cached in the Flutter shared_preferences.json for a long time (in my case ~3 weeks since the last successful sync, plus a 3.5-day machine sleep in the middle, plus the original Time-went-backwards panic from the top of this thread). When OpenBubbles re-enables CloudKit sync and replays these tokens to gateway.icloud.com/ckdatabase/..., Apple appears to return responses with status != Some(2) and a sync_continuation_token that does not advance in a productive way.

That feeds straight into the unbounded loop in src/icloud/cloudkit.rs#L744:

loop {
    let result = container.perform(&CloudKitSession::new(), Self::new(sync_token)).await?;
    responses.extend(result.changes);
    sync_token = result.sync_continuation_token;
    if result.status == Some(2) {
        break;
    }
}

…which has no max-iteration cap, no no-progress (same-token) guard, and no break on any error type. The await? only returns on actual transport errors, so each successful HTTP response with status != 2 just continues the loop, producing zero log output and zero errors, just a CPU-pegged worker thread. Same shape exists at L688 in FetchRecordZoneChangesOperation::do_sync.

Repro / fix that worked for me

After clearing all three Flutter sync tokens (set to "") and restarting OpenBubbles:

flutter.chatSyncToken       = ""
flutter.messageSyncToken    = ""
flutter.attachmentSyncToken = ""

iCloud started a fresh-from-page-1 sync and immediately began making real progress:

[INFO] Syncing group of 200 messages, total 7713
[INFO] Syncing group of 200 messages, total 8109
[INFO] Syncing group of 200 messages, total 8989
[INFO] Syncing group of 200 messages, total 9776
... (still climbing)

The messageSyncToken is now advancing monotonically with each page (verified by polling the prefs file every 30s). Memory is climbing as expected for bulk record load. ObjectBox DB on disk is growing. So clearing the tokens is the user-facing workaround.

Suggested defensive fixes

In addition to the panic fix in the original issue:

1. Add a same-token guard. If result.sync_continuation_token == previous_sync_token for N consecutive iterations, break with an Err(PushError::SyncStalled) so the caller can decide to reset and retry from scratch.
2. Add a max-iterations cap (or max-bytes / max-records cap) as a hard backstop.
3. Log every Nth iteration at info or debug level so an observer can tell the difference between "doing real work" and "wedged" without attaching a debugger.
4. On PushError::SyncStalled, automatically clear the sync token and retry once. Apple's CKDatabase tokens can apparently go stale on the server side; the client should self-heal rather than wedge.

Happy to send a PR for any combination of these if helpful. The same pattern applies to FetchRecordZoneChangesOperation::do_sync at L688.