[Tracking] Perceiving time-sensitive / transient UI (toasts, carousels) beyond a single static screenshot

[quanru]

Summary

This is a tracking/index issue for a shared root cause reported across multiple issues: Midscene perceives the UI from a single static screenshot taken at one moment. Any UI that is transient or time-sensitive — it appears, changes, or disappears between the moment of capture and the moment of action — cannot be reliably observed or acted upon.

这是一个用于索引和跟踪同一根因问题的元 issue。Midscene 目前基于单帧静态截图感知界面,对于在「截图时刻」与「操作时刻」之间出现、变化或消失的时效性 UI(如 Toast 提示、轮播 Banner),无法可靠地识别或操作。

Tracked issues

• [Feature]: Support for time-sensitive information suggestions (countdown disappearing elements) #1653 — [Feature] Support time-sensitive information (toast disappears before the next screenshot is sent to the model)
• Banner carousel element positioning #1728 — Banner carousel positioning (element located, but the carousel rotates before the click lands, so the wrong banner is clicked)
• [Bug]: H5 toast messages on Android are not detected #2100 — [Bug] H5 (non-native) toast messages on Android are not detected
• [Feature]: Quickly capture short-lived UI controls that appear briefly on screen #2272 — [Feature] Capture short-lived UI controls that auto-hide (e.g. video playback controls that disappear after ~5s)
• [Feature]: aiAssert supports dynamic scenario #962 — [Feature] aiAssert should support dynamic scenarios (animations etc.) by sampling several screenshots over a duration at a frameRate

Why they are the same problem

They all stem from the single-snapshot perception model:

• [Feature]: Support for time-sensitive information suggestions (countdown disappearing elements) #1653 / [Bug]: H5 toast messages on Android are not detected #2100 — a toast is shown right after an action, then auto-dismisses. By the time the next screenshot reaches the model, the toast is gone, so assertions / detection fail. ([Bug]: H5 toast messages on Android are not detected #2100 specifically concerns web/H5-rendered toasts, not native Android toasts.)
• Banner carousel element positioning #1728 — the target element is correctly located, but per-step latency (~4–5s) means the animated carousel advances before the click executes, so a different banner is hit.
• [Feature]: Quickly capture short-lived UI controls that appear briefly on screen #2272 — controls that appear only on tap and auto-hide after ~5s must be detected and acted on within a narrow window, which a single delayed screenshot cycle can miss.
• [Feature]: aiAssert supports dynamic scenario #962 — asks directly for the multi-frame assertion this index is about: feed aiAssert a short sequence of screenshots (duration + frameRate) so it can judge dynamic scenarios instead of a single static frame.

Possible directions

• Capture a short sequence of frames after an action (e.g. several frames at ~1s intervals, or only the changed local regions) and submit them together to the model, instead of a single frame — lets the model observe the transition. Trade-off: more tokens.
• Use streaming capture instead of one-shot screenshots to cut capture latency (e.g. scrcpy streaming rather than per-call adb screencap on Android).
• For animated/moving targets, reduce locate→act latency and/or re-verify the target at action time.

Individual discussion will be consolidated here; the original issues are being closed in favor of this index.