Skip to content

Improve iOS keyboard return reliability and perf without English label probing #646

@thymikee

Description

@thymikee

This was generated by AI during triage.

Problem

The iOS keyboard return path is both slower than nearby keyboard commands and biased toward English keyboard labels.

Current flow:

  • src/core/dispatch.ts maps keyboard enter / keyboard return to the XCTest runner command keyboardReturn.
  • ios-runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+Interaction.swift calls tapKeyboardReturnControl(app:) before trying app.typeText(XCUIKeyboardKey.return.rawValue).
  • tapKeyboardReturnControl probes English labels in order: return, Return, Enter, Go, Search, Next, Done, Send, Join.

That has two issues:

  • Non-English simulator/device languages may expose localized labels, so English label probing is not reliable as the primary path.
  • In the Settings search flow, the key is likely Search, so the runner pays several failed XCTest element lookups before the successful one. The updated iOS perf harness measured keyboard return at about 3833ms wall median / 3583ms daemon median, compared with type at about 2970ms wall / 2752ms daemon.

Perf report with the new harness row:

  • /private/tmp/agent-device-perf-pr643-series-full/perf-ios-2026-06-01T14-44-56-344Z.md

Likely Approach

Prefer a language-neutral return-key path before English label probing:

  1. If the iOS software keyboard is visible, try app.typeText(XCUIKeyboardKey.return.rawValue) first inside the existing Objective-C exception catcher.
  2. If that throws, try the existing single text-entry fallback: find the single text input, tap it, then typeText(.return).
  3. Keep visible keyboard-control tapping as a fallback, but do not make English label lookup the primary path.
  4. If label fallback remains, consider querying visible keyboard buttons/keys once and matching in memory rather than issuing repeated named XCTest queries for each candidate label.

Acceptance Criteria

  • keyboard return still works for common iOS return-key variants such as Return, Search, Go, Next, Done, and Send.
  • Add or update runner-level/manual validation notes for at least one non-English simulator language so the behavior is not English-label-dependent.
  • Run the iOS perf harness and compare keyboard return against type; the goal is to remove obvious label-probing overhead, not to make keyboard return faster than text entry in every run.
  • Preserve the existing unsupported-operation behavior when no keyboard/focused text input can accept return.

Relevant Code

  • src/core/dispatch.tshandleIosKeyboardCommand sends keyboardReturn.
  • ios-runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+CommandExecution.swift — runner command dispatch for .keyboardReturn.
  • ios-runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+Interaction.swiftpressKeyboardReturn, tapKeyboardReturnControl, and singleTextEntryElement.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions