Conversation
Android captures now carry the same keyboard fact the Apple runner publishes. The snapshot helper already lists every accessibility window with its type and screen bounds, so the band is read from the input method window in the captured tree with no extra adb call. A capture that could not list every window reports unmeasurable instead of absent.
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
All reported issues were addressed across 7 files
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
An input method window whose window-bounds did not parse, or parsed to non-finite numbers, used to fall through to absent. The capture saw the window but never measured it, so it cannot prove there is no keyboard. An input method window with parsed empty bounds still reads as absent.
…xture The keyboard capture fixture has two window roots and four nodes, so the helper metadata served with it now says so. Other captures keep the single-window defaults.
Reading the band from its own module added one module to what platform-android/src/mechanics.ts evaluates on import (177 against 176 at the merge-base). It now lives in snapshot.ts, which that closure already loads, so the import graph is the same as on main.
There was a problem hiding this comment.
All reported issues were addressed across 3 files (changes from recent commits).
Tip: Review your code locally with the cubic CLI to iterate faster.
Re-trigger cubic
…thod window is unmeasured
…kipped an input method window The helper drops a listed window whose root reads null or throws without a record, and windowCount counts only the roots it serialized, so the host could not tell an input method window it failed to read from one that was not on screen, and reported the keyboard absent. The helper now publishes the AccessibilityWindowInfo types of those windows as missingRootWindowTypes (empty when every listed window was read), and the host publishes it as androidSnapshot.missingRootWindowTypes. When it names an input method window (type 2), the keyboard band is unmeasurable with reason window-root-unavailable. An older helper omits the field and is trusted on the roots it serialized.
There was a problem hiding this comment.
All reported issues were addressed across 15 files (changes from recent commits).
Tip: Review your code locally with the cubic CLI to iterate faster.
Re-trigger cubic
… list
Number('') is 0 and Number('0x2') is 2, so a list like "1,,2" or
"0x2" passed as integers and could put a window type the helper never
sent into the keyboard decision. Each entry must now be plain decimal
digits, or the list is absent. The docs now describe the field as
windows the helper could not serialize, with a null or failing root as
examples, since a failure while reading the window's tree also lands
there.
|
The code in f028f3e looks right to me, but I'd like to see one more run before merge. The live emulator run was done at 907ed8c, and the only later change is the host-side parser, so please re-run it on the current head. Please also show that a tap behind the Gboard keyboard now refuses with tap_keyboard_occludes_target through the window-bounds fact, since today that is inferred from the band being the guard's input, not observed. Do the IME window bounds match the touchable keyboard region on non-Pixel keyboards, such as Samsung or a floating Gboard? A full-screen IME window would make the tap guard over-refuse. Not blocking: the local unionRects in https://github.com/callstack/agent-device/blob/f028f3e/packages/platform-android/src/snapshot.ts#L929 does the same job as the one in capture-kit screenshot-overlay-rects.ts, and there is a third copy in the iOS snapshot transitions, so one could move into @agent-device/kernel/rect next to isPositiveFiniteRect. Also ANDROID_WINDOW_TYPE_INPUT_METHOD at https://github.com/callstack/agent-device/blob/f028f3e/packages/platform-android/src/snapshot.ts#L872 sits apart from ANDROID_WINDOW_TYPE_APPLICATION in snapshot-content-recovery.ts, so the window-type constants could share one module such as ui-hierarchy.ts. Take or leave both. I looked for a smaller design and found none. The band has to come from the window list the helper already emits, and missingRootWindowTypes is the smallest protocol addition that tells a skipped IME window from an absent one. Is there a smaller way I missed? The Smoke Tests job failed in "Preflight iOS runner through public CLI" with daemon_startup_failed after a 15000 ms startup timeout. This diff does not touch daemon startup or the iOS runner, so I think the failure is unrelated. Please re-run that job. No conflicts. After the re-run and the evidence above, nothing else blocks this PR. |
This branch has not been deployed
2 participants
Summary
Android snapshots now carry the same
keyboardfact the Apple runner publishes (visiblewith a frame,absent, orunmeasurable). Before, onlykeyboard statuscould tell Gboard was up; the tree shows it as generic FrameLayout/TextView nodes.The helper already writes
window-typeandwindow-boundson each window root, so the band is the input method window's bounds (TYPE_INPUT_METHOD). No extra adb call. A capture that fell back to the active window, has a root without window metadata, was truncated, or has an IME window with unreadable bounds reportsunmeasurable. The test IME draws no window, so it readsabsent.The helper now also reports
missingRootWindowTypes, the window types it listed but could not serialize (null or failing root), published asandroidSnapshot.missingRootWindowTypes. If it names an input method window, the band isunmeasurable(window-root-unavailable). An older helper omits it.The tap guard already prefers a measured band, so Android taps now use the window bounds.
Downstream, tester-army/e2e reads the band to decide when to offer keyboard dismissal.
Validation
Commit 907ed8c.
pnpm check:affected --runpasses.pnpm test:replay:android: 6/6 on a Pixel 8 emulator (API 35).Live with the rebuilt helper, Settings search,
snapshot -i --json:open --no-test-ime), field focused:{"kind":"visible","frame":{"x":0,"y":1517,"width":1080,"height":883}}, matching the IME touchable region indumpsys window.keyboard dismiss:{"kind":"absent"}.{"kind":"absent"}.All three report
missingRootWindowTypes: []. Unit tests cover a null-root IME window, which can't be forced live.