FlagshipApr 2024 – present

Real-time mobile test recorder

Extended Appium's device drivers at the protocol level to stream live screen capture with element detection on real Android and iOS devices.

Special-access preview · Android + iOS + WebView

The problem

Recording test interactions on mobile is fundamentally harder than on web. Device communication is slower and asynchronous. Element hierarchies differ completely between iOS and Android — native elements and WebView elements coexist on the same screen with no unified access model.

There is no mobile equivalent of Chrome DevTools Protocol, the approach that made the browser Live Recorder possible. iOS adds another constraint: standard node traversal is deliberately throttled by the OS, making real-time cursor-tracking extremely slow if done naively.

What I built

Extended Appium's device drivers at the protocol level to support real-time recording sessions — a unique approach that doesn't exist in any standard Appium tooling.

Built an efficient neighbour-node traversal algorithm for element-under-cursor detection on iOS, bypassing the OS's slow standard traversal. Implemented a pure JavaScript approach for real-time element highlighting inside WebView contexts. Used the adapter design pattern to unify Android and iOS into a single modular recording pipeline — adding a new platform requires only a new adapter, not a rewrite.

Approaches considered

✕Image/pixel recognition — considered: too brittle, breaks on any visual change or device resolution difference

✕Appium Inspector — existing tool but captures static state snapshots, not a real-time recording session

✕Building directly on XCUITest / Espresso — too platform-specific; couldn't share code between Android and iOS without duplicating everything

✓Extending Appium drivers at the protocol level — chosen: real-time access to element hierarchy, single codebase for both platforms via adapters

Architecture

Extended Appium driver receives touch events → neighbour-node traversal finds element under cursor (iOS-optimised to skip slow full-tree traversal) → element metadata packaged and streamed over WebSocket to frontend → JavaScript injected into WebView renders real-time highlighting → adapter layer abstracts all platform differences between Android and iOS.

Result

Currently in special-access preview with select enterprise clients. Unique in market: real-time recording on local devices with both native and WebView element support, across Android and iOS, in a single unified flow.

AppiumAndroidiOSJavaJavaScript

Back to all work