Systems2022 – present

Automating the unautomatable — legacy & hybrid apps

DLL injection, UIA, JCEF, EdgeWebView2 — each problem category required a completely different low-level solution. Schneider Electric, Toyota, Macy's.

Schneider Electric in production · 0 other vendors succeeded

The problem

Enterprise clients had desktop applications that no automation framework could touch. Three distinct problem categories:

1. Legacy Windows apps with zero accessibility support — no UIA tree, no MSAA, nothing. Specifically the JADE application, which uses custom Windows GDI text-drawing calls invisible to any standard accessibility framework.

2. WinForm applications with canvas-based interactions and proprietary component trees — Schneider Electric's industrial application, which every automation vendor had attempted and given up on.

3. Hybrid desktop apps embedding web content — Electron-style apps using EdgeWebView2, Java desktop apps using JCEF (Java Chromium Embedded Framework) — where standard WebDriver couldn't reach the embedded browser.

What I built

Designed and implemented a separate solution for each problem category:

1. JADE Application (text-tracking via DLL injection): Injected a DLL using Microsoft Detour to intercept Windows GDI text-drawing APIs (TextOut, ExtTextOut) at the kernel interface level. This exposes every text element on screen — including those with no accessibility tree whatsoever — to the ACCELQ agent. Combined with regular UIA for standard controls.

2. Schneider Electric (WinForm + custom tree): Created user extension commands for their custom tree component and drag-and-drop canvas interaction. Implemented a hybrid approach merging UIA with WinForm-specific APIs through DLL injection, giving precise control over their custom component model.

3. JCEF / EdgeWebView2 (hybrid apps): Located the running embedded browser process, attached msedgedriver via the remote debugging port, and used standard Chrome DevTools Protocol commands as if it were a regular Chrome session. Zero additional instrumentation on the target app. Used by Toyota, Riva, Macy's, Schneider.

Approaches considered

✕Image/pixel recognition — rejected: too brittle, breaks on any visual change, no semantic meaning, extremely slow feedback loop

✕Standard UIA / MSAA — rejected: the root problem; these apps don't have an accessibility tree for UIA to read

✕Asking clients to modernise their apps — not viable: production systems used by thousands of people, years of migration effort

✓DLL injection via Microsoft Detour — chosen for JADE and Schneider: the only way to reach internal application state invisible to any external observer

✓EdgeWebView2 remote debugging port attachment — chosen for hybrid apps: standard CDP protocol, zero changes to target app, immediate WebDriver compatibility

Architecture

DLL injection approach: Microsoft Detour DLL injected at process startup → hooks target Windows APIs (GDI TextOut / UIA COM interfaces) → intercepts calls and forwards element data via named pipe → ACCELQ agent reads from pipe and translates to test commands.
Hybrid WebView approach: detect running EdgeWebView2/JCEF process → attach msedgedriver/chromedriver to remote debugging port → standard CDP session → test commands execute as if a normal browser.

Result

Schneider Electric's industrial application — the one every other vendor had given up on — is now in production. JADE application fully automated via GDI text-tracking. EdgeWebView2 approach used by Toyota, Riva, Macy's, and Schneider. PoC work covered 32 enterprise customers in the 2024–2025 period.

DLL InjectionUIA / MSAAJCEFEdgeWebView2WinForm

Back to all work