skip to content
all examples

Driving a UI SDK app through its agent bridge with e2e

A music app built with UI SDK, tested by the built-in agent and by a custom executor that uses the UI SDK agent bridge, with both runs measured.

Community examples aren't actively verified by the e2e team so exercise extra caution when running.

A score you can watch, built with UI SDK and tested with e2e. One test sentence, run by two agents: e2e's own, which reads the screen, and the same agent reaching the page through the UI SDK agent bridge.

Ricochet is a small music app. A score has four voices (melody, harmony, bass, drums) of 32 steps each. Every voice is a ball falling down its own lane, and every note is a soft pad the ball lands on at the moment the note sounds.

It exists to try one idea. UI SDK builds interfaces out of behaviors, so every component can tell an agent what it is, what state it is in and what it can do. e2e runs tests whose steps are sentences an agent carries out. So: what happens when an e2e agent uses a UI SDK app through that bridge instead of through the screen?

What is in here

e2e-ui-sdk/uiSdkBridge(): an e2e agent whose only tools are the three the UI SDK bridge defines (snapshot, query, act). One file, under eighty lines. Nothing in e2e or UI SDK is changed.
src/The app. Stock UI SDK components for the title, tempo, key and scale, and one custom component, the voice, that a person edits by pressing cells and an agent sets in one action.
tests/score.e2e.ts: the same sentence given to both agents. grid.e2e.ts: a person's way through the grid, with no model.
film/The film, rendered from e2e's own run reports. See film/README.md.

The test

typescript
describe('writing a score', { agent: ['native', 'bridge'] }, () => {
  test('an agent writes a given piece into the score', async ({ app, agent, screen }) => {
    await app.open('/');
    await agent.act(GOAL);   // "Write this piece into the score. … Set the title to "Tin roof" and the tempo to 92."
    await expect(screen.getByTestId('pattern-melody')).toHaveText(PIECE.melody);
    // … harmony, bass, drums and the summary line, all read from the real screen
  });
});

native is e2e's built-in agent. It reads the screen and presses cells of the grid, one note at a time. bridge is the same model with uiSdkBridge() as its executor: it asks the page what components it has and calls each voice's setValue. Whichever route the agent took, the result is checked note for note on the real screen.

What it measured

A 61-note, four-voice piece, DeepSeek V4.1 Flash for both agents, cache off, four runs.

Average of the 3 runs both got righte2e agentwith the UI SDK bridge
Time294.2 s (262.8 – 349.9)44.0 s (27.3 – 70.6)6.7× less
Model calls14.7 (13 – 17)4.3 (4 – 5)3.4× less
Tokens1,758,680 (1,607,583 – 1,863,733)35,300 (26,121 – 50,387)50× less

The bridge agent got the piece right in 4 runs of 4, the screen-reading agent in 3 of 4. In the fourth it held a repeated note instead of striking it again (in the grid, the cell after a note holds it, and a second press strikes it), reported the step as done, and the check on the real screen failed the test. That run is in film/runs/run5.json and is left out of the averages.

Read these with three things in mind:

  • The task suits the bridge. A piano-roll is the worst case for reading a screen: about a thousand cells, of which the agent must find and press some seventy. On an ordinary settings form (a name, two selects, three switches, a save button) the same comparison gave about 2× on time and 3.7× on tokens.
  • The runs vary. The bridge agent took between 27 and 71 seconds for the same sentence. The reports are in film/runs/; python3 film/extract.py film/runs/run*.json prints the figures.
  • The bridge does not act like a person. It calls a component's action directly, so it would not notice a button that is covered or off screen. That is why the test reads the result from the screen, and why this is a second way to drive a test, not a replacement for the first.

Use the bridge agent in your own project

Your app must be built with UI SDK and create the bridge before it renders:

typescript
import { createAgent } from '@ui-sdk/react/agent';

const agent = createAgent();
Object.assign(window, { agent });

Then copy e2e-ui-sdk/index.ts into your project and give an agent its executor:

typescript
import { web } from '@e2e-dev/web';
import { agentTools } from '@ui-sdk/react/agent';
import { uiSdkBridge } from './e2e-ui-sdk';

const engine = web();

export default {
  targets: [{ engine, app: { url: 'http://localhost:3000' } }],
  agents: {
    native: { model },
    bridge: { model, executor: uiSdkBridge({ engine, tools: agentTools }) },
  },
};

A test picks one with { agent: 'bridge' }, or runs once per agent with { agent: ['native', 'bridge'] }. The bridge agent needs a model reached through the AI SDK; e2e does not allow custom tools on its Claude Code and Codex executors.

Run it

You need Node 22.22 or newer.

terminal
npm install
npx e2e-web install chromium
npm run dev              # the app, at http://127.0.0.1:5174: press Play
npm run test:grid        # the test that needs no model

For the two agent tests, copy .env.example to .env.local and fill in a model with tool calling from any OpenAI-compatible endpoint, then:

terminal
npm test

One run of tests/score.e2e.ts takes about five minutes and, with the screen-reading agent, about 1.8 million tokens (half of them cached).

Writing a score by hand

Press a cell to put a note on that step. Press the cell right after a note to hold it through that step; press a held cell to strike the note again. As text, a voice is one token per step: a scale degree 1–10 (8 is the octave), . for a rest, - to hold. The drums take K, S, H in any combination. Degrees, not note names, so changing the key or the scale never breaks what is written.

Status

UI SDK is in beta (0.1.0-beta.1 here) and its agent key is marked experimental, so the bridge may change. Tested with e2e 0.23.0 on Chromium.

MIT licensed.

[ get started ]

your verificationlayerstarts withjust one command

npx e2e init