Community examples aren't actively verified by the e2e team so exercise extra caution when running.
A score you can watch, built with UI SDK and tested with e2e. One test sentence, run by two agents: e2e's own, which reads the screen, and the same agent reaching the page through the UI SDK agent bridge.
Ricochet is a small music app. A score has four voices (melody, harmony, bass, drums) of 32 steps each. Every voice is a ball falling down its own lane, and every note is a soft pad the ball lands on at the moment the note sounds.
It exists to try one idea. UI SDK builds interfaces out of behaviors, so every component can tell an agent what it is, what state it is in and what it can do. e2e runs tests whose steps are sentences an agent carries out. So: what happens when an e2e agent uses a UI SDK app through that bridge instead of through the screen?
What is in here
e2e-ui-sdk/ | uiSdkBridge(): an e2e agent whose only tools are the three the UI SDK bridge defines (snapshot, query, act). One file, under eighty lines. Nothing in e2e or UI SDK is changed. |
src/ | The app. Stock UI SDK components for the title, tempo, key and scale, and one custom component, the voice, that a person edits by pressing cells and an agent sets in one action. |
tests/ | score.e2e.ts: the same sentence given to both agents. grid.e2e.ts: a person's way through the grid, with no model. |
film/ | The film, rendered from e2e's own run reports. See film/README.md. |
The test
describe('writing a score', { agent: ['native', 'bridge'] }, () => {
test('an agent writes a given piece into the score', async ({ app, agent, screen }) => {
await app.open('/');
await agent.act(GOAL); // "Write this piece into the score. … Set the title to "Tin roof" and the tempo to 92."
await expect(screen.getByTestId('pattern-melody')).toHaveText(PIECE.melody);
// … harmony, bass, drums and the summary line, all read from the real screen
});
});native is e2e's built-in agent. It reads the screen and presses cells of the grid, one note at a time.
bridge is the same model with uiSdkBridge() as its executor: it asks the page what components it has and calls
each voice's setValue. Whichever route the agent took, the result is checked note for note on the real screen.
What it measured
A 61-note, four-voice piece, DeepSeek V4.1 Flash for both agents, cache off, four runs.
| Average of the 3 runs both got right | e2e agent | with the UI SDK bridge | |
|---|---|---|---|
| Time | 294.2 s (262.8 – 349.9) | 44.0 s (27.3 – 70.6) | 6.7× less |
| Model calls | 14.7 (13 – 17) | 4.3 (4 – 5) | 3.4× less |
| Tokens | 1,758,680 (1,607,583 – 1,863,733) | 35,300 (26,121 – 50,387) | 50× less |
The bridge agent got the piece right in 4 runs of 4, the screen-reading agent in 3 of 4. In the fourth it held a
repeated note instead of striking it again (in the grid, the cell after a note holds it, and a second press strikes
it), reported the step as done, and the check on the real screen failed the test. That run is in
film/runs/run5.json and is left out of the averages.
Read these with three things in mind:
- The task suits the bridge. A piano-roll is the worst case for reading a screen: about a thousand cells, of which the agent must find and press some seventy. On an ordinary settings form (a name, two selects, three switches, a save button) the same comparison gave about 2× on time and 3.7× on tokens.
- The runs vary. The bridge agent took between 27 and 71 seconds for the same sentence. The reports are in
film/runs/;python3 film/extract.py film/runs/run*.jsonprints the figures. - The bridge does not act like a person. It calls a component's action directly, so it would not notice a button that is covered or off screen. That is why the test reads the result from the screen, and why this is a second way to drive a test, not a replacement for the first.
Use the bridge agent in your own project
Your app must be built with UI SDK and create the bridge before it renders:
import { createAgent } from '@ui-sdk/react/agent';
const agent = createAgent();
Object.assign(window, { agent });Then copy e2e-ui-sdk/index.ts into your project and give an agent its executor:
import { web } from '@e2e-dev/web';
import { agentTools } from '@ui-sdk/react/agent';
import { uiSdkBridge } from './e2e-ui-sdk';
const engine = web();
export default {
targets: [{ engine, app: { url: 'http://localhost:3000' } }],
agents: {
native: { model },
bridge: { model, executor: uiSdkBridge({ engine, tools: agentTools }) },
},
};A test picks one with { agent: 'bridge' }, or runs once per agent with { agent: ['native', 'bridge'] }.
The bridge agent needs a model reached through the AI SDK; e2e does not allow custom tools on its Claude Code and
Codex executors.
Run it
You need Node 22.22 or newer.
npm install
npx e2e-web install chromium
npm run dev # the app, at http://127.0.0.1:5174: press Play
npm run test:grid # the test that needs no modelFor the two agent tests, copy .env.example to .env.local and fill in a model with tool calling from any
OpenAI-compatible endpoint, then:
npm testOne run of tests/score.e2e.ts takes about five minutes and, with the screen-reading agent, about 1.8 million
tokens (half of them cached).
Writing a score by hand
Press a cell to put a note on that step. Press the cell right after a note to hold it through that step; press a
held cell to strike the note again. As text, a voice is one token per step: a scale degree 1–10 (8 is the
octave), . for a rest, - to hold. The drums take K, S, H in any combination. Degrees, not note names, so
changing the key or the scale never breaks what is written.
Status
UI SDK is in beta (0.1.0-beta.1 here) and its agent key is marked experimental, so the bridge may change.
Tested with e2e 0.23.0 on Chromium.
MIT licensed.
