Caching
Replay verified agent actions without model calls, and let the agent take over when the app changes.
The replay cache records the actions of an agent.act() step after a later
check verifies the result. On the next run, the runner replays those
actions without a model call. When a control or the expected end state no
longer matches, the agent takes over from the current screen. See
Self-healing tests.
agent.assert, agent.waitFor, and agent.extract always run live. The
run summary counts the outcomes:
AI 4.1k tokens · 2 model calls · openai/gpt-6-luna-fast
Cache 4 replayed · 1 handed off · 1 missed · 1 not savednot saved counts the steps of passing and flaky tests whose
step.cache.write is unconfirmed,
no-change, or not-written in the attempt that passed.
When a recording is written
End the test with a check of its outcome. In read-write mode, the runner
saves the recording of each act at the end of the attempt when a later
verification step passed. One check at the end confirms every act before
it.
import { expect, test } from 'e2e';
test('adds an item', async ({ app, agent, screen }) => {
await app.open('/cart');
await agent.act('add one item to the cart');
await expect(screen.getByRole('status')).toHaveText('1 item');
});step.cache.write, in the report only, says what happened to the recording
when the attempt ended:
A step that took no action records nothing and reports no write.
A failing attempt still saves the steps verified before the failure. It
deletes unverified entries and the replays involved in the failure. To keep
setup goals recorded while you fix a failing assertion, end the setup with
a wait for its state, such as locator.waitFor(). A custom store needs a delete method for this. Cancelling
a run leaves the cache unchanged.
Failures where no model answered
An attempt whose only failures are ones where no model answered evicts nothing, since these say nothing about the app:
MODEL_UNAVAILABLE: no model is configured, or GitHub Copilot doesn’t serve it to the account.MODEL_PROVIDER_FAILED: the provider returned an error, such as a missing or rejected key, a rate limit, or a 5xx, or a request got no response within 120 seconds.
So a read-write run without model access keeps its recordings, including
a step that replayed before an agent.assert that could not reach its
model.
These count as a failure of the flow: an unusable model answer
(MODEL_OUTPUT_INVALID, CONTEXT_OVERFLOW), a refusal (MODEL_REFUSED),
STEP_TIMEOUT (even with a
request still open), and any other failure in the same attempt, such as an
afterEach assertion.
What must match
A cache entry belongs to one test, target, instruction, set of params, and agent. Each of these changes causes a miss:
- renaming the test, its
describegroup, or the target, or moving the test file, - changing the instruction or an ordinary param,
- running with another configured agent (
agent: ['buyer', 'admin']records one entry each), or changing the agent’scontextor the test’sagentContext(a secret in it enters the key as its name only), - switching to another engine, or changing the project id or the app’s identity.
Changing the model doesn’t cause a miss, and neither does upgrading the engine: a node the new version resolves differently misses at replay.
Use --no-cache to evaluate a new model. Replays expose
step.cache.recordedBy when the entry has provenance; older or unattributed
entries omit it.
A timestamp or a fresh email address would miss on every run. Wrap such
values in unique(), and the runner puts in the current value at replay:
import { unique } from 'e2e';
const name = `E2E ${Date.now()} Company`;
await agent.act('create a company named {name}', { params: { name: unique(name) } });
await expect(screen.getByRole('heading', { name })).toBeVisible();Use unique() for data the flow enters or checks, and ordinary params for
choices that change the flow, such as a plan name. The substitution also
covers the value as a URL spells it (%20, +) and a multi-line value as
the screen shows it, on one line. When a unique() value
equals or sits inside another param’s value, the step isn’t recorded:
step.cache.notRecorded reads param-collision, and the next run records
it.
For preview URLs that change on each deploy, set
app.identity to keep one cache
identity. On devices, replay uses the
screen title as the path.
Rerun a test while you fix it
While you fix a failing test, keep the cache on in read-write mode, the
local default, and outside strict mode,
which never writes. With CI set, an unset mode is read-only. A failed
run keeps the recordings a check
confirmed before the failure. The next run
replays those goals without model calls, as long as each replay still
matches the screen.
- Run only that test, by the line its
test(opens on:npx e2e run tests/cart.e2e.ts:3. - Don’t rename the test or reword a goal that passes. Each causes a miss: a rename for every goal in the test, a reworded goal for itself.
- When retries are on (
retriesin the config, orCIset), add--retries 0. Aretriesoption on the test or itsdescribewins over the flag. A retry doesn’t replay, so it runs every goal live. - When the test passes, run it one time with
--no-cache. Every goal runs live, which shows the agent still reaches it without a recording.
How a replay works
- Check the starting screen: it must be the same route.
- Find each recorded control by role, name, test id, and surrounding context, and repeat its action. The runner waits for missing controls and stops when a match is ambiguous.
- Check the final route and what the original step changed: the controls that appeared, with their checked, selected, expanded, or pressed state, and the ones that went away.
- Finish without a model call, or let the agent continue with a record of what ran and why replay stopped.
Typed values replay as recorded. A secret fill resolves the current value by name and goes through authorization again.
A replay finishes as soon as the recorded changes show, which can be before
content below them has loaded. To check content that loads after the step,
use agent.waitFor, which waits for it, rather than agent.assert, which
judges the screen once.
What counts as the same route
A route is the origin, the path, and the query. The record id and the
fragment don’t count. Ids, tokens, timestamps, and cache busters in the
path or the query read as an id placeholder, so /orders/42?t=1727780000
and /orders/7?t=1727780999 are one route.
Every other query value is part of the route: ?mode=safe and
?mode=unsafe differ, and so do /companies and /companies?tab=notes, or
a new tracking parameter. /products/summer-sneaker and
/products/winter-boot are different routes too.
A location on the app’s own origin is compared by its path, so a recording
follows the app to a preview deployment that shares its app.identity. A
page on any other origin is another route.
How the end check reads the screen
- At least one recorded change must happen during the replay. An outcome already on screen before the actions proves nothing. When the step’s last action moved to another route, that move is the evidence.
- An alert that the replay raised and the recording never saw stops the replay. The web engine does not observe Next.js App Router’s route announcer, so its title echo never counts.
- Text that differs on every run, such as a date, a duration, or an id,
is skipped while a stable change exists on either side of the step’s delta,
whether something appeared or disappeared. With nothing else, it is kept,
and a changed value hands the replay off. In an alert, it reads as a placeholder.
A node known only by a test id the app generates (
node-<uuid>) is skipped the same way. When kept, a node that appeared may carry any id with that pattern; one that went away is still checked by its exact id. - A count that names what it counts, such as
3 records imported, must read the same on replay when the step made it appear on the screen it acted on. A count that only moved (41 itemsto42 items), or one on a screen the step navigated to, is skipped only when its own side of the delta has a stable change. Otherwise the count stays, even when the opposite side has a stable change. A bare number such as a badge follows the text rule above. - A control with no name is found by its place among the unnamed controls of its kind inside the same named row, list item, or group. A control known only by a generated test id is found by the id pattern the same way, or when it was the only one with that pattern.
Why a step was not replayed
Each act step reports step.cache, absent when caching is off for the
step. agent.act() returns it as result.cache, without write, and the
trace page shows it under the step.
Both carry step.cache.reason, and step.cache.detail says where the
replay stopped and why (at action 2 of 3, tap button "Apply": ...). After
an end-mismatch it names the recorded nodes that did not show, the ones
that did not go away, and any alert the recording never saw, or says the
outcome already showed before the replay acted.
step.cache.entry is the entry’s key and its file name (<entry>.json) in
the file store.
To rule the cache out of a failure, run one time without it:
npx e2e run --no-cachepnpm exec e2e run --no-cachebunx e2e run --no-cacheLimits and gaps
A recording is truncated when the step:
- took more than 50 actions, which is possible only with
maxStepsabove 50, - typed a value or URL over 4,096 characters, or one that holds a registered secret,
- acted on a control with no usable description.
A gap is an action the cache can’t repeat: a mutating
project tool, or a value the agent read off the screen, typed
after it saw a screenshot, or worked out from the date. step.cache.derived
names the rule: whole-node (a control’s whole name or text),
minted-token (a code or id the app showed as its own word), date, or
pixels. Values from the instruction or params, and values the agent
composed itself, replay.
What a recording holds
Each entry is a JSON file under .e2e/cache/, named by a hash of its key.
It holds actions, target descriptions, typed text, a step summary, and the
end checks: the route and up to eight controls that appeared and eight that
went away. Entries hold no prompts, conversations, screenshots, observed
trees, or secret values. A secret fill stores only the secret’s name.
Other typed values are stored as typed. Review recordings before you share them. See cache trust.
Modes
cache: 'off',
// or
cache: { mode: 'read-write', dir: '.e2e/cache' },--no-cache overrides the config. In read-only mode, stale entries stay
until a read-write run replaces them. A custom cache.store replaces the
file store and doesn’t default to read-only in CI. e2e explore and
e2e mcp sessions don’t use the cache. A test a session runs
(open_session {test}) uses it as a run’s
first attempt does.
Share the cache across checkouts
Point every checkout of a project at one directory with E2E_CACHE_DIR, or
--cache-dir for one run. Either replaces cache.dir, resolves from the
project root (~ or a leading ~/ is your home directory), and leaves the mode
alone. With a custom cache.store, --cache-dir is an error and
E2E_CACHE_DIR is ignored. Parallel runs can share the directory: each write
replaces its file atomically. Entries are keyed by project, so give every
checkout the same projectId (or package.json name); without either,
each checkout’s path becomes its identity and nothing replays across them.
- Checkouts share entries: a failing run in any of them evicts the entry, and checkouts with the same app identity but different app code overwrite each other’s entries (the last write wins).
- While the override is set, a cache committed under
cache.dirisn’t read. - Use one directory per project. Two projects in one directory can fail with
a false
REPLAY_STALEundercache.strict.
E2E_CACHE_DIR=~/.cache/my-app/e2e npx e2e runCommit the replay cache
e2e init adds .e2e/cache/ to .gitignore, so each machine records its
own and CI starts with none. To share replays, remove that line and commit
the directory. CI stays read-only unless you set cache: 'read-write'.
Treat committed entries as test data, and review them in the pull request
that recorded them again.
Local runs leave a committed directory clean unless something changed. A replayed step never rewrites its entry, and a live recording with the same actions, paths, and anchors isn’t written. So an entry for a UI that drifts is repaired only when a control can no longer be found. A file changes when:
- a step records for the first time, or after a hand-off,
- a step fails after a replay, ends in an
end-mismatchthe agent had to repair, passes with nothing to record, or isn’t confirmed by the end of the attempt, which deletes the entry, - an
e2eupgrade changes how entries are matched, which gives every entry a new key, so the next run records them again.
Fail on a stale recording
By default, a committed recording that stops replaying fails nothing: the
agent takes the step over, and every CI run pays for model calls until
someone records it again. With --strict-cache (or
cache: { strict: true }), the step fails with REPLAY_STALE and makes no
model call:
CI=1 npx e2e run --strict-cacheCI=1 pnpm exec e2e run --strict-cacheCI=1 bunx e2e run --strict-cacheREPLAY_STALE exits 2 and is not retried.
A recording under another key comes from an e2e upgrade that changed the
keys, or a change to the project id, the app’s identity, or the agent’s
context. The message names that file and, for a recording made by a
current version, what changed. Delete the file when nothing replays it. This check doesn’t cover a
custom cache.store.
A strict run never writes or deletes an entry, even in read-write mode, so
a new step records only on a run without strict. Prefer --strict-cache in
CI over cache.strict in the config. To fix a stale entry:
- Run the test in
read-writemode without the flag. Turncache.strictoff if the config sets it. - Commit the changed entry. A custom
cache.storegets the new entry from that run instead.
Inspect the cache
npx e2e cache ls # one row per entry: test, target, instruction digest, age, actions
npx e2e cache stats # entry count and size
npx e2e cache clear # delete every entrypnpm exec e2e cache ls # one row per entry: test, target, instruction digest, age, actions
pnpm exec e2e cache stats # entry count and size
pnpm exec e2e cache clear # delete every entrybunx e2e cache ls # one row per entry: test, target, instruction digest, age, actions
bunx e2e cache stats # entry count and size
bunx e2e cache clear # delete every entryThe cached percentage in AI usage is the provider’s prompt cache. It
lowers the cost of a model call. The replay cache skips the call.
