> ## Documentation Index
> Fetch the complete documentation index at: https://e2e.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Test filters

> Run only the tests a pull request can affect.

With `--changed-since <ref>`, a test filter leaves out the tests the change
since that ref can't affect, and starts the files of the likeliest to fail first. The
decision filter asks a [decision model](https://e2e.com/docs/decision-models), such as Jev, about
each test. Without the flag, as on `main` and locally, every test runs.

<Video>
  <video src="/images/filters.mp4" aria-label="Two runs side by side. With --changed-since, the decision filter leaves out 25 of 27 tests and the run passes in 12 seconds. The full run is still going after 30 seconds." muted loop playsInline controls />

  With `--changed-since origin/main`, the decision filter leaves out 25 of 27 tests.
</Video>

## Add the decision filter

<CodeGroup>
```bash npm
npm install -D @e2e-dev/decision ai
```

```bash pnpm
pnpm add -D @e2e-dev/decision ai
```

```bash bun
bun add -d @e2e-dev/decision ai
```
</CodeGroup>

```ts title="e2e.config.ts"

export default {
  targets: [{ engine: web(), app: { url: 'http://localhost:3000' } }],
  filters: [decisionFilter({ model: gateway.decisionModel('typesafe-ai/jev') })],
} satisfies E2EConfig;
```

In CI, pass `--changed-since` on pull requests, and give the e2e step the
AI Gateway key:

```yaml title=".github/workflows/e2e.yml"
steps:
  - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
    with:
      fetch-depth: 0
  # ...
  - run: npx e2e run ${{ github.event_name == 'pull_request' && format('--changed-since origin/{0}', github.base_ref) || '' }}
    env:
      AI_GATEWAY_API_KEY: ${{ secrets.AI_GATEWAY_API_KEY }}
```

The flag goes only on `pull_request` runs, so a push, merge queue, or
scheduled run passes none and runs every test. Never pass it on
`pull_request_target`: that run holds your secrets, and a fork's diff
could talk the model into leaving tests out. `fetch-depth: 0` puts the
base branch and the merge base in the checkout. Without them, the run fails
with `INVALID_CHANGED_SINCE`.

The change is every file that differs from the merge base of the ref and
`HEAD`, uncommitted and untracked files included. The model reads the diff
of what the commits change, the test files, and the setup files that save
their sessions, with secrets redacted, except a value a secret provider
returns. It gets a file only the working tree changes, uncommitted or
untracked, by name alone, never its contents, so a file a CI step writes
doesn't reach it.

The filter leaves out a test when the model rates both the test and its
setup under 20% likely affected. Tests the model can't judge run, and the
filter's note says why: a diff over 60 kB, a test or setup file too large
to send with the diff, or some calls failing. When every call fails, as
without a key on a pull request from a fork, the filter fails: the run
warns and runs every test, starting first the ones whose test file, setup
file, or stored screenshots changed. When git can't measure the change, because it
is missing, the project is not in a repository, or the ref shares no
history with `HEAD`, the run fails with `INVALID_CHANGED_SINCE`. The
filter needs `ai` 7.0.134 or later.

Any [decision model](https://e2e.com/docs/decision-models) that answers `boolean` questions
works, such as Jev through TypeSafe directly:
`typeSafeAi.decisionModel('jev-latest')` from `@ai-sdk/typesafe-ai`, with
`TYPESAFE_AI_API_KEY` set. `providerOptions` goes with every call, such as
`{ gateway: { zeroDataRetention: true } }`.

## Preview locally

<CodeGroup>
```bash npm
npx e2e list --changed-since origin/main
```

```bash pnpm
pnpm exec e2e list --changed-since origin/main
```

```bash bun
bunx e2e list --changed-since origin/main
```
</CodeGroup>

`e2e list` prints each test a filter leaves out, with the reason. It starts
no app, and makes the same model calls as a run. Commit first: the decision
filter reads only what the commits change.

## What always runs

- Every test, when the run has no `--changed-since`, or when no file
  changed since the merge base.
- A `--last-failed` rerun, so `e2e run || e2e run --last-failed` is safe,
  and every `--watch` rerun after the first run.

The decision filter also runs:

- The tests in a test file that changed or whose stored screenshots
  changed, and the tests that use the sessions of such a setup file.
- Every test, when the config the run loaded changes, whatever its name,
  or when a file named `e2e.config.ts` (`.mts`, `.js`, `.mjs`),
  Gradle's `libs.versions.toml`, or a lockfile changes in the project's
  directory, inside it, or in a directory above it, such as the workspace
  root. Lockfiles are known by name, such as `pnpm-lock.yaml`,
  `Podfile.lock`, `gradle.lockfile`, `Gemfile.lock`, `go.sum`, or
  `Cargo.lock`. The model judges any other file, and those of a sibling
  project.

A filter that throws, returns something other than
`{ exclude, relevance?, note? }`, or runs past 5 minutes leaves nothing out,
and a warning says so. The next filter still judges every test it was given.
A failed filter scores nothing either, unless the error it threw carries
`relevance`, as in [Write your own](#write-your-own). A filter that runs past
5 minutes is abandoned, and nothing it throws after that counts.

On another CI service, pass `--changed-since origin/main` on pull request
builds only, and fetch enough history for the merge base. On GitLab, see
[Run only affected tests](https://e2e.com/docs/ci/gitlab#run-only-affected-tests). With
`--shard`, the filters on each machine judge only its slice.

## Start the likeliest first

The decision filter scores each test it keeps, and each target starts the
files of the likeliest tests first. A failure then shows early, which saves
the most with `--max-failures` and with one worker, the default in CI.

| Test | Score |
| --- | --- |
| Its test file, setup file, or stored screenshots changed | 100% |
| Every other test | The higher of its rating and its setup's, where a rating the model didn't give counts as 50% |

- Files start in four bands: 75% and up, 50% up to 75%, 25% up to 50%, and
  under 25%. A file takes the band of its likeliest test.
- With more than one worker, the longest file by the
  [timings history](https://e2e.com/docs/ci#cache-the-timings-history) starts first inside a
  band. A file with no time starts before the timed ones.
- Targets go by the band of their best file, ties in config order, and each
  target starts its own files in that order. A target's setup tests still
  finish before its first test file starts.
- A short likely file can start before a long unlikely one, so with several
  workers a run that finishes whole can end later. The report keeps its
  order, and `e2e list` shows none.

A filter you write scores tests with `relevance`, as in
[Write your own](#write-your-own).

## What you see

- **Terminal**: one line per filter, such as
  `decision kept 7 of 77 tests, left out 70 (1 call, 17.6k tokens, $0.0007)`.
  The decision filter's note ends with its spend: model calls, tokens, and
  the cost when the provider reports it, as the AI Gateway does. A line such
  as `ranked 7 of 7 tests by relevance, likeliest first` follows when
  the scores put tests in more than one band. A `Left out` row follows in the summary. When the
  filters leave nothing to run, the run passes and starts no app.
- **[Pull request comment](https://e2e.com/docs/github) and `summary.md`**: the headline
  `7 passed, 70 left out by filters`, and a folded table of the tests with
  their reasons.
- **`junit.xml` and `report.json`**: each left-out test is a skipped case,
  with `skip.filter` in the report, beside `run.filters` and `run.changes`.
  See [Reporters](https://e2e.com/docs/reference/reporters#test-filters).

## Adjust the decision filter

Wrap the decision filter to change what it judges. This one runs every
test when `infra/` changes, leaves `apps/mobile/`, which these tests don't
reach, out of the change, and always runs the `smoke` tests:

```ts title="tuned-filter.ts"

const decision = decisionFilter({ model: gateway.decisionModel('typesafe-ai/jev') });

export const tuned = {
  name: 'decision',
  filter(tests, context) {
    if (context.changes.files.some((file) => file.startsWith('infra/'))) return { exclude: {}, note: 'infra/ changed' };
    const judged = tests.filter((test) => !test.tags.includes('smoke'));
    const files = context.changes.files.filter((file) => !file.startsWith('apps/mobile/'));
    if (files.length === 0) return { exclude: Object.fromEntries(judged.map((test) => [test.id, 'only apps/mobile/ changed'])) };
    return decision.filter(judged, { ...context, changes: { ...context.changes, files } });
  },
} satisfies TestFilter;
```

Put `tuned` in `filters` in place of `decisionFilter(...)`. To run every
test on one pull request, leave out `--changed-since` when it has the
`all-tests` label. Add `labeled` to the events so adding the label starts a
run:

```yaml title=".github/workflows/e2e.yml"
on:
  pull_request:
    types: [opened, synchronize, reopened, labeled]
# ...
      - run: npx e2e run ${{ github.event_name == 'pull_request' && !contains(github.event.pull_request.labels.*.name, 'all-tests') && format('--changed-since origin/{0}', github.base_ref) || '' }}
```

## Write your own

A filter is `{ name, filter }`: `filter` gets the tests and the change, and
returns the tests to leave out with a reason for each. It can also score the
rest from 0 to 1 with `relevance`. This one maps changed paths to test tags,
and starts the tests whose area the change reaches first:

```ts title="areas-filter.ts"

/** The test tags each part of the app reaches, by path from the repository root. */
const AREAS: Readonly<Record<string, readonly string[]>> = {
  'src/todos/': ['todos'],
  'src/auth/': ['auth', 'smoke'],
  'src/checkout/': ['checkout'],
};
/** Every tag an area reaches; a test without one always runs. */
const AREA_TAGS: ReadonlySet<string> = new Set(Object.values(AREAS).flat());

export const areas = {
  name: 'areas',
  filter(tests, { changes }) {
    const reached = new Set<string>();
    let unmapped: string | undefined;
    for (const file of changes.files) {
      const area = Object.keys(AREAS).find((prefix) => file.startsWith(prefix));
      if (area === undefined) unmapped = file;
      else for (const tag of AREAS[area] ?? []) reached.add(tag);
    }
    const exclude: Record<string, string> = {};
    const relevance: Record<string, number> = {};
    for (const test of tests) {
      const tags = test.tags.filter((tag) => AREA_TAGS.has(tag));
      if (tags.some((tag) => reached.has(tag))) relevance[test.id] = 1;
      else if (tags.length > 0 && unmapped === undefined) exclude[test.id] = 'the change reaches none of its areas';
    }
    if (unmapped === undefined) return { exclude, relevance };
    return { exclude, relevance, note: `${unmapped} is in no area, so areas leaves nothing out` };
  },
} satisfies TestFilter;
```

Put it before the decision filter, so the model judges only the tests
`areas` kept:

```ts title="e2e.config.ts"

export default {
  targets: [{ engine: web(), app: { url: 'http://localhost:3000' } }],
  filters: [areas, decisionFilter({ model: gateway.decisionModel('typesafe-ai/jev') })],
} satisfies E2EConfig;
```

- Every test you don't name runs, and so does a test you name in a serial
  group another of its tests runs in.
- `relevance` takes test ids and numbers from 0 to 1, a test no filter names
  counts as 0, and a test several filters name keeps its highest. Anything
  else fails the filter, as for `exclude`.
- A filter that fails can still score. An error it throws with a `relevance`
  property, shaped and checked like the result's, scores the tests the same
  way, and the failure still warns and leaves nothing out. The decision
  filter does this, so a run without a key still starts the tests the change
  touches first. Scores that fail the check are dropped, and the warning
  says so.
- Each filter judges only the tests the one before it kept, and is not
  called when none are left.
- Pass `signal` to every call, and `redact` what you send to a model. It
  replaces each fixed-value secret and credential password, and any run of 8
  or more of its characters, so text that shares 8 characters with one, such
  as `Password` when a password is `Password123!`, is masked too. A value a
  secret provider returns passes through.
- Do the work inside `filter`: workers load the config too, and only the
  runner calls `filter`.

See [`TestFilter`](https://e2e.com/docs/reference/config#testfilter) for the types.

## Next

<CardGroup cols={2}>
  <Card title="Continuous integration" icon="rotate" href="/ci">
    Gate pull requests, shard, and rerun failures.
  </Card>
  <Card title="Reporters" icon="file-lines" href="/reference/reporters">
    Read `run.changes`, `run.filters`, and `skip.filter` from your own reporter.
  </Card>
</CardGroup>
