Test DSL

Test cases are stored as a small JSON DSL, validated by TestCaseSchema (packages/shared/src/schemas/test-dsl.ts). A test case is a name, an ordered list of steps, optional variable bindings (vars), and optional tags. Every step is an object whose op field selects exactly one of the operations below — unknown op values are rejected at parse time.

Test case shape

{
  "name": "Login — happy path",
  "tags": ["auth", "happy"],
  "vars": [
    { "name": "email", "value": "user@example.com", "secret": false },
    { "name": "password", "value": "s3cr3t", "secret": true }
  ],
  "steps": [
    { "op": "navigate", "url": "https://app.example.com/login" },
    { "op": "fill", "selector": "#email", "value": "{{email}}" },
    { "op": "fill", "selector": "#password", "value": "{{password}}" },
    { "op": "click", "selector": "button[type='submit']" },
    { "op": "assert.url", "contains": "/dashboard" },
    { "op": "assert.visible", "selector": "h1" }
  ]
}

• name — required, non-empty. When the DSL lives on a stored test case, the record's name column takes precedence at run time.
• steps — required array (may be empty); executed strictly in order.
• vars — optional, defaults to []. See Variables.
• tags — optional, defaults to []; free-form labels used for filtering.

Operations

op	Fields	Description
navigate	url	Open an absolute URL or root-relative path
navigate.dynamic	relativeTo? / source? (exactly one), path?, selector?, attribute?, urlTemplate?	Navigate to a runtime target (relative to the current page, clipboard, or a DOM value)
click	selector	Click an element
fill	selector, value	Type into an input
select	selector, value	Choose an option in a select
hover	selector	Hover an element
totp	selector, config	Fill a freshly generated 2FA/TOTP code
wait	ms?, selector?	Pause or wait for visibility
assert.url	equals?, contains?	Assert the current URL
assert.visible	selector	Assert an element is visible
assert.text	selector, equals?, contains?	Assert an element's text
assert.count	selector, equals	Assert how many elements match
assert.status	lessThan?, equals? (exactly one)	Assert the HTTP status of the last navigate
assert.consoleErrors	max	Assert at most max console errors
act	instruction	Agentic step from a natural-language instruction
output.save	name, selector?	Save an output value for dependency successors
upload	selector, default? / config? (exactly one)	Set a file into a file input
email.waitFor	inbox?, subjectContains?, fromContains?	Wait for a mail in the test inbox
email.fillOtp	selector, inbox?, subjectContains?, fromContains?	Extract an OTP from a mail and fill it
email.openLink	linkContains?, inbox?, subjectContains?, fromContains?	Open a link from a mail
assert.download	filenameContains?, minBytes?	Verify a file download triggered by a preceding step
api.call	config, url?, body?, extract?, save?	Fire a pre-configured server-side HTTP request and extract values
api.expect	status? / statusLessThan? (exactly one)	Assert the status of the last api.call response
api.dataAssert	apiName, selector, attribute?, equals? / contains? (exactly one)	Assert a UI element matches a value extracted by a preceding api.call
auth.basic	config	Answer an HTTP Basic Auth dialog with an assigned basic_auth config
clipboard.copy	name, selector, attribute?	Copy an element's text/attribute to the clipboard and store it as {{clipboard.<name>}}
clipboard.paste	name	Read the clipboard and store it as {{clipboard.<name>}}
assert.vision	instruction, failOn?	Evaluate a natural-language visual check on a screenshot via a vision LLM

navigate

Opens a page. url is either an absolute http(s) URL or a root-relative path starting with / (anything else fails validation).

{ "op": "navigate", "url": "https://app.example.com/login" }

{ "op": "navigate", "url": "/login" }

At run time the URL is resolved against the run's environment base URL: an absolute URL keeps its path/query but swaps its origin for the environment's origin; a relative path is resolved against the environment base. A relative path without a resolvable environment fails the run with a speaking message before any browser session starts.

click

Clicks the element matched by selector (non-empty string).

{ "op": "click", "selector": "button[type='submit']" }

fill

Types value into the element matched by selector. value may be empty and may contain {{var}} placeholders.

{ "op": "fill", "selector": "#email", "value": "user@example.com" }

select

Chooses the option with value value (non-empty) in the select element matched by selector.

{ "op": "select", "selector": "#topic", "value": "support" }

hover

Hovers the element matched by selector.

{ "op": "hover", "selector": "nav .menu" }

totp

Generates a time-based one-time password (TOTP, RFC 6238) and types it into the element matched by selector. config names a credentials_totp config that is assigned to the test case (see Configs in the project settings); its totp_secret seeds the code.

{ "op": "totp", "selector": "#code", "config": "main-login" }

• The code is generated lazily at step-execution time — never during pre-run interpolation — so it cannot expire between run start and the 2FA prompt (codes are only valid for ~30 seconds).
• config is a constant reference to the assigned config's name; unlike selector it is not {{var}}-interpolated.
• Both the generated code and the underlying secret are masked as *** in every text artifact (DOM dumps, error messages, logs).
• If the named config is not assigned to the test case, or is not of type credentials_totp, the step fails with a speaking error through the normal failure pipeline.

A typical login with 2FA combines the assigned config's credential vars with the totp step:

{
  "name": "Login with 2FA",
  "steps": [
    { "op": "navigate", "url": "https://app.example.com/login" },
    { "op": "fill", "selector": "#email", "value": "{{config.main-login.username}}" },
    { "op": "fill", "selector": "#password", "value": "{{config.main-login.password}}" },
    { "op": "click", "selector": "button[type='submit']" },
    { "op": "totp", "selector": "#code", "config": "main-login" },
    { "op": "click", "selector": "button[type='submit']" },
    { "op": "assert.url", "contains": "/dashboard" }
  ]
}

wait

Waits either a fixed ms (positive integer) or until selector becomes visible. If both are given, ms wins; with neither, the step is a no-op.

{ "op": "wait", "ms": 500 }

{ "op": "wait", "selector": ".spinner-done" }

assert.url

Asserts the current page URL. Supply equals (exact match), contains (substring match), or both.

{ "op": "assert.url", "contains": "/dashboard" }

assert.visible

Asserts that the element matched by selector is visible.

{ "op": "assert.visible", "selector": "[data-testid='welcome']" }

assert.text

Asserts the text content of the element matched by selector. Two modes — equals for an exact match:

{ "op": "assert.text", "selector": "h1", "equals": "Dashboard" }

…or contains for a substring match:

{ "op": "assert.text", "selector": ".success-banner", "contains": "sent" }

assert.count

Asserts that exactly equals elements (integer ≥ 0) match selector.

{ "op": "assert.count", "selector": ".error", "equals": 0 }

assert.status

Asserts the HTTP status of the main-document response of the last navigate step. Exactly one of the two fields must be set — lessThan (strictly less than, e.g. 400 for "no client/server error") or equals (exact status, e.g. 404 for an expected not-found state):

{ "op": "assert.status", "lessThan": 400 }

{ "op": "assert.status", "equals": 200 }

• Redirect chains resolve to the final response — intermediate 3xx hops never count as failures.
• Without a preceding successful navigate the step fails with a speaking "no navigation observed" message.
• Setting both fields or neither is rejected at parse time.

assert.consoleErrors

Asserts that at most max console entries of type error (integer ≥ 0) accumulated in the browser console since test start. max: 0 means "no console errors at all":

{ "op": "assert.consoleErrors", "max": 0 }

The failure message carries the actual error count plus the first error text, so a failing run is debuggable without opening the console artifact.

act

Executes one natural-language instruction through the browser agent (Stagehand act()). No selector is stored — the agent itself decides, based on the DOM and a visual model, which element to operate:

{ "op": "act", "instruction": "Log in with the demo account" }

• instruction is a non-empty string and may contain {{var}} placeholders; they are resolved before the browser session starts, exactly like every other string field (an unknown placeholder fails the run pre-session).
• The step behaves like any other step: pre/post screenshots, the per-step timeout, and the normal failure semantics (failedStepIndex, secret-masked error message) all apply.
• Every executed act step records a structured trace — the operation the agent performed, its reasoning, and success/failure — which is persisted per step and available on the run detail, including at the failure point.
• act steps are excluded from self-healing: there is no selector to heal, and re-running the instruction could double-perform the interaction.
• Secret-flagged vars interpolated into the instruction are masked as *** in every persisted trace and error text.

{
  "name": "Login via agent",
  "vars": [{ "name": "password", "value": "s3cr3t", "secret": true }],
  "steps": [
    { "op": "navigate", "url": "/login" },
    { "op": "act", "instruction": "Sign in as demo@example.com with password {{password}}" },
    { "op": "assert.url", "contains": "/dashboard" }
  ]
}

output.save

Saves an output value under the key name so that test cases depending on this one (in the same plan run) can consume it. With selector the visible text of the matched element is stored; without selector the current page URL is stored:

{ "op": "output.save", "name": "orderId", "selector": "#order-id" }

{ "op": "output.save", "name": "dashboardUrl" }

• name is a constant key (1–64 characters of letters, digits, _, ., -); unlike selector it is not {{var}}-interpolated (same rule as totp.config). Saving the same name twice = the last value wins.
• Caps: at most 20 distinct keys per run and 2048 characters per value. Exceeding either cap fails the step with a speaking error through the normal failure pipeline — nothing is silently truncated or dropped.
• Only a passed run persists its collected values (to runs.output_values); a failed or errored run never publishes outputs.
• Dependency successors in the same plan run receive each value as a variable under the output. namespace — e.g. {{output.orderId}} in any interpolated string field. This works for every dependency (wait_for and resume_from alike); a predecessor without output.save steps simply contributes no variables.
• A saved value that echoes a secret: true variable is masked as *** before it is persisted — outputs never leak secrets in the clear.

{
  "name": "Login saves the dashboard URL",
  "steps": [
    { "op": "navigate", "url": "/login" },
    { "op": "fill", "selector": "#email", "value": "{{email}}" },
    { "op": "click", "selector": "button[type='submit']" },
    { "op": "assert.url", "contains": "/dashboard" },
    { "op": "output.save", "name": "dashboardUrl" }
  ]
}

A successor with depends_on pointing at this test case can then start with { "op": "navigate", "url": "{{output.dashboardUrl}}" }.

upload

Sets a file into the <input type="file"> matched by selector. Exactly one of the two sources must be given — default for one of the built-in test files (a small PDF or a 1×1 PNG, deterministic, available without any configuration):

{ "op": "upload", "selector": "#attachment", "default": "pdf" }

…or config naming a file config that is assigned to the test case (custom file; see Configs in the project settings):

{ "op": "upload", "selector": "#attachment", "config": "sample-invoice" }

Field	Description
selector	The file input to set; {{var}}-interpolated like every selector.
default	"pdf" or "png" — a built-in default test file.
config	Name of an assigned file config (constant reference, not interpolated).

• Setting both sources or neither is rejected at parse time.
• config is a constant reference to the assigned config's name; unlike selector it is not {{var}}-interpolated (same rule as totp.config).
• Custom files are fetched lazily at step-execution time from the platform's file storage (the config's r2_key, capped at 25 MB). The config's url field is a reference/metadata only and is never fetched by the runner. A config that is not assigned, not of type file, or whose file cannot be fetched fails the step with a speaking error through the normal failure pipeline.
• The step only sets the input's value — combine it with click (submit) and an assertion (e.g. assert.text on a success banner) to verify the upload result, exactly like any other form interaction:

{
  "name": "Upload an attachment",
  "steps": [
    { "op": "navigate", "url": "/support/new" },
    { "op": "upload", "selector": "#attachment", "default": "png" },
    { "op": "click", "selector": "button[type='submit']" },
    { "op": "assert.text", "selector": ".upload-status", "contains": "default.png" }
  ]
}

Email-inbox steps (email.*)

Every run owns a system-managed test inbox: a single-use address per run (available as {{inbox.address}}) and a permanent project address ({{inbox.projectAddress}}). The tested app sends mail to one of these addresses (e.g. because a signup form was filled with {{inbox.address}}); the three email.* steps then verify receipt, extract OTP codes, or open links — strictly from the system inbox. External or private mailboxes are never accessible.

All three steps share the same matcher fields:

• inbox — "run" (default, the single-use run address) or "project" (the permanent project address). A constant reference, not {{var}}-interpolated.
• subjectContains / fromContains — optional substring matchers on the mail's subject resp. sender; {{var}}-interpolated like every string field.

Common semantics (mirroring the reference behaviour):

• The step polls the inbox for up to 3 minutes (fixed budget, regardless of the project's step timeout) and fails with a speaking error when no matching mail arrives in that window.
• Only mails received after the run started count.
• With several matches, the newest mail wins.
• The step works on the mail's text view; link extraction additionally considers href attributes of the HTML body.

email.waitFor

Passes as soon as a matching mail has arrived in the inbox:

{ "op": "email.waitFor", "subjectContains": "Willkommen", "fromContains": "noreply@" }

email.fillOtp

Waits for a matching mail, extracts the one-time code (the first 4–8 digit number, preferring lines that mention code/OTP/PIN/verify), and fills it into the element matched by selector:

{ "op": "email.fillOtp", "selector": "#otp", "subjectContains": "Code" }

• Extraction is deterministic (regex on the text view) — no AI involved.
• The extracted code is registered as a runtime secret: it appears as *** in every text artifact (DOM dumps, error messages, logs).
• A matching mail without an extractable code fails the step with a speaking error through the normal failure pipeline.

email.openLink

Waits for a matching mail, extracts the first absolute http(s) link (optionally filtered by linkContains), and navigates the browser there via the normal navigation mechanic — a following assert.status sees the status of exactly this navigation:

{ "op": "email.openLink", "linkContains": "/verify", "subjectContains": "bestätigen" }

A typical signup flow combines the inbox address variable with all three steps:

{
  "name": "Signup with email verification",
  "steps": [
    { "op": "navigate", "url": "/signup" },
    { "op": "fill", "selector": "#email", "value": "{{inbox.address}}" },
    { "op": "click", "selector": "button[type='submit']" },
    { "op": "email.waitFor", "subjectContains": "Code" },
    { "op": "email.fillOtp", "selector": "#otp" },
    { "op": "email.openLink", "linkContains": "/verify" },
    { "op": "assert.status", "lessThan": 400 }
  ]
}

• {{inbox.address}} / {{inbox.projectAddress}} are injected automatically when inbound email is configured for the deployment; they sit at the lowest variable priority, so config/DSL/run variables with the same name win.
• email.* steps are excluded from self-healing (polling steps carry no selector to heal).

navigate.dynamic

Navigates to a target that only exists at runtime, resolved against the live page (not against the environment base URL like the static navigate). Exactly one source must be set:

• relativeTo: "current" + path — a path relative to the current page ("../settings", "?tab=profile", "/dashboard", resolved against the current origin):

{ "op": "navigate.dynamic", "relativeTo": "current", "path": "../settings" }

• source: "clipboard" — reads the URL from the system clipboard (e.g. a verification link copied during the test) and navigates there:

{ "op": "navigate.dynamic", "source": "clipboard" }

• source: "domAttribute" + selector + attribute + urlTemplate — extracts a value from a page element (e.g. a dynamic user ID) and builds the target by replacing {value} in the template with the URL-escaped value:

{
  "op": "navigate.dynamic",
  "source": "domAttribute",
  "selector": "#profile-link",
  "attribute": "href",
  "urlTemplate": "{value}"
}

A following assert.status sees the status of exactly this navigation (the same navigation mechanic as the static navigate). path, selector and urlTemplate are {{var}}-interpolated; relativeTo, source and attribute are constant references and are not interpolated. Only http, https and file targets may be navigated — any other protocol fails the step with a speaking error, as do an empty clipboard or a selector that matches no element.

assert.download

Verifies that a browser download triggered by a preceding step (a click or act on a download button) completed, and optionally asserts the suggested filename and minimum byte size. The step waits up to 30 seconds for the download to finish (files up to 100 MB), then confirms the result — it never opens or inspects the file content (only name, size and existence are evaluated). Both fields are optional; a bare assert.download passes on any completed download:

{ "op": "assert.download" }

{ "op": "assert.download", "filenameContains": "report", "minBytes": 1024 }

• filenameContains — a substring that must appear in the browser-suggested filename; {{var}}-interpolated like every other matcher.
• minBytes — a lower bound on the downloaded file size in bytes.

A verified download is recorded on the run (filename + size, the filename secret-masked) and — when storage is configured — the file itself is kept so it stays viewable in the session. A download that does not arrive within 30 s, a filename/size mismatch, or a file over the 100 MB limit fails the step with a speaking error. Streaming media and e-mail attachments are out of scope (mail attachments are handled by the email.* steps).

API steps (api.call / api.expect)

These fire a server-side HTTP request during a test (outside the browser) — not the navigation the page makes itself. The request is pre-configured as an api_call config (HTTP method, URL, JSON headers, optional body) assigned to the test case; all header values are treated as secrets and are masked (***) everywhere they could surface.

api.call

Fires the request named by config. url/body optionally override the config (both {{var}}-interpolated); the config headers (including auth) are always preserved. extract pulls named values out of the JSON response via a flat dotted path; they become {{api.<name>}} (and {{api.<name>.<path>}} for a nested value) in later steps of the same run. save: true additionally writes the extracted values into the run output so dependency successors can read them as {{output.<name>}}.

{ "op": "api.call", "config": "billing-api", "extract": { "userId": "data.id", "token": "data.session.token" } }

{ "op": "api.call", "config": "billing-api", "body": "{ \"plan\": \"{{plan}}\" }", "save": true, "extract": { "orderId": "id" } }

• config — the assigned api_call config name (constant, never interpolated).
• url / body — optional step overrides ({{var}}-interpolated).
• extract — { "<name>": "<dotted.json.path>" }; a missing path fails the step.
• save — when true, extracted values also flow into the run output.

The URL must be public (http/https). Private, loopback, link-local and cloud-metadata addresses are blocked; the request never goes out for a blocked URL. The request has a fixed 15-second budget and a response size cap.

api.expect

Asserts the status of the last api.call response. Use exactly one of status (exact match) or statusLessThan:

{ "op": "api.expect", "status": 200 }

{ "op": "api.expect", "statusLessThan": 400 }

api.dataAssert

Asserts that a value a preceding api.call extracted (named by apiName) matches what the page actually renders — the UI↔API data-parity check. Without attribute the element's visible text is compared; with it, the named attribute. Use exactly one of equals (strict) or contains (the UI value contains the API value). The comparison is case- and whitespace-sensitive (the assert.text convention).

{ "op": "api.dataAssert", "apiName": "userName", "selector": "[data-testid='profile-name']", "equals": "Jane Doe" }

{ "op": "api.dataAssert", "apiName": "userId", "selector": "[data-testid='profile-card']", "attribute": "data-user-id", "contains": "u-1" }

• apiName — the name an earlier api.call extracted (constant, never interpolated).
• selector — the UI element to read ({{var}}-interpolated).
• attribute — optional; read this attribute instead of the element text.
• equals / contains — exactly one; a mismatch fails the step.
• api.dataAssert steps are excluded from self-healing — a data mismatch is a real failure, not a transient selector glitch.

auth.basic

Answers an HTTP Basic Auth challenge — the native browser dialog a server opens with a 401 WWW-Authenticate: Basic response. That dialog is not a DOM element, so fill / click / act cannot drive it; instead the assigned basic_auth config's username / password are applied as the browser context's HTTP credentials. config names a basic_auth config assigned to the test case:

{ "op": "auth.basic", "config": "staging-basic-auth" }

A full flow behind a Basic-Auth wall:

{
  "name": "Protected staging area",
  "steps": [
    { "op": "auth.basic", "config": "staging-basic-auth" },
    { "op": "navigate", "url": "/protected" },
    { "op": "assert.status", "equals": 200 },
    { "op": "assert.text", "selector": "h1", "contains": "Protected" }
  ],
  "vars": [],
  "tags": ["auth"]
}

• config is a constant reference to the assigned basic_auth config's name; it is never {{var}}-interpolated (the totp / api.call convention).
• The credentials apply to the whole session (set at browser-context creation), independent of the step's position — put auth.basic first, before the first navigate, like a login preamble.
• The password is masked (***) in every artifact; both username and password are registered as run secrets.
• auth.basic steps are excluded from self-healing (no selector, no replay context — the credential application lives in the session init).

clipboard.copy

Copies an element's value to the system clipboard and stores it run-locally under name, so later steps can reuse it as {{clipboard.<name>}}. Without attribute the element's visible text is copied; with it, the named attribute:

{ "op": "clipboard.copy", "name": "ref", "selector": "[data-testid='order-ref']" }

{ "op": "clipboard.copy", "name": "href", "selector": "a.share", "attribute": "href" }

• name is the run-local key (constant, never {{var}}-interpolated — the output.save / totp convention); selector is {{var}}-interpolated.
• attribute is optional and constant; absent = the element's visible text.

clipboard.paste

Reads the system clipboard and stores it run-locally under name, making it available as {{clipboard.<name>}} for later steps:

{ "op": "clipboard.paste", "name": "fromClipboard" }

A copy-then-reuse flow within one test:

{
  "name": "Carry the order reference into search",
  "steps": [
    { "op": "navigate", "url": "/orders/latest" },
    { "op": "clipboard.copy", "name": "ref", "selector": "[data-testid='order-ref']" },
    { "op": "navigate", "url": "/search" },
    { "op": "fill", "selector": "#q", "value": "{{clipboard.ref}}" },
    { "op": "click", "selector": "button[type='submit']" }
  ],
  "vars": [],
  "tags": ["clipboard"]
}

• name is the run-local key (constant, never interpolated).
• A passed run's clipboard values are persisted; a dependency successor in the same plan run receives them as {{clipboard.<name>}} (the output.save cross-run-injection convention).
• clipboard.copy / clipboard.paste steps are excluded from self-healing (no selector to re-point on a clipboard failure; a replay would double-write/read the system clipboard).

assert.vision

Captures a screenshot of the current page and asks a vision LLM (routed through the platform's LLM router on the cheap/Haiku tier — never a raw provider call) to evaluate a natural-language visual check. The model returns a constrained verdict (pass plus a short list of observations); the step passes or fails on that verdict. Use it for visual conditions a selector cannot capture — broken CSS, a misrendered layout, an unexpected error banner, a button that looks disabled:

{ "op": "assert.vision", "instruction": "the page renders with no broken CSS and no error banners" }

• instruction — the visual check, a non-empty constant sentence describing how the page should look. It is a constant description, not {{var}}-substituted for a DOM value (author it as plain prose).
• failOn — the polarity, defaulting to "mismatch":
  • "mismatch" (default): the step fails when the model does not confirm the instruction — the natural "the page should look like …" reading.
  • "match": the step fails when the model does confirm it — use it to assert the absence of a visual condition (e.g. instruction "a red error banner is visible" + failOn: "match" fails the test when the banner is actually there).

{ "op": "assert.vision", "instruction": "a red error banner is visible", "failOn": "match" }

• The model's observations are included in the failure message, so a failing visual check is debuggable without re-opening the screenshot. The screenshot the assertion was based on is the step's normal post-step artifact.
• assert.vision steps are excluded from self-healing — a visual assertion failure is a real failure, not a transient selector glitch.

Variables (vars)

Each entry binds a name to a string value:

{ "name": "password", "value": "s3cr3t", "secret": true }

String step fields (url, selector, value, equals, contains) may reference a binding via the placeholder syntax {{name}} (inner whitespace is allowed, e.g. {{ name }}). Placeholders are resolved before the first step runs; referencing an unbound variable fails the run instead of substituting a wrong value.

With "secret": true the variable's value is masked as *** in every text artifact that leaves the runner — DOM dumps, error messages, and logs. The flag defaults to false, so existing DSL without it stays valid.

Self-healing

When a step fails because a selector or assertion drifted, the self-heal engine classifies the failure, proposes a fix (a new selector or an equivalent assertion), and scores its confidence. Above the per-class confidence threshold the fix is applied automatically and the test re-runs in the same browser session; below it, the proposal is surfaced for review and approval.