Research guide

Natural Language Test Automation: Write E2E Tests in Plain English

Most product teams already have test cases. They just do not call them test automation.

By CueTest QA Team · Published 2026-09-02

They exist in acceptance criteria, Linear or Jira tickets, release checklists, Slack messages, QA spreadsheets, product specifications, and the sentence a developer says before deploying:

Make sure a customer can still upgrade from Starter to Launch and see the correct invoice before paying.

Traditional E2E automation requires somebody to translate that sentence into a programming language, a browser framework, selectors, waits, fixtures, assertions, helper functions, and CI configuration.

Natural language test automation attempts to remove much of that translation layer.

Instead of writing:

await page.getByRole('link', { name: 'Billing' }).click();
await page.getByRole('button', { name: 'Launch' }).click();
await expect(page.getByTestId('invoice-preview')).toContainText('Launch');

you can write something closer to the original business requirement:

Open Billing.
Choose the Launch plan.
Verify that the invoice preview shows the Launch plan and the expected monthly amount before payment.

In 2026, this is no longer only a recorder with English-looking labels. Platforms such as testRigor, Momentic, BrowserStack's low-code tooling, mabl, and CueTest use AI or semantic interpretation to turn human-readable intent into executable browser behavior.

But natural language is not automatically good test design.

A vague sentence written in English can be less reliable than a precise test written in code. The real value comes when a platform lets teams express product intent at the right level of abstraction while still enforcing explicit, observable outcomes.

This guide explains how natural-language testing works, how to write scenarios that stay useful, and when plain English is genuinely better than coded automation.

What is natural language test automation?

Natural language test automation lets people define executable software tests using human-readable instructions rather than directly writing browser automation code.

The system interprets statements such as:

Sign in as the administrator.
Open Team members.
Invite qa-new@example.test as a Member.
Verify that the new account appears in the pending invitations list.

and maps them to actions and checks against a running application.

Depending on the platform, this may happen in different ways.

Approach 1: natural language generates code

AI reads the instruction and creates Playwright, Selenium, Cypress, or another test artifact.

The generated code is then executed deterministically.

This is useful because test creation becomes faster while the resulting automation remains conventional and inspectable.

The downside is that the generated code still requires maintenance when the application changes.

Approach 2: natural language is the test artifact

The saved test remains human-readable. A runtime engine resolves instructions to application elements and actions.

testRigor is a prominent example of this model. Its public examples show high-level commands being translated into more specific interactions while the authored test stays in plain English.

Approach 3: natural language drives an agent

The instruction is treated as a goal. An AI agent observes the live application, determines the next action, adapts to the resulting state, and continues until the goal is achieved or cannot be verified.

Momentic exposes an explicit AI action for goal-driven scenarios. CueTest uses natural-language steps with bounded AI execution and expected-result verification.

These are related approaches, but they are not identical.

Why plain-English tests are attractive

The obvious benefit is that more people can understand the suite.

But the deeper benefit is reducing coupling between the business requirement and the implementation detail.

Consider the requirement:

A signed-in customer can download the latest invoice.

A low-level scripted test might depend on:

• /settings/billing route structure;
• a button's CSS selector;
• exact DOM hierarchy;
• a download event implementation;
• a helper that knows where invoices appear;
• a wait tuned to the current page behavior.

The business requirement does not care about most of those details.

If natural-language automation can locate “Billing,” identify the latest invoice, trigger the download, and verify that a file was produced, the test can stay aligned with the requirement even when the page is reorganized.

That is the ideal.

Natural language is an abstraction layer, not magic

Every abstraction hides complexity.

React lets developers think in components instead of manually updating DOM nodes. SQL lets developers describe data operations instead of writing storage-engine instructions. Playwright locators let test authors target elements through roles and labels instead of DOM traversal.

Natural-language testing moves the abstraction one level higher.

The system must solve questions that code normally answers explicitly:

• Which “Continue” button did the author mean?
• Does “open billing” mean click a sidebar item or navigate directly?
• What counts as “checkout succeeded”?
• Should a cookie banner be dismissed?
• Is a renamed button an acceptable change or a regression?
• If two products have the same label, which one should be selected?
• Can the agent retry an action?
• Is an unexpected modal safe to close?

A good platform resolves these ambiguities with context, constraints, and verification. A bad one merely makes failures harder to understand.

The difference between readable tests and vague tests

Natural-language testing fails when teams mistake brevity for clarity.

Too vague

Test registration.

What user state? Which registration method? What data? What should prove success? Is email verification required? What happens if the email already exists?

Better

Start from the public sign-up page as a logged-out visitor.
Create a new account using @env:NEW_USER_EMAIL and @env:TEST_PASSWORD.
Accept the required terms.
Submit registration.
Verify that the app opens the onboarding welcome screen for the new account.

The second version is still natural language, but it defines:

• starting state;
• data source;
• required action;
• scope;
• expected outcome.

That is what makes it executable.

The four parts of a strong natural-language E2E test

1. Starting state

Tell the test what must already be true.

Examples:

Start logged out.

Use the administrator account with an existing workspace and no pending invitations.

Start as a Starter-plan customer with an empty cart.

State is critical because E2E tests can accidentally inherit authentication, cookies, backend fixtures, or records from previous runs.

A test that passes due to leftover state is worse than a test that fails.

2. Intentional actions

Describe behavior in user/product language.

Good:

Open Team settings and invite the new member.

Bad:

Click the third item in the left nav and then click the blue button at x=940.

The latter reintroduces implementation coupling that natural-language testing is meant to remove.

3. Test data

Use controlled values.

Good:

Sign in using @env:TEST_EMAIL and @env:TEST_PASSWORD.

Avoid:

Make up an email and password that works.

Agents should not invent credentials, payment data, production customer identities, or destructive inputs.

4. Observable postconditions

Every meaningful action should end in something the system can verify.

Weak:

Submit the form.

Strong:

Submit the form and verify that the new beneficiary appears in the beneficiary list with status Active.

Natural language improves authoring only if success remains concrete.

Why selectors are not the real enemy

It is fashionable to say natural-language testing eliminates selectors. That oversimplifies the problem.

Modern Playwright tests can already avoid many brittle selectors by using semantic locators such as roles, labels, text, and test IDs. Playwright explicitly recommends user-facing attributes and warns against long CSS/XPath chains tied to DOM structure.

The true maintenance burden comes from broader assumptions:

• the page is reached through a particular route;
• an action happens in a particular order;
• a modal always appears;
• one component contains the control;
• a user starts in a specific state;
• a product workflow has not changed.

Natural-language and agentic systems can adapt at this higher level.

So the advantage is not “English is better than CSS.” The advantage is the test can be coupled to product intent rather than implementation path.

How modern tools approach natural-language testing

testRigor

testRigor's central model is plain-English executable automation. Its public examples show high-level instructions such as purchasing a product translated into specific interactions. This is attractive for organizations that want manual testers and business stakeholders to contribute directly.

Momentic

Momentic lets teams write E2E tests in natural language and supports both explicit steps and agentic actions. Tests can live as YAML and run in local/CI workflows. The explicit split between deterministic and agentic steps helps teams decide how much autonomy a scenario needs.

mabl

mabl's agentic authoring can build complete browser tests from an intent, use reusable workspace flows, create variables, and ask for missing information. This is natural-language input used to construct managed tests inside a broader QA platform.

BrowserStack Low Code Authoring Agent

BrowserStack documents an authoring agent that reads human-readable test steps and autonomously constructs low-code automation. It combines natural-language processing with self-healing and execution on BrowserStack infrastructure.

CueTest

CueTest stores browser journeys as natural-language steps and expected outcomes. It supports project-level environment variables such as @env:TEST_EMAIL, shared setup/teardown hooks, run evidence, visual tests, schedules, and CI/API triggers.

CueTest's public model emphasizes reusing a verified path when possible and invoking AI-assisted resolution when that path no longer proves the expected result. This is intended to keep common execution more predictable while retaining adaptation when the application moves.

Example: converting a product requirement into a natural-language test

Suppose a product ticket says:

As an account owner, I can invite a team member. The invite should show as Pending until accepted. Duplicate invitations should not create another row.

Do not create one massive vague test.

Break it into explicit behaviors.

Test 1: owner can invite a member

Starting as the workspace owner with no invitation for @env:INVITEE_EMAIL:

1. Open Team settings.
2. Invite @env:INVITEE_EMAIL with the Member role.
3. Verify that the invitation appears exactly once in Pending invitations.
4. Verify that the role displayed for the invitation is Member.

Test 2: duplicate invitation is prevented

Starting as the workspace owner with an existing pending invitation for @env:INVITEE_EMAIL:

1. Attempt to invite @env:INVITEE_EMAIL again with the Member role.
2. Verify that the application shows a duplicate/pending-invitation message.
3. Verify that there is still exactly one pending invitation for @env:INVITEE_EMAIL.

Notice that natural language does not eliminate test decomposition, test data, or assertions. It simply makes them readable.

Example: natural language for a dynamic checkout

A checkout can change based on location, inventory, plan, tax, and stored payment state.

An overly rigid test might encode every intermediate control. A goal-oriented test can focus on the contract:

Start as @env:TEST_BUYER with an empty cart and no saved payment method.

Add the Standard subscription to the cart.
Proceed through checkout using the test billing address.
Before submitting payment, verify that the order summary contains:
- Standard subscription
- the expected monthly price
- the expected tax for the test address

Do not submit the final payment.

The instruction gives the automation flexibility around layout while keeping the financially important postconditions exact.

Natural-language tests in CI/CD

Readable tests are not useful if they only run manually from a dashboard.

For CI, the challenge is controlling scope and runtime.

A good strategy is to tag natural-language tests by purpose:

smoke
checkout
billing
permissions
release-critical
post-deploy
exploratory

Then use different suites at different stages.

Pull request

Run a small set of deterministic or tightly bounded smoke journeys affected by the change.

Pre-deployment

Run release-critical E2E journeys and visual checks.

Post-deployment

Run adaptive smoke tests against the deployed environment.

Scheduled

Run broader natural-language journeys periodically against staging or production-safe accounts.

The fact that a test is easy to write does not mean it should run on every commit.

The biggest advantage: expanding test ownership

Traditional browser automation tends to create an ownership bottleneck.

A product manager writes acceptance criteria. A QA engineer rewrites them as manual cases. An automation engineer rewrites them again as code. A developer later debugs the code when CI fails.

Every translation creates information loss.

Natural-language testing can collapse some of these layers.

A product requirement can become closer to the executable test itself. QA can refine it. Developers can read it. A founder can understand why a release is blocked without opening a framework repository.

That does not mean “anyone can now do QA.” Testing skill still matters enormously. It means the artifact is accessible to more participants.

The biggest risk: false confidence from readable nonsense

A test can look beautifully understandable and still verify almost nothing.

Consider:

Make sure the dashboard works correctly.

A human reader feels like they understand it. An AI can probably perform some actions. But there is no precise contract.

Compare:

Sign in as @env:ANALYST_USER.
Open the dashboard.
Verify that Revenue, Active Customers, and Conversion Rate cards are visible.
Change the date range to Last 30 days.
Verify that the selected range is displayed and each metric finishes loading without an error state.

Natural-language tests must be reviewed with the same rigor as code.

Ask:

• Can two reasonable humans interpret this differently?
• Is the starting state defined?
• Is success observable?
• Is the data controlled?
• Does the test accidentally allow a bug to be worked around?
• Is the scenario small enough to diagnose?

When plain English is better than code

Natural-language automation is particularly strong when:

• the test describes a user/business outcome;
• the UI changes frequently;
• non-developers need to review or contribute;
• selector maintenance is disproportionate;
• the path can vary while the outcome stays constant;
• the workflow is important but not worth a large coded fixture structure;
• the team needs to automate cases currently trapped in manual regression.

When code is better than plain English

Use coded tests when:

• exact implementation behavior matters;
• assertions are low-level or highly technical;
• you need custom fixtures and helpers;
• test execution must be extremely fast and cheap;
• a large existing framework already works well;
• the sequence must not adapt;
• reviewers need explicit control flow rather than interpreted intent.

A mature team should not be ideological about either format.

A migration strategy that does not create chaos

Do not translate 500 Playwright tests into English just because a new platform exists.

Step 1: identify the expensive tests

Look at git history and CI failures. Which E2E tests receive the most maintenance that is not caused by real requirement changes?

Step 2: identify manual release checks

These are often the highest-value candidates because you are creating new automation rather than replacing working automation.

Step 3: rewrite the requirement, not the script

Do not mechanically translate:

page.locator('#foo').click()

into:

Click #foo.

Go back to what the product behavior is supposed to prove.

Step 4: define evidence

Decide what a failed run must capture for a developer to act: screenshots, trace, logs, console errors, network information, visual diff, failed step.

Step 5: run in parallel with the old process

Compare real release signal for several weeks before deleting trusted coverage.

Writing natural-language tests with CueTest

CueTest's documentation recommends keeping each test focused and naming the target state and expected result. Projects can store reusable values as environment variables, so credentials and environment-specific data do not have to be written directly into prompts.

A practical CueTest scenario could be:

Test: User can upgrade from Starter to Launch

1. Sign in using @env:TEST_BUYER_EMAIL and @env:TEST_BUYER_PASSWORD.
   Expected: The authenticated dashboard is visible.

2. Open Billing settings.
   Expected: Current plan shows Starter.

3. Choose the Launch plan.
   Expected: An invoice preview is shown before payment.

4. Verify the invoice preview.
   Expected: The plan is Launch and the recurring amount matches the configured Launch price.

If multiple tests require the same login or reset behavior, project lifecycle hooks can centralize setup rather than repeating it everywhere.

When a run fails, inspect the evidence before changing the prompt. A failure can be a product bug, a test issue, setup problem, infrastructure failure, application change, or AI error. Natural language makes editing easy, but “rewrite the prompt until it turns green” is not testing.

Frequently asked questions

Is natural-language testing the same as no-code testing?

Not exactly. No-code usually describes an authoring interface such as a recorder or visual builder. Natural-language testing specifically uses human-readable language as the test specification or input. Some tools combine both.

Does natural-language test automation use AI?

Modern platforms commonly use AI for interpretation, planning, element resolution, generation, or recovery, but natural-language interfaces can also be implemented with deterministic command grammars.

Can natural-language tests replace Playwright?

They can replace some product-facing E2E tests, especially where selector maintenance is high. Playwright remains better for many deterministic, code-level, high-frequency regression scenarios. The two approaches can coexist.

How specific should a natural-language test be?

Specific about starting state, test data, business actions, and expected outcomes; flexible about incidental implementation details that are not part of the requirement.

Can product managers write automated tests this way?

They can contribute much more directly, but good test design still requires QA thinking. Product managers may define scenarios and expected behavior while QA/engineering reviews state isolation, edge cases, safety, and assertions.

How do I prevent an AI test from doing the wrong thing?

Use controlled environments and accounts, explicit goals, constrained scopes, safe test data, stopping rules, clear postconditions, and a platform that records the actions and evidence used to reach the verdict.

Final takeaway

Natural-language test automation is valuable because product teams already think in behaviors, not selectors.

The opportunity is to preserve that language all the way into executable testing:

requirement
→ executable intent
→ browser evidence
→ release decision

instead of repeatedly translating it:

requirement
→ QA case
→ automation ticket
→ framework code
→ selector maintenance
→ CI output
→ human interpretation

Plain English will not remove the need for careful test design. It can remove a surprising amount of mechanical translation and accidental maintenance.

If your team has an important browser flow that is still tested manually because nobody wants to maintain another scripted E2E journey, that is the right place to experiment.

Run a natural-language browser test with CueTest and judge the result by the evidence it produces, not by how impressive the prompt looks.

Sources and further reading

• testRigor — Generative AI-based Test Automation
• Momentic Documentation
• Momentic — Agentic Testing
• mabl — Agentic Test Authoring
• BrowserStack — Low Code Authoring Agent
• BrowserStack — AI in Test Automation
• Playwright — Locators
• CueTest Documentation