Workflow example
A flaky Playwright test hid a real onboarding bug.
The team kept rerunning a failing signup check because it looked like another timing issue. CueTest reproduced the flow in a live browser and separated the product bug from the brittle script.
The failure had become background noise
The onboarding test failed two or three times a week. Sometimes it passed on rerun. Sometimes it timed out waiting for the workspace creation step. The team had learned to distrust it.
That distrust is expensive. Once a test gets labeled flaky, every real failure inside that area has to fight through the team’s existing skepticism.
This is why “flaky test” is not just an engineering hygiene problem. It changes release behavior. People start shipping around failures because the test has trained them to assume the tool is wrong.
What changed in the product
A new onboarding survey introduced a conditional required field. For some users, the field appeared after the workspace name step. For others, it did not. The Playwright test waited for the old success screen and failed with a timeout.
Because the error looked like familiar timing flake, the release team nearly ignored it.
What CueTest showed
CueTest ran the onboarding intent in a clean browser: create an account, name the workspace, answer required setup questions, and verify the dashboard opens. The agent reached the survey step and saw the required field blocking progress.
The run failed with the visible page state, the missing field context, and the exact step where onboarding stopped. That made it clear the product was blocking new users rather than the test runner being slow.
The result changed the conversation from “rerun the job” to “fix the onboarding condition.” That is the outcome a good QA tool should create: less debate about the failure and more attention on the user impact.
The triage workflow
The team compared three signals: the Playwright failure, the CueTest browser run, and the visible page state. The coded test had timed out waiting for the old success condition. CueTest showed the blocker that appeared before the success condition could ever happen.
That comparison helps readers searching for “flaky Playwright test” when the deeper issue is unclear. A workflow example can meet that need with a practical triage method instead of a generic list of retry settings.
The recommended workflow is simple: rerun the coded test once, run the user journey in CueTest, compare the failure step, and only then decide whether to fix the test, fix the app, or improve the setup data.
What the prompt should include
A strong onboarding prompt names the account state, the setup goal, and the final proof: “Create a new trial account, complete the required workspace setup questions, and verify that the user lands on the dashboard with the workspace ready.”
If the product has environment-specific test data, the prompt should reference env vars rather than hard-coded credentials. That keeps the case repeatable and avoids publishing sensitive details in docs, reports, or screenshots.
CueTest can turn this into a product-led tutorial later, but this workflow example should stay focused on the release lesson: a flaky label can hide a real onboarding bug.
Why this matters for early teams
Founders and small engineering teams often do not have a dedicated QA function. Flaky automation becomes something they work around instead of something they improve.
A browser-agent run can help by giving teams a second signal. If the coded test fails but CueTest completes the journey, the script may be brittle. If CueTest fails at the same user-facing step, the product likely needs attention.
Quarantine only after you classify the failure
Quarantine the coded test only when the current onboarding journey succeeds and the failure evidence points to the test harness. Keep the product issue open when both runs stop at the same user-visible state.
Record the owner, failure signature, and removal condition for every quarantined test. A quarantine without an exit condition turns missing coverage into permanent background noise.
Key takeaways
- Flaky tests can hide real product failures.
- A second browser-based signal helps separate brittle automation from broken UX.
- Onboarding deserves intent-based release smoke coverage because it is the first conversion moment.