THE SCIENCE OF TRUSTWORTHY TESTS

THE
RELIABILITY
LAB.

Green does not mean correct. Red does not always mean broken. You are the test engineer: manipulate timing, isolation, retries and assertions, then run an entire test suite. Learn to tell an actual product defect from a test that lies.

01 / EXPERIMENT SETUP

Build your hypothesis.

Choose what is wrong with this imaginary checkout feature. Then design how the test will observe it.

FAST / 1sPATIENT / 10s
02 / THE SIGNAL BOARD

One hundred runs. What do you know?

—CLEAN PASS
—PASS ON RETRY
—FINAL FAIL
Each square represents one simulated test. Tap a square to inspect its evidence. Green = first-attempt pass; amber = pass on retry; coral = final fail.

No evidence yet

Configure your test on the left and run the experiment to collect observations.

FAILURE TIME DISTRIBUTION / SIMULATION
The bars reveal why a timeout threshold may behave differently from a fixed sleep. Heights represent simulated response-time frequencies.
03 / THE DEEPER MODEL

The four ways tests deceive you.

FALSE GREEN

It passes. The bug remains.

A test checks that a click occurred, but never verifies that an order was saved. It reports success while the user journey is broken.

await page.getByRole('button', { name: 'Pay' }).click(); // No outcome assertion. What was proven?
FALSE RED

The product works. The test fails.

A correct asynchronous result arrives after a brittle fixed delay or an immediate check. Waiting for the actual outcome would avoid the race.

await page.waitForTimeout(1000); // brittle timing assumption await expect(page.getByRole('status')) .toHaveText('Paid');
FLAKY GREEN

The retry looks comforting.

A retry can recover from a transient problem. Playwright classifies tests that fail initially and pass on retry as flaky—not simply passed.

retries: 1, use: { trace: 'on-first-retry' } // Investigate the original failed run too.
CONTAMINATED SIGNAL

Tests share invisible state.

Two tests mutate the same account or inventory record. Parallel runs become order-dependent, even when every individual test is well written.

test('checkout', async ({ page }) => { // Create isolated test data. // Avoid shared mutable accounts. });
04 / CASE FILE

Make the call, SDET.

A customer pays successfully in production. In CI, the test sometimes times out waiting for a confirmation message. Enabling one retry makes the pipeline green. What is the best next step?