Experiments and how winners are (not) declared

Setting up an experiment

Experiments live under Experiments in each project. A test compares your current page, the control, with one or more variants on a primary metric chosen before it starts.

1. Write the hypothesis: what changes, for whom, and which metric should move. You can start from a CRO finding labeled Hypothesis.

2. Create the variant. The AI can draft it from a section you select, or you edit it by hand. Either way it is a patch you review and approve before it can run.

3. Choose the primary metric, such as qualified leads, confirmed bookings or paid orders. The result is judged on it.

4. Add guardrail metrics that must not get worse, such as qualified-lead rate, form errors or page speed.

5. Set the audience and eligibility, for example mobile visitors from Meta campaigns.

6. Set allocation weights, such as 50/50. Uneven splits need more traffic to reach a result.

7. Start the test yourself.

How many experiments can run at once depends on your plan. See pricing.

Experiment statuses

A test moves from draft to ready, running, paused if you pause it, and concluded, then ends as promoted or archived.

How visitors are assigned

• Assignment happens at a visitor’s first eligible session through a deterministic keyed hash, so the same visitor always gets the same variant.

• It is sticky across reloads and every step of a multi-step funnel, within what the visitor’s consent allows. Without a lasting identifier, it holds for the session.

• An exposure is recorded only when the variant actually rendered, not merely when the visitor was assigned.

• Your own QA previews are excluded, so checking a variant doesn’t affect results.

Health checks

Results only mean something when the setup is sound. Each test shows:

• Sample ratio mismatch check: if the observed split differs from your weights by more than chance explains, assignment or tracking is probably broken, and results shouldn’t be trusted until it is fixed.

• Sample size guidance: roughly how many visitors each variant needs to detect the smallest change you care about, and whether your traffic can get there.

• Affected population: exactly who is in the test, such as mobile visitors from Meta. The result applies to them only.

Bots, excluded internal traffic and QA previews are left out of the counts.

How results are analyzed

• No peeking. Checking daily and stopping when a test looks good produces false winners. Each test uses a fixed horizon, analyzed at the planned sample, or a sequential method declared before the start that stays valid when checked repeatedly.

• Ranges, not just a lift. Results show the estimated difference with a confidence interval.

• Multiple comparisons. With several variants or metrics, the analysis corrects for the extra chance of a fluke, or the test is marked exploratory.

• Maturity window. Qualified leads and paid orders arrive days after the visit, so the analysis waits rather than counting recent visitors as non-converters.

What doesn’t count as a win

• More raw form submissions while qualified leads fall. Raw submits are a diagnostic, not the goal.

• A test stopped early because it looked good.

• Too little data to tell the variants apart from chance.

• A gain on a metric that wasn’t the primary metric. It can inspire the next test, but it isn’t this test’s result.

• A primary-metric gain that came with a worse guardrail. Weigh the trade-off before you promote anything.

Result labels

Even a variant that is ahead is an estimate with a range, not a guarantee. Results can shift with seasonality, offers and the traffic your ads send.

Promote or roll back

You start, stop and promote tests yourself. The AI proposes experiments and prepares variants, but by default it never starts, stops or promotes one on its own. Promoting publishes the variant as a new page version, and the previous version is kept so you can roll back.

Automation is optional and stays off unless you opt in. Where it is available, it runs within limits you set, such as how many experiments may run at once, rolls back automatically if a guardrail regresses, can be switched off at any time and records every action in the audit log.

From CRO findings to tests

The CRO agent checks tracking and data quality first, then labels each finding by how much is known. Its numbers come from your analytics.

Only an experiment turns a hypothesis into a validated finding. You can also reject a finding and record why.