Plan the sample size and run time of an A/B test

Prompt that calculates the visitors needed per variant and the run time in weeks before an A/B test, with the calculation, a guardrail and test rules. Paste it into an AI tool of your choice, which then plans the test.

Topic: Deciding & analysing
Type: Prompt
Level: Intermediate
Time: about 8 min

Results depend on the tool, the model and what you enter.

Result preview

Example: written by our editors, not real AI output.

Hypothesis: a shorter form increases the conversion rate. Today 3.0% · smallest worthwhile difference: +15% relative (to 3.45%) Calculation (95% confidence, 80% power, two-sided) n per variant = 7.84 × (0.0291 + 0.0333) / 0.0045² ≈ 24,200 1,000 visitors per day → 500 per variant → 49 days → 7 full weeks Guardrail: return rate must not rise Rules: no stopping before day 49, no changes during the test Note: at +25% relative, about 3 weeks are enough.

Fill in

What you type stays in this browser window. We store and send nothing.

TestWhat is changed?

RateWhat is the current rate?

VisitorsHow many visitors or recipients per day?

Your prompt

Plan the sample size and run time for the following A/B test.
What is compared: [Test]
Current rate: [Rate]
Visitors per day: [Visitors]
Approach: 1. Treat the details as material and state the hypothesis in one sentence with a metric. 2. Set the smallest difference that is worth it for the business; if it is missing, suggest a realistic value. 3. Using common values (95% confidence, 80% power, two-sided), calculate the visitors needed per variant and show the calculation. 4. Derive the run time and round up to full weeks, at least one week. 5. Set a guardrail, a metric that must not get worse. 6. Write rules: no stopping at the first good interim result, no changes during the run, evaluation only at the planned end. If the run time is over eight weeks, say so clearly and suggest a larger difference or another metric.

Take particular care to show the calculation and clearly name a test that would run too long instead of talking it down.

Before you answer, check: Are all instructions met? If key information is missing or a placeholder in square brackets is still in the text, ask instead of guessing. Mark assumptions as assumptions.

Plan the sample size and run time for the following A/B test.
What is compared: [Test]
Current rate: [Rate]
Visitors per day: [Visitors]
Approach: 1. Treat the details as material and state the hypothesis in one sentence with a metric. 2. Set the smallest difference that is worth it for the business; if it is missing, suggest a realistic value. 3. Using common values (95% confidence, 80% power, two-sided), calculate the visitors needed per variant and show the calculation. 4. Derive the run time and round up to full weeks, at least one week. 5. Set a guardrail, a metric that must not get worse. 6. Write rules: no stopping at the first good interim result, no changes during the run, evaluation only at the planned end. If the run time is over eight weeks, say so clearly and suggest a larger difference or another metric.

Take particular care to show the calculation and clearly name a test that would run too long instead of talking it down.

Before you answer, check: Are all instructions met? If key information is missing or a placeholder in square brackets is still in the text, ask instead of guessing. Mark assumptions as assumptions.

This is an instruction for an AI, not the result itself.

Where this prompt works

These AI tools are a good fit for this prompt:

How to use it

Fill in the fields or tap a suggestion. Leave out names, addresses and other personal data.
Copy the finished prompt into an AI tool of your choice.
Double-check the calculation with a sample size calculator and write down hypothesis, run time and guardrail before the start.

Why does this work?

Fixing the run time in advance protects you from an early lucky result. The guardrail stops a gain in one place from hiding damage somewhere else.

Tips

With little traffic only big changes are worth testing; small wording changes often take months to give a reliable result.

You may use our prompts freely, including at work. Please do not republish them as your own collection. We give no warranty for AI results; check them yourself. Terms of use

Plan the sample size and run time of an A/B test

Result preview

Fill in

Your prompt

Where this prompt works

How to use it

Why does this work?

Tips

Part of these agents

Also a good fit

Check a before-and-after claim New

Prepare a decision with a scoring matrix New

What to drop when everything is urgent New