A/B Test Significance Calculator
Enter visitors + conversions for control and variant. Get both frequentist (classical p-value) and Bayesian (probability of being better) results, plus a plain-English "call it or keep running" verdict.
Experimentation Intelligence
Most A/B test calls are wrong: not because of the math, but because of when they're made.
A p-value of 0.049 at day 3 of a test is noise. Statistical significance doesn't mean the test is complete. This tool shows significance + power + sample adequacy and gives you one of three verdicts: Call it, Keep running, or Stop: no effect. Based on real decision rules used at mature experimentation programs.
Test data
Control (A)
–
Variant (B)
–
Verdict
Pending
Enter data above
Statistical results
Relative Lift
–
B vs A
p-value
–
Frequentist
P(B>A)
–
Bayesian probability
Statistical Power
–
Should be >80%
Confidence Interval
–
For the lift
Reading the result
Frequentist (classical)
What it answers: "If there were no real difference, how often would we see data this extreme by chance?"
Call it: p-value is below α AND power is above 80%.
Keep running: p-value is close to significance OR power is low.
Stop: no effect: power is high (>80%) but p-value stays above α.
Bayesian
What it answers: "Given the data, what's the probability that B is better than A?"
Call it: P(B>A) > 95% AND enough samples to be stable.
Keep running: P(B>A) between 70% and 95%.
Stop: no effect: P(B>A) stuck near 50%.
Next steps
Before launching your next test
Use the Sample Size calculator to find out how many visitors you need to detect your target lift, then start the test knowing exactly when it ends.
Frequently asked questions
What is statistical significance?
The probability that an observed difference between A and B isn't due to chance. 95% confidence (p < 0.05) is the common threshold.
What is the difference between frequentist and Bayesian?
Frequentist asks "what's the chance of this result if there's really no effect?" Bayesian asks "what's the probability B is better than A?" Bayesian is easier to communicate to non-statisticians.
Why 95% confidence specifically?
Convention from Ronald Fisher in the 1920s. Nothing magic about it. For high-stakes product changes some teams use 99%. For low-risk changes 90% is defensible.
What is a type I error?
A false positive. Calling a winner when there isn't one. 95% confidence = 5% type I error rate.
What is statistical power?
The probability of detecting a true effect of a given size. 80% power is standard. The Sample Size tool uses this.
When can I stop a test early?
Only if you planned sequential testing in advance. Peeking at results and stopping when the p-value hits 0.05 inflates false positives massively.
Should I split traffic 50/50 or 90/10?
50/50 is most efficient statistically. 90/10 is for risky changes where you want limited exposure to the variant.
How long should a test run?
At minimum, one full business cycle (usually a week) to capture weekly seasonality. For conversions over 7 days consider two cycles.
What is a minimum detectable effect (MDE)?
The smallest lift the test can detect reliably at your chosen power + significance. A test with 10% MDE won't spot a 3% real improvement.
Why do my test results flip over the course of the test?
Early results are noisy. Early winner / loser flipping is normal until you reach the planned sample size.
Does the calculator handle revenue tests?
For conversion rate yes. Revenue (continuous) needs a slightly different test (Welch's t-test). Use the sample size tool for that.
Free · no account
SEORCE for Chrome, Edge and Firefox
Audits the page you are on: headings, meta tags, schema, indexability, and whether AI crawlers can read it. Export the result as a PDF. Free, with no account.
It audits one page at a time and does not track AI answers. About the extension
Traffic from search and AI
See which visits came from search and AI.
SEORCE Web Analytics runs from one script tag, with Google Search Console connected. It shows visits from every AI source, from search and from campaigns, and it is free on every plan.
3,000+ users · Free forever plan, no card needed