PO Toolkit · Tool #11 of 12
Know if your test
actually worked
You ran a checkout flow test. Variant got +8.3% conversion on 2,400 users. Was it real? Enter your results and get a statistically rigorous ship / wait / discard verdict — in seconds.
No login · All calculations in your browser · Free forever
How it works
Statistical significance without the stats degree
Define your hypothesis
Write your hypothesis using the "We believe that X will cause Y because Z" format. Add the primary metric you're measuring.
Enter your results
Input visitors and conversions for both control and variant. Results can come from any A/B testing tool.
Get your verdict
See statistical significance, uplift, confidence, and a plain-language recommendation — calculated instantly in the browser.
Roadmap
What's coming next to this tool
Experiment management
Save experiments
PlannedSave multiple experiments locally and track them over time without a backend.
Experiment log
PlannedKeep a log of all past experiments with their results, decisions, and learnings.
Export to PDF
ConsideringGenerate a formatted experiment report PDF to share with stakeholders.
Statistics
Sample size calculator
PlannedCalculate the minimum sample size needed to detect a given uplift at 95% confidence before starting.
Sequential testing
ConsideringAdd support for sequential testing (SPRT) to allow early stopping without inflating type I error.
Multi-variant support
PlannedRun tests with more than 2 variants using chi-square with multiple degrees of freedom.
Integrations
PostHog import
ConsideringImport experiment results directly from PostHog feature flags and experiments.
Optimizely import
ConsideringPull results from Optimizely experiments via their API.
Under the hood
No black box — here's exactly what's being calculated
Chi-square test
Compares observed vs expected conversion counts between control and variant. If the difference is too large to be explained by chance, it's significant.
p-value < 0.05
The probability that the observed difference happened by random chance. Below 5% = the result is real with 95% confidence. Above 5% = keep collecting data.
Relative uplift
How much better (or worse) the variant performed vs control. A 3% absolute improvement on a 15% baseline = +20% relative uplift — much more informative than raw numbers.
Why I built this
Problem
Most POs don't validate A/B tests statistically. They call winners based on conversion rate alone, ignoring sample size and significance.
Solution
A chi-square calculator with plain-language verdicts — ship / wait / discard — that any PO can use without a stats background.
Impact
Fewer false-positive launches, more confident product decisions, and a team that builds the habit of data-driven validation.