A/B Testing: Statistical Rigor for CRO Specialists
Design statistically sound A/B tests with proper sample sizing and rigorous analysis
What You Can Do
You'll learn to design A/B tests that reliably detect real changes while minimizing false positives. This skill helps you calculate proper sample sizes, determine statistical power requirements, and interpret results using rigorous frequentist methods — ensuring every test delivers actionable insights and saves your team from chasing false signals.
Features
Compute optimal sample sizes based on baseline conversion rate, minimum detectable effect, significance level, and statistical power.
Determine the probability your test will detect a real effect given your sample size, baseline rate, and effect threshold.
Correctly interpret confidence intervals and understand what they reveal about the true effect size in your population.
Identify and adjust for inflated false positive rates when running multiple simultaneous tests or sequential peeks.
Translate business goals into statistical effect sizes (Cohen's h, lift percentage) to inform your test design.
Learn what p-values actually measure, common misinterpretations to avoid, and how to contextualize them with effect sizes.
Calculate how long your test should run to reach statistical power targets and detect the minimum effect size you care about.
Make confident launch decisions by separating statistical significance from practical significance and business impact.
Example Output
Example 1: Sample Size Calculation
You ask: "We have a 3% baseline conversion rate. We want to detect a 0.5% absolute lift (to 3.5%). What sample size do we need with 80% power and 5% significance?"
Claude returns:
- Minimum sample size per variant: 9,223 users
- Total test duration: ~5 days (at 2,000 users/day)
- Power curve visualization: Shows how power increases with sample size
- Trade-offs: "If you reduce power to 70%, you need only 6,800 users per variant (3-4 days)."
Example 2: Interpreting Ambiguous Results
You submit test data:
- Control: 100/3,500 conversions (2.86%)
- Variant: 120/4,100 conversions (2.93%)
- P-value: 0.18
Claude diagnoses:
- ✓ Not statistically significant (p > 0.05)
- ✓ Confidence interval for lift: [-1.2%, +4.8%]
- ✓ Test was underpowered; need 12,000 users per variant to detect 0.5% lift reliably
- ✓ Recommendation: Continue test or redesign with larger effect threshold
Example 3: Multiple Comparison Correction
You say: "I'm running 5 A/B tests simultaneously on different pages."
Claude advises:
- ✓ Uncorrected false positive rate: 22.6%
- ✓ Bonferroni-corrected threshold: p < 0.01 (not 0.05)
- ✓ Alternative: FDR control for less conservative approach
- ✓ Best practice: Prioritize 1-2 tests and run others sequentially
What's Included
- Sample Size Calculator Template: Pre-built formulas and lookup tables for conversion rate tests, continuous metrics, and count data scenarios.
- Statistical Power Curves: Visual relationships between sample size, effect size, power, and significance level to guide experiment design.
- Multiple Comparison Checklist: Step-by-step guidance on identifying when correction is needed and how to apply Bonferroni, Holm, or FDR methods.
- Confidence Interval Interpretation Guide: Templates for correctly stating what confidence intervals mean and translating them into business decisions.
- P-Value and Significance Reference: Common misconceptions about p-values, correct interpretations, and how to avoid misuse in reports.
- Effect Size Threshold Lookup: Industry benchmarks and guidance on what constitutes a practically meaningful effect in your domain.
Who It's For
- CRO Specialists
- Product Managers
- Growth Engineers
- Marketing Analysts
- UX Researchers
Best For
- Designing experiments before launch
- Calculating required sample sizes and test duration
- Interpreting test results and detecting false positives
- Determining practical vs statistical significance
- Troubleshooting inconclusive test results
You might also like
Upload your Google Ads account data and receive a comprehensive strategic audit that identifies performance gaps, competitive positioning issues, and high-impact optimization opportunities. Claude analyzes keyword quality, bidding strategies, budget allocation, and landing page performance to generate a prioritized 90-day roadmap with detailed ROI projections for each recommended change.
Analyze historical customer journeys across multiple touchpoints to determine each channel's true ROI contribution. You can model different attribution approaches, identify the most effective conversion paths, and generate data-driven recommendations for reallocating programmatic media budgets to maximize return on ad spend.
You can quickly structure competitor research, map competitive positioning across markets, and identify white space opportunities. The skill synthesizes company data, product analysis, and market trends into actionable strategic recommendations that inform your go-to-market strategy and product roadmap.
You create structured guerrilla marketing campaigns that deliver outsized impact without big budgets. The skill designs tactics from concept through execution plan, identifies and mitigates legal and reputational risks, and stress-tests ideas for feasibility before you invest time or money. You get actionable playbooks with timelines, resource estimates, contingency plans, and success metrics.
Automate commission validation to catch calculation errors, tier mismatches, and overpayments before payment. Reconcile payer-affiliate disputes by comparing transaction records side-by-side and recommending resolution amounts with full justification. Generate compliance-ready payment reports with timestamped audit trails that satisfy finance teams and regulatory requirements.
You can architect sophisticated audience segments within your DMP, optimize them for campaign performance, and map audiences across marketing channels. This skill helps you classify customer data into actionable segments, define activation rules, analyze segment quality metrics, and recommend optimization strategies based on your business objectives and data constraints.
Build structured audience segments from raw data, validate taxonomies for consistency and platform compatibility, and optimize segment hierarchies for programmatic advertising campaigns. This skill helps you create campaign-ready audience definitions, identify and eliminate overlap, and generate comprehensive documentation.
This skill helps you design, document, and troubleshoot integrations between your marketing platforms. You get API mapping strategies, workflow automation guidance, and ready-to-use integration templates for popular tools like Salesforce, HubSpot, Marketo, and Zapier. Whether you're building native connections or orchestrating multi-step workflows, this skill guides you through requirements gathering, authentication setup, and data validation.