Experiment Design & Statistical Analysis for Research Engineers
Design sound experiments and analyze results with statistical rigor
What You Can Do
You can design statistically valid experiments with proper power analysis, choose the right statistical tests for your data type, analyze results while controlling for multiple comparisons, and generate publication-ready reports with accurate interpretation of findings. Claude helps you avoid common statistical pitfalls and ensures your experimental claims are well-supported by evidence.
Features
specify sample size, effect size, and significance level to get sample size recommendations
get recommendations for t-tests, ANOVA, chi-square, Mann-Whitney U, and others based on your data structure
compute Cohen's d, odds ratios, and other effect measures with plain-English explanation
apply Bonferroni, FDR, and other adjustments when testing multiple hypotheses
verify normality, homogeneity of variance, independence, and other test prerequisites
describe charts that best communicate findings (forest plots, violin plots, confidence intervals)
structure methods, results, and discussion sections with proper statistical reporting
Example Output
Example 1: Experiment Design
You ask: "Design an A/B test to detect a 10% improvement in click-through rate with 95% confidence and 80% power."
Claude returns:
- Required sample size per group: 1,847
- Total participants: 3,694
- Recommended test: Two-proportion z-test
- Effect size (Cohen's h): 0.21
- Study design considerations and critical assumptions
Example 2: Statistical Analysis
You provide: "Control group: mean = 42.3 (SD = 8.1, n=50), Treatment: mean = 46.8 (SD = 7.9, n=50)"
Claude analyzes:
- Test selection: Independent samples t-test
- Results: t(98) = 2.34, p = 0.021
- Effect size (Cohen's d): 0.66 (medium effect)
- 95% CI for difference: [0.73, 8.23]
- Interpretation: Statistically significant with meaningful practical improvement
Example 3: Report Generation
You request: "Generate methods and results sections for a two-arm experiment testing treatment vs. control."
Claude provides complete text including study design, randomization, statistical methods with justification, results with test statistics and p-values, and interpretation aligned with pre-specified hypotheses.
What's Included
- SKILL.md: Full skill with experiment design workflow, statistical test decision trees, and analysis templates
- Experiment design templates: Power analysis checklist, sample size guidance, study protocol template
- Statistical analysis workflows: Decision trees for test selection, assumption-checking procedures, interpretation guidelines
- Report templates: Methods section template, results reporting structure, discussion framework
- Statistical test reference: When to use t-tests, ANOVA, chi-square, Mann-Whitney U, Kruskal-Wallis, and effect size formulas
- Assumption checking guides: Procedures for verifying normality, homogeneity, independence, and sphericity
- Effect size interpretation: Cohen's conventions and practical significance thresholds across research domains
Who It's For
- Research engineers — Design experiments to validate algorithm improvements and system changes
- Data scientists — Run rigorous A/B tests and interpret results with statistical confidence
- Product managers — Conduct user experiments and make data-driven product decisions
- ML engineers — Validate model improvements against baselines with proper statistical testing
- Academic researchers — Design studies and report results following publication standards
Best For
- Designing A/B tests and experiments — Specify your hypothesis and constraints, get sample size and test recommendations
- Analyzing experimental results — Input summary statistics or raw data, get statistical test results and interpretation
- Choosing statistical tests — Describe your data type and research question, get recommendations with clear justification
- Multiple hypothesis testing — Learn when and how to apply Bonferroni, FDR, and other adjustments
- Writing statistical methodology — Generate publication-ready methods and results sections with proper reporting conventions
You might also like
This skill helps you systematically diagnose failures in distributed ML training and serving infrastructure. You provide system logs, metrics, and error traces, and Claude performs structured root-cause analysis to identify the underlying issue—whether it's resource exhaustion, distributed system deadlock, data pipeline corruption, or model serving misconfiguration. You get a detailed diagnosis with remediation steps ranked by likelihood and implementation effort.
by moonwell-fi
You can systematically diagnose why your RAG system isn't returning relevant results and identify specific misalignments between queries and context. This skill walks you through analyzing retrieval quality, scoring context relevance, and optimizing prompt-retrieval alignment to improve your RAG accuracy. You'll get concrete, actionable recommendations for fixing bottlenecks in your pipeline.
by moonwell-fi
You'll handle difficult guest conversations with structured communication templates that de-escalate tension while protecting your property. Generate professional damage assessments, complaint responses, and resolution protocols that document interactions clearly and mitigate liability. Create personalized communication strategies tailored to each guest situation, from minor maintenance requests to serious damage claims.
by moonwell-fi
Systematically evaluate your Claude prompts using quantitative scoring frameworks to identify weaknesses in clarity, task alignment, and output quality. You'll receive detailed optimization recommendations with A/B testing guidance to improve prompt effectiveness and consistency across different use cases and input variations.
by moonwell-fi
You can prepare your domain-specific datasets, train fine-tuned Claude models optimized for your use case, benchmark performance against base models, and deploy production-ready custom models. This skill automates the entire fine-tuning pipeline from raw data to live inference, complete with cost-benefit analysis and version control.
by moonwell-fi
You get structured guidance for designing controlled experiments, selecting appropriate metrics, and analyzing results with statistical rigor. Claude helps you identify confounding variables, calculate required sample sizes, interpret statistical significance, and document methodology for reproducible benchmarking that stands up to scrutiny.
by moonwell-fi
Design experiments with statistical rigor from the ground up. You'll generate power analyses, create randomization strategies, write preregistration documents, and get AI-assisted reviews of your experimental design to catch biases before data collection. Whether you're planning a clinical trial, A/B test, or academic study, you'll have a complete, defensible experimental protocol.
by moonwell-fi
This skill transforms high-level feature requests into actionable implementation plans by conducting deep requirement interviews, analyzing existing codebases (or planning greenfield projects), and producing comprehensive handoff documents. It distinguishes between brownfield development (with existing code to integrate) and greenfield projects (starting from scratch), ensuring your technical strategy accounts for your actual constraints and patterns.
by moonwell-fi