Experiment Design Assistant for Data Scientists
Design statistically rigorous experiments with preregistration and power analysis
What You Can Do
Design experiments with statistical rigor from the ground up. You'll generate power analyses, create randomization strategies, write preregistration documents, and get AI-assisted reviews of your experimental design to catch biases before data collection. Whether you're planning a clinical trial, A/B test, or academic study, you'll have a complete, defensible experimental protocol.
Features
Determine sample size needed for 80% statistical power with customizable effect sizes and significance levels
Create stratified, blocked, or clustered randomization schemes tailored to your study design
Auto-generate Open Science Framework (OSF)-compatible preregistration documents with all required sections
Get quick reference tables for common experimental designs (t-tests, ANOVA, correlation)
Get recommendations on which statistical tests match your hypotheses and experimental structure
Identify selection bias, confounding, measurement bias, and other threats to validity
Structure your research questions using the SMART hypothesis framework
Get detailed feedback on your study design with specific, actionable improvements
Example Output
Power Analysis Output
Study: A/B Test on Email Subject Lines
Effect Size (Cohen's d): 0.3 (small-to-medium)
Target Power: 80%
Significance Level (α): 0.05
Sample Size Per Arm: 176
Total Sample Size: 352
Duration: ~4 weeks (assuming 45 conversions/day)
Randomization Schema
Method: Stratified Random Assignment
Strata: By user region (US, EU, APAC)
Block Size: 4 (to maintain balance within regions)
Software: Use Python's random.seed(42) or R's blockrand
Assignment Table:
Region | Control | Treatment | Total
US | 88 | 88 | 176
EU | 58 | 58 | 116
APAC | 30 | 30 | 60
TOTAL | 176 | 176 | 352
Preregistration Snippet (OSF Format)
Study Title: Impact of Subject Line Personalization on Email Open Rates
Primary Hypothesis: Personalized subject lines increase open rates by ≥10% vs. generic lines.
Statistical Test: Two-sample proportion test (χ²)
Analysis Plan: Intent-to-treat analysis with logistic regression controlling for user region and account age.
What's Included
- SKILL.md: Core experiment design system prompt and workflows
- Power Analysis Calculator: Template with effect size interpretations and sample size tables
- Preregistration Template: OSF-ready document covering all 22 AsPredicted.org fields
- Randomization Strategy Guide: Stratified, blocked, and clustered assignment methods with code examples
- Bias Detection Checklist: 15-point validity review covering internal, external, construct, and statistical conclusion validity
- Statistical Test Reference: Quick matrix of tests by hypothesis type (correlation, comparison, regression)
- Hypothesis Framework Worksheet: SMART hypothesis structure guide
Who It's For
- Academic researchers designing dissertation studies or journal submissions
- Data scientists in healthcare and biotech planning clinical or observational studies
- Product managers and UX researchers designing A/B tests and user experiments
- Clinical trial coordinators creating protocols that meet regulatory standards
- Social scientists conducting behavioral experiments or field studies
Best For
- Planning experiments before data collection — Set up your study design and get feedback before you invest time and resources
- Calculating sample sizes — Ensure you have enough statistical power to detect your effect of interest
- Creating randomization procedures — Eliminate selection bias with defensible assignment methods
- Writing preregistration documents — Meet journal requirements and boost credibility with Open Science practices
- Reviewing designs for validity threats — Catch confounds, measurement bias, and generalizability issues early
You might also like
This skill guides you through rigorous experiment design from hypothesis to analysis plan. You'll compute the statistical power needed for your tests, identify confounding variables to control for, design optimal randomization strategies, and generate pre-registered analysis plans that prevent p-hacking and publication bias. Whether you're running a product A/B test or investigating causal effects in observational data, you get structured, validated designs that account for real-world constraints.
by moonwell-fi
You can build production-grade vector search systems that retrieve semantically similar documents in sub-millisecond time using AgentDB's optimized HNSW indexing. The skill enables you to implement RAG pipelines, semantic search engines, and intelligent knowledge bases with configurable embedding dimensions, distance metrics (cosine, Euclidean, dot product), and similarity thresholds—all with built-in quantization and caching for massive performance gains.
by moonwell-fi
This skill transforms high-level feature requests into actionable implementation plans by conducting deep requirement interviews, analyzing existing codebases (or planning greenfield projects), and producing comprehensive handoff documents. It distinguishes between brownfield development (with existing code to integrate) and greenfield projects (starting from scratch), ensuring your technical strategy accounts for your actual constraints and patterns.
by moonwell-fi
Diagnose root causes of model performance failures, select optimal evaluation metrics based on your use case and data characteristics, and design comprehensive evaluation strategies. You'll get systematic workflows for analyzing failure modes, statistical significance testing, threshold optimization, and cross-validation planning—turning vague model problems into actionable fixes.
by moonwell-fi
This skill helps you systematically diagnose failures in distributed ML training and serving infrastructure. You provide system logs, metrics, and error traces, and Claude performs structured root-cause analysis to identify the underlying issue—whether it's resource exhaustion, distributed system deadlock, data pipeline corruption, or model serving misconfiguration. You get a detailed diagnosis with remediation steps ranked by likelihood and implementation effort.
by moonwell-fi
Systematically evaluate your Claude prompts using quantitative scoring frameworks to identify weaknesses in clarity, task alignment, and output quality. You'll receive detailed optimization recommendations with A/B testing guidance to improve prompt effectiveness and consistency across different use cases and input variations.
by moonwell-fi
You can rapidly compile, organize, and format disparate health economic evidence—from clinical trials to cost-effectiveness analyses—into structured, regulatory-compliant dossiers. This skill maps your evidence to specific payer and HTA requirements, automatically generates evidence hierarchies, and produces submission-ready dossier outlines with formatting that meets regulatory standards for NICE, EUnetHTA, and other major bodies.
by moonwell-fi
You can confidently navigate complex statistical analyses by working through structured workflows that match your research question and data type to the right test. The skill guides you through assumption validation, helps you interpret results accurately without common pitfalls, and walks you through regression diagnostics with full transparency about what your findings actually mean.
by moonwell-fi