Experiment Designer
Design statistically valid experiments with hypothesis testing and sample size calculations
What You Can Do
You can design experiments that meet statistical validity standards by defining clear null and alternative hypotheses, calculating required sample sizes based on effect sizes and power, identifying confounding variables to control, and pre-specifying analysis plans that prevent p-hacking and bias. This skill produces complete experiment designs combining scientific rigor with practical feasibility for product testing, marketing validation, and research studies.
Features
creates null and alternative hypotheses with clear statistical framing
determines required participants using effect size, power, and significance levels
maps variables that could bias results and specifies control methods
designs assignment schemes that ensure unbiased group allocation
structures primary and secondary outcomes before data collection to prevent selective reporting
justifies sample sizes and provides confidence in effect detection
estimates execution timeline and required resources for feasibility assessment
Example Output
Example 1: Product A/B Test
- Hypothesis: Adding social proof to checkout increases conversion by 8% (effect size)
- Sample size: 2,400 users per variant (n=4,800 total) to achieve 80% power at 5% significance
- Confounders controlled: device type, traffic source, time of day
- Primary metric: Checkout completion rate; Secondary: revenue per user
- Timeline: 2-week test window
Example 2: Marketing Campaign Study
- Null hypothesis: Email subject line A and B have equal open rates
- Calculation: 6,500 recipients per variant to detect 2% lift with 90% power
- Randomization: Random split at subscriber level, stratified by engagement tier
- Analysis plan: Chi-square test on primary outcome; subgroup analysis by user tenure as pre-specified secondary outcome
What's Included
- SKILL.md instruction file with full experiment design methodology:
- Hypothesis statement template: structured format for null/alternative hypotheses with effect size specifications
- Sample size calculation worksheet: interactive framework with power, significance, and effect size inputs
- Confounder control checklist: systematic list of variables to identify and control strategies by experiment type
- Pre-analysis plan template: captures hypotheses, metrics, subgroups, and analysis procedures before data collection
- Power analysis documentation guide: explains trade-offs between sample size, power, and effect magnitude
Who It's For
- Product managers — designing A/B tests for feature launches and user experience optimization
- Research scientists — planning rigorous studies with statistical validity and reproducibility
- Data scientists — structuring experiments that prevent bias and support causal inference
- Marketing analysts — validating campaign strategies with appropriate sample sizes and controls
- UX researchers — designing behavioral experiments with confounding variable management
Best For
- A/B tests for product features, checkout flows, and user interface changes
- Marketing campaign validation and channel comparison studies
- Randomized controlled trials and quasi-experimental designs
- Sample size justification for funding applications and research proposals
- Pre-analysis plan documentation for open science and reproducibility standards
You might also like
You provide production data and process details, and this skill systematically analyzes the information to identify root causes of quality issues, yield losses, or equipment failures. It develops prioritized corrective actions with implementation timelines, compliance documentation, and performance metrics to track improvement effectiveness.
by kylezantos
You can build interactive UI components with purposeful motion—transitions, hover states, micro-interactions, and enter/exit animations—or audit existing motion design and receive detailed feedback. The skill operates in two focused modes: Create mode helps you implement motion that serves a clear purpose, while Audit mode evaluates your animations against industry-standard principles and reports specific findings with improvement recommendations.
by kylezantos
This skill guides you through systematic root cause analysis for manufacturing defects, helping you move beyond surface symptoms to uncover underlying failure mechanisms. You'll build investigation workflows, interpret production data, develop corrective actions backed by evidence-based reasoning, and prevent recurrence through structured follow-up.
by kylezantos
You can systematically analyze process upsets, equipment failures, and quality issues to pinpoint root causes and recommend corrective actions. This skill guides you through structured troubleshooting methodologies, evaluates process parameters against specifications, and identifies safety risks associated with failures. You'll generate documentation suitable for incident reports and continuous improvement initiatives.
by kylezantos
This skill parses experiment metadata and outputs to extract critical metrics from completed machine learning experiments. It reads experiment_summary.yaml to identify runs, pulls final training loss from SLURM logs, extracts accuracy metrics from inspect-ai evaluation files, and generates a clean summary.md report—all while logging the extraction process for reproducibility.
by kylezantos
You can structure and run Kaizen events from start to finish, guiding your team through waste analysis, root-cause identification, and rapid improvement design. The skill generates detailed event agendas, countermeasure documentation, and performance metrics—enabling you to close improvement opportunities within days rather than weeks.
by kylezantos
This skill structures and accelerates improvement initiatives using the PDCA (Plan-Do-Check-Act) framework combined with 5 Why analysis and standardized work templates. You'll build data-driven problem statements, identify root causes systematically, and create actionable improvement plans with built-in controls. It enables teams to move from reactive firefighting to proactive, continuous optimization grounded in evidence.
by kylezantos
You can diagnose root causes of yield loss, identify process drift in real time, and optimize critical parameters using structured troubleshooting frameworks. The skill helps you prioritize corrective actions based on statistical analysis, accelerate problem resolution, and build repeatable optimization workflows that your team can execute consistently across production runs.
by kylezantos