ML Model Evaluation & Debugging
Debug ML performance issues with structured evaluation workflows
What You Can Do
Diagnose root causes of model performance failures, select optimal evaluation metrics based on your use case and data characteristics, and design comprehensive evaluation strategies. You'll get systematic workflows for analyzing failure modes, statistical significance testing, threshold optimization, and cross-validation planning—turning vague model problems into actionable fixes.
Features
Systematically identify root causes like data drift, class imbalance, or feature leakage
Choose evaluation metrics aligned with your business goals and data characteristics
Create comprehensive evaluation plans with cross-validation and statistical tests
Diagnose and recommend solutions for imbalanced classification problems
Interpret confusion matrices to uncover specific failure patterns
Find optimal decision thresholds and analyze precision-recall tradeoffs
Select appropriate CV methods for your data and model type
Determine if performance improvements are statistically meaningful
Example Output
Root Cause Diagnosis
- Issue: 95% accuracy but 0.45 F1 score
- Root Cause: Severe class imbalance (99% negative class)
- Evidence: Precision 0.92, Recall 0.29 → model predicts mostly negatives
- Recommendation: Switch to stratified k-fold CV, use F1/AUC-PR metrics, apply class weighting
Evaluation Plan
- Primary metrics: AUC-PR, macro-F1, per-class precision/recall
- Cross-validation: Stratified 5-fold CV (preserves class distribution)
- Baseline: Train naive classifier to compare against
- Statistical test: McNemar's test for comparing model improvements
What's Included
- `SKILL.md`: Core evaluation & debugging skill with decision trees
- Evaluation metric selection checklist:
- Failure mode diagnosis template:
- Cross-validation strategy guide:
- Class imbalance mitigation workflows:
- Statistical significance test reference:
- Confusion matrix interpretation guide:
- Production monitoring recommendations:
Who It's For
- ML/Data Scientists debugging production models
- ML Engineers selecting evaluation metrics
- Model Researchers analyzing failure modes
- Analytics Engineers validating data pipeline models
- ML Ops professionals monitoring model performance
Best For
- Diagnosing unexpected model performance degradation
- Selecting appropriate evaluation metrics for your use case
- Analyzing failure modes in classification or regression
- Optimizing decision thresholds and precision-recall tradeoffs
- Planning comprehensive evaluation strategies for new models
You might also like
This skill guides you through rigorous experiment design from hypothesis to analysis plan. You'll compute the statistical power needed for your tests, identify confounding variables to control for, design optimal randomization strategies, and generate pre-registered analysis plans that prevent p-hacking and publication bias. Whether you're running a product A/B test or investigating causal effects in observational data, you get structured, validated designs that account for real-world constraints.
by moonwell-fi
You can build production-grade vector search systems that retrieve semantically similar documents in sub-millisecond time using AgentDB's optimized HNSW indexing. The skill enables you to implement RAG pipelines, semantic search engines, and intelligent knowledge bases with configurable embedding dimensions, distance metrics (cosine, Euclidean, dot product), and similarity thresholds—all with built-in quantization and caching for massive performance gains.
by moonwell-fi
This skill transforms high-level feature requests into actionable implementation plans by conducting deep requirement interviews, analyzing existing codebases (or planning greenfield projects), and producing comprehensive handoff documents. It distinguishes between brownfield development (with existing code to integrate) and greenfield projects (starting from scratch), ensuring your technical strategy accounts for your actual constraints and patterns.
by moonwell-fi
This skill helps you systematically diagnose failures in distributed ML training and serving infrastructure. You provide system logs, metrics, and error traces, and Claude performs structured root-cause analysis to identify the underlying issue—whether it's resource exhaustion, distributed system deadlock, data pipeline corruption, or model serving misconfiguration. You get a detailed diagnosis with remediation steps ranked by likelihood and implementation effort.
by moonwell-fi
Systematically evaluate your Claude prompts using quantitative scoring frameworks to identify weaknesses in clarity, task alignment, and output quality. You'll receive detailed optimization recommendations with A/B testing guidance to improve prompt effectiveness and consistency across different use cases and input variations.
by moonwell-fi
You can rapidly compile, organize, and format disparate health economic evidence—from clinical trials to cost-effectiveness analyses—into structured, regulatory-compliant dossiers. This skill maps your evidence to specific payer and HTA requirements, automatically generates evidence hierarchies, and produces submission-ready dossier outlines with formatting that meets regulatory standards for NICE, EUnetHTA, and other major bodies.
by moonwell-fi
You can systematically diagnose why your RAG system isn't returning relevant results and identify specific misalignments between queries and context. This skill walks you through analyzing retrieval quality, scoring context relevance, and optimizing prompt-retrieval alignment to improve your RAG accuracy. You'll get concrete, actionable recommendations for fixing bottlenecks in your pipeline.
by moonwell-fi
You can confidently navigate complex statistical analyses by working through structured workflows that match your research question and data type to the right test. The skill guides you through assumption validation, helps you interpret results accurately without common pitfalls, and walks you through regression diagnostics with full transparency about what your findings actually mean.
by moonwell-fi