ML Infrastructure Failure Analysis & Optimization
Diagnose ML infrastructure failures with structured root-cause analysis
What You Can Do
This skill helps you systematically diagnose failures in distributed ML training and serving infrastructure. You provide system logs, metrics, and error traces, and Claude performs structured root-cause analysis to identify the underlying issue—whether it's resource exhaustion, distributed system deadlock, data pipeline corruption, or model serving misconfiguration. You get a detailed diagnosis with remediation steps ranked by likelihood and implementation effort.
Features
aggregates logs from training jobs, resource managers, databases, and serving infrastructure to surface patterns
detects deadlocks, timeouts, communication failures, and split-brain scenarios in distributed training setups
identifies CPU/memory/GPU exhaustion, disk I/O bottlenecks, and network saturation from metrics
traces data corruption, schema mismatches, and ingestion failures from end-to-end logs
analyzes inference latency, cache misses, hardware underutilization, and load-balancing issues
ranks potential root causes by likelihood, impact, and fix complexity
generates specific actionable steps tailored to your infrastructure type
identifies minimal steps to reproduce the failure for testing fixes
Example Output
Training Job Timeout Analysis:
ROOT CAUSE: GPU memory exhaustion after epoch 3 (95% utilization)
CONFIDENCE: High (logs + metrics align)
IMPACT: All 8 worker nodes OOM-killed after 2h 14m
REMEDIATION:
1. Reduce batch size from 256 to 128 (10 min)
2. Enable gradient checkpointing in model (30 min)
3. Increase swap on worker nodes (5 min)
Serving Latency Issue:
ROOT CAUSE: Single-threaded model loading blocking requests (p99 latency = load time)
CONFIDENCE: High (latency spikes correlate with model reloads)
REMEDIATION:
1. Implement async model loader (1 hour)
2. Pre-warm cache on deployment (30 min)
3. Scale from 2 to 4 replica pods (5 min)
EXPECTED: p99 latency < 200ms
What's Included
- SKILL.md: Complete failure diagnosis workflow with decision trees and analysis templates
- Log parsing templates: Pre-structured prompts for Kubernetes, PyTorch, TensorFlow, and cloud provider logs
- Failure diagnosis checklist: Systematic investigation guide covering resource utilization, networking, data integrity, and model configuration
- Remediation runbook: Template for documenting and prioritizing fixes
- Root-cause prioritization matrix: Decision framework to rank hypotheses by likelihood and impact
- Post-mortem template: Incident documentation format for team learning and prevention
- Infrastructure reference: Common failure patterns for Kubernetes, distributed schedulers, and cloud platforms
Who It's For
- MLOps engineers — Debugging distributed training clusters and model serving infrastructure
- ML platform engineers — Troubleshooting production incidents affecting multiple users
- Data scientists — Understanding why training jobs fail or models underperform in production
- DevOps engineers — Supporting ML workloads on Kubernetes or cloud infrastructure
- Site reliability engineers — Root-cause analysis for ML system outages
Best For
- Diagnosing unexpected training job failures or hangs — Timeouts, OOM errors, deadlocks
- Troubleshooting model serving latency or throughput issues — Slow inference, high p99 latency, uneven load distribution
- Identifying resource constraints — Understanding bottlenecks in distributed systems (CPU, GPU, memory, network, I/O)
- Post-incident root-cause analysis — Systematic investigation for incident reports and prevention planning
- Infrastructure optimization — Using failure patterns to guide configuration and scaling decisions
You might also like
You can systematically diagnose why your RAG system isn't returning relevant results and identify specific misalignments between queries and context. This skill walks you through analyzing retrieval quality, scoring context relevance, and optimizing prompt-retrieval alignment to improve your RAG accuracy. You'll get concrete, actionable recommendations for fixing bottlenecks in your pipeline.
by moonwell-fi
You'll handle difficult guest conversations with structured communication templates that de-escalate tension while protecting your property. Generate professional damage assessments, complaint responses, and resolution protocols that document interactions clearly and mitigate liability. Create personalized communication strategies tailored to each guest situation, from minor maintenance requests to serious damage claims.
by moonwell-fi
Systematically evaluate your Claude prompts using quantitative scoring frameworks to identify weaknesses in clarity, task alignment, and output quality. You'll receive detailed optimization recommendations with A/B testing guidance to improve prompt effectiveness and consistency across different use cases and input variations.
by moonwell-fi
You can prepare your domain-specific datasets, train fine-tuned Claude models optimized for your use case, benchmark performance against base models, and deploy production-ready custom models. This skill automates the entire fine-tuning pipeline from raw data to live inference, complete with cost-benefit analysis and version control.
by moonwell-fi
You get structured guidance for designing controlled experiments, selecting appropriate metrics, and analyzing results with statistical rigor. Claude helps you identify confounding variables, calculate required sample sizes, interpret statistical significance, and document methodology for reproducible benchmarking that stands up to scrutiny.
by moonwell-fi
Design experiments with statistical rigor from the ground up. You'll generate power analyses, create randomization strategies, write preregistration documents, and get AI-assisted reviews of your experimental design to catch biases before data collection. Whether you're planning a clinical trial, A/B test, or academic study, you'll have a complete, defensible experimental protocol.
by moonwell-fi
This skill guides you through systematic feature engineering to identify and validate features that drive real model performance. You'll leverage Claude to analyze your data, generate feature hypotheses, validate statistical significance, and quantify business impact—all with workflows designed to catch pitfalls before production deployment.
by moonwell-fi
This skill transforms high-level feature requests into actionable implementation plans by conducting deep requirement interviews, analyzing existing codebases (or planning greenfield projects), and producing comprehensive handoff documents. It distinguishes between brownfield development (with existing code to integrate) and greenfield projects (starting from scratch), ensuring your technical strategy accounts for your actual constraints and patterns.
by moonwell-fi