ETL Pipeline Design & Optimization
Design and optimize data ETL pipelines for production performance
What You Can Do
Analyze your data pipeline architecture, identify performance bottlenecks, and receive detailed optimization strategies. You'll get architectural recommendations for scalability, data quality assurance patterns, and concrete implementation guidance for improving throughput and reducing latency in your ETL workflows.
Features
Assess current ETL design and identify structural improvements for better performance
Pinpoint which stages consume the most time and resources
Design pipelines that handle growing data volumes without performance degradation
Implement validation, reconciliation, and error handling at each stage
Get concrete techniques to reduce latency and increase throughput
Evaluate tools and formats for your specific pipeline requirements
Identify resource-intensive operations and suggest cost-effective alternatives
Example Output
Pipeline Performance Analysis:
- ✅ Current state: Your daily ETL processes 2.5M records in 4 hours with 12% failure rate
Identified Bottlenecks:
-
JSON Serialization (Est. 40% of runtime)
- Root cause: Parsing 1.2M intermediate records as JSON
- Fix: Switch to Parquet format → 2.1x speedup
- Implementation: Add Parquet encoder before aggregation stage
-
Unindexed Lookups (Est. 32% of runtime)
- Root cause: Sequential scans of 500K reference table per transform
- Fix: Pre-load reference table into memory hash map
- Expected improvement: 3x faster join operations
Recommended Architecture:
Ingest → Validate → Transform → Aggregate → Parquet Sink
(fail fast) (cached refs) (vectorized) (batch I/O)
Data Quality Checklist:
- Row count reconciliation before/after each stage
- Field-level type and null validation
- Duplicate detection on primary key
- Late-arriving dimension handling
What's Included
- SKILL.md: Complete ETL design framework with analysis workflows, optimization techniques, and architectural patterns
- Pipeline Architecture Template: Structured format for documenting current and proposed pipelines
- Bottleneck Detection Checklist: Common performance anti-patterns and where to look
- Optimization Workbook: Proven techniques for throughput, latency, and cost reduction
- Data Quality & Validation Guide: Testing strategies, reconciliation queries, and alerting patterns
Who It's For
- Data engineers — Optimize production ETL pipelines and reduce processing time
- Data architects — Plan scalable data integration strategies from scratch
- Analytics engineers — Improve data warehouse load performance and reliability
- Platform engineers — Debug slow pipelines and implement resilient workflows
- Performance engineers — Analyze bottlenecks and measure optimization impact
Best For
- Diagnosing why ETL pipelines are slow or failing
- Designing new data integration architectures
- Reducing pipeline execution time and infrastructure costs
- Planning data quality validation and error handling
- Evaluating technology choices (Spark, Airflow, tools, formats)
You might also like
Rapidly analyze production incidents by correlating observability signals—logs, metrics, traces, and events—to identify the true root cause instead of symptoms. You'll reconstruct incident timelines, quantify impact scope, and generate both immediate remediation steps and long-term prevention measures.
Claude analyzes your code diffs and generates standardized commit messages that follow the Conventional Commits specification. The skill automatically determines the correct commit type, scope, and description based on the changes you've made, ensuring your messages are parseable by automation tools while remaining human-readable for code reviewers.
You'll transform high-level feature requirements into a complete development plan covering API design, frontend implementation, testing strategy, and documentation. The skill helps you identify dependencies, parallelize work across frontend and backend teams, and establish quality gates that ensure features ship bug-free with proper error handling and logging.
Transform schema and data changes into production-ready migration packages that protect data integrity and minimize downtime. You can handle column additions, renames, type conversions, and backfills using proven patterns like expand-contract and shadow tables. The skill generates explicit rollback scripts, validation checklists, and phased execution plans that keep applications stable throughout the migration.
You can create comprehensive performance testing strategies tailored to your application's architecture, analyze test results to identify bottlenecks, and generate prioritized recommendations for optimization. The skill helps you design load tests, interpret metrics like latency and throughput, and create action plans for performance improvements.
You can design comprehensive performance test strategies tailored to your application architecture, analyze performance metrics and logs to identify bottlenecks, and receive specific optimization recommendations. The skill guides you through creating realistic load scenarios, interpreting performance data, and prioritizing fixes based on impact analysis.
You can design end-to-end NGS pipelines tailored to your experiment type (WGS, WES, RNA-seq, targeted), troubleshoot pipeline failures by analyzing QC metrics and logs, and optimize performance by fine-tuning parameters and identifying bottlenecks. This skill handles the full workflow from raw sequencing data quality assessment through variant interpretation and annotation.
You can accelerate your entire antenna design workflow—from initial concept to optimized specifications—by leveraging Claude's knowledge of electromagnetic theory, antenna types, and design trade-offs. Claude helps you parameterize designs, identify optimal configurations for your frequency band and application, and prepare detailed briefs for simulation and manufacturing. You'll generate design reviews, impedance-matching solutions, and validation checklists without context-switching to multiple tools.