Data Pipeline Architect
Design scalable ETL/ELT data pipelines with orchestration and quality checks
What You Can Do
You can design complete data pipeline architectures that reliably move data from source systems to analytical or operational destinations. This skill helps you choose between ETL vs. ELT patterns, select appropriate orchestration tools (Airflow, Prefect, dbt), map complex data flows, handle schema evolution, embed data quality checks, and create maintainable pipeline specifications that scale with your organization.
Features
Choose the right architecture based on your source systems, transformation complexity, and destination requirements
Evaluate and select from Airflow, Prefect, dbt, or custom solutions based on team skill, infrastructure, and pipeline complexity
Document source systems, transformation stages, dependencies, and target schemas with clear lineage
Manage breaking changes, backward compatibility, and versioning for evolving data models
Embed validation rules, monitoring, alerting, and anomaly detection throughout pipeline stages
Design resilient pipelines with proper handling of failures, timeouts, and idempotency
Create auditable, maintainable pipeline specifications for handoff and team collaboration
Example Output
Pipeline Architecture Example
For a multi-source analytics pipeline:
- Architecture Decision: ELT pattern chosen (transform in warehouse) due to large data volume and existing Snowflake infrastructure
- Tool Stack: Prefect for orchestration (flexible task dependencies), dbt for transformations, Snowflake for compute
- Data Flow: Raw data lands in staging → dbt models apply business logic → marts feed BI tools
- Quality Checks: Row count validation at each stage, null checks on key columns, referential integrity tests
- Monitoring: Failed task alerts to Slack, daily lineage reports, 4-hour SLA enforcement
Schema Evolution Plan
- Versioning Strategy: New columns added as
v2suffixes during transition, deprecated columns tracked in metadata - Backward Compatibility: Maintain mapping layer for 2 releases before removing old schemas
- Breaking Change Protocol: 30-day notice to downstream teams, automated migration scripts provided
What's Included
- SKILL.md: Complete data pipeline architecture framework with decision trees
- Pipeline Design Template: ETL/ELT architecture canvas with source-transform-destination mapping
- Tool Selection Checklist: Evaluation matrix for Airflow, Prefect, dbt, and custom solutions
- Data Quality Specification: Validation rule templates and anomaly detection patterns
- Schema Evolution Playbook: Breaking change management and versioning strategies
- Architecture Documentation Template: Standardized specification for pipeline handoff and governance
Who It's For
- Data Engineers — Design and implement production-grade pipelines with proper orchestration and monitoring
- Data Architects — Plan enterprise-scale data infrastructure and integration strategies
- Analytics Engineers — Architect ELT workflows that combine data loading with transformation in the warehouse
- Data Leaders/Managers — Evaluate tool options and design team processes around data pipeline development
- Platform Engineers — Build internal data infrastructure and self-service pipeline platforms
Best For
- Multi-source data integration projects combining APIs, databases, and file systems
- ETL/ELT pipeline design and architecture decisions
- Tool selection and evaluation for data orchestration
- Schema evolution and breaking change management strategies
- Data quality framework and validation rule design
- Pipeline documentation and governance specification
You might also like
This skill guides you through rigorous experiment design from hypothesis to analysis plan. You'll compute the statistical power needed for your tests, identify confounding variables to control for, design optimal randomization strategies, and generate pre-registered analysis plans that prevent p-hacking and publication bias. Whether you're running a product A/B test or investigating causal effects in observational data, you get structured, validated designs that account for real-world constraints.
by moonwell-fi
You can build production-grade vector search systems that retrieve semantically similar documents in sub-millisecond time using AgentDB's optimized HNSW indexing. The skill enables you to implement RAG pipelines, semantic search engines, and intelligent knowledge bases with configurable embedding dimensions, distance metrics (cosine, Euclidean, dot product), and similarity thresholds—all with built-in quantization and caching for massive performance gains.
by moonwell-fi
This skill transforms high-level feature requests into actionable implementation plans by conducting deep requirement interviews, analyzing existing codebases (or planning greenfield projects), and producing comprehensive handoff documents. It distinguishes between brownfield development (with existing code to integrate) and greenfield projects (starting from scratch), ensuring your technical strategy accounts for your actual constraints and patterns.
by moonwell-fi
Diagnose root causes of model performance failures, select optimal evaluation metrics based on your use case and data characteristics, and design comprehensive evaluation strategies. You'll get systematic workflows for analyzing failure modes, statistical significance testing, threshold optimization, and cross-validation planning—turning vague model problems into actionable fixes.
by moonwell-fi
This skill helps you systematically diagnose failures in distributed ML training and serving infrastructure. You provide system logs, metrics, and error traces, and Claude performs structured root-cause analysis to identify the underlying issue—whether it's resource exhaustion, distributed system deadlock, data pipeline corruption, or model serving misconfiguration. You get a detailed diagnosis with remediation steps ranked by likelihood and implementation effort.
by moonwell-fi
Systematically evaluate your Claude prompts using quantitative scoring frameworks to identify weaknesses in clarity, task alignment, and output quality. You'll receive detailed optimization recommendations with A/B testing guidance to improve prompt effectiveness and consistency across different use cases and input variations.
by moonwell-fi
You can rapidly compile, organize, and format disparate health economic evidence—from clinical trials to cost-effectiveness analyses—into structured, regulatory-compliant dossiers. This skill maps your evidence to specific payer and HTA requirements, automatically generates evidence hierarchies, and produces submission-ready dossier outlines with formatting that meets regulatory standards for NICE, EUnetHTA, and other major bodies.
by moonwell-fi
You can confidently navigate complex statistical analyses by working through structured workflows that match your research question and data type to the right test. The skill guides you through assumption validation, helps you interpret results accurately without common pitfalls, and walks you through regression diagnostics with full transparency about what your findings actually mean.
by moonwell-fi