Databricks ETL Pipeline Architect
Design and optimize production Databricks ETL pipelines
What You Can Do
You can design, build, and optimize production-grade Databricks ETL pipelines that handle large-scale data transformations reliably. Get architectural guidance on pipeline design patterns, performance tuning strategies, data quality frameworks, and troubleshooting approaches that follow Databricks and Apache Spark best practices.
Features
Create scalable, maintainable ETL structures using Delta Lake, medallion patterns, and incremental processing
Identify and fix inefficient transformations, optimize shuffle operations, and tune cluster configurations for your workload
Implement validation rules, schema enforcement, anomaly detection, and data freshness monitoring
Design retry logic, idempotency patterns, dead letter queues, and failure recovery strategies
Analyze transformation logic for correctness, performance, and maintainability across notebooks and jobs
Investigate pipeline errors using logs, metrics, and execution plans to identify root causes and solutions
Reduce compute spend through job clustering, auto-scaling tuning, caching strategies, and resource-efficient code patterns
Integrate Kafka, APIs, cloud storage, and databases with proper error handling and schema evolution
Example Output
Pipeline Design Recommendation:
Design a three-layer medallion architecture:
- Bronze: Raw data ingestion with schema validation
- Silver: Cleaned, deduplicated data with data quality checks
- Gold: Business-ready aggregations for analytics
Performance Diagnosis:
Identified shuffle bottleneck in join operation:
- Current: 45-minute runtime for daily job
- Issue: Broadcasting large dimension table (8GB) instead of smaller fact table
- Solution: Reverse join order + partition pruning
- Expected: 8-minute runtime (82% improvement)
Data Quality Implementation:
from great_expectations.datasource import PandasDatasource
Validation rules:
- No NULL values in customer_id (required)
- Transaction amount > 0 (range check)
- Created_date ≤ today (temporal check)
- Duplicate detection on (order_id, line_item)
What's Included
- SKILL.md: Complete Databricks ETL architecture and optimization guide
- Pipeline Design Template: Medallion pattern boilerplate with Delta Lake configuration
- Spark Tuning Checklist: Step-by-step performance diagnostic workflow
- Data Quality Framework: Reusable validation patterns and monitoring SQL
- Error Handling Patterns: Retry logic, idempotency, and failure recovery code samples
- Cost Optimization Worksheet: Analysis template for cluster sizing and job costing
- Troubleshooting Decision Tree: Flowchart for diagnosing common pipeline failures
Who It's For
- Data Engineers — Building and maintaining ETL pipelines at scale
- Analytics Engineers — Designing data models and transformation logic
- ETL Developers — Transitioning from traditional platforms to Databricks
- Data Platform Architects — Designing enterprise data infrastructure
- Solutions Architects — Advising customers on Databricks best practices
Best For
- Designing scalable ETL pipelines from scratch
- Optimizing underperforming or slow-running jobs
- Implementing comprehensive data quality checks and monitoring
- Troubleshooting pipeline failures and debugging Spark errors
- Migrating legacy ETL workloads to Databricks with performance targets
You might also like
You can submit product formulations, ingredients, and labeling details to receive comprehensive compliance assessments across major international food regulatory bodies. The skill identifies specific gaps between your product and target market requirements, prioritizes remediation efforts by market opportunity, and generates actionable market-entry strategies with realistic timelines and resource estimates for each jurisdiction.
by adeline117
You can rapidly assess ingredient lists for nutritional completeness, calculate protein quality scores (DIAAS, PDCAAS), evaluate nutrient-nutrient interactions, identify fortification opportunities, and generate compliant nutrition labels. The skill helps you optimize formulations against USDA, FDA, EFSA, and Codex Alimentarius standards before lab testing, saving time and resources in product development.
by adeline117
You can transform raw sensory data and consumer feedback into standardized flavor architectures that drive formulation decisions. This skill systematizes descriptive analysis results, models how ingredient combinations create emergent flavor characteristics, and translates preference data into actionable formulation adjustments. You'll create evidence-based flavor briefs that align R&D and marketing stakeholder expectations while documenting flavor architecture for regulatory compliance.
by adeline117
You can conduct rigorous allergen risk assessments that map allergen flow paths through facility design, equipment shared-use patterns, and personnel movement. This skill quantifies cross-contamination probability at each control point, prioritizes intervention investments based on risk severity, and produces audit-ready documentation demonstrating due diligence to regulators and third-party auditors.
by adeline117
You can submit marketing copy, label claims, or advertising language and receive a structured compliance assessment that maps claims against FDA/FTC guidance, state regulations, and case law. The skill identifies claim type (health claim, structure-function claim, comparative claim, etc.), flags regulatory risks across jurisdictions, assesses substantiation strength, and recommends revisions to reduce enforcement exposure. This accelerates claim review workflows and prevents costly post-launch violations.
by adeline117
You can systematically evaluate accelerated aging studies (Arrhenius, humidity chamber, ASLT protocols) and extrapolate results to real-world ambient conditions. Claude helps you extract degradation patterns from multiple test scenarios, quantify safety margins, and build scientifically defensible shelf life predictions that satisfy regulatory requirements for FDA, EFSA, FSSAI, and other food authorities.
by adeline117
You can create operation-specific HACCP plans that satisfy regulatory requirements (FSMA, GFSI, SQF, BRC) by analyzing your unique process flows, identifying biological/chemical/physical hazards, establishing science-based critical limits, and documenting defensible monitoring and corrective action procedures. This skill helps you translate regulatory principles into operational reality that your team can implement and auditors will accept.
by adeline117
You can design and execute professional texture profiling studies that decompose complex mouthfeel characteristics into measurable, reproducible dimensions. This framework guides you through panelist training with calibrated reference scales, standardized texture vocabulary, data collection protocols, and translation of sensory results into actionable product specifications that R&D and quality control can implement and monitor.
by adeline117