Databricks Query Optimization & Pipeline Debugging
Optimize Databricks queries and resolve pipeline issues
What You Can Do
You can optimize Databricks SQL queries for performance, debug pipeline failures with root-cause analysis, and recommend cluster configurations. Claude analyzes execution plans, identifies bottlenecks, and provides concrete refactoring suggestions to reduce query runtime and lower infrastructure costs.
Features
Analyze execution plans to identify expensive operations, data shuffles, and scanning inefficiencies
Refactor queries with partitioning strategies, index hints, and join optimization techniques
Trace pipeline failures through logs, identify data quality issues, and recommend fixes
Right-size worker nodes, adjust shuffle partitions, and optimize memory allocation for your workload
Map data flow across jobs and notebooks to identify bottlenecks and failure propagation
Compare query performance before/after optimization and quantify resource savings
Identify fragmented data files, recommend OPTIMIZE/VACUUM operations, and analyze table statistics
Design SLAs for pipelines and configure thresholds to catch performance degradation early
Example Output
Input: Slow query execution plan with 12 shuffles
Output:
-- Optimized query with pre-filter and join reordering
SELECT customer_id, SUM(amount)
FROM sales
WHERE year = 2026 -- Filter first
JOIN customers USING(customer_id)
GROUP BY customer_id
Execution time: 45s → 8s | Shuffle operations: 12 → 2 | Estimated savings: 92% reduction
What's Included
- SKILL.md: Complete Claude skill with workflows for query optimization, pipeline debugging, and cluster tuning
- Query Analysis Checklist: Step-by-step guide for diagnosing slow queries
- Optimization Templates: Reusable SQL patterns for joins, aggregations, and window functions
- Debug Workflow: Decision tree for pipeline failure diagnosis with remediation steps
- Configuration Reference: Cluster sizing formulas and parameter tuning guide
Who It's For
- Data Engineers — Optimize ETL pipeline performance and reduce cloud spending
- Analytics Engineers — Tune SQL queries powering dashboards and reports
- Database Administrators — Manage cluster resources and enforce performance standards
- Data Scientists — Accelerate feature engineering pipelines on large datasets
- MLOps Engineers — Debug training data pipelines and optimize model training workflows
Best For
- Diagnosing slow Databricks queries and recommending optimizations
- Debugging failed Delta Lake jobs with detailed error analysis
- Right-sizing Databricks clusters and predicting cost savings
- Analyzing execution plans to identify data shuffles and scanning bottlenecks
- Optimizing SQL patterns for Databricks-specific features (Photon, Delta cache, Z-order)
You might also like
You can design culturally appropriate survey instruments tailored to international development contexts, analyze mixed-methods data by integrating quantitative findings with qualitative insights, and implement robust data quality management systems for your field operations. Claude helps you create survey templates with adaptive logic, develop validation protocols, and produce data quality reports that highlight issues before analysis begins.
by skillslib
You create comprehensive DCAT (Data Catalog Vocabulary) metadata profiles and RDF descriptions that standardize your dataset documentation, ensure compliance with open data directives, and dramatically improve discoverability. The skill generates machine-readable metadata in multiple formats, validates against international standards, and produces human-readable cataloging profiles that meet GDPR, FAIR data, and open government requirements.
by skillslib
You can diagnose warehouse bottlenecks using structured data analysis, then generate optimized layouts and flow improvements. The skill models space utilization, labor deployment, and throughput constraints to identify quick wins and long-term improvements, delivering quantified cost savings and performance gains for your specific facility.
by skillslib
You can systematically code qualitative data against your evaluation framework, surface patterns and themes across multiple responses, extract compelling impact stories with supporting evidence, and generate summary matrices that link narrative findings to your outcomes claims. This skill maintains analytical rigor while reducing the 40-60 hours typically spent manually reviewing transcripts and open-ended survey responses per evaluation cycle.
by skillslib
Automatically parse RFP requirements and generate compliant, structured proposal responses that address every question. You'll reduce proposal turnaround time from weeks to hours while ensuring all requirements are met and compliance is validated. Eliminate manual rework by identifying gaps and inconsistencies before submission.
by skillslib
This skill helps you transform raw accident investigation data into organized, actionable reports. You can extract key findings, build timelines, identify root causes, and generate structured summaries that comply with safety and legal standards. It systematically analyzes incident patterns, contributing factors, and recommendations to support thorough investigations.
by skillslib
You can convert raw discovery call notes into polished, multi-section sales proposals that include executive summaries tailored to buyer personas, needs analysis mapped to your solution architecture, tiered pricing options with competitive positioning, ROI projections with financial justification, and realistic implementation timelines. The skill preserves personalization and strategic positioning while automating structural components, letting you respond to RFPs and time-sensitive opportunities without sacrificing quality or consistency.
by skillslib
Turn project descriptions, timelines, and structured data into professional draw.io diagrams instantly. You can generate flowcharts, org charts, Gantt charts, swimlane diagrams, and process flows by describing what you need. Claude creates draw.io XML that you can open directly in the editor or modify further.
by skillslib