Databricks Query Optimization & Pipeline Debugging
Optimize Databricks queries and resolve pipeline issues
What You Can Do
You can optimize Databricks SQL queries for performance, debug pipeline failures with root-cause analysis, and recommend cluster configurations. Claude analyzes execution plans, identifies bottlenecks, and provides concrete refactoring suggestions to reduce query runtime and lower infrastructure costs.
Features
Analyze execution plans to identify expensive operations, data shuffles, and scanning inefficiencies
Refactor queries with partitioning strategies, index hints, and join optimization techniques
Trace pipeline failures through logs, identify data quality issues, and recommend fixes
Right-size worker nodes, adjust shuffle partitions, and optimize memory allocation for your workload
Map data flow across jobs and notebooks to identify bottlenecks and failure propagation
Compare query performance before/after optimization and quantify resource savings
Identify fragmented data files, recommend OPTIMIZE/VACUUM operations, and analyze table statistics
Design SLAs for pipelines and configure thresholds to catch performance degradation early
Example Output
Input: Slow query execution plan with 12 shuffles
Output:
-- Optimized query with pre-filter and join reordering
SELECT customer_id, SUM(amount)
FROM sales
WHERE year = 2026 -- Filter first
JOIN customers USING(customer_id)
GROUP BY customer_id
Execution time: 45s → 8s | Shuffle operations: 12 → 2 | Estimated savings: 92% reduction
What's Included
- SKILL.md: Complete Claude skill with workflows for query optimization, pipeline debugging, and cluster tuning
- Query Analysis Checklist: Step-by-step guide for diagnosing slow queries
- Optimization Templates: Reusable SQL patterns for joins, aggregations, and window functions
- Debug Workflow: Decision tree for pipeline failure diagnosis with remediation steps
- Configuration Reference: Cluster sizing formulas and parameter tuning guide
Who It's For
- Data Engineers — Optimize ETL pipeline performance and reduce cloud spending
- Analytics Engineers — Tune SQL queries powering dashboards and reports
- Database Administrators — Manage cluster resources and enforce performance standards
- Data Scientists — Accelerate feature engineering pipelines on large datasets
- MLOps Engineers — Debug training data pipelines and optimize model training workflows
Best For
- Diagnosing slow Databricks queries and recommending optimizations
- Debugging failed Delta Lake jobs with detailed error analysis
- Right-sizing Databricks clusters and predicting cost savings
- Analyzing execution plans to identify data shuffles and scanning bottlenecks
- Optimizing SQL patterns for Databricks-specific features (Photon, Delta cache, Z-order)
You might also like
Turn project descriptions, timelines, and structured data into professional draw.io diagrams instantly. You can generate flowcharts, org charts, Gantt charts, swimlane diagrams, and process flows by describing what you need. Claude creates draw.io XML that you can open directly in the editor or modify further.
by skillslib
You can design culturally appropriate survey instruments tailored to international development contexts, analyze mixed-methods data by integrating quantitative findings with qualitative insights, and implement robust data quality management systems for your field operations. Claude helps you create survey templates with adaptive logic, develop validation protocols, and produce data quality reports that highlight issues before analysis begins.
by skillslib
You can diagnose warehouse bottlenecks using structured data analysis, then generate optimized layouts and flow improvements. The skill models space utilization, labor deployment, and throughput constraints to identify quick wins and long-term improvements, delivering quantified cost savings and performance gains for your specific facility.
by skillslib
You can systematically code qualitative data against your evaluation framework, surface patterns and themes across multiple responses, extract compelling impact stories with supporting evidence, and generate summary matrices that link narrative findings to your outcomes claims. This skill maintains analytical rigor while reducing the 40-60 hours typically spent manually reviewing transcripts and open-ended survey responses per evaluation cycle.
by skillslib
Get complete, ready-to-deploy lead magnets tailored to your ideal customer profile. Claude designs the concept, writes the persuasive copy, builds the email sequences, and creates deployment strategies that attract qualified prospects and build engaged lists fast. You'll capture 3-5x more subscribers with magnetic offers that actually resonate.
by skillslib
You can submit UI mockups, wireframes, or detailed descriptions of mobile game screens to Claude for comprehensive interaction pattern analysis. Claude evaluates touch target sizing, gesture hierarchies, screen orientation handling, accessibility compliance (WCAG 2.1 AA), and player engagement friction points. You'll receive specific, actionable recommendations for optimizing layouts across phones, tablets, and foldables—plus competitor pattern benchmarking and tutorial flow refinement.
by skillslib
You can convert raw discovery call notes into polished, multi-section sales proposals that include executive summaries tailored to buyer personas, needs analysis mapped to your solution architecture, tiered pricing options with competitive positioning, ROI projections with financial justification, and realistic implementation timelines. The skill preserves personalization and strategic positioning while automating structural components, letting you respond to RFPs and time-sensitive opportunities without sacrificing quality or consistency.
by skillslib
This skill generates optimized facility layouts by analyzing material and personnel flow patterns, validating against operational constraints, and comparing multiple design options with quantified metrics. You provide your facility's dimensions, equipment locations, and operational requirements, and Claude produces detailed layouts with flow efficiency scores, space utilization analysis, and cost-benefit comparisons. Each layout option includes specific recommendations for improving throughput, reducing congestion points, and enhancing safety compliance.
by skillslib