Kafka Pipeline Architect
Design production Kafka pipelines with verified performance and scale
What You Can Do
You can architect production-ready Kafka pipelines with schema validation, partition strategies, and lag management. Claude helps you identify bottlenecks, optimize throughput, and design fault-tolerant systems with concrete performance guarantees. You'll get architecture diagrams, configuration templates, and troubleshooting playbooks for real-world streaming challenges.
Features
Balance load across brokers based on key cardinality and throughput
Validate message format, handle schema evolution, prevent data corruption
Design metrics dashboards and alert thresholds for consumer lag
Design replication, exactly-once semantics, and crash recovery workflows
Identify bottlenecks in producers, brokers, and consumers with concrete tuning parameters
Calculate broker disk, memory, network requirements from message volume projections
Design cross-region replication, disaster recovery, and migration strategies
Example Output
Example 1: Partition Strategy Analysis
Topic: user-events
Current partitions: 4
Recommended partitions: 12
Rationale: Key cardinality ~2M users, target throughput 50K msgs/sec
Partition assignment: Round-robin on user_id hash
Expected rebalance time: 2.3s
Example 2: Consumer Lag Diagnosis
Consumer group: analytics-processor
Lag by partition:
- P0: 1.2M messages (12 mins behind)
- P1: 450K messages (4.5 mins behind)
- P2: 50K messages (30 secs behind)
Root cause: Consumer thread on P0 processing at 1.6K msgs/sec vs 2.1K msgs/sec ingest
Recommendation: Increase consumer parallelism or add 2 more consumer instances
Example 3: Schema Evolution Checklist
✓ New field: user_country (string, optional)
✓ Default value: 'UNKNOWN'
✓ Backward compatibility: Old producers can read new schema
✓ Forward compatibility: New consumers can read old messages
✓ Deployment order: Schema registry → consumers → producers
What's Included
- SKILL.md: Complete Kafka architecture framework with decision trees for partitioning, serialization, and failure recovery
- Pipeline design template: Configuration scaffold for producers, brokers, consumers with tuning parameters
- Partition strategy worksheet: Cardinality analysis, throughput estimation, and rebalancing guide
- Lag monitoring checklist: Metrics to track, alerting thresholds, and consumer health dashboard template
- Troubleshooting decision tree: Root cause analysis for common issues (slow consumers, broker saturation, rebalancing loops)
- Multi-cluster playbook: Cross-region setup, failover procedures, and testing strategies
- Performance tuning reference: All broker, producer, and consumer configs with impact analysis
Who It's For
- Data engineers architecting streaming data pipelines
- Platform engineers designing multi-tenant Kafka infrastructure
- DevOps engineers optimizing Kafka cluster performance
- Backend engineers troubleshooting production consumer lag
- Streaming architects designing disaster recovery and multi-region systems
Best For
- Designing partition and replication strategies for new pipelines
- Diagnosing and fixing consumer lag and throughput issues
- Planning capacity for projected message volumes
- Validating schema evolution and backward compatibility
- Migrating or rebalancing topics across clusters
You might also like
Rapidly analyze production incidents by correlating observability signals—logs, metrics, traces, and events—to identify the true root cause instead of symptoms. You'll reconstruct incident timelines, quantify impact scope, and generate both immediate remediation steps and long-term prevention measures.
Claude analyzes your code diffs and generates standardized commit messages that follow the Conventional Commits specification. The skill automatically determines the correct commit type, scope, and description based on the changes you've made, ensuring your messages are parseable by automation tools while remaining human-readable for code reviewers.
You'll transform high-level feature requirements into a complete development plan covering API design, frontend implementation, testing strategy, and documentation. The skill helps you identify dependencies, parallelize work across frontend and backend teams, and establish quality gates that ensure features ship bug-free with proper error handling and logging.
Transform schema and data changes into production-ready migration packages that protect data integrity and minimize downtime. You can handle column additions, renames, type conversions, and backfills using proven patterns like expand-contract and shadow tables. The skill generates explicit rollback scripts, validation checklists, and phased execution plans that keep applications stable throughout the migration.
You can create comprehensive performance testing strategies tailored to your application's architecture, analyze test results to identify bottlenecks, and generate prioritized recommendations for optimization. The skill helps you design load tests, interpret metrics like latency and throughput, and create action plans for performance improvements.
You can design comprehensive performance test strategies tailored to your application architecture, analyze performance metrics and logs to identify bottlenecks, and receive specific optimization recommendations. The skill guides you through creating realistic load scenarios, interpreting performance data, and prioritizing fixes based on impact analysis.
You can design end-to-end NGS pipelines tailored to your experiment type (WGS, WES, RNA-seq, targeted), troubleshoot pipeline failures by analyzing QC metrics and logs, and optimize performance by fine-tuning parameters and identifying bottlenecks. This skill handles the full workflow from raw sequencing data quality assessment through variant interpretation and annotation.
You can accelerate your entire antenna design workflow—from initial concept to optimized specifications—by leveraging Claude's knowledge of electromagnetic theory, antenna types, and design trade-offs. Claude helps you parameterize designs, identify optimal configurations for your frequency band and application, and prepare detailed briefs for simulation and manufacturing. You'll generate design reviews, impedance-matching solutions, and validation checklists without context-switching to multiple tools.