Kafka Pipeline Architect
Design production Kafka pipelines with verified performance and scale
What You Can Do
You can architect production-ready Kafka pipelines with schema validation, partition strategies, and lag management. Claude helps you identify bottlenecks, optimize throughput, and design fault-tolerant systems with concrete performance guarantees. You'll get architecture diagrams, configuration templates, and troubleshooting playbooks for real-world streaming challenges.
Features
Balance load across brokers based on key cardinality and throughput
Validate message format, handle schema evolution, prevent data corruption
Design metrics dashboards and alert thresholds for consumer lag
Design replication, exactly-once semantics, and crash recovery workflows
Identify bottlenecks in producers, brokers, and consumers with concrete tuning parameters
Calculate broker disk, memory, network requirements from message volume projections
Design cross-region replication, disaster recovery, and migration strategies
Example Output
Example 1: Partition Strategy Analysis
Topic: user-events
Current partitions: 4
Recommended partitions: 12
Rationale: Key cardinality ~2M users, target throughput 50K msgs/sec
Partition assignment: Round-robin on user_id hash
Expected rebalance time: 2.3s
Example 2: Consumer Lag Diagnosis
Consumer group: analytics-processor
Lag by partition:
- P0: 1.2M messages (12 mins behind)
- P1: 450K messages (4.5 mins behind)
- P2: 50K messages (30 secs behind)
Root cause: Consumer thread on P0 processing at 1.6K msgs/sec vs 2.1K msgs/sec ingest
Recommendation: Increase consumer parallelism or add 2 more consumer instances
Example 3: Schema Evolution Checklist
✓ New field: user_country (string, optional)
✓ Default value: 'UNKNOWN'
✓ Backward compatibility: Old producers can read new schema
✓ Forward compatibility: New consumers can read old messages
✓ Deployment order: Schema registry → consumers → producers
What's Included
- SKILL.md: Complete Kafka architecture framework with decision trees for partitioning, serialization, and failure recovery
- Pipeline design template: Configuration scaffold for producers, brokers, consumers with tuning parameters
- Partition strategy worksheet: Cardinality analysis, throughput estimation, and rebalancing guide
- Lag monitoring checklist: Metrics to track, alerting thresholds, and consumer health dashboard template
- Troubleshooting decision tree: Root cause analysis for common issues (slow consumers, broker saturation, rebalancing loops)
- Multi-cluster playbook: Cross-region setup, failover procedures, and testing strategies
- Performance tuning reference: All broker, producer, and consumer configs with impact analysis
Who It's For
- Data engineers architecting streaming data pipelines
- Platform engineers designing multi-tenant Kafka infrastructure
- DevOps engineers optimizing Kafka cluster performance
- Backend engineers troubleshooting production consumer lag
- Streaming architects designing disaster recovery and multi-region systems
Best For
- Designing partition and replication strategies for new pipelines
- Diagnosing and fixing consumer lag and throughput issues
- Planning capacity for projected message volumes
- Validating schema evolution and backward compatibility
- Migrating or rebalancing topics across clusters
You might also like
You can accelerate your entire antenna design workflow—from initial concept to optimized specifications—by leveraging Claude's knowledge of electromagnetic theory, antenna types, and design trade-offs. Claude helps you parameterize designs, identify optimal configurations for your frequency band and application, and prepare detailed briefs for simulation and manufacturing. You'll generate design reviews, impedance-matching solutions, and validation checklists without context-switching to multiple tools.
You can convert raw API code, OpenAPI specs, GraphQL schemas, and implementation details into production-ready reference documentation. The skill generates systematically organized endpoint documentation with parameter tables, real request/response examples, error code catalogs, authentication details, rate limits, and SDK usage snippets in multiple programming languages—creating the authoritative developer reference for REST APIs, GraphQL APIs, WebSocket APIs, and webhook schemas.
Rapidly analyze production incidents by correlating observability signals—logs, metrics, traces, and events—to identify the true root cause instead of symptoms. You'll reconstruct incident timelines, quantify impact scope, and generate both immediate remediation steps and long-term prevention measures.
This skill guides you through designing Kalman filters for your specific state estimation and sensor fusion problems. You'll learn to diagnose filter divergence, tune covariance matrices (Q and R), and validate observability—turning mathematical theory into production-ready filters. The skill provides step-by-step workflows, diagnostic tools, and practical templates for common applications like GPS/IMU fusion, SLAM, and aircraft navigation.
Claude analyzes your code diffs and generates standardized commit messages that follow the Conventional Commits specification. The skill automatically determines the correct commit type, scope, and description based on the changes you've made, ensuring your messages are parseable by automation tools while remaining human-readable for code reviewers.
You'll transform high-level feature requirements into a complete development plan covering API design, frontend implementation, testing strategy, and documentation. The skill helps you identify dependencies, parallelize work across frontend and backend teams, and establish quality gates that ensure features ship bug-free with proper error handling and logging.
You get a comprehensive security and gas optimization review of your Solidity smart contracts. The skill identifies common vulnerabilities like reentrancy attacks, overflow/underflow bugs, and unchecked external calls, while also pinpointing expensive operations and suggesting specific gas optimizations. Use this as a pre-audit screening to catch issues early and prepare your code for professional security review.
You can rapidly design, optimize, and validate chromatographic and LC-MS/MS methods for analyzing drugs and metabolites in biological matrices. This skill guides you through analytical method development workflows—including parameter selection, sensitivity optimization, and regulatory strategy—while ensuring ICH Q2(R2) and FDA compliance throughout the development process.