Grafana Alert Rule Engineering
Engineer production-grade Grafana alert rules with intelligent escalation
What You Can Do
Design sophisticated multi-threshold Grafana alert rules that intelligently route incidents to the right teams based on severity and context. Reduce false positive alerts by 40-60% through dynamic thresholding, baseline-aware triggers, and alert quality evaluation. Generate complete incident response runbooks and escalation workflows that integrate directly with your notification channels and on-call systems.
Features
Automatic severity escalation based on conditions
Route alerts to specialists based on service ownership
Pre-written remediation steps for each alert type
Quantify and reduce alert noise systematically
Slack, PagerDuty, email, and webhook routing rules
Document alert rule decisions and evolution
Test alert behavior with synthetic data
Align alerts with service level objectives
Example Output
Alert Rule: High API Latency with Escalation
AlertName: APILatencyHigh
Thresholds:
- Warning: p95_latency > 200ms (5 min)
- Critical: p99_latency > 500ms (2 min)
Escalation:
- Warning → Slack #platform-oncall
- Critical → PagerDuty (team: API Platform)
Runbook: https://wiki.company.com/runbooks/api-latency
Escalation Policy Template
Level 1 (15 min): Slack notification to team channel
Level 2 (20 min): PagerDuty escalation to primary on-call
Level 3 (30 min): Escalate to team lead
Level 4 (45 min): Escalate to manager + VP Engineering
False Positive Reduction Report
✓ Alert: HighMemoryUsage
- Baseline: 150 false positives/month
- Dynamic threshold applied (p99-based)
- Result: 12 false positives/month (92% reduction)
- ROI: 98% fewer notifications, 0 missed incidents
What's Included
- SKILL.md: Complete alert rule engineering workflow with decision trees for threshold selection, escalation design, and runbook creation
- Alert Rule Templates: Ready-to-customize rules for CPU, memory, disk, latency, error rates, and availability
- Escalation Policy Templates: Time-based and severity-based escalation workflows with team routing
- Runbook Template: Standardized format for incident response playbooks with diagnosis and remediation steps
- Alert Quality Evaluation Checklist: Criteria for scoring and ranking alerts by business impact vs. noise ratio
- False Positive Reduction Playbook: Systematic approach to identifying and eliminating noisy alerts
- Prometheus Query Optimization Guide: Best practices for writing efficient, stable alert queries
Who It's For
- SRE and DevOps engineers designing monitoring systems for production services
- Platform engineers building alerting infrastructure and runbook automation
- On-call engineers managing alert fatigue and incident response workflows
- Systems architects implementing resilience patterns and incident management processes
Best For
- Designing multi-tier alert rules with intelligent escalation policies
- Reducing false positive alert rates and on-call fatigue
- Building comprehensive incident response playbooks and runbooks
- Integrating monitoring systems with on-call and incident management platforms
- Creating organizational alert rule standards and engineering best practices
You might also like
You can instantly generate engaging captions, story arcs, and pitch-ready copy for your sports photography. Whether you're covering a championship game, developing a photo essay series, or submitting work to media outlets, this skill crafts deadline-ready narratives that showcase your images and hook editors. Transform raw photos into publication-quality coverage with professional storytelling that brings readers into the defining moments of athletic competition.
by manwithstyle
You can design comprehensive Datadog monitoring strategies that transform raw metrics into actionable intelligence. Claude helps you architect dashboards that surface critical insights, configure intelligent alerts that reduce false positives, and build escalation workflows that speed up incident response. The result: measurably lower MTTR and a team that trusts your alerts.
by manwithstyle
You can transform complex datasets into compelling interactive narratives that maintain coherence while inviting exploration. This skill helps you architect data-driven stories where insights unfold naturally through guided discovery points, keeping readers engaged without limiting their agency. Readers navigate your narrative with purpose, uncovering patterns and relationships that would remain hidden in static presentations.
by manwithstyle
Transform breaking news briefs into compelling visual story frameworks that guide photographers toward impactful coverage. You'll receive multiple narrative angles, detailed shot lists organized by scene and purpose, and complete assignment briefs with sourcing guidance. Each output ensures comprehensive emotional and contextual storytelling from first frame to final edit.
by manwithstyle
Analyze Docker container logs instantly to identify root causes of failures, crashes, and resource bottlenecks. Get actionable recommendations for resource allocation, network configuration, and architectural improvements that eliminate production issues before they cascade.
by manwithstyle
Systematically analyze your AWS accounts to identify cost optimization opportunities, security gaps, and performance bottlenecks. You'll receive prioritized, actionable recommendations tailored to your architecture, from EC2 right-sizing to unused resource cleanup and security hardening.
by manwithstyle
This skill walks you through a structured data analysis workflow—from initial exploration and claim verification to identifying compelling story angles and building evidence-backed narratives. You'll learn to spot anomalies, validate statistics, and construct defensible arguments that stand up to scrutiny. It's designed for journalists who want to move beyond surface-level data work into serious investigative storytelling.
by manwithstyle
Accelerate your beat reporting by organizing research, identifying story angles, discovering sources, and developing narrative frameworks. You can quickly synthesize complex topics, spot emerging trends, and prepare comprehensive story outlines that anchor your reporting. This skill helps you go deeper on your beat with less time spent on research logistics.
by manwithstyle