# Connect Grafana to PagerDuty > Bridge your observability data and on-call workflows to resolve incidents faster and cut alert fatigue. **Canonical page:** https://tray.ai/connectors/grafana-pagerduty-integrations/ **Grafana connector:** https://tray.ai/connectors/grafana-integrations/ **Grafana documentation:** https://tray.ai/documentation/connectors/service/grafana ## Overview Grafana is the go-to platform for visualizing and analyzing metrics, logs, and traces across your infrastructure stack. PagerDuty handles intelligent incident management, routing critical alerts to the right on-call engineers at the right time. Connecting Grafana with PagerDuty creates a direct pipeline from metric anomaly detection to structured incident response, so no critical threshold breach gets ignored. Operations and SRE teams rely on Grafana dashboards to track system health, but a dashboard alone can't wake up an engineer at 2 a.m. or coordinate a cross-team incident response. Without a tight connection between Grafana and PagerDuty, teams end up manually checking dashboards, copy-pasting alert details into tickets, and guessing who's on call. By connecting these two platforms through tray.ai, every Grafana alert — whether triggered by CPU spikes, error rate surges, latency anomalies, or infrastructure failures — can automatically create, update, acknowledge, or resolve PagerDuty incidents with full context attached. This cuts toil, accelerates mean time to response (MTTR), and makes sure your observability investment actually drives faster recovery. ## Use cases ### Automatic Incident Creation from Grafana Alerts When a Grafana alert fires and crosses a defined threshold — say, a 5xx error rate exceeding 2% or p99 latency spiking above SLA — tray.ai opens a new PagerDuty incident with the full alert payload, dashboard link, and affected service metadata attached. This closes the gap between detection and escalation that costs teams critical minutes during outages. On-call engineers get a rich, actionable notification rather than a raw metric dump. - Reduces MTTR by triggering PagerDuty escalation the moment a Grafana threshold is breached - Attaches dashboard snapshots and runbook links directly to the PagerDuty incident - Eliminates manual incident creation and the human error that comes with it ### Auto-Resolve PagerDuty Incidents When Grafana Alerts Recover When a Grafana alert transitions from firing to resolved, tray.ai automatically resolves the corresponding PagerDuty incident, preventing stale incidents from cluttering your queue and confusing on-call responders. This bidirectional status sync keeps both platforms aligned throughout the full alert lifecycle. Teams spend less time manually closing incidents and more time confirming that systems are genuinely stable. - Keeps PagerDuty incident queues accurate and free of ghost incidents - Reduces alert fatigue by closing noise once underlying issues are resolved - Provides a clean audit trail linking Grafana recovery events to PagerDuty resolution timestamps ### Escalate High-Severity Grafana Alerts to Specific PagerDuty Services Not all alerts need the same urgency or team. With tray.ai, you can route Grafana alerts to different PagerDuty services and escalation policies based on alert labels, severity tags, or the originating data source — database alerts go to the DBA on-call team, Kubernetes alerts go to the platform engineering squad, and application errors notify the backend development team. Each team gets only the incidents relevant to their domain, not everything. - Eliminates mis-routed alerts that slow down incident response - Maps Grafana alert labels and annotations to PagerDuty service-level routing rules - Supports multi-team environments with distinct on-call schedules and escalation policies ### Enrich PagerDuty Incidents with Grafana Dashboard Context A bare alert notification rarely gives an on-call engineer enough to act on immediately. Using tray.ai, when a PagerDuty incident is created, the workflow can simultaneously query Grafana for a rendered dashboard snapshot or a direct deep-link to the relevant panel, then append it to the incident as a note or custom field. Engineers open their PagerDuty mobile notification and immediately see the metric trend that caused the incident — no dashboard hunting during a high-pressure outage. - Cuts time-to-context for on-call engineers from minutes to seconds - Attaches live Grafana panel links to PagerDuty incident notes automatically - Improves post-incident reviews with visual context embedded in the incident record ### Sync PagerDuty Incident Acknowledgments Back to Grafana Annotations When an on-call engineer acknowledges or resolves a PagerDuty incident, tray.ai writes a Grafana annotation onto the relevant dashboard panel, marking exactly when the incident was noticed and resolved. This puts a visible, time-stamped overlay on your metric graphs that ties human response actions to system behavior. Over time, these annotations build a historical record of operational events directly inside your observability layer. - Correlates incident response timelines directly on Grafana metric charts - Builds an automatic operational log without requiring manual annotation - Improves retrospectives and SLA reporting by aligning PagerDuty events with metric data ### Suppress PagerDuty Alerts During Grafana-Scheduled Maintenance Windows Planned maintenance, deployments, or load tests shouldn't flood your PagerDuty queue with spurious incidents. With tray.ai, when a Grafana silence or maintenance window is created, an automated workflow simultaneously sets a PagerDuty maintenance window for the affected services, preventing unnecessary pages to on-call engineers. When the Grafana silence expires, the PagerDuty maintenance window lifts automatically. Both systems stay consistent without anyone needing to update them separately. - Prevents on-call engineers from being paged during known maintenance periods - Keeps Grafana silences and PagerDuty maintenance windows in sync - Reduces alert noise that erodes trust in monitoring systems over time ### Generate Weekly Incident and Alert Trend Reports Combining Grafana's metric history with PagerDuty's incident data gives operations leaders a complete picture of system reliability. tray.ai can run a scheduled workflow that queries PagerDuty for weekly incident counts, MTTR, and responder data, then pushes summary statistics as annotations or data points into a Grafana dashboard or a connected reporting tool. Teams get visibility into alert volume trends, recurring failure patterns, and the operational cost of reliability issues — without manual data extraction. - Automates weekly SRE and ops reliability reporting across both platforms - Surfaces recurring alert patterns that point to systemic issues needing permanent fixes - Combines infrastructure health metrics with incident response KPIs in a single view ## Templates ### Grafana Alert Firing → Create PagerDuty Incident Monitors an incoming Grafana webhook for alert state changes and automatically creates a structured PagerDuty incident with severity mapping, affected service, alert labels, and a link back to the originating Grafana panel whenever an alert transitions to the Firing state. Connectors used: Grafana, PagerDuty ### Grafana Alert Resolved → Auto-Resolve PagerDuty Incident Listens for Grafana alert resolution events and automatically sends a resolve action to PagerDuty using the stored incident ID, closing the incident and appending a resolution note with the recovery timestamp and Grafana alert name. Connectors used: Grafana, PagerDuty ### Route Grafana Alerts to PagerDuty Services by Label Inspects Grafana alert labels and annotations to dynamically route incoming alerts to the correct PagerDuty service and escalation policy, supporting multi-team on-call environments where different services own different infrastructure domains. Connectors used: Grafana, PagerDuty ### PagerDuty Incident Acknowledged → Write Grafana Annotation Triggers when a PagerDuty incident is acknowledged or resolved and writes a corresponding time-stamped annotation to the relevant Grafana dashboard panel, creating a persistent operational record overlaid on metric visualizations. Connectors used: PagerDuty, Grafana ### Sync Grafana Silence → PagerDuty Maintenance Window Detects when a Grafana alert silence is created or updated and automatically creates a matching PagerDuty maintenance window for the specified services, then removes the window when the Grafana silence expires. Connectors used: Grafana, PagerDuty ### Weekly PagerDuty Incident Summary → Grafana Annotation Dashboard Runs on a weekly schedule to pull incident counts, MTTR, and top alerting services from PagerDuty, then pushes summary annotations into a designated Grafana operations dashboard so teams can track reliability trends over time. Connectors used: PagerDuty, Grafana ## Challenges Tray.ai solves ### Alert Payload Structure Inconsistency Across Grafana Versions Grafana's alerting system changed substantially between legacy alerting and the Unified Alerting engine introduced in Grafana 8+, resulting in very different webhook payload formats. Teams running different Grafana versions or migrating from legacy to unified alerting run into broken integrations when payload field names and structures change unexpectedly, causing missed PagerDuty incidents or malformed alert data. **How Tray.ai helps:** tray.ai's visual workflow builder lets teams build conditional data transformation logic that detects the incoming payload format and normalizes it to a consistent structure before passing it to PagerDuty. Field mapping and JSONPath expressions can be updated in the tray.ai interface without redeploying code, so adapting to a Grafana version change takes minutes rather than a sprint. ### Deduplicating Alerts to Prevent PagerDuty Incident Storms When a single infrastructure failure triggers multiple correlated Grafana alerts — a database outage cascading into application errors, latency spikes, and health check failures — each alert can independently create a separate PagerDuty incident, overwhelming on-call engineers with duplicate pages for what is effectively one root cause. Alert storms like this erode trust in the monitoring system and slow incident response. **How Tray.ai helps:** tray.ai workflows handle deduplication by using the Grafana alert fingerprint or a shared label value as a PagerDuty dedup_key when calling the Events API, so multiple correlated alerts collapse into a single PagerDuty incident. tray.ai's built-in data store tracks active fingerprints so the workflow updates an existing incident rather than creating a new one. ### Maintaining Bidirectional Lifecycle Sync Without Duplicate Actions Keeping Grafana alert states and PagerDuty incident statuses synchronized in both directions is genuinely tricky. Resolving an incident in PagerDuty shouldn't re-trigger a Grafana alert, and a Grafana recovery event shouldn't close an incident that was manually escalated by an engineer for further investigation. Without careful state management, bidirectional workflows can enter feedback loops or overwrite deliberate human actions. **How Tray.ai helps:** tray.ai's workflow logic supports conditional branching and state checks before taking any action. Workflows can query the current PagerDuty incident status before resolving it, skipping resolution if the incident has been manually escalated or moved to a different status. tray.ai's data store provides lightweight state persistence to track which actions were system-initiated versus human-initiated. ### Mapping Grafana Alert Severity to PagerDuty Escalation Policies at Scale Large organizations can have dozens of Grafana alert rules with varying severity labels and hundreds of PagerDuty services with distinct escalation policies. Manually maintaining a mapping between these two systems is error-prone and time-consuming, and a misconfigured routing rule can mean a critical production alert goes to the wrong team or gets assigned the wrong urgency level. **How Tray.ai helps:** tray.ai lets teams define routing logic in a centralized workflow using lookup tables, conditional branches, or even a connected Google Sheet or Airtable as a dynamic routing configuration source. When routing rules change, operations teams update the configuration source rather than rebuilding the workflow, making large-scale routing management both scalable and auditable. ### Handling Grafana API Authentication and Rate Limits in High-Volume Environments In high-traffic production environments, Grafana webhooks may fire hundreds of alerts per hour during degraded conditions. Programmatically fetching dashboard snapshots or annotation data for each incident can quickly exhaust Grafana API rate limits or run into authentication token expiry, leaving incomplete incident context in PagerDuty. **How Tray.ai helps:** tray.ai includes built-in retry logic, configurable request throttling, and error handling steps that gracefully manage API rate limit responses with exponential backoff. Authentication credentials for both Grafana and PagerDuty are stored securely in tray.ai's credential manager and can be rotated without modifying workflow logic, so high-volume alert scenarios don't result in data loss or broken integrations. ## Learn more - Intelligent Integration: https://tray.ai/platform/intelligent-ipaas/ - Merlin Agent Builder: https://tray.ai/platform/merlin-agent-builder/ - Agent Gateway for MCP: https://tray.ai/platform/agent-gateway/ - Book a demo: https://tray.ai/contact/