# Databricks + Datadog integration > Bring your lakehouse analytics and observability stack together to catch failures faster, cut downtime, and keep data workflows running at full speed. **Canonical page:** https://tray.ai/connectors/databricks-datadog-integrations/ **Databricks connector:** https://tray.ai/connectors/databricks-integrations/ **Databricks documentation:** https://tray.ai/documentation/connectors/service/databricks **Datadog connector:** https://tray.ai/connectors/datadog-integrations/ **Datadog documentation:** https://tray.ai/documentation/connectors/service/datadog ## Overview Databricks handles some of the most complex data engineering and machine learning workloads in the enterprise. Datadog gives you real-time observability across infrastructure, applications, and services. Together, they're a natural pairing for data teams who need full visibility into the health and performance of their pipelines, notebooks, and ML models. Integrating Databricks with Datadog means every job failure, resource spike, or data quality anomaly gets surfaced, tracked, and acted on — no manual log reviews, no ad hoc alerting. Data pipelines are only as reliable as your ability to monitor them. When Databricks jobs fail silently, clusters autoscale unexpectedly, or ML model performance degrades, the downstream impact is real: broken dashboards, stale reports, and expensive reprocessing runs. Connecting Databricks to Datadog through tray.ai lets data engineering and operations teams automatically stream job run metrics, cluster health data, and notebook execution statuses into Datadog dashboards and alert workflows. That closes the gap between your lakehouse and your monitoring stack. Teams can correlate infrastructure events with pipeline failures, enforce SLAs on data freshness, and kick off automated remediation workflows from a single observability plane. ## Use cases ### Automated Databricks Job Failure Alerting in Datadog When a Databricks job or pipeline run fails, tray.ai automatically pushes a structured event and error details into Datadog, triggering an alert to the on-call engineer. Failures surface in seconds rather than getting discovered hours later during downstream reporting. Teams can correlate job failures with concurrent infrastructure events to find root causes faster. - Reduce mean time to detect (MTTD) for failed Databricks jobs - Centralize all pipeline failure events alongside infrastructure alerts in Datadog - Eliminate manual log polling and delayed failure discovery ### Databricks Cluster Cost and Resource Monitoring Stream Databricks cluster utilization metrics — DBU consumption, worker node counts, autoscaling events — into Datadog as custom metrics. Data platform teams can build Datadog dashboards that track spend against budget thresholds and get proactive alerts when clusters exceed defined resource limits. FinOps and engineering teams share a single view of cost efficiency. - Prevent runaway cluster costs with real-time DBU consumption alerts - Visualize autoscaling behavior alongside application performance metrics - Let FinOps teams act on cost anomalies without accessing Databricks directly ### ML Model Performance Drift Detection and Alerting As Databricks ML models serve predictions in production, tray.ai can periodically extract model evaluation metrics and feature drift scores from MLflow and forward them to Datadog as custom metrics. When model accuracy drops below a configurable threshold, Datadog fires an alert to the ML engineering team. The feedback loop between model serving and operational monitoring stays closed. - Detect and alert on ML model performance degradation in near real-time - Track model health alongside the infrastructure it runs on in a single pane - Cut the time between drift detection and model retraining trigger ### Data Pipeline SLA Monitoring and Breach Notification Define SLA windows for Databricks pipeline runs — nightly ETL jobs that must complete before business dashboards refresh, for example. tray.ai monitors job completion times and automatically posts SLA compliance events to Datadog, triggering escalations when pipelines run late. Data engineering and BI teams both get visibility into data freshness SLAs without building custom tooling. - Enforce data freshness SLAs with automated monitoring and escalation - Give BI and analytics stakeholders confidence in report accuracy - Retire manual SLA tracking spreadsheets and status update meetings ### Databricks Audit Log Ingestion for Security Monitoring Continuously pull Databricks audit logs — user access events, cluster creation, data access patterns — into Datadog Log Management via tray.ai. Security and compliance teams can build Datadog detection rules on top of Databricks activity, identifying anomalous access patterns or policy violations in real time. No custom log pipelines needed to meet zero-trust data governance requirements. - Centralize Databricks audit events in Datadog for unified security monitoring - Trigger Datadog security signals on suspicious Databricks access patterns - Speed up compliance reporting with queryable, structured audit log data ### Incident-Driven Databricks Job Retry Orchestration When Datadog detects a recoverable infrastructure issue — a transient cloud provider outage or an overloaded worker node — tray.ai listens for the resolved event and automatically triggers a retry of the affected Databricks job. This self-healing pattern cuts the need for manual intervention during overnight or weekend pipeline runs. Teams wake up to completed jobs rather than stalled pipelines. - Automate job retries triggered by Datadog incident resolution events - Reduce on-call burden for recoverable, transient pipeline failures - Improve data availability SLAs through self-healing pipeline orchestration ### Unified Data Platform Health Dashboard Population Aggregate Databricks job run summaries, cluster health snapshots, and Delta Lake table statistics into Datadog via scheduled tray.ai workflows. These metrics populate a centralized data platform health dashboard used by engineering leadership and data ops teams. Stakeholders get a real-time view of platform reliability, throughput, and capacity without needing Databricks workspace access. - Give leadership real-time visibility into data platform health via Datadog - Reduce ad hoc status requests to data engineering teams - Consolidate lakehouse metrics with broader infrastructure KPIs in one dashboard ## Templates ### Databricks Job Failure → Datadog Event + Alert Monitors all Databricks job runs and automatically creates a Datadog event with full error context whenever a job enters a failed state. The event triggers a configured Datadog monitor alert, notifying the on-call team via their preferred channel. Connectors used: Databricks, Datadog ### Databricks Cluster Metrics → Datadog Custom Metrics Pipeline Runs on a recurring schedule to collect Databricks cluster utilization data — active workers, DBU usage, autoscaling events — and submits them to Datadog as custom metrics for dashboarding and threshold alerting. Connectors used: Databricks, Datadog ### MLflow Model Metrics → Datadog Performance Monitoring Extracts the latest model evaluation metrics from Databricks MLflow experiment runs and forwards them to Datadog as time-series metrics, enabling model performance trend monitoring and degradation alerting directly within Datadog. Connectors used: Databricks, Datadog ### Datadog Incident Resolved → Databricks Job Retry Listens for Datadog incident resolution webhooks and automatically triggers a retry of a pre-mapped Databricks job, enabling self-healing pipeline orchestration after recoverable infrastructure failures. Connectors used: Datadog, Databricks ### Databricks Audit Logs → Datadog Log Management Continuously syncs Databricks audit log entries into Datadog Log Management on a scheduled basis, enabling security teams to build detection rules, anomaly monitors, and compliance reports on top of Databricks activity data. Connectors used: Databricks, Datadog ### Databricks Pipeline SLA Monitor → Datadog Escalation Workflow Tracks expected completion windows for Databricks Delta Live Tables or job pipelines and creates a Datadog event and alert escalation when a pipeline misses its SLA deadline, notifying stakeholders before downstream systems are impacted. Connectors used: Databricks, Datadog ## Challenges Tray.ai solves ### Bridging Databricks API Complexity with Datadog's Metrics Model Databricks exposes job, cluster, and MLflow data through multiple distinct REST APIs with different authentication patterns and pagination models, while Datadog expects metrics and events in specific structured formats with proper tagging. Building and maintaining this translation layer by hand requires significant custom engineering effort. **How Tray.ai helps:** tray.ai's pre-built Databricks and Datadog connector operations handle API authentication, pagination, and data transformation out of the box. Teams can map Databricks API responses to Datadog metric and event schemas using tray.ai's visual data mapper — no custom code required. ### Maintaining Sync Reliability for High-Frequency Monitoring Data Cluster metrics and job status data need to flow into Datadog frequently — sometimes every few minutes — to be useful for on-call teams. A brittle polling workflow that fails silently under load or during API rate limiting creates dangerous gaps in observability coverage exactly when teams need it most. **How Tray.ai helps:** tray.ai workflows include built-in error handling, automatic retry logic, and configurable rate limiting to keep data flowing reliably even during API throttling events. Workflow execution logs and failure alerts mean teams always know if their monitoring pipeline itself needs attention. ### Handling Dynamic Databricks Cluster and Job Topologies Enterprise Databricks environments don't sit still — clusters spin up and down, new jobs get added frequently, and workspace configurations vary across environments. A static integration that hardcodes job IDs or cluster names goes stale fast and misses new pipelines that need monitoring coverage. **How Tray.ai helps:** tray.ai workflows can dynamically discover active Databricks jobs and clusters at runtime using list API calls, applying monitoring logic to all discovered resources rather than a hardcoded set. New pipelines get automatically included in monitoring coverage without any workflow updates. ### Correlating Databricks Failures with Broader Infrastructure Context Databricks job failures are rarely isolated — they're often caused by underlying cloud infrastructure events, network issues, or upstream data quality problems. Without correlating Databricks events with the rest of the infrastructure stack, data teams end up investigating symptoms rather than root causes. **How Tray.ai helps:** By routing Databricks events into Datadog alongside infrastructure metrics, tray.ai lets teams use Datadog's correlation and anomaly detection features across the full stack. Tagging Databricks events with cluster IDs, cloud regions, and environment names enables precise cross-signal correlation in Datadog. ### Securing Credentials Across Databricks and Datadog Environments Integrating Databricks with Datadog means managing multiple sensitive credentials — Databricks personal access tokens or service principal OAuth tokens, plus Datadog API and application keys — across dev, staging, and production environments. Hardcoding or mismanaging these secrets puts both platforms at risk. **How Tray.ai helps:** tray.ai stores all connector credentials in an encrypted secrets vault and supports environment-specific credential configurations, so production Databricks and Datadog credentials never appear in workflow definitions or logs. Role-based access controls in tray.ai restrict which team members can view or modify connector authentication settings. ## Learn more - Intelligent Integration: https://tray.ai/platform/intelligent-ipaas/ - Merlin Agent Builder: https://tray.ai/platform/merlin-agent-builder/ - Agent Gateway for MCP: https://tray.ai/platform/agent-gateway/ - Book a demo: https://tray.ai/contact/