Stream Real-Time Data from Kafka into Databricks at Scale

Continuously polls a configured Kafka topic, batches incoming messages, and writes them to a target Delta Lake table in Databricks using the REST API — including error handling, dead-letter routing, and offset commit confirmation.

Steps:

• Subscribe to a specified Kafka topic and collect messages up to a configurable batch size or time window
• Transform and validate the message payload, routing malformed records to a dead-letter queue
• Write the valid batch to a Databricks Delta Lake table via the Databricks REST API and commit the Kafka offset

Connectors Used: Kafka, Databricks

Listens for a qualifying message on a Kafka topic, extracts relevant parameters from the event payload, and uses those parameters to trigger a specific Databricks job run — then monitors the run status and publishes the result back to Kafka.

Steps:

• Consume an incoming Kafka message and apply filter conditions to confirm it should trigger a job
• Call the Databricks Jobs API to start the configured job, passing extracted message fields as parameters
• Poll the Databricks run status until completion, then publish a result event back to an output Kafka topic

Connectors Used: Kafka, Databricks

Monitors Databricks job and pipeline run statuses on a scheduled interval and publishes a structured failure event to a designated Kafka topic whenever a run exceeds duration thresholds or ends in an error state.

Steps:

• Query the Databricks Jobs API on a scheduled interval to retrieve recent run statuses and durations
• Evaluate each run against configurable success and SLA thresholds to identify failures or slow runs
• Publish a structured alert message to a Kafka topic for downstream consumers such as incident management systems

Connectors Used: Kafka, Databricks

Reads events from a Kafka topic, constructs the required feature payload, calls a Databricks Model Serving endpoint for real-time inference, and writes the scored output — including original event plus prediction — back to a Kafka results topic.

Steps:

• Consume messages from the input Kafka topic and extract the feature fields required by the ML model
• POST the feature payload to the Databricks Model Serving REST endpoint and capture the prediction response
• Merge the original event with the model prediction and publish the enriched record to the output Kafka topic

Connectors Used: Kafka, Databricks

Aggregates Kafka consumer group lag metrics and Databricks cluster utilization data on a recurring schedule, then writes a combined health snapshot to a Delta table for use in operational dashboards and alerting workflows.

Steps:

• Retrieve current consumer group lag values from the Kafka Admin API for monitored topic partitions
• Fetch active cluster utilization and job queue depth from the Databricks Clusters and Jobs APIs
• Write the combined health record to a Delta Lake monitoring table and trigger an alert if any metric exceeds its threshold

Connectors Used: Kafka, Databricks

Orchestrates a controlled replay of archived Kafka messages into a Databricks Delta table, managing checkpointing, rate limiting, and cluster scaling instructions to complete the backfill without impacting live pipeline capacity.

Steps:

• Accept a backfill configuration specifying the Kafka topic, partition offsets, time range, and target Delta table
• Read message batches from the specified historical offset range, writing each batch to Databricks and logging the last successful offset as a checkpoint
• On completion or interruption, publish a backfill status summary to a Kafka audit topic and release cluster resources

Connectors Used: Kafka, Databricks