Data operations integrations and automations

Getting data out of the systems that hold it and back into the ones that need it, with history kept and row counts reconciled.

5 guides  ·  Built with Tray Headless

What this work has in common

The warehouse is only as trustworthy as the least careful process writing to it. These guides cover getting data out of operational systems without losing deletes or ordering, checking it for the failures that do not raise an error, and sending modelled data back to the systems where people work.

The common theme is that a pipeline reporting success and a pipeline being correct are different things. A job can run green every night while a source schema change drops a column, a deleted account keeps counting, or a merge in the CRM leaves two rows for the same customer. The checks that catch those cases are row counts, freshness and tests on the fields decisions depend on, run every time and routed to a person who owns the source.

The usual systems are a warehouse such as Snowflake, a transformation layer such as dbt, a BI tool such as Looker, the CRM and the help desk as sources, and the same CRM and help desk as destinations for reverse ETL.

The guides cover CRM to warehouse sync, reverse ETL, master data and a customer 360 record, data quality monitoring, and schema change management. Change data capture lives under platform engineering, because it is a pattern several teams build on.

Where data operations breaks

Syncing current state only

A sync that copies today's version of every record cannot answer what the pipeline looked like last quarter. Capture changes as history and model current state on top of it.

Losing deletes

Most API extracts return the records that exist. A record deleted in the source quietly stays in the warehouse and keeps counting in every report. Capture deletes explicitly, or reconcile row counts on a schedule.

A failed test nobody owns

A data quality alert posted to a shared channel is read by nobody in particular. Route each test to the owner of the source, and say which dashboards and syncs depend on the table that failed.

Overwriting a person's edit

Reverse ETL that writes a modelled value over a field a rep just changed by hand loses that change without telling anyone. Decide which fields the warehouse owns, and leave the rest alone.

The data operations guides

Each one is the design, the sequence it runs in, the prompts that build it, and what changes when it runs in production.

How to build a CRM to warehouse sync

Capture history instead of current state, handle deletes and field changes, land raw then model, and prove the row counts.

Integration

SalesforceSnowflakeSlackLooker

How to build reverse ETL from the warehouse

Push modelled data back into the tools people work in, syncing deltas, respecting field ownership, and never overwriting a human.

Integration

SnowflakeSalesforceMarketoZendesk

How to build a customer 360 and master data sync

Pick a survivorship order per attribute, resolve identity on more than email, and publish a golden record every system can point at.

Integration

SalesforceNetSuiteSnowflakeClearbit

How to build data quality monitoring

Test what breaks decisions, alert the owner rather than a channel, and say what depends on a failure.

Automation

SnowflakeLookerSlackOkta

How to build schema change management

Detect a source change before it breaks a model, resolve what depends on it, and tell the owner in time to act.

Automation

SalesforceSnowflakeLookerSlack

The systems involved

The applications these guides read from and write to, most used first. Each links to its connector page.

Connections these guides build

Solutions for data operations

The solution pages for this work, with the customer stories behind them.

Guides for other teams