How to build data quality monitoring

What matters here

• Write checks against what would break a decision, not against everything you can measure.
• Alert a person, never a channel. An alert with no owner gets read by everybody and acted on by nobody.
• Say what breaks downstream. "Row count anomaly on orders" is noise; "the revenue dashboard is wrong" is an incident.
• Freshness is the check that matters most. A stale table looks correct and is the hardest failure to notice.
• Track alerts that led to a fix. A check nobody ever acts on should be deleted instead of tuned.

Who this is for

You run data or analytics engineering. Dashboards are trusted until they are obviously wrong, and the gap between a pipeline breaking and somebody noticing is measured in weeks.

How it works in practice

From data going wrong to somebody fixing it, in order.

1. 1

   Checks are written from decisions backwards

   What number would somebody act on, and what would make it wrong.

2. 2

   Every table has a named owner

   Not a team. A check on an unowned table is a check nobody will action.

3. 3

   Freshness is checked first

   A stale table is the failure that looks correct and survives longest.

4. 4

   Failures resolve their blast radius

   Which dashboards, which models and which reverse ETL syncs depend on it.

5. 5

   The owner is alerted with what breaks

   Not with a check name and a threshold.

6. 6

   Downstream consumers are told or paused

   A dashboard built on failed data should say so instead of showing the number.

Paste these into Claude Code or Codex with the Tray Headless plugin installed. Each stage runs on its own. The systems named in them are the worked example rather than a requirement, and every prompt says so.

Once per project, run /tray-workflows:set-workspace to pick the workspace these build in. Point it at a sandbox first.

1. 1

   First, work backwards from decisions

   Testing everything measurable produces noise nobody can rank.

   Headless skills build-workflow tray-patterns

   Copy

   Use build-workflow. The systems in play are Snowflake, dbt and Looker,
   or whatever we run in those seats.
   
   Do not start by listing tables. Start with the decisions people make on
   this data: the revenue number in the board pack, the pipeline number in
   the forecast, the churn figure in the QBR.
   
   For each, trace back the tables it depends on, then write checks against
   what would make that specific number wrong.
   
   Checks written this way are few and they matter. Checks written by
   enumerating tables are many and they get muted.

2. 2

   Resolve an owner for every table

   An alert addressed to everybody is addressed to nobody.

   Headless skills build-workflow

   Copy

   Use build-workflow. Build an ownership register: table, owner, deputy,
   the decisions it feeds, and its expected refresh cadence.
   
   Resolve owners to real people through the directory, and flag any whose
   owner has left.
   
   Any table with no owner does not get checks. That sounds backwards and it
   is deliberate: a check on an unowned table produces alerts nobody
   actions, and after a fortnight everybody mutes the channel including for
   the tables that do have owners.
   
   Make unowned tables a visible list instead. That list is the actual
   problem and it is more useful than a hundred unactioned alerts.

   Refusing to monitor unowned tables is the single decision that keeps a monitoring programme alive past its first quarter.

3. 3

   Check freshness first, then correctness

   Stale data looks correct and is trusted.

   Copy

   Freshness is the most important check and the one most often missing.
   
   For every table, check it updated within its expected window. Compute the
   expected window from history rather than asking somebody to declare it,
   and account for weekends and month end where the pattern differs.
   
   Then correctness checks, sized to the decision:
   
     Row count against a range derived from history, not a fixed number
     Null rate on fields that must be populated
     Uniqueness on anything used as a key
     Referential integrity on joins that matter
     Distribution shifts on numeric fields that feed a headline figure
     Cross-system reconciliation, such as warehouse revenue against the ERP
   
   A stale table returns yesterday's answer confidently, and nobody
   questions a number that looks plausible.

4. 4

   Attach the blast radius to every failure

   A check name and a threshold is not an incident.

   Headless skills tray-gotchas

   Copy

   Use tray-gotchas, then resolve what depends on each table before
   alerting: which dashboards, which models, which reverse ETL syncs, and
   which downstream tables.
   
   Alert the owner with that list, not with a check name. "Row count anomaly
   on fct_orders" is a puzzle. "The revenue dashboard and the exec KPI view
   are showing wrong numbers, caused by fct_orders" is an incident somebody
   can size in seconds.
   
   Then act on it: pause any reverse ETL that would push failed data into
   the tools people work in, and mark affected dashboards as stale rather
   than letting them show a number.
   
   Pushing data that failed its own tests into a CRM is worse than pushing
   nothing, because it looks current and gets believed.

5. 5

   Measure whether checks lead to fixes

   A check nobody acts on should be deleted, not tuned.

   Copy

   Record an outcome for every alert: fixed, known and accepted, or false
   positive.
   
   Report: alerts by table and by check, the share that led to a fix, median
   time to acknowledge, checks that never fire, and checks that fire
   constantly.
   
   Delete anything with a low fix rate. A check that fires weekly and is
   dismissed weekly trains people to dismiss everything, including the one
   that mattered.
   
   Also report data incidents found by a person instead of by a check. That
   number is the honest measure of coverage, and it is the one that tells
   you which decision to work backwards from next.

6. 6

   Validate, then hand the register over

   Because ownership changes and checks follow the decisions.

   Copy

   Run the per-step schema checks and the whole-workflow audit before this
   touches production.
   
   Then open the same workflow in Tray Build so analytics engineering can
   maintain the ownership register, add checks and adjust thresholds in the
   visual canvas as the decisions people make on this data change.

What it connects to

Checks run where the data is and alerts have to reach a person.

Named systems are the ones most teams run, not the only ones that work. Each is an authentication in your Tray workspace, referenced by name, so the workflow never holds a credential. Where we have a connector page, the name links to it.

Questions people ask

Related guides