Data quality monitoring: who sees bad data first

The revenue dashboard has been wrong for three weeks. A load failed partway on a Sunday, the table has not been refreshed since, and every number on the dashboard is accurate to the end of that Sunday. The first person to notice is the VP presenting it to the board.

There was an alert. It went to the #data-alerts channel on the first day, along with 40 others that week, and nobody owns that channel.

Monitoring quality and freshness is the last of the seven steps of data integration. It’s the step that tells you whether the other six worked.

What is data quality monitoring?

Data quality monitoring runs checks on the tables that drive decisions after every load: whether the data is fresh, complete and plausible. A good setup gives every table an owner, says what breaks downstream when a check fails, and treats freshness as the most important check.

Why bad data reaches the dashboard first

1. Alerts go to a channel. An alert with no owner is read by everybody and acted on by nobody. Within a month the channel is muted.
2. Checks cover what is easy to measure. Teams check every column for nulls and every table for row counts, and generate hundreds of alerts that don’t matter.
3. Freshness isn’t checked. A stale table looks correct. Every value in it was true at some point, so nothing looks wrong.
4. The alert doesn’t say what breaks. “Row count anomaly on orders” means nothing to the person who reads it. “The revenue dashboard is wrong” is an incident.
5. Checks are never deleted. Checks nobody acts on stay forever and bury the ones that matter.

What it costs

The cost is trust. Once a board deck goes out with a wrong number, every number after it gets questioned, and analysts spend their time defending figures instead of explaining them.

It also costs every decision made in the meantime. Three weeks of stale data is three weeks of forecasts, targets and spend decisions made on last month’s picture.

What to fix first

1. Start from the decisions. List the dashboards leadership and finance use, and the tables behind them. Monitor those first.
2. Check freshness on every one of those tables. Alert when a table has not loaded on schedule, before anyone looks at it.
3. Give every table an owner. A named person, who receives the alert directly.
4. Say what breaks. Include the dashboards and reports that read the table in every alert.
5. Write checks against what would change a decision. A missing region, a total that moves by more than a set amount, a key with no match.
6. Delete checks nobody acts on. Track which alerts led to a fix, and remove the rest.

The guide to building data quality monitoring has the prompts for each step. For the alerting side, reducing error fatigue with an error reporting dashboard covers how we grouped workflow errors at Tray.ai.

Where it fits

Monitoring catches what the other steps miss. A CDC pipeline that drops deletes, a file feed that arrives truncated and a renamed field all produce a successful run, and a reconciliation or freshness check is often the first place they show up.

For all seven steps, see the data integration process.