Data Quality That Fails Closed: A Reconciliation Program for Agencies

The problem nobody budgets for

Agencies spend heavily on dashboards, data lakes, and increasingly on AI pilots, but very little on the unglamorous work of checking whether the data feeding those systems is actually correct. A field office submits case records with a typo'd SSN. A legacy mainframe export drops leading zeros on a ZIP code. Two systems disagree on whether a grant closed in FY23 or FY24. None of this looks like a crisis in the moment. It becomes one when a report to an oversight committee, a benefits eligibility check, or a model prediction is built on top of it.

NIST's AI Risk Management Framework is explicit that data quality is a governance function, not an afterthought: the 'Map' and 'Measure' functions both assume you know where your data came from, how complete it is, and what its known limitations are. If you cannot answer those questions, you cannot honestly answer whether an AI system built on that data is trustworthy. The same logic applies without any AI in the picture. Bad inputs produce bad decisions whether a human or a model is making the call.

What a real validation program looks like

A validation program is not a single tool. It is a layered set of checks applied at ingestion, at transformation, and before publication, each catching a different class of problem. The goal is to make data quality a property of the pipeline, not a task someone does manually after the fact in a spreadsheet.

Structural rules catch the obvious: required fields present, correct data types, values within valid ranges, referential integrity between tables (a case ID in one system actually exists in the source system). Domain rules encode agency-specific logic: an award amount cannot exceed the appropriation ceiling for that program, a benefit start date cannot precede an eligibility determination date, a facility code must map to an active site. Cross-source rules compare the same fact reported in two systems and flag divergence rather than silently picking one.

None of these rules are exotic. What matters is that they run automatically, on every load, and that failures are visible to a human rather than quietly logged and ignored.

Reconciliation: making disagreeing systems tell the truth

Most agencies run parallel systems of record for the same entities: a case management system and a financial system both track the same grant, a state eligibility system and a federal reporting system both track the same recipient. Reconciliation is the process of comparing these sources on a schedule, quantifying how much they disagree, and routing the disagreement to someone who can resolve it rather than letting each downstream report pick whichever number is convenient.

A working reconciliation process has three parts. First, matching keys that let you confidently say record A in System 1 is the same real-world entity as record B in System 2, even when identifiers are formatted differently. Second, tolerance thresholds, because not every penny-level rounding difference needs a human review, but a five-figure variance in reported obligations does. Third, an aging report: unresolved discrepancies that sit open for 30, 60, or 90 days need to escalate, because stale reconciliation breaks are exactly the kind of finding an IG audit surfaces later.

This is also where a lot of well-intentioned data warehouses fail quietly. They load from both systems, present both numbers side by side or, worse, average them, and nobody is accountable for the discrepancy itself. Reconciliation makes the discrepancy the thing you manage.

Anomaly detection: catching what rules cannot

Rules catch known failure modes. Anomaly detection catches the ones you did not think to write a rule for: a sudden 40 percent spike in a category of transactions, a field that has reported the same default value for six months because an upstream form changed and nobody noticed, a distribution shift in an input feature that quietly degrades a model's predictions without throwing an error.

This does not require exotic machine learning. Simple statistical process control, control charts, and volume/velocity checks against historical baselines catch the majority of real-world data drift. Reserve more sophisticated anomaly detection models for high-volume, high-stakes pipelines where the pattern of normal behavior is complex enough that static thresholds produce too many false positives or misses.

The output of anomaly detection should never be a silent adjustment. It should be a flag, timestamped and attributed, that a data steward reviews. Auto-correcting anomalies without review is how a formatting fix for one recipient's address quietly becomes a policy about how all addresses in that ZIP code are treated.

Fail-closed pipelines: the design choice that actually matters

The single highest-leverage architectural decision in a government data pipeline is whether it fails open or fails closed. A fail-open pipeline, when it hits bad data it cannot validate, passes it through anyway with a warning logged somewhere nobody reads. A fail-closed pipeline stops, quarantines the record or batch, and requires a human decision before that data reaches a report, a dashboard, or a model's training set.

Fail-closed costs something real: latency, and occasionally an angry program office asking why yesterday's numbers are not in today's report. That cost is the entire point. It converts an invisible data quality failure into a visible operational one, at the moment it is cheapest to fix, rather than months later when it surfaces in a GAO finding or an incorrect eligibility determination. NIST 800-171 and FedRAMP control families already push agencies toward this posture for security events; the same discipline belongs in the data quality layer, because a bad record reaching a live system is its own kind of incident.

Fail-closed only works with clear ownership: someone specific has to be paged when the pipeline halts, with a defined SLA for triage. Without that, teams disable the checks instead of fixing the data, which is worse than not having the checks at all.

How VAERESOURCE builds this in practice

We build validation, reconciliation, and anomaly detection as first-class pipeline components, not bolt-on scripts, so agencies can show an auditor exactly which rule caught which record and when. Every pipeline we deliver is fail-closed by default: bad data is quarantined with a full audit trail, not silently passed through or silently dropped.

Because a rule set is only as good as the humans who maintain it, we keep a human-in-the-loop at the points that matter most: reconciliation breaks, anomaly flags, and any rule override. That is the same posture NIST's AI RMF asks for in the systems built on top of this data, and it is a lot easier to satisfy when the underlying data pipeline was designed for scrutiny from day one rather than retrofitted for it after an incident.