Part of building a new analytical environment involves an AI-assisted pass through legacy pipelines to propose dbt models, tests, and documentation. It is genuinely fast: code archaeology that once took days can surface a useful first draft in an afternoon.
It is also a specific kind of fast that requires a specific kind of review.
A plausible rule that was only a coincidence
While generating documentation for a recent transformation, an agent inferred a business rule from a naming pattern. Records with a particular status code appeared always to originate from one source system, so the agent proposed documenting that relationship as a rule and writing a test to enforce it.
The correlation was real in the historical sample. It was not a rule. It was a coincidence of how one source happened to be configured during that period. Nothing enforced it, nothing guaranteed it would continue, and a newer source system already in the pipeline did not follow the pattern.
A test would have turned an observation into a guarantee. It would also have started failing as soon as the new data arrived.
The output was not careless. It was a reasonable inference from the evidence available. That is what makes this mistake harder to catch than a broken query: within its evidence, it reads as correct.
The review test: decision, constraint, or correlation?
Syntax checks and passing tests catch almost none of this. Every proposed rule needs three questions:
- Decision: Did an accountable person or process explicitly choose this behavior?
- Constraint: Does a system, contract, or validated control require it to remain true?
- Correlation: Is it merely what the available historical sample happened to contain?
Code alone often cannot distinguish among them. That requires knowing whether a pattern reflects intent, enforcement, or accident—the judgment that remains with the person accountable for the model’s meaning.
What responsible acceleration looks like
Agents work on bounded problems with relevant context, explicit standards, and reviewable outputs. Every generated rule, test, and definition is reviewed specifically for whether it represents a real decision or an observed coincidence. Nothing reaches the shared model without a person accountable for that answer.
The speed is real and worth having. It does not change who is responsible for knowing what the business meant.