Provenance Isn't Evaluation
Provenance records how a requirement was produced. Evaluation judges whether it is right. Much of what has been added under AI governance this year is provenance. We think a check that provides independent, repeatable evidence of quality needs four properties. It
- sits outside the generating step,
- applies a rule that existed before the artifact,
- gives the same finding when run again, and
- leaves a result an outsider can inspect.
Knowing exactly how a requirement was produced tells you nothing about whether it is correct. That gap sits underneath most of this year's governance work, in our corner of the market and in the ones next to it. Governance is the umbrella term, and it is far from the only label in circulation. We are also seeing engineering intelligence, context layers, and traceability now pitched as a live record of change rather than the static link graph the term has meant historically.
The common move across all of them has been to record more about the generating step, capturing which model produced a requirement, what it was asked, which sources it drew on and who approved the result. That is real work and an improvement on what came before. They share the same failure mode. A system can govern itself, know its own context, keep its own live record, and still have the generation of the artifact and the evaluation of the artifact happening in the same place. A thorough live trace of a self-generating, self-evaluating system is just a more detailed self-report.
Regulation is another driver, moving quickly as rules are updated to account for AI in regulated industries. We track those changes closely ourselves, and the need behind them is genuine. A feature built for a particular regulation records compliance with that regulation. It does not evaluate the requirement, and compliance with one regulation does not establish compliance with another.
A requirement can have perfect provenance and still be wrong. A history of how a requirement was produced describes what happened. It does not assess what was produced. A useful record goes beyond authorship and change. It lets you see the requirement at any point in its lifecycle, what changed, which issues were resolved, what state it was in, and the findings it returned against defined quality rules. That record connects the history of a requirement to its evaluation. But the two often get filed under the same heading. They are not the same job, and evaluation only works as an independent check when it is kept separate from the system that generated it.
This is what the engineers we talk to describe most often, usually in plain terms. One systems engineer at a medical device manufacturer put it this way:
“You have the generative aspect to generate requirements, but then you also have the judge to actually measure the quality. Those two should be two different things.”
The mechanism that satisfies the four properties above is the Separation Principle, which puts evaluation outside the system that produced the artifact. “Outside” is not only the human reviewer. Humans can still participate, and often should. Separation is the starting point, not the whole test. The check must also apply rules defined before the artifact exists, return the same finding when run again, and leave a result an outsider can inspect. A human reviewer can provide independence from the generator; independence alone does not guarantee repeatability. A deterministic check inside the generating system can provide repeatability without independence. The four properties have to hold together. The Requirements Operating Model, the framework QRA is currently publishing in chapter form, is built on this principle throughout. Chapter 5 is the Governance Layer, where the evaluation side of the principle is developed in detail.
The strongest answer we see to this today is still a human reviewer. A person reads the output and a gate sits after generation, which comes closer to a check than any amount of logging. On the programs we have looked at it is often the only thing standing between a generated requirement and the record. A human verdict does not inherently guarantee the same finding when the review is repeated. Who was on the gate, how much time they had, and when the deadline fell can all shape the result.
The speed of generation has created a bottleneck in human review, especially where there are no agreed standards for the generated artifact. Breaux and Singhal, in a 2026 paper in Requirements Engineering, argue that as AI agents make implementation less expensive, engineering effort shifts toward expressing, curating, verifying and evaluating requirements.
The same pressure is visible in software engineering, where the effects of AI coding agents can be measured across hundreds of firms.
- Chen and Stratton, in a 2026 Harvard working paper, Artificial Intelligence in the Firm: Bottlenecks in Software Production, draw on engineering telemetry from 718 firms and report:
- a 49 percent increase in the time between submitting and merging a pull request after AI coding agent adoption
- a 35 percent increase in review comments per pull request
- pull-request time includes queueing and rework alongside active review.
- A 2026 research paper from JetBrains, in collaboration with Lund University, puts the shift in plain terms. The bottleneck is no longer generation; it is review. The paper argues that line-by-line diffing fails at the scale and heterogeneity of AI-generated changes, and that a reviewer cannot infer confidence or correctness from the polished appearance of the output.
- A systematic literature review by Cheng and colleagues, published in Software: Practice and Experience in 2026, synthesizes 238 papers on generative AI in requirements engineering. Challenges identified across the studies include:
- reproducibility (66.8 percent)
- hallucinations (63.4 percent)
- interpretability (57.1 percent)
- evaluation methods described as fragmented, with gaps in standardized benchmarking, and only 1.3 percent of the studies reaching production-level integration.
An engineering leader at a safety-critical infrastructure firm described this on a recent call.
“It looks very credible, what they’ve produced, but actually what they may be just generating spam. And now in the safety critical things we’re putting together, obviously that’s very dangerous. You can’t actually spot the spam anymore.”
The evidence from adjacent engineering disciplines points to the same problem. Human review is becoming a constraint as generated output increases. Where review depends on individual judgment rather than agreed standards and repeatable checks, producing more artifacts does not necessarily mean getting more of them vetted.
The pattern is not confined to engineering. A 2026 detection study classified approximately 20 percent of ICLR 2025 peer reviews and 12 percent of Nature Communications 2025 reviews as fully AI-generated. These are detector-based estimates, not verified authorship figures, and should be interpreted with caution. The study measures only reviews generated in full, so it also does not capture the larger share of reviews with partial AI involvement.
The question this raises is whether an independent check can itself be produced by a model. The same question runs in reverse. When the expectation of keeping pace falls on humans alone, with generation moving faster and no agreed standard for its output, more review is not the answer either. The four properties above apply either way. A model-produced check qualifies if it meets all four. One that fails any of them is a second generator, not a check.
Chapter 4 of the Requirements Operating Model, out in October, examines the layer where generation happens and why that layer cannot certify its own output. However complete the record of generation, it is not an evaluation. Chapter 5 develops the Governance Layer, where the Separation Principle operates.