Somewhere around month nine of a stalled modernization, a status deck will show a green checkmark next to "conversion complete." The COBOL has been translated. The new codebase builds. It runs. Nobody in the room asks the only question that matters, which is whether it does the same thing the mainframe does — because "it compiles" has quietly become the milestone, and behavior was never on the schedule as something separate to prove.
This is not a semantic nitpick. It is the single most common way a modernization budget gets spent on the wrong thing. Compiling is a syntax check. It tells you the target language parsed the input without choking. It tells you nothing about whether a rounding rule survived, whether an edge case in a decades-old batch job got dropped in translation, or whether the new system produces the same output as the old one on the transaction that actually matters — the one from three years ago that hit a branch nobody has looked at since the programmer who wrote it retired.
Why artifacts are the easy thing to bill for
Artifacts are countable. Lines of code converted, modules migrated, test cases passing, sprints closed — these are the numbers that go in a status report, because they're unambiguous and they always go up. Behavior is not countable in the same way. You cannot put "matches production on every code path we haven't thought to check yet" on a burndown chart. So the incentive, on both sides of the contract, quietly shifts toward the thing that's easy to measure and away from the thing that was actually being paid for.
A vendor billing time and materials has no structural reason to prefer "provably correct" over "looks done." A fixed-price vendor has an active reason to prefer it: a build that compiles and passes a curated set of test cases can be presented as delivery, and delivery triggers payment. Whether that build reproduces thirty years of accumulated business logic — including the parts nobody remembers the reason for, which is most legacy financial and payroll logic — is a question that only surfaces later, usually during whatever pass for parallel run, and by then the artifact has already been accepted.
What "paying for behavior" actually requires
Paying for behavior instead of artifacts means the acceptance criteria has to be defined before the work starts, in terms that can't be gamed by a clean build:
- Byte-exact comparison against real production data, not synthetic test fixtures the vendor generated itself. A test suite the same team wrote to validate its own conversion is not independent evidence — it's a claim wearing a green checkmark.
- Coverage of the paths nobody remembers, not just the paths in the requirements doc. Legacy systems accrete edge-case handling for regulatory changes, one-off fixes, and business rules that predate anyone currently on the project. A conversion that only reproduces documented behavior silently drops the undocumented behavior, and undocumented behavior is usually where the money-handling logic lives.
- A reproducible run, not a demo. If parity can only be shown once, in a controlled walkthrough, with a specific dataset the vendor picked, it isn't proof — it's theater. Real proof is something the buyer can re-run against a dataset the buyer picks.
- Negative controls. A comparison harness that would pass even against a deliberately broken build isn't measuring anything. If nobody has ever verified the test suite actually fails on a wrong answer, a passing suite tells you the harness works about as much as it tells you the conversion does.
The audit-office pattern
This is not a hypothetical risk. It's the pattern that shows up, repeatedly, in the public post-mortems that government audit offices publish after a modernization runs over budget or gets cancelled outright: a system reported as substantially complete, milestones marked delivered, and then a much later discovery — often during user acceptance testing that should have happened far earlier — that the "complete" system doesn't actually reproduce what the old one did. The gap between "it compiles and demos" and "it behaves correctly on the business's real data" is exactly where those projects lose their second and third years. The compiling part was never the hard part. It was just the part that was easy to report as done.
What to ask for instead
If you're the buyer, the fix isn't more process — it's a different question at every milestone. Not "does it build," not "does it pass the demo," but: show me the byte-for-byte comparison against a dataset I picked, on every path, including the ones you didn't know existed until the comparison ran. If a vendor can't produce that comparison on demand, at any point in the project, what you're being sold is an artifact — and artifacts are not the thing you're paying for. The system doesn't need to compile cleanly to be worthless to the business. It needs to compile cleanly and be wrong in a way nobody caught, which is exactly what "it compiles" as a milestone guarantees you'll eventually find out the hard way.