Every modernization vendor will show you a green test suite. Almost none of them will show you what generated the inputs. That gap is where parity theater hides, because a test suite built on synthetic data can pass at 100% and still tell you nothing about whether the replacement system behaves like the one it's replacing. Synthetic data is data the vendor's own team wrote, or generated from a schema, to exercise the paths the vendor's own team thought to exercise. It is, by construction, a test of whether the new system agrees with the vendor's mental model of the old one. It is not a test of whether the new system agrees with the old one.
Production data is different in a way that matters specifically because it's inconvenient. Thirty years of a mainframe system in daily use accumulates inputs nobody would ever choose to write by hand: fields that were repurposed mid-decade and now hold values that don't match their own definition, dates that predate the epoch the schema assumes, sentinel values some long-gone programmer used to mean "not applicable" that still flow through downstream logic today, packed-decimal fields that drifted out of their documented range because a validation routine upstream had a bug nobody ever fixed because the output still looked plausible. A synthetic test generator, however sophisticated, samples from what its author believes is possible. Production data contains what actually happened, including the parts nobody believed were possible until they found them in a batch file.
Why the theater works on buyers
Parity theater is effective precisely because a synthetic-data test suite is easier to build, easier to demo, and easier to make pass than a real one. A vendor can hand-pick clean, representative-looking cases, run them through both systems, get a match, and present that match as proof of parity. It photographs well. It is also close to meaningless as a risk signal, because the clean cases were never where the risk was. The risk was always concentrated in the minority of records that don't look like anything the vendor's synthetic generator would have thought to produce — and by definition, those are exactly the records a synthetic suite can't surface, because surfacing them requires already knowing they exist.
This is why "we have a comprehensive test suite" and "we ran your actual production data through both systems and diffed the output" are different claims, and only the second one is evidence. A comprehensive synthetic suite proves coverage of imagined scenarios. Byte-for-byte comparison against real production batches proves the replacement handles what the business actually does, including the parts the business itself may have forgotten it does, because some rule from 1994 is still firing on some subset of accounts and nobody remembers why.
What real parity testing requires
Testing against production bytes is more work, which is exactly why it gets skipped. It means running full historical batches — payroll runs, claims cycles, settlement files, whatever the system actually processes — through the legacy system and the candidate replacement side by side, on the same inputs, and comparing every output field, not a sampled subset. It means treating any mismatch as a defect to explain, not a rounding difference to wave off. It means doing this across enough historical volume and enough calendar time to catch the inputs that only occur on a fiscal year boundary, or during a leap year, or during whatever seasonal batch runs once a quarter and therefore doesn't show up if the test window is short.
None of this requires the production data to be exposed irresponsibly. Real inputs can be used under the same access controls and data-handling agreements that already govern the legacy system, with output comparison run in a controlled environment. The point isn't to loosen data governance — it's to refuse to let data governance become the excuse for testing against a synthetic stand-in instead. A vendor that says real production data testing isn't feasible for compliance reasons is usually telling you they haven't built the controlled environment to do it responsibly, not that it can't be done. Plenty of regulated environments — payroll, claims, settlement — manage exactly this tradeoff routinely for audits and disaster-recovery testing. It's a solved operational problem, not a novel one.
What to ask for
A buyer evaluating parity claims should ask a specific question: what fraction of the comparison so far has run against real production batches, as opposed to synthetic or hand-built test cases? A vendor with genuine confidence in their conversion will have a number, will know exactly which production periods it covers, and will be able to show you a mismatch they found and how they resolved it — because on a system old enough to need modernizing, a synthetic-only test pass finding zero mismatches on the first try isn't a good sign. It's a sign the test never touched the parts of the system that were actually going to be hard.
A green synthetic suite tells you the vendor understood their own test data. A green production-byte comparison tells you the vendor understood your system.
The uncomfortable version of this for a buyer is that parity theater isn't always dishonest — sometimes it's a vendor who genuinely believes their synthetic suite is representative, because building a proper production-byte harness is expensive and slow and nobody asked them to justify skipping it. That's still a buyer's problem to catch, not the vendor's to volunteer. Ask for the real numbers, on real bytes, before the green checkmark becomes the thing you're paying for instead of the thing you were promised.