Demand proof before you sign: a buyer's checklist for COBOL modernization

Most failed modernizations were lost before kickoff — in the RFP, in the demo that demoed nothing, in the contract that paid for effort instead of outcomes. Here is the checklist we would use if we were the buyer: eight demands, in order of importance. They are not exotic. That is the point — a serious vendor can satisfy every one of them, and a grifter can satisfy none.

The eight demands

1. A byte-exact parity demo on YOUR code, before signing. Not a slide, not a reference call, not a demo of somebody else's payroll. Give the vendor a representative slice — real programs, real copybooks, production-shaped data — and require the modernized version to produce byte-for-byte identical output. Days, not months. This single demand filters out most of the market, which tells you what you need to know about most of the market.
2. Determinism in the conversion path. Ask one question: "if you translate the same program twice, do you get the same result?" If the answer involves AI anywhere in the translation itself, the honest answer is no — and every translated line becomes a line someone must manually verify. Assistants for analysis, fine. Dice in the pipeline, no.
3. A reproducible proof environment. The parity demo should arrive as something you can rerun yourself — a container, a script, your inputs, their outputs, a diff that reads zero. If proof only ever runs on the vendor's laptop, it isn't proof; it's theater.
4. Numbers with denominators. "High automation" is marketing. 12,847 of 12,900 programs converting cleanly is engineering. Demand conversion rates, test counts, and parity percentages with real denominators — and ask what happens to the remainder, because the remainder is where your budget goes.
5. Negative controls. Any test suite can be green if it only contains tests that pass. Ask how they prove their tests can fail — deliberately broken inputs that the harness must catch. A vendor who has never thought about this has a test suite built to reassure, not to verify.
6. Runtime equivalence, not compile success. "It compiles on the target" is the cheapest milestone in modernization and the one most often sold as progress. The system's value is its behavior: batch outputs, screen flows, edge-case arithmetic, the weird date logic from 1994. Payment milestones should attach to behavioral parity, never to compilation.
7. You own the output, unconditionally. The translated source, the build scripts, the test harness — yours, in escrow or in hand, from day one. Some vendors keep translated code hostage inside proprietary runtimes so that leaving them means modernizing twice. Read that clause before you praise the price.
8. Parallel-run exit criteria in the contract. Define, up front and in writing: how long old and new run side by side, what "matching" means (we would argue: byte-exact), what discrepancy rate triggers what response, and who pays for reconciliation. The projects that rot for years are the ones that never defined what done means.

Red flags that predict the blown budget

• Proof is scheduled after contract signature — "in the validation phase."
• The estimate was produced from lines-of-code counts without running anything.
• The plan requires a multi-year freeze on a system your business changes weekly.
• "Fixed price" with a change-order mechanism wide enough to drive a mainframe through.
• References who can't discuss specifics, on projects that are mysteriously all "in progress."
• Rescue-rate pricing for fixing their own earlier phase.
• Any sentence containing "AI-powered translation" and "money-handling code" without the word "verification" between them.

The one-week test

Everything above compresses into a single procurement move: give every bidder the same code slice and the same week. Whoever returns a reproducible, byte-exact parity demonstration is your shortlist. Whoever returns a slide deck about their transformation journey framework has told you exactly how the next three years would go.

Modernization does not have to be a leap of faith. The technology to prove equivalence — deterministically, byte for byte, before commitment — exists. The only reason the industry still sells faith is that faith bills better. Stop buying it.