When you test a shader translator, the tempting thing is to compile the output and check it against last week's output. That is what a golden file is: a snapshot of your own belief. It catches regressions against yourself. It says nothing about whether either version was right.
The industry knows this, which is why the incumbents are careful about what they claim. SPIRV-Cross's tests compare emitted text against golden files. naga's docs famously ask you not to run the output through other generators as a check, because the disagreements are noise. Both are defensible positions. Neither proves the output computes the right pixels.
That is the gap zioshade tries to close: the only shader cross-compiler I am aware of, at the time of writing, that renders its own output as part of verification. Every backend* is pixel-diffed before it ships.
zioshade is a young cross-compiler with one maintainer, and that changed the design constraint: I cannot ask anyone to trust me, so the output has to be judged by things that are not me, preferably things that would enjoy watching me fail. This is the stack, in the order it was built, each layer existing because the previous one let something through.
One oracle: render-diff against the incumbent
The base layer renders zioshade's MSL and the MSL from an independent glslang-plus-SPIRV-Cross pipeline on a real Metal GPU, and diffs the pixels. Fragments by framebuffer, vertex shaders by captured position, compute by output buffers. 1300+ shaders, zero divergences, one command to re-run.
A differential against one reference has a blind spot you can state exactly: if both compilers misread the spec the same way, they render identically wrong and pass. Any single-oracle differential has this hole. It is not a reason to skip the oracle; it is a reason to add a second one.
Two oracles: make them disagree
So the second layer adds naga as a second reference, with its own independent SPIR-V-to-MSL backend, unrelated to SPIRV-Cross. Now a shared misreading needs three implementations to coordinate the same mistake. Where all three render the same pixels, that is the strongest claim I know how to make short of a formal proof.
And the day the oracle was wrong
The row-major matrix work gave the whole stack its best moment. A row_major decoration describes how bytes sit in a buffer. It does not describe the matrix you are holding in a register after a load, but code paths kept treating it as if it did, and a whole-struct load followed by a member extract slipped past every compensation.
When I fixed it, one oracle flagged the fix as wrong: naga rejected the output. On inspection, naga itself was internally inconsistent on this exact shape: its own WGSL path transposed the loaded copy while its own Metal path did not. Two naga backends disagreed with each other, and the pixel diff, not the validator, settled which one was right. The fixed code renders identically to SPIRV-Cross and to naga's Metal on the GPU; naga's WGSL leg was the bug.
That is the argument for rendering over validating, in one anecdote. Validators check shape. Pixels check meaning. Same instinct as when I shipped stunt: fake the physics, never the plot.
Classification: not every diff is a bug
Once you render-diff at scale you meet a new problem: shaders that are supposed to be nondeterministic-adjacent. Mandelbrots whose escape condition sits on a knife edge, fract-of-sin hash chains, anything where reordering two multiplies moves a pixel by one ULP and the whole image avalanches. A naive harness reports these as failures. I have burned whole afternoons on Mandelbrot pixels that were never wrong, just dramatic. At some point you start talking to the harness. It does not answer back, which is probably for the best. That is how you learn to distrust the harness, and then how you learn to classify what it is actually telling you.
So divergences get classified, not just counted: deterministic miscompiles, single-ULP chaos amplified by the shader's own math, and proxy artifacts introduced by the check itself. The rule is that classification has to be provable. The five remaining diffs in the D3D12 WARP sweep, for instance, are all fp-contraction: recompile both sides with strict IEEE and they render pixel-identical. Benign, with a receipt, not benign because I was tired.
What is still not proven
MSL renders natively on Metal. HLSL renders on D3D12 WARP, which is a software rasterizer: real D3D12 semantics, no vendor hardware. GLSL and WGSL are compile-checked and verified through a Metal proxy, because I do not have a native GL or Vulkan leg wired yet. Those gaps are written on the front page of the docs, which is where gaps belong.
The verification does not make the compiler correct. It makes the failure modes visible, specific, and expensive to hide, which is the property you actually need before you trust a compiler with your pixels.