TEST · 10 SEP 2026

I ran two image-generation pipelines head to head — five prompts, six seeds each, 60 images generated in total — and judged the results myself, by hand, through 59 head-to-head rounds (the pairing isn't one round per image; a matchmaking system decides who faces whom). The overall score came out close to a tie: 28 wins to 23, with 8 too close to call. If I'd stopped there, the honest conclusion would have been "no real difference."

Splitting the same results by subject told a completely different story. On one prompt — a fisherman on an overcast day — one pipeline won 12 times out of 12. On another — a mountain ridge shot into backlight — the other pipeline won 14 times out of 15. Two landslides, pointing in opposite directions, cancelling out into what looked like a coin flip. The real finding wasn't "which pipeline is better," it was "which pipeline for which kind of image" — and the average had been quietly erasing that the whole time.

To scale past hand-judging every pair, I tried an 8-billion-parameter local vision model as an automatic judge on the same 60 image pairs. Its internal consistency looked fine — it agreed with itself 77% of the time when the same pair was shown twice, and never contradicted itself outright. That looked like a healthy signal.

It wasn't. Across 80 judged pairs, the model picked the same pipeline 63 times, the other pipeline 17 times, and never once called a tie — regardless of subject, including the prompt where a human judge had gone 14-1 the other way. When I had it re-score the exact 20 pairs I'd already judged myself, it matched my calls only half the time. That's chance-level agreement. A judge that always gives the same answer will look perfectly consistent and still be measuring nothing — high agreement with itself is not the same thing as being right. The model's stated reasons were vague, too — "more coherent composition," "better lighting" — and almost never pointed at a concrete flaw like a broken hand or malformed text.

A judge that always gives the same answer will look perfectly consistent and still be measuring nothing.

This is an excerpt. The full write-up was first published on Haenginro Lab.