We test on photographs, not on text files
Better Translate Team ·
The convenient way to test a translation pipeline is to feed it clean text and score
the output. We do not do that, because it measures the easy half.
What the easy version misses
By the time a menu is clean text, the hard work already happened. On a real board the phone's
recogniser splits a dish name across several fragments, in the wrong order, with strokes missing
from the characters. Handing a model a tidy string skips all of that and scores a problem nobody
actually has.
Our set is five photographed menus: a French set menu, a Hakata ramen book, an izakaya board shot
at night and the same board in daylight, and a kaiseki card. Real photographs, including bad ones.
What that exposed
- The same board recalls 0.63 of its dishes at night and 0.32 in daylight glare. A text-file
benchmark cannot see that at all.
- Handing the recogniser five languages instead of two changed nothing on any menu.
- Automatic language detection took the French menu from 0.14 recall to 0.57.
- Extra reasoning time helps a damaged photograph and does nothing to a clean one.
Every one of those changed what the app does, and none of them would have shown up on clean
input.
Why publish it
Most translation apps describe the camera feature and stop. The honest version is that reading
the page is the hard half, it happens on your phone before anything is sent anywhere, and on a bad
photograph no amount of model quality rescues it.
The full matrix is here.