We ran 32 models over the same Japanese menu
The Better Translate team ·
Before a translation model sees a menu, the phone has to read it. We benchmarked that
step, and the result changed what the app does by default.
The test set is five photographed menus: a French set menu, a Hakata ramen book, an izakaya
board shot at night and again in daylight, and a kaiseki card. Each one went through Apple's Vision
text recogniser under four language settings, then through 32 model configurations, all on the same
images.
What the numbers say
Recall below is the share of items on the menu whose exact characters survive
the recogniser, measured before any model is involved. It is the ceiling. No translator, however
good, recovers a character the phone never read.
Three things fell out of it
Listing more languages does nothing. We expected that handing the recogniser
five languages at once would help it cope with a mixed menu. It was identical to pinning two, on
every single row. The language list is a hint applied to the image as a whole, not tried line by
line, so a French line under a Japanese-first list stays broken.
Automatic detection is free. It matched or beat the pinned setting on every
Japanese menu, and it turned the French menu from unreadable into the cleanest scan in the set:
recall from 0.14 to 0.57, confidence from 0.38 to 0.98. There was no case where pinning won. The
app detects automatically now, and this table is why.
The same board is a different problem in daylight. The izakaya board scanned
at night recalls 0.63 of its items. The same board in daylight recalls 0.32. Glare and angle move
the number further than any model choice we tested.
Why we are publishing this
Most translation apps describe the camera feature and stop. The honest version is that reading
the menu is the hard half, the phone does it before anything is sent anywhere, and on a bad
photograph of a handwritten board no amount of model quality rescues it. If a scan comes back thin,
retake the photo straighter and closer. That does more than anything happening on our end.
The benchmark harness and the raw runs live in the app's own repository. The numbers above are
ours, from our own test set, and we would rather show them than claim a percentage with nothing
behind it.