The failure that looks like success
The Better Translate team ·
The dangerous failure in menu translation is not a garbled line. It is a line that
never appears.
These numbers come from our own menu benchmark: five real photographed menus, run through candidate models under the same conditions. It is a model-selection test, not a statement about what any given request uses today.
On our test menus, the best configuration kept 93% of the add-ons: toppings,
course components, the small print under a dish that says what comes with it. Another kept
22% at low effort and 60% at high.
Why that is worse than it sounds
It did not mangle them. It dropped them. There is no half-translated line to squint at, no
obvious mistake. The menu comes back looking clean and correct, and the thing you would have
ordered is simply not on it.
A wrong translation announces itself. A missing one does not. You cannot ask about a dish you
were never shown, and you have no way to know the list is short.
What we do about it
- The original text stays beside every line, so a page that came back thin is visibly thin next
to the photograph you took.
- The overlay mode keeps each translation where the dish sat on the page, which makes a gap
obvious in a way a written-out list does not.
- Completeness is scored in the benchmark as its own number, separately from whether the English
is any good. A configuration that writes beautiful English about half a menu does not pass.
The general version
When you are evaluating anything that summarises or extracts, ask what it does with the parts it
is unsure about. Dropping them scores well on every quality measure that only looks at the output.
It is also the failure mode that a person is least able to catch.