The slower model returned less of the menu
The Better Translate team ·
On the hardest menu in our test set, one configuration ran for 1,029 seconds and
returned less of the board than another returned in 94.
These numbers come from our own menu benchmark: five real photographed menus, run through candidate models under the same conditions. It is a model-selection test, not a statement about what any given request uses today.
That is not a small gap. It is eleven times the wall clock for a worse answer, on the same
photograph, with the same text already recognised on the device before either of them saw it.
What that means when you are standing in a bar
A menu scan is not a background job. Somebody is holding a phone over a board with a person
waiting. Seventeen minutes is not a slow feature, it is a broken one, and a result that is also
less complete is the worst of both.
So the model that ships is not the one that scores best given unlimited time. It is the one that
comes back before anybody puts the phone down. We measured both halves of that rather than assuming
the bigger number wins.
Speed and quality were not a trade here
The thing worth knowing is that we did not have to choose. The faster configuration was also the
more complete one, on recall and on exact character match. The slow one was not buying accuracy
with time; it was just slow.
Which is a useful warning about picking a model by reputation. On the specific job of reading a
damaged handwritten board, the ranking did not match the general ranking at all.
Why we run this at all
Every time the pipeline changes we re-run five real photographed menus through it, including one
board shot at night and the same board shot in daylight. It takes an afternoon and it has changed
what the app does at least twice.
The full matrix is here.