Why one model for every language is a bad idea for customer feedback
We tried the obvious approach first: one large multilingual model, fine-tuned. It scored well on benchmarks and disappointed on real messages. Here is what it kept missing.
By Howzer Team, Research ·
When we started, the obvious path was one big multilingual model, fine-tuned on feedback. It would cover German and English out of the box and every other language for free. We built it. The benchmark numbers were fine and the behaviour on real messages was not.
A model can score 92 percent on an academic benchmark and still not tell a frustrated German customer from a furious one.
Four things it kept getting wrong
- German builds whole ideas into single words. Vertragskündigungsbestätigung is a confirmation of a contract cancellation, and a shared model chops it into fragments that mean nothing individually.
- German complaints are often not phrased as complaints. I would have expected that the appointment held is a complaint, and a multilingual model reads it as neutral.
- The choice between formal and informal address says something about the relationship, and therefore about how the message should be answered. A model trained across languages has no reason to notice.
- English works the opposite way. The last straw and could not care less mean nothing if you read them literally, which is exactly what a model does when idioms were a rounding error in its training data.
What we do instead
One model per language, each trained on customer feedback rather than on encyclopaedia articles and news, and each calibrated separately, because the point at which a German message becomes worrying is not the point at which an English one does.
- Shared across all languages
- Trained on general-purpose text
- The same thresholds everywhere
- Word splitting that suits no language exactly
- Dedicated per language
- Trained on real customer feedback
- Thresholds calibrated per language
- Word splitting native to the language
Both produce identical output, so everything downstream, root cause, replies, routing, works the same regardless of the language it came from. The only difference is that each model is better at its own language than any single model would be at both. Any language we add later gets the same treatment.