A fix for the strawberry problem: retrofitting LLMs to read letters
Ask many chatbots how many r's are in strawberry and they say two. The reason is tokens.
Autor: The desk, News · · 1 min czytania
Tagi: language models · tokenization · research · nature
Tego tekstu nie ma jeszcze w języku Polski, więc czytasz po angielsku.
- Wiadomość
- Ask many chatbots how many r's are in strawberry and they say two.
- Dlaczego to ważne
- The reason is tokens.
- Co jest potwierdzone
- The photograph is generated for this story.
Powiązane z nazwanym źródłem. Nieoznaczone jako zweryfikowane. Nature News and Views: Retrofitted LLM can count the letter i's in artificial intelligence
The news
A paper in Nature by Minixhofer and colleagues reports a technique called byteification that retrofits existing token-based large language models to operate at the byte level, giving them access to individual characters. The authors show byteified models achieve competitive performance while keeping the ability to read individual characters, according to an accompanying News and Views piece published on October 7.
Why it matters
Character blindness causes errors in spelling, code, names and languages with rich morphology. Retrofitting existing models is far cheaper than training new byte-level ones.
The details
Most LLMs encode text as tokens that stand for sequences of letters, so they cannot see the bytes that make up each word. That is why many models say strawberry contains two r's instead of three, the News and Views authors Zhao Zhang and Yingfei Xiong explain.
Byteification converts a trained model rather than starting over, which matters given the cost of pre-training.
What we could not check
- We read the News and Views summary, not the full paper.
- Performance comparisons depend on benchmarks chosen by the authors.
Źródła
The photograph is generated for this story. It does not show the people named in it.