A fix for the strawberry problem: retrofitting LLMs to read letters
Ask many chatbots how many r's are in strawberry and they say two. The reason is tokens.
Yazan The desk, News · · 1 dk okuma
Yayın yönetmeni: Dimitri Gärtner
Etiketler: language models · tokenization · research · nature
Bu haber henüz Türkçe dilinde yok; İngilizce okuyorsunuz.
- Haber
- Ask many chatbots how many r's are in strawberry and they say two.
- Neden önemli
- The reason is tokens.
- Ne doğrulandı
- The photograph is generated for this story.
Adı geçen bir kaynağa bağlı. Doğrulanmış olarak işaretlenmedi. Nature News and Views: Retrofitted LLM can count the letter i's in artificial intelligence
The news
A paper in Nature by Minixhofer and colleagues reports a technique called byteification that retrofits existing token-based large language models to operate at the byte level, giving them access to individual characters. The authors show byteified models achieve competitive performance while keeping the ability to read individual characters, according to an accompanying News and Views piece published on October 7.
Why it matters
Character blindness causes errors in spelling, code, names and languages with rich morphology. Retrofitting existing models is far cheaper than training new byte-level ones.
The details
Most LLMs encode text as tokens that stand for sequences of letters, so they cannot see the bytes that make up each word. That is why many models say strawberry contains two r's instead of three, the News and Views authors Zhao Zhang and Yingfei Xiong explain.
Byteification converts a trained model rather than starting over, which matters given the cost of pre-training.
What we could not check
- We read the News and Views summary, not the full paper.
- Performance comparisons depend on benchmarks chosen by the authors.
Kaynaklar
The photograph is generated for this story. It does not show the people named in it.