A fix for the strawberry problem: retrofitting LLMs to read letters
Ask many chatbots how many r's are in strawberry and they say two. The reason is tokens.
Oleh The desk, News · · 1 menit baca
Disunting oleh Dimitri Gärtner
Tag: language models · tokenization · research · nature
Tulisan ini belum ada dalam Bahasa Indonesia, jadi Anda membaca bahasa Inggris.
- Beritanya
- Ask many chatbots how many r's are in strawberry and they say two.
- Mengapa ini penting
- The reason is tokens.
- Yang terkonfirmasi
- The photograph is generated for this story.
Terikat pada sumber yang disebut. Tidak ditandai terverifikasi. Nature News and Views: Retrofitted LLM can count the letter i's in artificial intelligence
The news
A paper in Nature by Minixhofer and colleagues reports a technique called byteification that retrofits existing token-based large language models to operate at the byte level, giving them access to individual characters. The authors show byteified models achieve competitive performance while keeping the ability to read individual characters, according to an accompanying News and Views piece published on October 7.
Why it matters
Character blindness causes errors in spelling, code, names and languages with rich morphology. Retrofitting existing models is far cheaper than training new byte-level ones.
The details
Most LLMs encode text as tokens that stand for sequences of letters, so they cannot see the bytes that make up each word. That is why many models say strawberry contains two r's instead of three, the News and Views authors Zhao Zhang and Yingfei Xiong explain.
Byteification converts a trained model rather than starting over, which matters given the cost of pre-training.
What we could not check
- We read the News and Views summary, not the full paper.
- Performance comparisons depend on benchmarks chosen by the authors.
Sumber
The photograph is generated for this story. It does not show the people named in it.