A fix for the strawberry problem: retrofitting LLMs to read letters
Ask many chatbots how many r's are in strawberry and they say two. The reason is tokens.
Của The desk, News · · 1 phút đọc
Chịu trách nhiệm: Dimitri Gärtner
Thẻ: language models · tokenization · research · nature
Bài này chưa có bản Tiếng Việt, nên bạn đang đọc tiếng Anh.
- Tin
- Ask many chatbots how many r's are in strawberry and they say two.
- Vì sao đáng kể
- The reason is tokens.
- Điều đã được xác nhận
- The photograph is generated for this story.
Gắn với một nguồn được nêu tên. Chưa đánh dấu là đã kiểm chứng. Nature News and Views: Retrofitted LLM can count the letter i's in artificial intelligence
The news
A paper in Nature by Minixhofer and colleagues reports a technique called byteification that retrofits existing token-based large language models to operate at the byte level, giving them access to individual characters. The authors show byteified models achieve competitive performance while keeping the ability to read individual characters, according to an accompanying News and Views piece published on October 7.
Why it matters
Character blindness causes errors in spelling, code, names and languages with rich morphology. Retrofitting existing models is far cheaper than training new byte-level ones.
The details
Most LLMs encode text as tokens that stand for sequences of letters, so they cannot see the bytes that make up each word. That is why many models say strawberry contains two r's instead of three, the News and Views authors Zhao Zhang and Yingfei Xiong explain.
Byteification converts a trained model rather than starting over, which matters given the cost of pre-training.
What we could not check
- We read the News and Views summary, not the full paper.
- Performance comparisons depend on benchmarks chosen by the authors.
Nguồn
The photograph is generated for this story. It does not show the people named in it.