A fix for the strawberry problem: retrofitting LLMs to read letters
Ask many chatbots how many r's are in strawberry and they say two. The reason is tokens.
글 The desk, News · · 1분 읽기
태그: language models · tokenization · research · nature
이 기사는 아직 한국어로 없습니다. 영어를 읽습니다.
- 이 소식
- Ask many chatbots how many r's are in strawberry and they say two.
- 왜 중요한가
- The reason is tokens.
- 확인된 것
- The photograph is generated for this story.
이름을 밝힌 출처에 묶여 있다. 검증됨으로 표시하지 않았다. Nature News and Views: Retrofitted LLM can count the letter i's in artificial intelligence
The news
A paper in Nature by Minixhofer and colleagues reports a technique called byteification that retrofits existing token-based large language models to operate at the byte level, giving them access to individual characters. The authors show byteified models achieve competitive performance while keeping the ability to read individual characters, according to an accompanying News and Views piece published on October 7.
Why it matters
Character blindness causes errors in spelling, code, names and languages with rich morphology. Retrofitting existing models is far cheaper than training new byte-level ones.
The details
Most LLMs encode text as tokens that stand for sequences of letters, so they cannot see the bytes that make up each word. That is why many models say strawberry contains two r's instead of three, the News and Views authors Zhao Zhang and Yingfei Xiong explain.
Byteification converts a trained model rather than starting over, which matters given the cost of pre-training.
What we could not check
- We read the News and Views summary, not the full paper.
- Performance comparisons depend on benchmarks chosen by the authors.
출처
The photograph is generated for this story. It does not show the people named in it.