An interpretability researcher argues neural networks are cleaner than we think
If he is right, a large part of interpretability research is solving a smaller problem than assumed. It is a hypothesis, not a paper yet.
The desk द्वारा, News · · 1 मिनट का पठन
चिह्न: interpretability · goodfire · research · qwen
यह कथा अभी हिन्दी में नहीं है, इसलिए आप अंग्रेज़ी पढ़ रहे हैं।
- खबर
- If he is right, a large part of interpretability research is solving a smaller problem than assumed.
- यह क्यों मायने रखता है
- It is a hypothesis, not a paper yet.
- जो पुष्ट है
- The photograph is generated for this story.
एक नामित स्रोत से जुड़ा। सत्यापित के रूप में चिह्नित नहीं। Goodfire Research
The news
Nick Cammarata of interpretability startup Goodfire published an essay titled Neural networks are clean, arguing that most neurons in modern networks are individually understandable and that superposition, networks packing many unrelated concepts into one neuron, is far rarer than the field assumes. In preliminary audits of Qwen3-32B he reports that about 90 percent of neuron poles have one main job, about 7 percent two related jobs and 2 to 4 percent are not yet understood.
Why it matters
Sparse autoencoders built to untangle superposition dominate interpretability. If neurons are mostly clean, direct neuron-level audits of frontier models become practical.
The details
Cammarata also argues that classic examples of mixed-up neurons in InceptionV1's last layer came from architecture and tooling problems, and that apparent polysemanticity is mostly a neuron growing related offshoots during training.
He frames it as a working hypothesis. The percentages depend on thresholds, the samples are small, and a companion paper is pending. The essay drew more than 560 points on Hacker News.
What we could not check
- No peer-reviewed paper or replication exists yet.
- Percentages depend on the author's own thresholds.
स्रोत
The photograph is generated for this story. It does not show the people named in it.