Sam Altman· @sama · X·· 3 小时前AI 评分62
AI 导读
Sam Altman 转发 Nick Cammarata 关于可解释性的观点,称很高兴看到这一内容。Nick Cammarata 认为可解释性可能是当下最重要的科学问题,因为不了解所构建的系统就无法解决对齐;他曾在 OpenAI 参与机制可解释性早期工作并因此悲观,后来启动 OpenAI 首个大规模自动化可解释性项目,当时用原始工具只能识别约 2% 的神经元。他表示借助现代智能体工具,参数量达万亿级的模型中绝大多数神经元可读、回路清晰,许多著名的纠缠神经元案例实为误读,并认为随着自动化科学家不断进步,雄心勃勃的可解释性研究有望在最需要时被解决。
正文
Very happy to see this from Nick!
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works https://already-clean.goodfire.com/在 X 查看被引用的帖子
来源:Sam Altman · x.com