斯坦福研究：50 条 LLM 结论中 44% 已失效，平均寿命 15.7 个月

Stanford researchers Owen Queen and James Zou tested 50 widely cited claims about large language models against 20 models and found that 44 percent have already "expired." Among the 22 findings with an estimated expiration date, the average lifespan was just 15.7 months.

The two examined the claims using models released between July 2024 and September 2026 by eight different developers. The original effect behind the expired findings has largely disappeared, while another 34 percent have weakened significantly. Only 16 percent still hold, and 6 percent have actually gotten stronger. After one year, 16 percent had expired. After two years, that number climbed to 39 percent.

What models "can't do" expires fastest

Claims about what models can't do expire fastest, according to the study. Five of nine reasoning-related findings are considered expired, and none have held up or gotten stronger. They include well-known weaknesses like counting letters in "strawberry," arithmetic in unfamiliar number systems, losing information in the middle of long contexts, and giving up correct answers when users push back. The researchers suspect these kinds of errors have a "correct" outcome, giving developers a clear target to train toward.

Claims about preferences and tendencies prove longer-lived. Of the 11 findings that have held or gotten stronger, three involve LLM behavior as evaluators. They prefer their own answers, rate accepted and rejected papers similarly, and are swayed by linguistic spin in medical abstracts. Three more involve sycophancy and agreement toward users. Moreover, there's a left-leaning political bias, near-identical answers to open-ended questions, and a non-uniform distribution when generating random numbers.

Hallucinated references dropped from 55 percent to zero, but not everywhere

As a case study, Queen and Zou looked at the fabrication of scientific citations. In the original 2023 study by Walters and Wilder, GPT-3.5 made up 55 percent of its references. The Stanford researchers sent the same prompts to current models. GPT-6.1 Sol fabricated 0 percent, and Claude Sonnet 5.5 came in at 1 percent. But Qwen3.7-Flash, also a 2026 model, still fabricated 69 percent of its references in the same test.

The study warns against reading these results as an all-clear. On so-called two-hop questions, where a model has to combine two known facts, Claude Sonnet 5.5 still fails in about 10 percent of cases where it correctly answers both individual questions. The original paper reported 43 percent for ChatGPT. The "Reversal Curse" and contradictory answers to paired time-related questions have weakened too, but they're still detectable.

Social biases are a mixed case, according to the researchers. The left-leaning political bias persists, while gender stereotypes and implicit association biases have weakened. Biases against speakers of African American English have actually flipped direction in the trend data.

Model-to-model gaps remain wide even within the same generation

Results vary considerably across models, even within the same release window. Among models released between July and September 2026, the number of expired findings ranges from 11 for DeepSeek V4 Pro to 30 for GPT-6.1 Sol. Neither release date, provider, nor model size reliably predicts the pattern.

Queen and Zou draw a broad conclusion. Empirical findings about LLMs should be presented as observations about specific models at a specific point, not as general truths. Papers should document the model, version, and date, and publish prompts and analysis code so findings can be re-tested on future models.

The researchers point to a post by Andrej Karpathy that describes how perception gaps about LLMs' actual behavior affect trust in AI and how often people use it. Karpathy, an OpenAI co-founder who has since joined Anthropic, also recently said that currently available models are just a preview of future capabilities.