Stanford Study Finds 44% of Tested LLM Findings Have Expired

Research claims about large language models can age faster than the papers that describe them. Stanford researchers Owen Queen and James Zou tested 50 widely cited findings against 20 models and found that 44 percent no longer held. Another 34 percent had weakened significantly; 16 percent remained stable, and 6 percent had grown stronger.

Among the 22 findings for which the researchers estimated an expiration date, the average lifespan was 15.7 months. The share that had expired rose from 16 percent after one year to 39 percent after two. For developers and researchers, the results make model version and test date part of the finding itself: a reported behavior may not generalize to newer systems, or even to another model released around the same time.

Reasoning weaknesses faded fastest

The fastest-changing claims concerned tasks models supposedly could not handle. Five of nine findings about reasoning had expired, and none had held steady or strengthened. Examples included counting the letters in “strawberry,” doing arithmetic in unfamiliar number systems, retrieving information from the middle of long contexts, and maintaining a correct answer when a user challenges it.

Queen and Zou suggest that these failures may be especially likely to fade because they have clear right answers. That gives developers a direct training target: if a model repeatedly misses a letter count or calculation, training can reward the correct response. The trend doesn’t establish exactly which changes caused each result, but it does show why a weakness measured on one generation shouldn’t automatically be treated as a lasting property of language models.

Findings about tendencies and preferences were more likely to persist. Of the 11 findings that held or strengthened, several concerned how models evaluate text: they favored their own answers, rated accepted and rejected papers similarly, and responded to persuasive framing in medical abstracts. Other persistent results involved models agreeing with users, producing highly similar answers to open-ended questions, and showing a non-uniform pattern when asked to generate random numbers. The study also found that a left-leaning bias persisted.

Better citation accuracy doesn’t settle broader reliability

The authors revisited a 2023 study by Walters and Wilder that found GPT-3.5 fabricated 55 percent of its scientific references. Using the same prompts, Queen and Zou found no fabricated references from GPT-6.1 Sol and a 1 percent rate from Claude Sonnet 5.5. Qwen3.7-Flash, also released in 2026, fabricated 69 percent in that test.

That spread is a useful reminder that an improvement on one model doesn’t establish a general capability across a model family or release period. Citation accuracy also tests a narrower skill than factual reasoning as a whole. The paper’s two-hop question results illustrate the distinction: Claude Sonnet 5.5 still failed in about 10 percent of cases where it answered both component questions correctly. The original paper reported a 43 percent failure rate for ChatGPT.

Other errors have become less common without disappearing. The “Reversal Curse,” in which a model’s knowledge of one relation doesn’t reliably carry over when the relation is reversed, has weakened. So have contradictory responses to paired questions about time. Both remain detectable, so improvement on a benchmark shouldn’t be read as proof that the underlying failure mode is gone.

Bias results don’t move in one direction

The study’s social-bias results were mixed. Gender stereotypes and implicit association biases weakened, while the observed bias against speakers of African American English changed direction in the trend data. That shift makes careful test design and reporting especially important: a single score can hide changes in which groups are favored or disadvantaged, and results from one model may not describe another.

The researchers found wide variation even among models released between July and September 2026. The number of expired findings ranged from 11 for DeepSeek V4 Pro to 30 for GPT-6.1 Sol. Release date, developer and model size did not reliably predict which findings would hold. A model’s place in a release cycle, by itself, therefore offers little basis for assuming that a particular result applies.

Treat each result as dated evidence

Queen and Zou argue that papers should describe empirical results as observations about a specific model at a specific point in time, rather than as universal claims about LLMs. They recommend documenting the model, version and test date, and sharing prompts and analysis code so others can repeat the evaluations on later systems.

That approach would make research easier to interpret as models change. It also gives adopters a more practical question to ask: does the evidence apply to the system and version we plan to use, under conditions like our own? As the study’s citation and reasoning tests show, the answer may differ sharply between models—and a once-reliable result can lose relevance within a couple of years.

The takeaway is not that earlier findings were useless. They describe evidence collected under particular conditions. Their value depends on making those conditions visible and checking whether the result still holds when the model changes.