Every Benchmark Launched 2023-2024 Has Fallen — The METR / SWE-Bench / CORE-Bench / MLE-Bench / PostTrainBench Sequence

📊 Full opportunity report: Every Benchmark Launched 2023-2024 Has Fallen — The METR / SWE-Bench / CORE-Bench / MLE-Bench / PostTrainBench Sequence on ThorstenMeyerAI.com — validation score, market gap, and execution plan.

All six major benchmarks launched in 2023-2024 to measure AI research and development capabilities have either been saturated or are on the verge of saturation within a few months, according to recent analysis by Thorsten Meyer. This pattern suggests AI progress is accelerating faster than many models previously predicted, with potential implications for AI deployment timelines and policy considerations.

Thorsten Meyer’s analysis, based on data from Jack Clark’s recent report, highlights that each of the six benchmarks—covering areas from software engineering to model training efficiency—has experienced rapid saturation. For example, the SWE-Bench, which measures real-world software engineering skills, improved from 2% to 93.9% in 30 months, reaching a saturation point earlier than expected. Similarly, the METR time horizons benchmark, assessing the duration of AI tasks, shrank from 30 seconds to 12 hours over four years, indicating exponential growth in AI speed and efficiency.

All six benchmarks, which were specifically designed to be challenging for AI systems, show a consistent pattern: saturation or near-saturation within a short timeframe of months. This includes the CORE-Bench, which measures research reproduction capabilities and was declared solved by its authors after reaching 95.5% in December 2025, and the MLE-Bench, tracking end-to-end machine learning engineering, which is progressing toward early saturation by early 2027.

Implications of Rapid Benchmark Saturation for AI Development

The rapid saturation of these benchmarks signals that AI systems are rapidly approaching or surpassing human-level capabilities in key research and engineering tasks. This accelerates the timeline for deploying advanced AI models and could influence policy, workforce planning, and investment strategies. It also raises questions about the remaining challenges in AI safety, robustness, and generalization, which may now become the focus as capabilities plateau in these specific benchmarks.

Recent Trends in AI Benchmark Progress and Expectations

Prior to 2023, AI benchmarks generally showed steady but incremental improvements over several years. The launch of challenging benchmarks in 2023 aimed to measure the true progress of AI research, with expectations that saturation might take years. However, recent data reveals these benchmarks have all been saturated within a few months to a year, indicating an acceleration in AI research capabilities. This pattern aligns with other indicators of rapid AI advancement, such as the exponential growth in AI training speed and task completion horizons.

Experts like Jack Clark and Thorsten Meyer have emphasized that these benchmark saturations provide a structural argument that AI progress is moving at a pace consistent with forecasts of reaching significant capability milestones by 2028, if not sooner.

“The pattern across these six benchmarks is the structural argument: saturation within months indicates an acceleration in AI research capabilities.”

— Thorsten Meyer

Remaining Challenges and Unconfirmed Aspects of Benchmark Saturation

While the saturation of these benchmarks indicates rapid progress, it remains unclear how this translates to broader AI capabilities in real-world applications. Some experts caution that benchmarks can be saturated through overfitting or data contamination, and may not fully reflect true general intelligence or robustness. Additionally, the long-term implications for AI safety, policy, and regulation are still under discussion, with uncertainty about how these rapid advances will influence future developments.

Next Steps in Monitoring AI Progress and Policy Responses

Researchers and policymakers will likely focus on developing new benchmarks that challenge current AI systems beyond their saturation points. Further analysis is expected to assess whether these rapid improvements translate into practical, reliable AI capabilities. Additionally, discussions around regulation, safety, and ethical deployment are expected to intensify as AI approaches these new performance thresholds.

Key Questions

What does benchmark saturation mean for AI development?

It indicates that AI systems have achieved or exceeded human-level performance in specific tasks measured by these benchmarks, suggesting rapid progress and potential approaching a plateau in these areas.

Are these benchmarks representative of real-world AI capabilities?

While they measure important aspects of AI research, benchmarks may not fully capture the complexity, robustness, or safety of AI systems in real-world scenarios. Saturation suggests progress in specific tasks but not necessarily in general intelligence or reliability.

How soon could we see practical impacts from this rapid progress?

Potentially within the next few years, as AI systems become capable of performing complex research, engineering, and deployment tasks at or beyond human levels, influencing industry and policy decisions.

What are the risks associated with rapid benchmark saturation?

Risks include overestimating AI capabilities, underestimating safety challenges, and the possibility of AI systems being less robust than their benchmark performance suggests. It underscores the need for careful evaluation and regulation.

Source: ThorstenMeyerAI.com