📊 Full opportunity report: Engineering Is Automated. Research Is the Residual. on ThorstenMeyerAI.com — validation score, market gap, and execution plan.
TL;DR
AI systems are increasingly capable of automating core engineering tasks in AI development, reaching near-saturation on key benchmarks. Research, however, remains less automated and more uncertain. This shift could reshape AI R&D processes in the coming years.
Recent empirical data show that AI systems have achieved near-complete automation of core engineering tasks in AI development, while research automation remains less advanced, according to Thorsten Meyer’s analysis of Jack Clark’s recent essay.
Clark’s six benchmarks, which measure AI capability in core science and engineering tasks relevant to AI R&D, reveal a pattern of rapid progress. For example, the CORE-Bench, which assesses research reproduction, improved from 21.5% in September 2024 to 95.5% in December 2025, with the benchmark author declaring it ‘solved.’ Similarly, the MLE-Bench, evaluating performance on Kaggle competitions, advanced from 16.9% to 64.4% over sixteen months, approaching mid-tier human performance. These trajectories suggest that AI can now handle many engineering tasks with reliability comparable to skilled post-docs or engineers.
Meanwhile, progress in research-related tasks is less clear-cut. Clark notes that there is no single benchmark for research automation but highlights ongoing advances in kernel design, with models generating optimized GPU kernels and automating code conversion processes. These developments indicate that engineering is increasingly automated, but research remains a residual challenge, possibly because some aspects of research are inherently more complex or less well-defined for automation.
Engineering is automated.
Research is the residual.
Six skill benchmarks. Edison’s framing. The question Clark leaves open is whether research is just engineering at scale.
Jack Clark’s Import AI #455 catalogs six benchmarks measuring AI capability on AI R&D tasks and concludes “AI can today automate vast swatches, perhaps the entirety, of AI engineering.” The residual question is research. The structural read on the residual: it may not be a permanent moat.
Six skills. One trajectory.
Clark catalogs six benchmarks measuring AI capability on AI R&D-relevant tasks. Each individual benchmark could be noise. Six benchmarks moving together is a curve. The pattern is the cascade observed across the broader Clark series — visible here in the specific R&D-skill domain.
GPU-Accelerated Computing with Python 3 and CUDA: From low-level kernels to real-world applications in scientific computing and machine learning
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Three data points. Mixed signal.
Clark provides three data points on the creative-spark question. Yes-evidence: Erdős-1051, centaur math discovery, sporadic Move-37-style moments. No-evidence: low yield, framing dependence, absence of acceleration. The mixed signal is the honest read.
The data supports two readings. Pessimistic: rare moments suggest creative insight is qualitatively distinct from engineering work. Optimistic: rare moments are an artifact of low-volume exploration; more shots on goal yields more discoveries. Both readings are consistent with Clark’s “vast swatches, perhaps the entirety” claim. They differ on the residual.
CLAUDE AI UNLEASHED From First Prompts to Pro: The Complete Guide to Claude AI for Writing, Research, Coding, and Business (The Claude AI Mastery Series)
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Five dimensions Clark gestures at but leaves underdeveloped.
Clark’s section is rigorous on the empirical evidence. Five strategic dimensions matter for the institutional response that the Clark series synthesis argues is structurally inadequate.
![Claude AI for Beginners Bible: [5 in 1] The Ultimate Guide to Automate Your Work, Save Hours Every Week, and Use AI for Real-World Results](https://m.media-amazon.com/images/I/415+fSJacsL._SL500_.jpg)
Claude AI for Beginners Bible: [5 in 1] The Ultimate Guide to Automate Your Work, Save Hours Every Week, and Use AI for Real-World Results
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Two readings. Different equilibria.
The structural question Clark leaves open: is research a permanent moat that bounds automated AI R&D, or is it engineering at scale that dissolves with more shots on goal? Both readings are consistent with the current data. They differ by orders of magnitude in consequences.
Productivity multiplier years
Recursive loop operational
Kaggle competition AI tools
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Five audiences. Asymmetric cost of being wrong.
The institutional response should not bet on inspiration being a permanent moat. If the distinction holds, capacity built is still useful. If it closes, capacity is necessary. Asymmetric cost-of-being-wrong points toward building now.
IN INDUSTRY
IN ACADEMIA
POLICYMAKERS
INVESTORS
EVERYONE ELSE
Engineering is automated. The residual is the question. The institutional response should not bet on inspiration being a permanent moat.
Implications of Engineering Automation for AI Development
The rapid automation of engineering tasks suggests that future AI development could become more efficient and less reliant on human engineers for routine tasks. This could accelerate innovation cycles and reduce costs. However, the residual nature of research automation indicates that fundamental scientific discovery and creative problem-solving may still require human input, potentially creating a new division of labor in AI R&D. Understanding this shift is crucial for policymakers, industry leaders, and researchers planning for the next phase of AI development.
Recent Advances in AI Capabilities and Benchmarks
Over the past year, multiple benchmarks measuring AI’s ability to perform core R&D tasks have shown rapid progress. The CORE-Bench, evaluating research reproduction, has nearly reached full automation, while the MLE-Bench, testing AI on Kaggle competitions, is approaching human-level performance. These benchmarks, along with ongoing research in kernel design and code automation, reflect a broader trend of AI capabilities reaching or surpassing practical thresholds for engineering tasks. Historically, AI has struggled with automating research processes, but recent developments suggest a paradigm shift is underway, driven by advances in large language models and specialized AI systems.
“The pattern across multiple benchmarks indicates that AI can now automate vast swaths of AI engineering, with research remaining a residual challenge.”
— Thorsten Meyer
Unresolved Questions About Research Automation Limits
While engineering tasks are nearing full automation, it remains unclear how much of AI research—such as scientific discovery, hypothesis generation, and novel problem-solving—can be automated. Clark leaves this as an open question, and current benchmarks do not fully capture the complexity of research activities. The pace at which research automation might catch up with engineering remains uncertain, as does the potential need for human creativity in scientific breakthroughs.
Next Steps in Monitoring AI R&D Automation Progress
Researchers and industry will likely focus on developing new benchmarks for research automation, exploring the limits of current AI systems, and investigating how automation impacts the scientific discovery process. Monitoring the evolution of kernel design and code automation, as well as potential breakthroughs in automating hypothesis generation, will be key. Policy discussions around AI’s role in research and engineering should also consider these emerging capabilities and uncertainties, shaping future R&D strategies.
Key Questions
What does it mean that engineering is now automated?
It means AI systems can now handle many core engineering tasks involved in AI development, such as reproducing research, optimizing code, and building infrastructure, with reliability approaching that of skilled human engineers.
Why is research still considered residual?
Research involves creative, hypothesis-driven activities that are less well-defined and harder to automate, making it a more complex challenge for current AI systems.
How might this shift affect AI development in the future?
Automation of engineering tasks could accelerate AI development cycles, reduce costs, and shift the human role towards more creative and scientific activities, although the full automation of research remains uncertain.
What are the main uncertainties now?
It is unclear how much of scientific research can be automated, and whether future AI breakthroughs will fully bridge the residual gap in research automation.
What should industry and policymakers do in response?
They should prepare for a potential acceleration in AI R&D, invest in developing benchmarks for research automation, and consider the implications of shifting roles between humans and AI systems.
Source: ThorstenMeyerAI.com
