🔍 Read the full analysis: What Happens To Consultants’ Data Science Skills Without AI? on ThorstenMeyerAI.com
Get hardware and tech essentials delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
A randomized trial at Boston Consulting Group found that consultants using ChatGPT performed better on coding, statistics and prediction tasks than a control group using conventional online resources. When both groups later took technical questions without AI, researchers found no evidence that the ChatGPT group had learned more.
A randomized trial involving consultants at Boston Consulting Group found that those given ChatGPT performed better on data science tasks outside their usual skills, but showed no detectable learning advantage when later tested without the AI tool. The study, published October 26 in Nature Human Behaviour, suggests that task performance while using AI should not be treated as proof that workers can perform the same work independently.
The trial assigned consultants to use either GPT-4 after 15 to 20 minutes of training or conventional online resources, including Google and Stack Overflow, for technical tasks. Researchers report that the ChatGPT group scored 49 percentage points higher than the control group on a Python coding task, 20 points higher on statistics and 17 points higher on prediction. The scores were measured against a benchmark of 44 BCG data scientists who completed the tasks without AI.
The AI-assisted consultants still averaged below the data scientists on all three tasks. On coding, for example, the control group scored an average of 63 percentage points below the expert benchmark, compared with 14 points below for the ChatGPT group. The authors also report that the AI group completed coding and prediction about nine and 14 minutes faster, respectively; they found no significant time difference on statistics.
After the work tasks, participants answered technical questions about probabilities, machine learning and coding without ChatGPT. The researchers say they “find no evidence” that the treated group did better than the control group on this follow-up test. The paper describes the performance gain as temporary. It also reports that the ChatGPT group became more confident that the tool could solve problems it could not; the authors characterize that finding as exploratory.
The trial in motion · BCG consultants · 2024
With ChatGPT: close to the experts. Without it: no measurable gain.
A randomized trial with a control group that used Google and Stack Overflow instead of ChatGPT.
- 1 · Training15–20 minutes on prompting ChatGPT
- 2 · TasksTwo 90-minute data science tasks, with ChatGPT
- 3 · ScoresGap to 44 data scientists closes, not fully
- 4 · Tool removedNo ChatGPT from here on
- 5 · TestFive technical questions
Coding task score
0 = data scientists’ average
ChatGPT on+49 pp on codingStill below the data scientists on average
Knowledge test without ChatGPT
ChatGPT group vs. control, five questions
ChatGPT off−7.7 to +4.8 ppNone significant · joint test p = 0.531
Source: Wiles, Krayer et al., “Generative AI and the temporary upskilling of knowledge workers”, Nature Human Behaviour (2026); authors’ manuscript, Sections 2.1 and 2.3, Tables B7 and B9. Bars scaled to the five published differences.
Graphic: Powered by Thorsten Meyer AI
AI Raises Output, Not Tested Skills
The results distinguish between doing a task with AI and learning to do it without AI. For employers, that difference affects how they interpret productivity gains, staff training and workers’ technical capabilities. A person who produces a stronger analysis with a chatbot may still need support to check, explain or reproduce the work independently.
The study does not establish that AI cannot teach technical skills. It tested a particular short training session, a specific set of tasks and a later assessment. But its findings caution against assuming that access to a capable tool automatically builds lasting expertise. For work where errors carry consequences, companies may need to assess both the AI-assisted result and the worker’s ability to evaluate it.
Products Worth Considering
Logitech MX Ergo S Advanced Wireless Trackball Mouse USB-C – Graphite
Ledger Flex Crypto Wallet with 2.8-Inch E Ink Touchscreen
Logitech MX Keys S Combo Wireless Keyboard and Mouse, Graphite
As an affiliate, we earn on qualifying purchases.
How the BCG Trial Worked
The experiment took place at BCG in late March and early April 2024 and was preregistered before data collection. Of 986 consultants allocated to the two groups, 573 started the survey and 487 submitted work on both assigned tasks; the authors used that 487-person group for their main analysis. Participants had at least a bachelor’s degree, and most reported no coding experience.
Each consultant completed two of three 90-minute exercises: cleaning and combining datasets in Python, evaluating a statistical model, or building a model related to the predictability of international football matches. The researchers say the tasks were designed so ChatGPT could provide useful guidance but could not independently answer every question correctly. The study was published more than two years after the trial began.
The comparison with data scientists offered an expert reference point, while the control group helped distinguish the effect of ChatGPT from access to online information generally. The design therefore tested more than whether a chatbot could supply answers: it also examined whether brief exposure to the tool was followed by improved performance when the tool was unavailable.
““find no evidence””
— The study authors
Limits of the Follow-Up Test
The study does not show how long any learning might take to develop with more extensive training, repeated practice or different teaching methods. The reported follow-up used five technical questions, so its results address that assessment rather than every form of knowledge retention or later job performance.
The findings also come from one employer, a specific group of consultants and tasks designed with input from OpenAI’s Economic Impacts Research team. The source material does not provide a basis for generalizing the results to every profession, AI system or workplace. The confidence finding is described by the authors as exploratory, not as a definitive measure of how workers assess AI in general.
What Employers Should Measure
The study reports no follow-up experiment or planned next milestone. Its immediate implication for workplace evaluations is to distinguish AI-assisted output from independent skill: employers can test whether staff can explain, check and reproduce work after the tool is removed, rather than relying only on results produced with AI.
Further research would be needed to establish whether longer or differently designed instruction produces lasting gains, and whether the pattern holds across jobs and tasks. Until then, this trial supports a measured conclusion: ChatGPT improved performance on the tasks tested while available, but researchers found no corresponding advantage on the short, tool-free assessment.
Key Questions
What did the BCG study find?
Consultants using ChatGPT scored higher than a control group on coding, statistics and prediction tasks. On a later technical test without ChatGPT, researchers found no evidence that the AI group performed better.
Did the ChatGPT group perform as well as data scientists?
No, not as a whole. The group averaged below the 44 BCG data scientists who completed the tasks without AI, although it moved closer to the expert benchmark on all three exercises.
How many consultants were included in the main analysis?
The main analysis included 487 consultants who submitted work on both assigned tasks. A total of 986 consultants were allocated to the groups, and 573 began the survey.
Does the study prove AI cannot teach workers new skills?
No. It tested brief training, a specific group of consultants and a short tool-free follow-up assessment. The authors’ finding is that this trial showed no evidence of a learning advantage on that assessment, not that longer or different instruction could never work.
Source: ThorstenMeyerAI.com
