🔍 Read the full analysis: How AI Affected Consultants’ Data Science Skills After Use on ThorstenMeyerAI.com
Get smart everyday buys delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
A randomized trial involving Boston Consulting Group consultants found that ChatGPT improved performance on coding, statistics and prediction tasks when participants could use it. In a follow-up without the tool, researchers found no evidence that the AI group performed better on technical questions than the comparison group.
A randomized trial involving Boston Consulting Group consultants found that access to ChatGPT raised performance on three data-science tasks, but researchers found no evidence of better technical knowledge when participants later answered questions without the AI tool. Published in Nature Human Behaviour on October 26, the study distinguishes improved work with AI from skills workers can demonstrate on their own.
The trial assigned consultants to use either GPT-4 after 15 to 20 minutes of prompting training or conventional online resources, including Google, Stack Overflow and Khan Academy. Participants completed two of three 90-minute tasks: Python data cleaning, statistical analysis or a prediction exercise. The study was preregistered before data collection, which took place at BCG in late March and early April 2024.
In the main analysis, based on 487 consultants who submitted work on both tasks, the ChatGPT group scored 49 percentage points higher than the control group on coding, 20 points higher on statistics and 17 points higher on prediction. The researchers compared results with the average performance of 44 BCG data scientists who did the tasks without AI. The ChatGPT group remained below that expert benchmark on average across all three tasks.
After the work tasks, participants answered five technical questions without ChatGPT. The authors reported no evidence that the AI group did better than the control group on that assessment. They also found the ChatGPT group was more confident that the tool could solve problems it could not; the authors described that finding as exploratory.
The trial in motion · BCG consultants · 2024
With ChatGPT: close to the experts. Without it: no measurable gain.
A randomized trial with a control group that used Google and Stack Overflow instead of ChatGPT.
- 1 · Training15–20 minutes on prompting ChatGPT
- 2 · TasksTwo 90-minute data science tasks, with ChatGPT
- 3 · ScoresGap to 44 data scientists closes, not fully
- 4 · Tool removedNo ChatGPT from here on
- 5 · TestFive technical questions
Coding task score
0 = data scientists’ average
ChatGPT on+49 pp on codingStill below the data scientists on average
Knowledge test without ChatGPT
ChatGPT group vs. control, five questions
ChatGPT off−7.7 to +4.8 ppNone significant · joint test p = 0.531
Source: Wiles, Krayer et al., “Generative AI and the temporary upskilling of knowledge workers”, Nature Human Behaviour (2026); authors’ manuscript, Sections 2.1 and 2.3, Tables B7 and B9. Bars scaled to the five published differences.
Graphic: Powered by Thorsten Meyer AI
Better Task Performance, Not Proven Skill
The results matter for employers deciding whether AI access counts as workplace upskilling. In this trial, consultants using ChatGPT completed unfamiliar technical work more successfully than people using conventional web resources. That has immediate implications for how organizations allocate tasks and support staff, even if the workers do not acquire independent expertise.
The follow-up sets a boundary on what the performance gains establish. It found no evidence of a corresponding advantage when the tool was unavailable, so the study does not show that brief AI training taught participants to perform the tasks unaided. For companies, that distinction affects training plans, hiring decisions and quality checks: successful AI-assisted output should not automatically be treated as proof of a worker’s standalone capability.
The researchers’ findings also do not establish that AI never supports learning. They measure one short intervention and a particular set of tasks and follow-up questions. The narrower takeaway is that this experiment detected task gains with the tool, but no detectable independent gain on its post-task test.
Popular right now
Bose QuietComfort True Wireless Earbuds with Active Noise Cancellation, Bluetooth 5.3, USB-C Charging, IPX4, Black
Sony BRAVIA 6 65-Inch OLED 4K HDR Smart Google TV with Gemini (K-65A60)
Vitamix 5200 Blender with 64 oz Container, Black
As an affiliate, we earn on qualifying purchases.
How the BCG Trial Was Designed
The study, titled “Generative AI and the temporary upskilling of knowledge workers,” was conducted at BCG in March and April 2024 and published in 2026. Of 986 consultants allocated to the study, 573 began the survey and 487 submitted work on both assigned tasks. All had at least a bachelor’s degree; the authors said most reported no coding experience.
The tasks were designed by BCG data scientists with OpenAI’s Economic Impacts Research team. According to the study, ChatGPT could help participants but could not correctly answer the questions independently, so copying a response was not sufficient. The researchers also examined task completion time: the ChatGPT group finished coding and prediction faster on average, while the time difference for statistics was not significant.
Among participants the paper classified as competent coders, the AI group scored five percentage points below the data scientists on coding and statistics; the authors found no evidence of a difference from experts on those two tasks. That subgroup result does not change the overall finding: the full ChatGPT group did not match the expert average.
““Generative AI and the temporary upskilling of knowledge workers””
— The study authors
Limits of the Skill Test
The follow-up consisted of five technical questions on related topics, rather than a repeat of the full work tasks. The available results do not establish whether longer or different training, repeated practice, or continued use would produce durable independent skills. The study also took place at one consulting firm and examined selected tasks, so its findings do not automatically apply to every job or AI system.
The source material does not provide details on how long after the work tasks the follow-up occurred. Nor does the reported finding of no evidence of better performance prove that participants gained no knowledge at all; it means the researchers did not detect an advantage for the ChatGPT group on that assessment. The confidence finding was explicitly described by the authors as exploratory, and should not be treated as a definitive measure of how users judge AI across settings.
Questions for Workplace Training
The reported study does not identify a scheduled next experiment or a follow-up milestone. For organizations applying the results, the practical next step is to distinguish AI-assisted output from unaided skill when evaluating training. Employers can test both: whether staff can complete a task with the system and whether they can explain or reproduce the work without it.
Further evidence would be needed to determine whether longer training, repeated tasks or different forms of instruction change the post-tool result. Until then, the trial supports a limited conclusion: ChatGPT helped participants perform better on the studied tasks while available, but the researchers found no post-task advantage without it.
Key Questions
What did ChatGPT change in the trial?
Compared with the group using conventional web resources, consultants with ChatGPT scored 49 percentage points higher on coding, 20 points higher on statistics and 17 points higher on prediction.
Did consultants retain the skills after ChatGPT was removed?
On five technical questions completed without ChatGPT, the authors found no evidence that the AI group did better than the control group. That result does not prove that no individual learned anything.
Did the ChatGPT group perform as well as data scientists?
No, not on average. The study compared consultants with 44 BCG data scientists who completed the tasks without AI, and the overall ChatGPT group remained below their average on all three tasks.
How many consultants were included in the main analysis?
The main analysis included 487 consultants who submitted work on both assigned tasks. A total of 986 were allocated to the study, and 573 began the survey.
Does the study show that AI cannot teach data science?
No. It tested a short prompting training, particular tasks and a five-question assessment without the tool. The findings show no detectable advantage on that follow-up, not that longer or different instruction could never build lasting skills.
Source: ThorstenMeyerAI.com
