The gap between output and understanding

A A statistical method that combines results from multiple independent studies to estimate an overall effect. A single study might be too small to be conclusive; a meta-analysis pools the evidence across many studies.meta-analysis of 69 ChatGPT experiments from 2022 to 2024 (Deng et al., 2025) finds that students using ChatGPT achieve higher academic performance, feel more motivated, and score better on higher-order thinking tasks, all while exerting less mental effort. More learning for less effort? Sounds great, right?!

When researchers remove the AI from students, the gains disappear. Akgun & Toker (2025) found that advantages measured on immediate tasks had vanished entirely three weeks later. Darvishi et al. (2024) found that students who had relied on AI produced lower-quality work after its removal than students who never had it. Even Bastani et al.’s pedagogically constrained GPT Tutor, which withheld direct answers and pushed students to reason through problems, only managed to avoid harm and did not outperform the no-AI control on the subsequent exam (Bastani et al., 2025).

Interestingly, Bastani et al. also ran an NLP classification of student messages and found that in 95% of GPT Base conversations, students asked for the answer at least once. The error pattern in the data tells us something about how they treated those answers. GPT-4 was correct only 51% of the time, making logical errors on 42% of problems and arithmetic errors on 8%. A student who was reading and evaluating the solutions would catch arithmetic mistakes more easily than logical ones, since checking a calculation is simpler than evaluating a line of reasoning. But both error types reduced practice scores by similar amounts, suggesting students were accepting the output wholesale rather than, say, internalising the wrong concepts because they ‘learned’ from the AI. It is also telling that the students themselves never reported feeling that they had learned less, even though their exam scores had dropped by 17%.

Learning science predicted this

Generative AI provides exactly the conditions that the cognitive science research would predict to be harmful. GenAI reduces difficulty, provides constant guidance, and makes work feel fluent. The apparent fluency itself is a huge part of the problem because students may believe they have learned because the task felt easy, when in fact they have not built durable memory or transferable skills.

Several mechanisms contribute to this, from the straightforward (the model does the cognitive work so the student never practises it) to the subtler (the student stops evaluating the model’s output, and then stops noticing when it is wrong). I unpack the different labels researchers have given to these processes, and how they relate to each other, in a new blog post. The short version is that these labels describe the same problem at different levels, from the general mechanism to the behaviour you see in a student’s chat log.

The decades of research that Soderstrom and Bjork review show that ease during practice is a weak guide to what people retain weeks later and to how they fare on a genuinely new task. The mismatch between performance and learning existed long before GenAI, but a chatbot can now produce the polished output that used to require understanding. When we grade what students can generate in a single sitting, with or without a model whispering in the tab, we are mostly auditing that output.

Assessment built on the wrong theory

Most university courses and the studies I described above share the same assumption about how to find out whether students learned: teach them, let them practise, then remove the support and judge how they tackle a similar task on their own. Bransford & Schwartz (1999) call this “sequestered problem solving” (SPS).

Bransford & Schwartz argue that SPS, and the “direct application” theory of transfer that accompanies it, are responsible for much of the pessimism about whether education produces transfer at all. Under SPS, transfer looks rare. Students tested in isolation frequently fail to produce adequate solutions (1999, pp. 66–68), and the conclusion is that their education did not prepare them.

But consider what SPS misses. When Bransford & Schwartz (1999, pp. 66–67) asked fifth graders and college students to create recovery plans for bald eagles, neither group produced adequate plans. Under SPS, both failed. But when asked what they would need to research, the groups diverged: fifth graders asked about individual eagles (“How big are they?”), while college students asked structural questions about ecosystems, historical threats, and the kinds of specialists needed. Their prior learning had not given them the answer, but it had prepared them to ask better questions.

Bransford & Schwartz call this “preparation for future learning” (PFL). Where direct application asks “can you apply what you know?”, PFL asks “has what you know prepared you to learn new things?” The evidence for PFL is found in process: the sophistication of questions asked, the quality of hypotheses formed, the ability to seek and use resources effectively, the trajectory of improvement when given the chance to revise.

Nobody would assess a newly qualified teacher by locking her in a room and testing whether she can recall her education courses from memory. You would watch her teach over time: how she adapts to her students, how she seeks feedback, how her practice improves. That is PFL assessment. Yet for most students, in most courses, we give them the locked-room version and call the result “evidence of learning.”

SPS cannot tell us whether GenAI-mediated learning changed what students “know with,” whether it shifted their questions or their readiness to learn the next thing. Our conventional assessments cannot tell us either. As Bransford & Schwartz write: _“Despite the value of the SPS methodology, it often comes with a set of unexamined assumptions about what it means to know and understand. The most important assumption is that ‘real transfer’ involves only the direct application of previous learning; we believe that this assumption has unduly limited the field’s perspective.”

We still treat one solo shot at the problem as the verdict on learning. Preparation for future learning looks elsewhere: at how students approach problems they have not seen before, and whether they get better when given the chance to revise.

Watching how students learn

Bransford & Schwartz (1999) gave a concrete example in 1999: students attempted a geometry challenge, rated how confident they were, and chose how much help they wanted (from a brief definition up to an interactive simulation). They then tried an analogous problem. What mattered was how they responded to difficulty: did the student recognise they were stuck, pick help that addressed the gap, and improve on the second attempt?

Current assessment rarely captures any of this. A student who uses ChatGPT to produce correct code and a student who struggled with the problem, asked the AI specific questions, and modified what it returned would receive the same mark on most rubrics. The first student may not notice a gap in their understanding if the problem changed. The second student already showed that they can.

Observing learning trajectories takes time. It requires reading students’ process alongside their products, examining how they interact with tools, tracking their questions and revisions. This is labour-intensive work that does not scale through automation (yet?), because the judgment required is precisely the kind that cannot be delegated to an algorithm without recreating the problem we are trying to solve. If we outsource the evaluation of learning processes to AI, we risk the same gap at the assessment level that we identified at the student level which is the appearance of rigour without the substance.

Educators who want to assess learning rather than performance need time to observe processes, design dynamic assessments, and evaluate trajectories. This means smaller class sizes, with more time and motivation to investigate the learning process of their students when grading their work, or more teaching staff, or both. It means administrators recognising that AI-resilient assessment is not a technology problem with a technology solution. The answer may be, uncomfortably to some, more humans, not more AI.