Building AI tutors that preserve your course’s sequence and your way of teaching

Dr Jon Cardoso-Silva
Associate Professor (Education)
šŸ“§ J.Cardoso-Silva@lse.ac.uk

Department of Methodology, LSE

20 Nov 2026

About me

  • Associate Professor (Education), LSE Department of Methodology
  • PhD in Computer Science, King’s College London (2018): optimisation and network science
  • One of ten LSE AI and Education Fellows: a programme giving academics a term’s teaching buyout, over two years, to redesign teaching, learning, and assessment around AI tools
  • My Fellowship grew out of GENIAL, a study of how students use generative AI during coursework

The GENIAL study

GENIAL: When do Generative AI tools act as a catalyst for learning?

  • A collaborative project across several LSE departments
    • Data Science Institute
    • Department of Statistics
    • Department of Management
    • School of Public Policy
  • Timeline: July 2023 to April 2024
  • Funding: internal (LSE Eden Centre for Educational Enhancement, LSE Data Science Institute)

We asked students to share their chat logs and in the case of my data science courses, I also collected the git histories of their assignments.

DS105 - Data For Data Science

  • The course has many assessment cycles and is very experiential by design.

A contrastive example

Looking at the data, it was very clear just how much of a double-edged sword GenAI could be: two second-year undergraduates doing the same assignment, both using ChatGPT but with very different approaches to their independent study practices.

Student A Second year, International Social and Public Policy. No prior programming experience.

The task involved writing code to:

  • Navigate to a page
  • Grab the entire content of that page
  • Select just the relevant information
  • Clean it up
  • Store it

Student A copy-pasted links and HTML by hand, because ChatGPT’s example output looked like the finished task and never asked for the scraping code.

Student B Second year, Economics. Had coded in Python before.

ChatGPT often produced overcomplicated code. The student went back to the teaching material and told the AI:

ā€œyour stuff is not in line with what I learnedā€

Our findings and the GENIAL framework

  • Students treat GenAI differently for learning than for assessment, and focus more on the output than on the learning journey itself.
  • GenAI seems most beneficial to learning when students take the initiative (self-agency) and direct the tool themselves.
  • The performance paradox: chatbots give students an illusion of competence, and without process tracking we cannot really know what the student knows anymore (something that is now common sense I’d dare say).
  • Our GENIAL framework, developed from 2023 to 2025, lets educators map out how a course activity should unfold, and then lets researchers check, from chat logs, whether that is what happened.

Sallai, Cardoso-Silva, Barreto, Panero, Berrada & Luxmoore (2024). Approach Generative AI Tools Proactively or Risk Bypassing the Learning Process in Higher Education. LSE Public Policy Review.

Cardoso-Silva, Sallai, Kearney, Panero & Barreto (2025), the GENIAL framework, SSRN.

Khan Academy’s Socratic tutor

Sal Khan on stage at the LSE Data Science Institute event in June 2024

Brave new words: how AI will revolutionise education, with Salman Khan Ā· DSI Event Ā· Wed 26 Jun 2024

šŸ“• Read my book review.

  • Khanmigo is perhaps the most famous pedagogical AI tutor in the market. Sal Khan, founder of Khan Academy, called AI tutoring the biggest positive transformation education has ever seen in his 2024 book.
  • Khanmigo is perhaps the most famous pedagogical AI tutor in the market. Sal Khan, founder of Khan Academy, called AI tutoring the biggest positive transformation education has ever seen in his 2024 book.
  • In an interview in April 2026, though, Sal Khan was saying that for many students Khanmigo had been a non-event (so far?).

Khosravi et al. paper, pages 13 and 14, each thumbnail with an arrow to its underlined sentence zoomed beside it

Khosravi, H., Gasevic, D., Sadiq, S., et al. (2026). Building AI Companions that Prioritise Learning over Performance. Preprint, arXiv, pp. 13–14. doi.org/10.48550/arXiv.2605.04816

  • Khanmigo is perhaps the most famous pedagogical AI tutor in the market. Sal Khan, founder of Khan Academy, called AI tutoring the biggest positive transformation education has ever seen in his 2024 book.
  • In an interview April 2026, though, Sal Khan was saying that for many students Khanmigo had been a non-event (so far?).

  • It started out as a ā€œSocraticā€ bot, but seeing the frustration of students, Khanmigo developers adjusted it: the tutor now offers worked examples and hints that fit what each student requires, alongside its questions.

Limits of Socratic mode

Default ā€œLearning modeā€ / Socratic mode, in use by all major AI companies, is NOT a good pedagogical feature.

Claude responding to a student’s question used as a study tool, with an ever-sprawling number of sub-questions.

  • We followed one postgraduate student using Claude’s Learning mode during an entire term for a proof-based mathematics course (MSc Level).
  • In several occasions, the student asked very specific and narrow questions about an exercise uploaded to the chatbot, and Claude would go on to throw 20-30 numbered or lettered blocks of questions across to the student several messages in a row.

It gives me five really great conceptual questions that I should think about, but then it gets so lost in the tiny little sub-questions that I don’t even get to talk about the five big ones.

Lena (pseudonym), postgraduate student, interview after the course

Cardoso-Silva, J., & Ruf, J. (2026). Coexistence of mathematical reasoning and cognitive offloading: A longitudinal case study of one student’s chatbot use across a term of proof-based mathematics. Preprint, SSRN. doi.org/10.2139/ssrn.7608220

How I’m building AI tutors

…while trying to overcome the limitations mentioned before

A pedagogic and time-aware AI tutor

(simplified high-level view) The goal is to give the LLM enough context on the why we do things in our courses and a particular order and how, that is the way we expect students to learn in our disciplines.

Content
🧭
šŸ—“ļø
🌐
šŸ‘¤
Why does \(s^2 = \frac{1}{n-1}\sum (x_i - \bar{x})^2\) divide by \(n-1\), not \(n\)?
🧩 Router
Takes the course's content and the specific protocols into account to match each message
šŸ“‹ starting_profile_requirement
Seeds the learning profile before routing any request
šŸ”€ single_task_conversation
Keeps one conversation to one task
🧮 formula_intuition
Explains what a formula means before the symbols
šŸŒ‰ everyday_scene
Builds an everyday scene around a concept the student does not get
šŸ› ļø assignment_guidance
Coaches planning and debugging, never writes code
šŸ’¬ general
Default answer mode when no other protocol matches
✨ Responder
Writes the reply from the chosen protocol and the content
✨
The reason is in something we saw in the šŸ–„ļø W03 Lecture: \(\bar{x}\) is built from your own \(n\) data points, so it already uses up one degree of freedom...

I built and deployed this as a Claude Project. Let’s have a look ā­ļø

The four components of the tutor

While this software is not ready, I was able to replicate this behaviour using only prompts and text files.

How this ā€˜prompts-only’ version works

  • I wrote those components in plain text, following a specific structure and uploaded it to the Project Instructions of the AI platform.
  • Each week I had to edit CURRENT WEEK and CURRENT PHASE manually. Claude’s clock inference was unreliable.
  • weekly/ and assignments/ encode what the course has covered so far, up until the current moment in the course.

āš ļø This is an old template, shown here just so you have a more concrete idea of how it works! We are going to publish a preprint and a tutorial, along with some of our newer prompt templates you could adapt to Claude, ChatGPT or Google Gemini for your own students.

How students used it

DS105W in Winter Term 2025/26

  • We deployed this in the DS105W - Data for Data Science course in Winter Term 2025/2026 (January to April 2026), a course with 103 students.
  • My course philosophy follows the logic of semi-continuous assessment and a sort of flipped classroom: I give students formative tasks that involve skills and concepts I only introduce afterwards.
  • 60 students agreed to share their chat logs and git histories with the study. 53 of them shared at least one chat log, and 40 shared at least one chat log from our AI tutor.
  • Although we don’t have access to official data, I suspect ~80% of students used my AI tutor in the course.

DS105W (Winter 2025/26): formative practice W01 to W04, then Mini Projects across the term.

Daily message volume across the teaching term. Counts include only the 40 students who shared at least one chat log from our AI tutor. Background bands mark assessment periods; dashed lines mark Thursdays (lecture day).

Guiding questions on coursework

Every time a student asked about a concept, the AI tutor pointed them to where and when in the course they had already met it, before asking what approach they would take.

Direct help with setup

Questions about platform, set up, or more operational in nature didn’t go ā€œSocraticā€ but were delivered as a straight answer to the student.

Withholding topics not yet taught

Questions about off topic content, in relation to where we were in the course, were dealt with correctly.

The AI tutor correctly indicate to the student in Week 04 that they should not (yet) use the pandas library at this stage in the course. The tutor reinforced the fact I wanted people to explore different tools at different points in the course.

Later in the term, answering to another student about suitability of libraries, the AI tutor correctly indicated that at that point in the course, the pandas library was the most adequate choice, as indicated in the course.

The course leader’s guidance

The AI tutor (and the students) sometimes referred to me by name.

  • I found 60 occurrences of my name in the dataset of chat logs šŸ™ƒ

  • Thankfully, every one of those replies was accurate to what I actually instructed the chatbot to say. It didn’t fabricate anything.

    Note though that fabricated attributions are indeed a risk of this type of technology. We can’t forgo AI literacy training.

  • Students themselves mentioned my name in 16 messages, across 22 chat logs.

Protocol activation

  • Unsurprisingly, most student queries were about help with coursework. During this study, we only collected chat logs at the points of coursework submission.
  • One potential side-effect: office hours for technical support dropped. Students seemed to come more for conceptual questions instead.

Each square represents 20 messages exchanged with the AI tutor.

Limitations of this design (in Claude)

We could not control programmatically how Claude was responding to students: the longer a conversation ran, the more likely the LLM was to drift from the instructions.

The student should be the one justifying its choices. Here, the AI tutor offers a justification when the student didn’t even ask for it!

In a later implementation, we added a protocol (always on) to make sure conversations don’t change topics or go longer than 15 to 20 messages on the same topic.

Specific prompt engineering techniques

1 Write model questions and answers, not ā€œwhen a student asks X, do Y.ā€ An LLM given the second form parrots the instruction back to the student, word for word.
2 Keep each protocol’s trigger distinct from every other protocol’s. If two protocols could both plausibly match the same message, the LLM has to guess which one you meant.
3 Start with one system prompt, then test it by playing a student, before splitting it into separate files.

Courses using the design

  • Instructors translate their own tacit knowledge of how they teach into protocols and weekly briefings, written in plain text, no programming required.
  • The tutor then answers only from what the course has taught so far, in the order the course leader chose to teach it, and switches between direct help and Socratic questioning to fit the student’s situation.
  • Tested on:
    • DS105A and DS105W (data science and programming courses)
    • MY202A (machine learning)
    • MG317 (organisational learning)
    • AC102 and AC105 (introductory accounting courses), amongst others

Knowledge tracing

What I am testing now: give the chatbot the skills we expect each student to demonstrate awareness of or proficiency in each week.

MY202A, Week 01.

What I’m building now

Content
🧭
šŸ—“ļø
🌐
šŸ‘¤
Why does \(s^2 = \frac{1}{n-1}\sum (x_i - \bar{x})^2\) divide by \(n-1\), not \(n\)?
🧩 Router
Takes the course's content and the specific protocols into account to match each message
šŸ“‹ starting_profile_requirement
Seeds the learning profile before routing any request
šŸ”€ single_task_conversation
Keeps one conversation to one task
🧮 formula_intuition
Explains what a formula means before the symbols
šŸŒ‰ everyday_scene
Builds an everyday scene around a concept the student does not get
šŸ› ļø assignment_guidance
Coaches planning and debugging, never writes code
šŸ’¬ general
Default answer mode when no other protocol matches
✨ Responder
Writes the reply from the chosen protocol and the content
✨
The reason is in something we saw in the šŸ–„ļø W03 Lecture: \(\bar{x}\) is built from your own \(n\) data points, so it already uses up one degree of freedom...

The current prototype (preliminary!)

You would fill out a form describing your course and how you want the tutor to behave.

Course description

Teaching approach

  • How do you expect students to learn in this course?
  • How is your course taught?
  • What role do you want the AI tutor to have?
  • Platforms, links and references useful throughout the course
  • How should the tutor respond if a student tries to get round its rules?

Writing protocols

Trying it as a student

Thank you

Cardoso-Silva, J., Sallai, D., Kearney, C., Panero, F., & Barreto, M. E. (2025). Mapping Student-GenAI Interactions onto Experiential Learning: The GENIAL Framework. SSRN Electronic Journal. (Under Review)

Sallai, D., Cardoso-Silva, J., et al. (2024). Approach Generative AI Tools Proactively or Risk Bypassing the Learning Process in Higher Education. LSE Public Policy Review, 3(3), 7.

Cardoso-Silva, J., & Ruf, J. (2026). Coexistence of mathematical reasoning and cognitive offloading: A longitudinal case study of one student’s chatbot use across a term of proof-based mathematics. Preprint, SSRN.

Cardoso-Silva, J., & Williams, S. (2026). Beyond Socratic modes: designing a chatbot tutor around the course’s sequence and pedagogy. Preprint coming soon.

Contact me if you’d like to exchange ideas, or to test these ideas and this analysis on your own courses or research.

The whole ā€œhuman in the loopā€ concept, borrowed from software engineering, doesn’t make sense in education. We should be thinking of ā€œAI in the loopā€ instead.

Blog: jonjoncardoso.github.io/blog

šŸ“§ J.Cardoso-Silva@lse.ac.uk