The emergence of generative artificial intelligence in late 2022 fundamentally altered the educational landscape, forcing a reckoning between technological potential and pedagogical reality. While early iterations of chatbots like ChatGPT were swiftly adopted by students as shortcuts for completing homework, the unintended consequence was a documented decline in critical thinking and math proficiency. In response to this trend, education technology leaders sought to pivot, proposing that AI could move beyond the role of an "answer machine" to become a "Socratic tutor." Among the most prominent efforts was Khan Academy’s release of Khanmigo in 2023. However, a comprehensive two-year study conducted between 2024 and 2026 by researchers at the University of Toronto suggests that the existence of an AI tool is insufficient to improve student outcomes if student engagement remains minimal.
The Evolution of the AI Tutor Model
Following the initial, chaotic rollout of Large Language Models (LLMs) in classrooms, the educational sector faced a significant challenge: how to harness the speed of AI without sacrificing the rigor of learning. Khan Academy, an organization long synonymous with digital learning tools, introduced Khanmigo to fill this void. Unlike basic chatbots, Khanmigo was programmed to act as an instructional guide, withholding direct answers while prompting students with hints and leading questions—the hallmarks of the Socratic method.
The hypothesis was straightforward: if students could access a personalized, 24/7 tutor that offered real-time feedback, they would be better equipped to master complex mathematical concepts. However, the reality of classroom integration proved more complex than the theory suggested. The study, led by researchers Philip Oreopoulos and Nina Low, examined the interaction patterns of students across 18 middle schools in Tennessee, tracking usage and performance metrics over two academic years.
Chronology of the Tennessee Experiment
The study was structured as a randomized controlled trial (RCT) involving students who were identified as needing remedial support—specifically, those performing at least one grade level behind their peers. These students were already enrolled in dedicated remedial math courses.
- 2023: Khanmigo is released to the public, designed to function as an AI-powered assistant for homework and skill practice.
- 2024: Researchers begin tracking student interactions in 18 Tennessee middle schools. Half of the cohort is provided with access to the Khan Academy ecosystem, including Khanmigo, while the other half continues with existing digital remedial tools like Zearn, IXL, and DeltaMath.
- 2025: Data collection continues, revealing a clear trend: while students were initially curious about the AI tool, consistent usage plummeted over time.
- 2026: Final analysis of the data indicates that while the Khan Academy curriculum provided moderate academic benefits, the presence of the Khanmigo AI tutor provided no measurable, statistically significant improvement over traditional practice methods.
Analyzing the Engagement Gap
The most critical finding from the NBER-circulated paper is the discrepancy between the availability of technology and its utilization by the student population. The researchers observed that while nearly all participants engaged with the AI assistant during the initial stages of the study, the frequency of use dropped sharply once students realized the bot would not provide immediate answers to their math problems.
"Seeking help with one’s own confusion remained a choice, and most students declined it most of the time," noted Oreopoulos and Low. This "engagement gap" highlights a fundamental psychological barrier in educational technology: students often prioritize efficiency—finishing the assignment as quickly as possible—over the deeper, often frustrating process of learning. When the AI refused to act as a shortcut, the students perceived it as an obstacle rather than an aid.
Furthermore, the data showed that students who did use the tool often attempted to "game" the system, testing the AI to see if it would eventually break its programmed constraints and provide the solution. When Khanmigo remained firm, the students simply stopped logging in.
Official Responses and Strategic Shifts
Sal Khan, the founder of Khan Academy, has been transparent about the findings, choosing to view them as a necessary data point in the iterative process of software development. In a public blog post and subsequent interviews, Khan acknowledged that the internal metrics at Khan Academy mirrored the findings of the University of Toronto study.
"It’s allowed us to learn and hopefully make the new version even more helpful," Khan stated. He emphasized that the primary focus during the early phase of the rollout was not just efficacy, but safety. The organization implemented strict guardrails to protect student data and prevent the "hallucinations" that characterize many generative AI models.
Responding to the data, Khan Academy has begun a strategic shift in how Khanmigo is integrated into the user experience. The tool is no longer a separate, voluntary tab; it is now woven into the fabric of the platform. If a student answers a problem incorrectly, the AI is programmed to proactively initiate a conversation, rather than waiting for the student to ask for help. Additionally, the organization is testing "gamified" incentives, such as allowing students to earn "credit" for using the tutor to work through a mistake, which then counts toward their progression requirements.
Implications for the Future of AI in Education
The findings from the Tennessee study carry significant implications for school districts and policymakers who are currently rushing to integrate AI into their curriculum.
The Myth of Passive Learning
The study serves as a reminder that AI is a tool, not a panacea. The mere presence of an advanced AI in the classroom does not guarantee learning. Success depends on the integration of the tool into the existing pedagogical workflow and, crucially, the motivation of the learner. If students view the tutor as a barrier to completion, the technology will fail regardless of how advanced its underlying algorithms may be.
The Role of Teacher Intervention
A key takeaway from the report is that technology is most effective when it complements, rather than replaces, the human element of instruction. The study suggests that for tools like Khanmigo to be truly effective, teachers must play an active role in encouraging their use and modeling how to interact with the AI in a way that promotes cognitive struggle rather than avoidance.
Policy and Data Privacy
As school districts move forward, the "no harm" approach championed by Khan Academy remains a gold standard for implementation. While the efficacy of the tool is still being refined, the commitment to protecting student privacy and preventing cheating remains the primary prerequisite for any widespread adoption of AI in public education.
Looking Ahead: What Comes Next?
The next phase of AI in the classroom will likely be defined by "active intervention" models. By forcing the AI to engage when a student struggles—rather than waiting for a request for help—developers hope to bridge the engagement gap. However, the academic community will remain vigilant. Future studies will need to move beyond simple engagement metrics to determine if these new, more proactive AI models actually result in better retention and higher test scores over the long term.
For now, the lesson for educators and parents is clear: the integration of AI into schools is not a sprint, but a long-term experiment. The technology is evolving, and the methods for teaching students how to use it as a tool for inquiry, rather than a crutch for completion, are still in their infancy. As researchers continue to monitor the impact of these changes, the focus will remain on whether these sophisticated digital tutors can eventually foster the level of cognitive engagement required to boost academic performance in a measurable, meaningful way.
