In this experiment, an unrestricted AI assistant improved practice performance but lowered subsequent unaided scores. A teacher-informed tutor largely avoided that harm, without demonstrating an independent-test advantage over the control group.
The paper
Hamsa Bastani and colleagues published “Generative AI without guardrails can harm learning: Evidence from high school mathematics” in PNAS on June 25, 2025. DOI: 10.1073/pnas.2422633122.
What the researchers tested
Nearly 1,000 students at one Turkish high school participated. Classrooms were randomized to ordinary resources, a GPT-4 chat assistant, or a GPT-4 tutor using teacher-informed hints and solutions. Four sessions combined review, practice, and an unaided exam.
What they found
Both AI groups improved during assisted practice. On unaided exams, the unrestricted-assistant group scored about 17% lower relative to the control group. The guarded tutor showed no statistically significant difference from control. That is not evidence of superior independent learning.
What this study cannot tell us
This was one school, mathematics, and a specific GPT-4 implementation. Immediate exams do not establish long-term retention. The findings cannot settle how every AI tutor performs across subjects, ages, or newer models.
Questions educators can take into the classroom
The following are our practical suggestions, rather than additional findings from the experiment. Before adopting a tool, define the skill you want learners to carry away. Ask how you will notice that skill when the tool is unavailable.
For a small classroom pilot, have students attempt a problem, use help, and then tackle a fresh problem independently. Keep the original attempts and explanations alongside the final answers. This can give you something concrete to discuss with students; it does not reproduce the study or establish a causal effect.
When reviewing an AI product, ask for examples of what it does when a learner requests the answer immediately. Can a teacher inspect or configure the help? What evidence supports its learning claims? Look for an evaluation that matches your students and intended use.
Treat a polished answer as the start of a conversation. Ask the learner to explain a step, identify a mistake, or apply the idea in another context. Choose those follow-up tasks around the lesson objective.