Two interfaces differed in what students had to do first
The study ran in an undergraduate proof-based mathematics course covering logic, sets, relations, and proof writing. Both AI tools called the OpenAI API without fine-tuning. The AI chatbot accepted mathematics questions at any point and allowed repeated exchanges; its system prompt told it to avoid final answers and complete proof steps in favor of Socratic questions, scaffolds, and partial hints, but students could still press for more direct help. The homework-feedback interface instead required a written proof attempt before returning critique, using the same prompt, instructor reference solution, and structured feedback format.
That design difference is the paper's central contrast. Proof writing requires constructing and explaining a chain of reasoning rather than recognizing a plausible answer, so whether a student attempts reasoning before receiving help matters for what the help can develop. The authors frame this through the assistance dilemma: too little help leaves a student stuck, while too much can replace the activity needed for learning. Cognitive offloading, in this setting, means obtaining proof steps from the AI instead of generating them.
Randomized access lifted homework but not the next exam
After Midterm 1, 153 students with grade and usage records were randomly assigned as individuals to receive both interfaces earlier (75 students) or later (78 students). During the staggered-access window, only the early group could use them for homework. The two groups started close: Midterm 1 means were 85.45 and 83.88, and Homework 1–4 averages were 85.09 and 83.59.
During that window, 59 of 75 early-access students and none of the 78 later-access students logged at least one chatbot message or homework-feedback request. Adjusting for prior homework average and assignment, the early group scored 2.56 points higher on Homework 5–6 (SE 1.22, 95% CI [0.17, 4.94], p = .036; 306 observations from 153 learners). Raw means were 98.16 versus 95.77 on Homework 5 and 100.48 versus 95.91 on Homework 6; bonus items allowed scores above 100.
The next exam did not follow. On Midterm 2, completed without AI, raw means were 69.95 for the early group and 71.52 for the later group; adjusting for Midterm 1 gave an estimate of −2.47 points (95% CI [−5.60, 0.66], p = .122; N = 152). Because the exam used similar problem types but different questions, the authors read this as no detectable transfer from supported homework to independent proof performance. Randomization covered access timing only, not which interface students chose or how much they used it, so this comparison cannot separate the two interfaces' effects.
Lower confidence predicted heavier use of both tools
After Midterm 2, all students had access and chose their own usage. Among 152 students with self-efficacy and Midterm 2 data, 106 used the chatbot and 90 requested homework feedback. Baseline self-efficacy came from four survey items rated 1–7 and averaged, with high internal consistency (Cronbach's α = 0.91).
At the same Midterm 2 score, a one-point higher self-efficacy score predicted 39% fewer student messages to the chatbot (p < .001) and 31% fewer homework-feedback requests (p = .002). Higher prior exam scores were associated with lower predicted use of both interfaces, but neither association was statistically significant (incidence rate ratio .982 for chatbot messages, p = .056; .989 for feedback requests, p = .393). In short, less confident students sought more help from both tools, which is why the usage analysis controls for self-efficacy and prior performance.
More chatbot messages tracked with lower exam scores
Between Midterms 2 and 3, controlling for baseline self-efficacy, Midterm 2, and homework-feedback use, each additional student message to the chatbot was associated with a .0302-point lower Midterm 3 score (95% CI [−.0567, −.0036], p = .026; N = 148), or −3.02 points per 100 messages. The homework-feedback association was positive but not statistically reliable (B = 0.084, p = .131).
Two robustness checks kept the chatbot association negative: adding original access group gave B = −.0284 (95% CI [−.0543, −.0026], p = .031), and using log(1+x) counts gave B = −1.53 (95% CI [−2.64, −.41], p = .007). The log-scale coefficient is on a different scale and should not be compared numerically with the per-message figure. These are associations from self-selected use, not randomized effects of the interfaces, and the count models pool users and nonusers, so they do not separate taking up an interface from using it more once adopted.
A confidence path ran through chatbot use
The mediation analysis on 148 students traced an indirect association from baseline self-efficacy to Midterm 3 through chatbot messages: B = .622 exam points per one-point increase in self-efficacy (95% bootstrap CI [.121, 1.560]; Holm-adjusted p = .021). Higher self-efficacy predicted fewer messages after accounting for Midterm 2 (B = −20.62, β = −.254, p = .004), and more messages predicted lower Midterm 3 scores after accounting for self-efficacy, Midterm 2, and feedback requests (B = −.0302, β = −.180, p = .015).
Read in the other direction, a one-point lower self-efficacy score corresponded to 0.62 fewer exam points through chatbot use. The path through Midterm 2 alone was positive (1.975 points, 95% CI [1.028, 3.226]), and the direct self-efficacy coefficient in the Midterm 3 equation was 4.18 points (β = .307, p < .001). The serial path through Midterm 2 and chatbot use, and both paths involving homework feedback, had intervals including zero. The authors present cognitive offloading as a likely interpretation, not a measured mechanism: mental effort and students' use of AI-generated reasoning were not directly measured, and unmeasured differences among students could contribute to the indirect association.
Dialogues show learning goals alongside delegated steps
Two four-turn sessions illustrate the contrast. In Case A, a student asks the chatbot to solve a congruence problem while saying they will try to make sense of the solution, then responds to an invitation to verify the formula with "Could you do the next step." The chatbot's construction also swaps the two given inverses and assumes a relation that was not given. In Case B, a student shares an attempted counting solution; the chatbot incorrectly describes the method as overcounting, directing attention away from the actual error.
The authors use these cases to argue that asking a student a question is not sufficient: the question must target the actual error in the student's reasoning, and systems should check the mathematical basis of their feedback. Neither session includes a later student explanation that resolves the difficulty. The dialogue analysis is illustrative, drawn from a labeling corpus of 659 sessions from 106 users, and the intent classifier labeled only the first student message, so a session that began as help-seeking could later become answer-seeking.
The design priority is feedback on attempted reasoning
The paper's practical split follows from its own comparison: chatbot use was negatively associated with later exam scores, while homework-feedback use, which required a submitted proof, was not negatively associated. The authors recommend against using an open-ended AI chatbot as a tutor and suggest exploring a tutor embedded in course activities that requires students to attempt and explain their reasoning before receiving further help. They also propose a future randomized study with three arms: chat that provides answers, chat that provides hints without answers, and feedback on submitted proof attempts, holding the model and course content constant and testing both immediate performance and later performance without AI.
The scope is one proof-oriented course, with consented students who may differ from nonparticipants, and the Midterm 2 assessment measured ordinary course performance rather than a purpose-built transfer or delayed-retention test. Data collection ended at Midterm 3 for budget reasons, so the final exam was not analyzed. Within those bounds, the decision-relevant threshold is not whether AI raises homework scores, which it did by 2.56 points, but whether the help arrives after the student has produced reasoning of their own.
Sources used in this answer
Supporting or Replacing Students' Reasoning? Cognitive Offloading in AI-Assisted Proof Learning
Earlier randomized access to an AI chatbot and homework-feedback interface raised homework scores by 2.56 points but produced no significant exam advantage; after universal access, each 100 additional chatbot messages were associated with 3.02 fewer exam points, while feedback requests were not negatively associated.