Measuring the Impact of AI-Driven Activities on Problem Solving and Conceptual Understanding in Undergraduate Physics Courses

Background:This full research paper aims to measure the impact of using IA chatbots on the development of problem-solving skills and conceptual understanding among sophomore engineering students enrolled in physics courses. Although the use of AI chatbots is increasing in educational environments, there is still a need to objectively measure their impact on student performance following a structured educational approach.Purpose/Research Questions:(a) How much does the use of AI-driven educational activities impact student performance? (b) What are the overall students’ perceptions about the use of AI chatbots in classroom activities?Methods:We implemented a classical pre-test/post-test design with 14 experimental and control groups at Institution X, with seven interventions in first-year physics courses during Spring-2024, Fall-2024, and Spring-2025. 175 students participated in the experimental groups and 156 in the control groups. The students in the experimental groups completed one or two structured AI-driven activities of selected themes during the course. In contrast, the students in the control groups followed regular activities without the professor explicitly using AI support. The themes covered with the AI were (a) conservation of energy, (b) thermodynamics, and (c) magnetism. The pre-test was administered during the first week of classes, and the post-test at the end of the 5-week courses. Different professors participated in the experimental and control groups. An online perception questionnaire was also administered to all experimental students to inquire about their experience working with the AI chatbots.Findings:The experimental groups achieved higher average learning gains than the control groups (48 vs. 33 points; p = 0.016). However, the difference in the average relative learning gain between the experimental and control groups was not significant (p = 0.173). The perception questionnaire showed strong student endorsement of AI support, with 77% of them reporting improved understanding of course concepts when using IA chatbots.Implications:We found that integrating AI-driven activities into a guided, verification-focused workflow may yield higher physics learning gains among first-year engineering students. However, we suggest increasing the sample size of students participating in the intervention using a rigorous research methodology to measure the impact of AI-driven educational activities effectively. In any case, the professor should guide students to use AI tools wisely and effectively to avoid brain-drain and to foster real students’ critical thinking by developing evaluation and curator skills.