This paper presents the evaluation of the educational impact of two conversational AI applications, one designed to support student self-study (Study Buddy) and the other to assist teachers in instructional planning (Teacher Mate) within the context of the AI4EDU project. The evaluation was conducted during the second piloting phase across four European countries in real classroom settings. In Greece, the evaluation involved 148 participants, 3 teachers and 145 students. The evaluation protocol included training workshops, pre- and post-intervention surveys, system interaction data analysis, and classroom-based implementation. Greek participants were assigned to experimental and control groups for comparative analysis. During a multi-week period, students used Study Buddy during self-study both at school and at home, while teachers integrated Teacher Mate in lesson planning and student monitoring tasks. Quantitative data from motivation and engagement questionnaires and interaction logs were triangulated with qualitative data from semi-structured interviews and classroom observations. The analysis provided insights into the impact of AI tools on users’ educational performance, engagement and motivation and helped identify possible drawbacks and key aspects of AI implementation that can be exploited to demonstrate the issues of ethics, transparency and inclusive use of AI in real educational settings. In general, participants from both groups expressed favorable opinions of the AI-driven applications. The combination of our study’s empirical results emphasizes the need to align AI tools with user needs, deliver customized training, and cultivate trust and openness of AI applications. These initiatives are critical for fostering inclusive, stimulating, and impactful AI-supported educational settings.

Authors:
Anna Vacalopoulou, Athena Research Center, Greece
Spyridoula Stamouli, Athena Research Center, Greece
Georgios Paraskevopoulos, Athena Research Center, Greece
Dimitra Dimitrakopoulou, Ellinogermaniki Agogi, Greece
Gregory Milopoulos, Ellinogermaniki Agogi, Greece
Marios Papaevripidou, University of Cyprus, Cyprus
Theodoros Karafyllidis, University of Cyprus, Cyprus
Sofoklis Sotiriou, Ellinogermaniki Agogi, Greece
Vassilis Katsouros, Athena Research Center, Greece

About the Presenter(s)
Anna Vacalopoulou is scientific associate at the Institute for Language and Speech Processing of Athena Research Center. Her current project is AI4EDU: Conversational AI assistant for teaching and learning.

Connect on Linkedin
https://www.linkedin.com/in/anna-vacalopoulou-48872832/

See this presentation on the full schedule – Thursday Schedule

Bookmark