BILT Education Development Project holder Rich Pyle and his group of Student Partners have created an infographic sharing their findings from their research this year.

You can read a text version of the infographic below.

Building Students’ AI Literacy through critical review of sample reports and their own work. 

Authors: Academic Staff: Rich Pyle, Joel Ross and Aydin Naseehi. Student Partners: Yinuo Li and Tirenioluwa Omigbodun 

Unit Information 

Name of Unit: Engineering Communication, Measurement and Data Analysis (ECMDA)

Unit Size: About 920 First-Year Engineering Students 

Unit Assessment Structure: 65% Lab Report, 20% Code and 15% Reflection Portfolio

Introduction

Previously, we gave the ECMDA students AI-generated feedback on their lab reports and ran a live session to encourage review and reflection. What if the students generate feedback on their own reports using AI and they critically review AI-generated lab reports? Could this build their critical thinking skills and AI literacy? The BILT EDP ’25/56 pilot investigates whether students actively engaging with generative AI platforms in live sessions can foster critical thinking and AI literacy.

BILT EDP ’25/26 Pilot Intervention Tools 

1. AI-generated Lab Report Suite: This suite consists of two lab reports generated by Microsoft Copilot and a task guide that aids students in reflecting as they read them. The lab reports were generated from the same source but had varying scores (high and low) according to the unit’s rubric. 
2. Local LLM Critique Assistant: This tool, developed with the Faculty of Science and Engineering Technical Services, provides feedback on uploaded lab report drafts. Characteristics of the Local LLM Critiquing Assistant
   1. No Data Storage: It does not store prompts or outputs, ensuring data security. 
   1. Dictated Behaviour: It has rules for the output format and contextual information, such as the Unit Rubric.
   1. Energy Use Monitoring: It monitors the energy use while generating an output to prompt.

BILT EDP ’25/26 Pilot Activities

1. Pre-session activities 
   1. Optional AI-generated feedback report. 
   1. AI-generated lab reports.
   1. A didactic session on large language models (LLMs).
2. Facilitated live session activities hosted at the Intervention Site: Room G.01, Ivy Gate Building, University of Bristol. 
   1. Peer Reviews.
   1. A didactic session on the importance of critically reviewing AI-generated outputs.
   1. 1-hour session using the Local LLM Critiquing Assistant. 
3. Post-session Activities
   1. Surveys on student perceptions of the pilot.
      1. Likert Scale Questions. 
      1. Semi-structured Questions.

Student Perceptions of the BILT EDP ’25/26 

Likert Scale Questions

1. Out of 8 respondents, 5 students agree that they prefer interactive engagement with AI over ready-made AI-generated feedback, 2 students are neutral, while 1 student disagrees. 
2. Out of 8 respondents, 3 students agree that interacting with the LLM requires more thinking than reviewing AI/peer feedback, 4 students are neutral, while 1 student disagrees
3. Out of 8 respondents, 4 students agree that interacting with the LLM helped them understand their own work better, 2 students are neutral, while 2 students disagree
4. Out of 8 respondents, 6 students agree that they can identify weaknesses/limitations in AI-generated feedback, 1 student is neutral, while 1 student disagrees

Opinions expressed by Students 

1. “The AI did pick up the parts that needed improvement from my report.”
2. “A major limitation is only having access in Ivy Gate and not at home”
3. “… not enough time was given”

Reflections 

Students expressed that they found the LLM local critique assistant useful; its being local also increased their confidence around data safety while informing them on energy use.  Although the LLM’s responses helped identify weaknesses in the report, some students felt they were ‘unnatural’. 

The pilot proves students’ preference for active engagement in feedback over passive interaction. It also sheds light on the increasing interest among students in local LLMs for learning and assessment. Looking ahead, we plan to make the tool more accessible to students and to use the local LLM earlier in other facilitated live sessions. 

Acknowledgement 

We want to acknowledge the support of the  University of Bristol IT for supporting the Data Protection Impact Assessment, the AI Education Team (ai-education@bristol.ac.uk) for communication and project development support, Axel Montout and Steve Leaback from FSE Technical Services for development of the local LLM platform, and the Research Ethics Team (research-ethics@bristol.ac.uk) for advice regarding the survey and ethical approval processes.