Measure what students think?
Our AI feedback pilot covers some uncharted territory, so it felt essential to understand how students experienced it. But that raised some immediate questions:
- What did we really want to know?
- Which questions would help us find that out?
- How should we present those questions to students?
The first step was getting clear on our goals. Were we trying to evaluate the effectiveness of the AI tool? Record how students made use of it? Understand its impact on student learning? Compare it to peer review? All of the above? Working this out was an iterative process, but ultimately, we felt the most important questions to be:
- Can AI provide feedback that is useful for student’s writing development?
- When delivered in a structured setting, can engaging with AI feedback build critical thinking and AI literacy?
With these goals in mind, we began planning an approach to measuring success. A survey seemed the best choice, scaling well to large cohorts like ours and capable of capturing a broad range of experiences, even with the low response rates that surveys often face. Design of our survey was guided by two useful channels: sage advice from colleagues in Bristol Institute for Learning and Teaching (BILT) and a Research Ethics Application. The ethics application initially seemed a scary thing to someone used to ‘classical’ engineering research like me, but turned out to be a clear process which encouraged productive reflection on both how and what we should be asking.
The final iteration of our survey consisted of 11 (mostly multiple choice) questions, gathering anonymous responses, distributed by email to students who agreed to receiving it (via a Blackboard survey) and flagged up via QR code in the live session. With a measure of hope and confidence we felt we had a route to measuring success of this pilot. More on the results later!
