Going live, in T minus…
Having set out our objectives, survey and ethics it was time to make this idea a reality! The remit for our AI feedback tool was to locally (i.e., without sending anything beyond University systems) run a large set of draft reports through an AI model and produce (hopefully) useful feedback. While writing software to achieve this does require some coding knowledge, writing the portion that talks to the AI model was the easy part! Ollama (and other similar tools) allows for downloading and prompting publicly available AI models on a local computer with just a couple of lines of code. The hard work turned out to be in working out what to ask it and how to nicely format its inputs and outputs.
I found that getting useful feedback involved a fair amount of iteration on testing different AI models and prompt engineering. The system we ended up with iterates through our rubric, asking the AI for:
- Whether or not the report fulfils this requirement
- Some reasoning to support this
- A relevant quote from the report
and with some careful data handling, these combined into a document for each student that opted into AI feedback. Some learning points from this process were:
- Rubbish in = rubbish out: our rubric is written concisely, for humans to understand, and assumes a lot of prior knowledge about the assessment. This knowledge must be in the prompt to get sensible results.
- Simplify the task where you can: because we give the students a report template, the headings can be used to section the reports. Giving the AI only the section it needs to see helped produce relevant quotes.
- Bigger is not always better: some of the larger AI models (for this relatively simple task) gave more useful feedback than smaller ones
With our software written the last thing to do was find somewhere to run it. Whilst a regular Dell laptop can just about load up a small AI model running 100s of reports was going to take almost a week to compute. On the other end of the spectrum, University supercomputer capacity is abundant, but in demand and not always easy to get a hold of for pilot projects like ours. In the end, we got off the ground with time on a kind colleague’s workstation, giving us enough compute power to run through the entire rubric for a report in just 45s. Now all that remains is the fuel for the fire: the reports!
