Date: 16th July 2026
Authors: Marios Kremantzis, Hua Jin, Anthi Chondrogianni, Aniekan Essien, Sophie Lythreatis.
Theme: Generative AI theme
Subject area: Curriculum Framework dimensions Inspiring and innovative
Faculty: Faculty of Arts, Law and Social Sciences
The Practice
Students increasingly use general-purpose AI tools for academic support, but these tools are not connected to individual units, their learning materials or assessment requirements and can there-fore provide generic or inaccurate guidance (Kayalı et al., 2023; Kremantzis et al., 2026). We ex-plored whether unit-specific chatbots, grounded in approved teaching materials and embedded in Blackboard Ultra, could provide more relevant, accessible and trustworthy support.
Working with unit directors and student partners, we developed chatbots to help students locate resources, navigate assessment information and identify appropriate support. The chatbots were piloted across three units with approximately 750 registered students: Quantitative Analysis in Management, Economic Principles and Modelling Analytics. Three comparable units without the project chatbot provided a comparison group. Importantly, these students were not “AI-free”; many already used ChatGPT and other external tools. The meaningful comparison was therefore between a governed, unit-specific chatbot and the existing support ecosystem alongside stu-dents’ informal AI use.
The evaluation combined a baseline survey of 197 students, analytics from 430 anonymous chat-bot sessions and 6,722 messages, and eight focus groups involving 27 students from chatbot-enabled and comparison units. The groups were broadly comparable at baseline, while chatbot readiness among intervention students was high (mean 4.11/5). The pilot required a software licence, close collaboration with unit directors, regular content review and student-partner input into testing and usability.
Findings
Four key lessons emerged.
First, the chatbot’s main value was practical rather than transformational. It reduced the effort required to search across Blackboard pages, handbooks, emails and recordings. Students used it primarily to find, check and organise information, not as a replacement for lectures, tutorials or independent study.
Second, availability mattered. Around 60% of sessions occurred outside weekday working hours, demonstrating the value of immediate support when staff were less likely to be available. Stu-dents also valued having a low-pressure place to ask routine or “obvious” questions.
Third, assessment was both the highest-value and highest-risk use case. It generated the greatest demand, including 43% of coded free-text questions. Although 67.5% of valid answer-level feed-back was positive overall, satisfaction varied: deadline information received 81.3% positive feed-back, while summative-assessment responses received only 48.8%. Assessment guidance there-fore requires particularly careful design, updating and oversight.
Fourth, trust was conditional. Blackboard embedding and University association increased cred-ibility, but students still expected accurate, current and source-linked answers. They wanted the chatbot to state its limitations clearly and direct complex, personal or marks-critical questions to staff.
The Impact
The chatbot’s main impact was on the student experience. In the focus groups, students in the chatbot units valued the tool most for reducing the effort of finding information. Rather than searching across Blackboard, emails and other sources, they could ask the chatbot directly and be pointed to what they needed, such as tutor information, resources and assessment details.
Students also valued the chatbot as a low-pressure place to ask routine or “obvious” questions they might have hesitated to take to a lecturer. This was particularly useful around busy periods, such as the start of the unit and the run-up to assessments, when quick answers mattered most.
Students also had a clear sense of what the gaps the chatbot was expected to bridge, and where it was not the right tool. For difficult concepts, and personal matters, they still turned to their lecturers, personal tutors and professional support services. Their message was that the chatbot worked best as a first point of contact for everyday questions, complementing rather than re-placing their lecturers and tutors (Antony & Ramnath, 2023).
Overall, students experienced the chatbot as a valuable complement to existing teaching and support, providing timely, unit-specific guidance that made studying more accessible while rein-forcing, rather than replacing, human support.
Next Steps
Our immediate priority is to strengthen summative-assessment guidance before scaling the ini-tiative. We will develop unit-specific response cards covering task requirements, weighting, deadlines, submission processes, marking criteria and links to authoritative sources.
We will also improve source-linking, ensure that unit and topic context passes correctly into free-text responses, introduce clearer recovery routes following negative feedback, and escalate high-risk questions to the appropriate staff or services.
Future evaluation will assess not only usage but also accuracy, trust, satisfaction and reductions in students’ search burden. With institutional support and collaboration with the University’s Digital Education Office, this human-centred model could provide a practical and scalable layer of student support.
Contact
Dr. Marios Kremantzis: marios.kremantzis@bristol.ac.uk
See an example
Antony, S. and Ramnath, R., 2023. A Phenomenological Exploration of Students’ Perceptions of AI Chatbots in Higher Education. IAFOR Journal of Education, 11(2), pp.7-38.
Kayalı, B., Yavuz, M., Balat, Ş. and Çalışan, M., 2023. Investigation of student experiences with ChatGPT-supported online learning applications in higher education. Australasian Journal of Educational Technology, 39(5), pp.20-39.
Kremantzis, M., Chondrogianni, A. and Essien, A., 2026. Evaluating the impact of AI chatbots on student support and engagement in UK higher education. Journal of Further and Higher Education, pp.1-23.
Authors:
Marios Kremantzis, Hua Jin, Anthi Chondrogianni, Aniekan Essien, Sophie Lythreatis.
Leave a Reply