Tag: assessment

Development in assessment practice in HE – Joint SIG RAISE Online Meeting

The Engaging Assessment and Research & Evaluation Special Interest Groups from RAISE would like to invite you all to a joint online SIG meeting taking place on Friday 5th June 2020 (10:30am – 3:30pm). The purpose of the meeting is to profile the work of early-career researchers/ academics and practitioners in the area of student engagement and assessment. The meeting is open to all from students to university staff members.

*Note this meeting will take place online, a link will be sent to all delegates ahead of the event.*

We hope to see many of you there!

For more information about RAISE Network: http://www.raise-network.com/get-involved/special-interest-groups-sigs/

Register your interest to attend the free SIG online meeting and view the timetable on the Eventbrite page.

News

The Big Scary Word Beginning with C (not that one)

robytoberts

This might not seem like a time of opportunity. Everything is cancelled or postponed, and it seems like our worlds are shrinking (both metaphorically, and physically – something I’m acutely aware of as I’m currently working out of my dad’s bike shed). But there’s a chance here to take a huge stride towards something the university has been inching towards for some time. And it’s more than just a chance – I think in these extraordinary circumstances, there’s a serious need for it.

Co-creation is using students’ feedback, opinions and skills to develop learning and teaching. Like I say, it’s nothing extremely new to the university, but it’s usually on a much smaller scale. These aren’t business as usual times. I’m asking any heads of year, heads of school or anyone else involved in decision making around assessment and teaching to co-create like you’ve never co-created before. I’m mostly talking about final year students, as I think this is the most pressing concern, but this applies to end of year assessment for all students, and the transition to digital teaching for the last few weeks of term.

There’s a mountainous task ahead. Re-designing the in-person, timed, high-pressure, exam-based assessment that a lot of subjects use as a heavy proportion of a student’s final degree classification seems almost impossible. Or finding a way to account for a lack of support and teaching for students who don’t have exams. And not to mention it’s during a time of incredible mental stress on academics and students alike. 

There’s plenty of literature out there about moving to online assessment – what works, what doesn’t, how to mitigate against plagiarism, loads of fun stuff. And as academics, it might be tempting to look through research and case studies, talk to other academics in other universities, and come up with a robust plan, backed up by literature and experts alike. But there’s a huge human element here that is never going to get captured without getting students heavily involved in the decision making process from the start. 

So please, as soon as possible, start thinking about how you and your students are going to face this together. There’s an endless list of tools you can use to find out what your students want, and generate ideas, without anyone having to leave their house. You can send out polls, you can run q&as, you can use padlet to collect ideas and comments, you can use discord or skype to organise small group discussions. You can even use tools like Blackboard Collaborate to run online workshops. There’s a community of students who are scared, nervous, uncertain about their future, and feeling like huge decisions are about to be made that they have no control over but will have a massive effect on them, in the short and long term. You’ve got the tools to turn that anxiety into real solutions that work for staff and students and might even be able to set a precedent for student-led decision making in the future. 

I assume the idea of workshops doesn’t fill you with glee. I know how hard it is to get students to engage with them, and you often only hear from the same sort of students. But I promise you, if you advertise through as many channels as you can, and are honest with students, and tell them you aren’t sure what to do, but want to work with them to figure it out, you will be inundated with students wanting to be involved (and not just the annoying ones who write blogs..).

Everything feels out of control at the minute, but this is our future, and if there’s a chance to have a say in it, we won’t miss it. We want our degree classification to be fair. We want everyone to have a chance to get what they are aiming for. We want it to represent everything we’ve done over the last three years. Every lecture, seminar, lab, late night in the library, hours spent searching for just the right paper; every moment when something finally clicked into place, every lecture watched five times on replay at half speed until we got it; the experiments that went right, the experiments that went wrong; the 9am monday lectures which we really wish we’d gone to more of, the 9am lectures we dragged ourselves into with a lounge stamp on our arm and a pounding headache, and all the other parts of the three or more years that we’ve given to this degree.


Please listen to your students. Trying to manage this chaotic situation must be a nightmare, and just like us, you’re only human and you can only do so much. In your shoes, I can’t even imagine what I’d do. But I know where I’d start.

Toby Roberts, BILT Student Fellow

News

The Examined Casualty

BILT

Last week I attended an ‘Emergency First Aid at Work’ training course. I had minor concerns about whether I would be able to use the defibrillator correctly (nailed it) and whether we’d be made to look at gory images of lacerations (we were, but my eyes were averted). What I wasn’t concerned about, however, was whether I was going to walk out with a certificate in First Aid or not – the University had paid for me to attend this course so therefore, naturally, I should pass, right?

Wrong.

Fifteen minutes into the course, Terry (a charming 68-year old Brummy who had spent the last forty years training people in First Aid) let us all know that at the end of the day, there was going to be an exam. It had twenty questions and it was multiple choice. And yes, people had failed it, and if we did, we would have to come back next week and sit it again.

Panic immediately set in. Suddenly, what had become a semi-jolly from work took me back ten years to my time at university when I clung onto every word the lecturer said when he was covering something on the exam. My hand sprang into action and I wondered why they had only given us three pages to write notes on.

Every time something on the exam was mentioned, Terry did his little hand gesture telling us to write it down. I spent my lunch break revising the notes I had already written. As the end of the day approached, I stopped listening to Terry and started re-reading over my ‘exam notes’. We sat the exam and, lo and behold, we all passed (in fact, most of us got 100%).

As I drove home, I thought about the switch that had occurred when we learnt of our exam-based fate. Would I have worked as hard if the exam hadn’t been mentioned until lunch time? Or just sprung on us at the last minute? Would I still have taken notes? The answer to these questions is ‘yes’- I had signed up to learn about First Aid and I was keen and ready.

I know a lot of academics feel strongly that exams are a good form of assessment. And, a well-written and designed exam can be – but one is never taught how to write an exam. Like all things, some are good at it and some aren’t. Most of the exams I had personally experienced have been a regurgitation of information. Is it possible to teach teachers how to design a good exam?

A final thought: for the vast majority of people, exams cause some level of panic. I accept that they are an efficient way to assess students, but what can we do to reduce the tension they bring? I came across this article yesterday which provides a brilliant example of how a teacher ‘instills the belief that they have practised the hardest maths that they have ever had to face, so why be scared of an exam?’, which is a novel approach and something I feel we should start putting into practice with our students.  

Amy Palmer

bristol conversations in education logo

Computer-based Diagnostic Assessment of Young Learners with Automated Feedback – an International Trial

This event is part of the School of Education’s ‘Bristol Conversations in Education’ seminar series. These seminars are free and open to the public.

Speaker: Tony Clark, Cambridge Assessment English

An effective diagnostic test can play a key role in the language learning process, allowing specific strengths and weaknesses in students’ linguistic development to be identified and then addressed (Jang, 2012). This paper describes the development of an online diagnostic test by Cambridge Assessment English that assesses English grammatical knowledge at A2 level.   
 
As most language tests to date have been proficiency or achievement tests, there has been relatively little research done in the field of diagnostic language assessment and there is no real agreement on exactly what it entails (Alderson, 2005; Alderson, Brunfaut, & Harding, 2015; Davies, 1999; Lee, 2015). It was decided to create something based on the learning-oriented assessment framework (Jones & Saville, 2016) and in response to this lack of research in the field of diagnostic testing. Another aim was to trial a faster, more iterative way of working to better respond to the continuous rapid changes in technology by producing an initial prototype to trial which we could later improve based on the trial results. 
 
Aimed at learners of approximately 15 years old, the test provides detailed diagnostic feedback on seven grammar categories at both individual and class levels, aiming to improve curriculum and lesson planning and accommodate students’ learning needs. The test was trialled internationally and surveys and focus groups were designed to investigate student and teacher perspectives. As well as discussing the results of the trial, the paper also outlines planned modifications for the next version of the Diagnostic Grammar Test and the implications of this research for wider pedagogical practice.

CLICK TO REGISTER

bristol conversations in education logo

Extrapolating the widening participation agenda to the recruitment of underserved groups in medical research: Assessment, ethnicity, and language

This event is part of the School of Education’s ‘Bristol Conversations in Education’ seminar series. These seminars are free and open to the public.

Speaker: Dr Talia Isaacs, UCL Institute of Education, University College London

Widening participation has long been a strategic objective in UK higher education, with government targets for increasing the diversity of student intake in university admissions (e.g., HEFCE; see Rose et al., 2019). The argument for catering to a wider demographic naturally extends to the healthcare sector, including health intervention research, which tests the safety and effectiveness of different medical treatments for patients (Bartlett et al., 2005). Although examining the composition of the recruited sample in a study and the extent to which it is representative of the target population to which the results will be extrapolated has not traditionally been a focus in health intervention research in the UK (Brown et al., 2014), there are some signs of change. One informal indicator is ongoing work of a National Institute for Healthcare Research (NIHR) Clinical Research Network (CRN) on including underserved/underrepresented groups in the context of clinical trials (Rochester et al., in progress). This project is likely to inform future requirements for research funding applications.

In this talk, couched under the broader theme of trends in the field of language assessment, interdisciplinarity, and the role of our professional associations, I will discuss why this specific-purposes topic is relevant. Language testers need to be engaging and lending their expertise to different stakeholder groups, including domain experts from different fields, as part of what has been termed as “indigenous assessment” (Jacoby & McNamara, 1999). This is particularly important to improve the quality of assessments that are used for gatekeeping purposes to promote social justice (principles of inclusion, equitability, fairness, etc.; Shohamy, 2001). By way of an example, I will focus specifically on the role of language as a criterion for including or excluding patients from participating in trials (Isaacs et al., 2016). I will argue that adequate operationalization of the language proficiency construct is potentially high stakes for patients in this context and should be a research priority, notwithstanding barriers to conducting interdisciplinary research.

CLICK TO REGISTER

Teaching Stories

The Primary Experience: What Can We Learn about Cross-Institutional Changes?

BILT

The following post was written by Dr. Isabel Hopwood-Stephens, a TESTA Researcher.

As one of the TESTA researchers attached to BILT, I’m going to be involved in collecting and analysing data about Bristol undergraduates’ experience of assessment. The aim of TESTA is to provide an evidence-based starting point for discussions among Programme Teams about how students’ experience of assessment might be improved, thereby increasing their engagement with their study and satisfaction with the course.

This is done by sharing any issues identified in the analysis and providing ideas which are likely to involve teaching staff making changes to aspects of the assessment experience; for example, offering detailed verbal feedback on a draft of an essay, which the student can use to improve it, before the essay is submitted for grading, or explicitly discussing and exemplifying the marking criteria with students to help them internalise standards.

Having a good idea about how to improve students’ experience of assessment is one thing, though; making the required modifications to working habits to enact those ideas is another. My recent research into the factors that enable or inhibit changes to assessment practice among primary school teachers has provided some interesting pointers.

As part of my study into primary teachers changing their assessment practice, I looked at the main vehicle for teachers’ professional development in primary schools: the staff meeting. I was expecting to find that staff meetings with particular characteristics – where teachers could discuss how they worked, were encouraged to raise questions, and where the focus on learning was clear – would be significantly linked to subsequent reports of school-wide changes to their assessment practice.

Instead, I found out that the characteristics of the wider workplace seemed more influential. Teachers who felt that their workplaces encouraged collaborative, cross-departmental working and innovation were more likely to also report school-wide changes to how they carried out assessment.

This made me think that the kind of professional learning that helps primary teachers to change the ways that they do their job takes place during the wider working day, through ongoing conversation with colleagues, rather than within the confines of a staff meeting. When I looked at communication style between teaching colleagues, I also found that the activities which school-wide changes to teaching practice seemed to entail – negotiation and agreement of shared goals; reflection upon and review of progress; sharing of best practice; questioning and clarification of aims – were underpinned by an open and dynamic communication style that facilitated the involvement of all in discussion and decision-making. This research was conducted with primary teachers in state-maintained primary schools, a working environment which we might consider somewhat removed from the more selective and purposeful atmosphere of a university. However, it will be interesting to see whether the characteristics of the working environment and the interpersonal communication style experienced by academic staff plays a role in enabling programme-wide changes to aspects of practice as a result of participating in TESTA.

500 Words, News

My Retirement from Competitive Baking

BILT

Yesterday, after an excruciating three-week wait, it was the Education Services Charity Bake Off Final. I had made it through to the final after winning my heat (cheese and rosemary scones, if you must know) and I had been practising for my chance at winning the title ever since.

I was as happy with my cake as a novice baker could be, having opted for a chocolate and passionfruit cake, and eagerly awaited the results as the morning went on. By the time it came to 1pm, when colleagues from across the office gathered around waiting our Director to announce the winner, I was actually nervous.

I didn’t win. I didn’t expect to win – there were some amazing cakes on offer from some equally amazing bakers – but no one likes to lose do they? I spend the afternoon texting my husband about how I was never going to bake again and fanaticising about throwing my rolling pin away when I got home.

And I don’t plan on entering another baking competition; I didn’t like the waiting around for weeks not knowing what the result is going to be – yet this is exactly what so many 17 and 18-year olds are going through today.

Having sat their exams months ago, they have spent their summer nervously awaiting the results that will determine their future. Whether they go to university or not, and whether, if they do choose on university, that university is their ‘first choice’, or whether they have to go though ‘clearing’ (an awful process and even more awful word to use for it – surely there is a better way it can be done?*).

But there is no option for a university student to ‘never bake again’ – doing a degree is like a three-year baking competition. For the few students who do well in all of their assessments this is fine (read: smash the soufflé), but for the majority of students who struggle though at least some of their degree, the process of endlessly awaiting the next result is hugely detrimental for their wellbeing – and yet we continue to assess in this way.

As an adult, we don’t experience this same kind of stress. The wait to hear if you’ve been accepted for a mortgage, or if your latest paper has been accepted in to journal, is about as close as we come. But these are annual occurrences at best and, as adults, we have the experience of know we can always resubmit a paper or apply for a different mortgage. I wonder if we experienced the continual insecurity and nerves that students face around assessment that we would still choose to assess in this way?

One way to reduce this insecurity could be a move towards more formative assessments and less summative assessment may be one approach, or a move away from numerical grading may be another, but it is difficult to know what balance could be reached between keeping students motivated while still removing the carrot of a grade they are happy with.  

So, while I’ll be hanging up my apron for the foreseeable future, I’ll be thinking of all the students starting in September (and coming back) who will be facing another year of blind bakes and wondering what we can do to help reduce the anxiety around results and assessments this causes.  

*If this area interests you, I highly recommend this WonkHE piece on making university admissions truly inclusive – including two very viable recommendations.  

Amy Palmer

Teaching Stories

Strategic Students and Question Spotting

BILT

The following piece was written by Helen Heath, a BILT Fellow, Reader in Physics and (soon to be!) University Education Director (Quality).

Why do we think that students being strategic in their learning is a bad thing? Is this an example of emotive conjugation as brilliantly illustrated by Anthony Jay and Johnathan Lynn in the “Yes Minister” series, “I give confidential security briefings. You leak. He has been charged under section 2a of the Official Secrets Act.” ?

“I only have time for important things, you have concentrated on the wrong things, students are question spotting rather than learning.”

Academics are very strategic in the tasks they decide to undertake. They pick tasks that will result in promotion, they tune their lectures to give students what they want to get those good questionnaire responses and they leave jobs undone that they have decided are not worth the time and effort. Yet we seem to criticise students for the same behaviour. We decide not to read the majority of the 200 papers in the Senate pack. Quickly reviewing the headings and deciding what matters to us. This is sensible use of precious time. A student decides they don’t have time to read and understand the whole textbook so they will look at previous examinations and see what topics are more likely to come up and this is “question spotting”.

But is “question spotting” such a bad idea? There is some sense academically. If a question (or a variation of a question) about the same topic appears every year then the examiner is giving a message that this is a topic they regard as important. We might hope that students had realised what were the key topics in other ways. We might stress these key topics in our lectures. We might like to think our students were able to just “get” what is key but that’s a high-level skill and the key topics may only be obvious when they have reappeared in subsequent years. When students are struggling with the nuts and bolts of a subject it’s not surprising that they can’t manage to see the wood for the trees.

Many weaker students are known to find difficulty with scaffolding their learning and identifying the key elements that will enable them to succeed later. They use every piece of information they can to work out what these key topics are and that includes judging what we regard as important by what we assess them on. The topics we choose to place an emphasis on in our final assessment must be import so question spotting is a way of understanding what it is that academics regard as important.

I’d suggest that this strategic planning is not only useful for passing examinations but it’s a useful life skill. The difficulty arises where students question spot and learn by rote with no understanding. The symptom of this in Physics is often a good response to a question that looked like the one that was asked but was slightly different.

The HEA training materials used in the programme focussed assessment training for the pilot project encouraged academics to consider what are the threshold topics in their area. There is much written about threshold topics in physics a recent paper even suggests that there are too many threshold concepts in physics to count them (“Identifying Threshold Concepts in Physics: too many to count” R. Serbanescu 2017). If this is the case, we need to guide the students by deciding what we think is key. If we fail to do that then we shouldn’t blame the students for looking at what we indicated was key by our assessment. Assessment does drive learning and if we are assessing the same topic repeatedly then it is driving the students to learn that topic.

One mechanism we have tried in physics which has some advantages is giving the students a list of questions of which a subset will be a people be guaranteed to appear on the paper and make up ~40% of the material. These direct students towards the bare bones of the course. If they can answer this set of questions they should at least be able to reproduce the basic information in the course Looking our definition of what constitutes a third class performance in assessment (“some grasp of the issues and concepts underlying the techniques and material taught” UoB 21 point scale 40-50 descriptor) the ability to simply regurgitate with reasonable accuracy some basic concepts could be seen to meet these. Ideally students would want to go further but, in some cases, they haven’t had the time to absorb that particular piece of knowledge and digest it in the depth we would expect. While there are still time constraints on the acquisition of knowledge in a Higher Education programme inevitably almost everyone will come up against a concept that they are unable to grasp before the assessment.

And is learning by rote so bad? I do not set out to prove Pythagoras’ theorem every time I need to use it for a question.

Forms of assessment should have a range of tasks that test both use of tools and deeper concepts, but students should not be criticised for directing their learning towards topics they think are likely to come up in an examination. By putting these topics on the examination regularly we have declared them to be important.

Teaching Stories

Assessing Celebrity Cultures

BILT

Rumour had it that both the teaching and assessment on the third-year English Literature Celebrity Cultures module was pushing boundaries to introduce students to new ways of thinking. Intrigued, I arranged a meeting with its unit leaders, Rowena Kennedy-Epstein and Andrew Blades, to find out more about what they were up to.

The Celebrity Cultures unit has been running for just one academic year, but already word has got around that this unit is one worth taking. Andrew and Rowena came up for the concept of the course through a desire for students to reflect on course materials in a more “personal, idiosyncratic” way. They recognised a disconnect between the way academics thought and the way students were encouraged to think.

“… as scholars we are deeply involved in the emotional life of our material. And I think we felt that the students here didn’t quite understand kind of their political positions within how to engage with our texts and cultures, and this is set up, I guess, in some ways to think about that.”

The course material covers gender studies, cultural studies, critical race studies and queer studies but it’s also about how students find materials. Andrew and Rowena use celebrities as the central concept, thinking about how we, as an individual and as a society, create icons; how we obsess over certain things, how we look at things, what and how we expect things to be as opposed to how they are. Ideas about the political world that are then interrogated through the idea of celebrity.

In terms of planning the course, Rowena and Andrew sat down and did all the thought about its structure and assignments simultaneously, making the transition between materials and assessment seamless and organic. There are several things that set this unit apart from others on the degree.

Each week, students were tasked with writing a 250-word lecture reflection, considering what had struck them the most about the content. Students could either do this in the time between the lecture and the seminar, or at the beginning of the seminar, where the first 15 minutes of each session was handed over to students to either write this reflection or discuss the lecture with others in their group.

The lecture reflection also had additional benefits – lecture theatres were full; in part this is down to the reflective piece, but also the fact that lectures are delivered by multiple speakers, with a number guest academics from across the Faculty of Arts taking the lectern each week, turning each session into a mini-conference, with lectures being a mix of scripted material, reflection and discussion between academics, film clips, etc. This didn’t come without its organisation difficulties, but the benefits for students were huge – Andrew observed that in his entire career he had not seen lecture theatres so full! Students were not aware of what the lecture each week so they would have to come.

These lecture reflections formed part of a portfolio of work across the unit, in which students chose their best two reflections to make up alongside a traditional essay (75% for the portfolio), with a group presentation too (25%). Students continued to write throughout the course, creating a sense of continual reflection, which removed the emphasis on the ‘final’ assessment. Andrew and Rowena both said how high the quality of work was across the board, and this was undoubtedly because the students were given their own voice to reflect on what they had learnt. As well as the 2 lecture responses and essay, there’s a 500 word piece they call a ‘meditation’ – on a particular celebrity figure or phenomenon. This is a one-off creative-critical piece, and each of the three seminar tutors produced their own and presented it at a lecture at the beginning of term.

“Students will often hide behind a kind of what they think to be a scholarly style and behind certain buzz phrases… which are often ways of clouding the very things that they want to express. Academic, scholarly language is a learned artificial language, none of us speak like that. And in fact, it can often be really inarticulate in what it’s trying to say and deliberately obscure [it]. And I think, in a way, you’re sort of parting the clouds over that, and demystifying that, to some extent, brought out at this time better, better quality of writing, which had fewer of different types of technical terms, and fewer of some of the technical terms that are actually often misused.”

The majority of students on the unit enjoyed this way of learning and being assessed, yet a few found the academic freedom difficult. Rethinking education in this way won’t always feel comfortable for every student, and ‘Celebrity Culture’ definitely addresses some of the problems students currently find with more traditional units – heavy emphasis on a final, summative assessment without much room for practice and difficulty engaging with lectures and course materials are both solved through the design and delivery of this unit. Although the study of celebrity isn’t applicable to all, the educational elements certainly are.

Amy Palmer

Assessment in higher education conference

Assessment in Higher Education Conference 2019

This event will be the seventh international Assessment in Higher Education conference. This research and academic development conference is a forum for critical debate of research and innovation focused on assessment and feedback practice and policy. The themes for our 2019 conference will invite a wide range of papers, practice exchanges and posters. Themed poster presentations, accompanied by a short pitch from the authors, have been a particular strength of the conference and have encouraged networking by delegates.

Keynote Speakers

Phil Dawson: Associate Professor at Deakin University

Bruce Macfarlane: Professor of Higher Education at University of Bristol