News

Moving Assessment Online: Key Principles for Inclusion, Pedagogy and Practice

This AdvanceHE webinar was chaired by Patrick Baughan with presenters David Carless, Jess Moody and Jess Stokes discussing different aspects. The format of the webinar was that each presenter gave a 10-15 minute presentation (some followed these guidelines more closely than others) and at the end questions were taken and the panel had a discussion.

Screengrab of the three speakers and panel chair on Zoom.

David Carless was the first to speak, covering assessment and feedback in online learning environments. His recent tweets (@CarlessDavid) cover a lot of the material discussed below, but I’ve summarised the main points he addressed below.

Assessment principles:

  • Flexibility and choice to enable – we want to give students opportunity to show best knowledge and performance.
  • Assessment needs to be a partnership with students, rather than something that is done to students.
  • Assessment during this period should be of ‘no detriment’ to our students. We need to provide alternative assessments that can meet the learning outcomes we are looking for – David offered examples of these but you can see a similar list on this DEO page.

Feedback principles:

  • Pedagogy even more than technology should guide planning feedback.
  • Students need to be active in the feedback processes, making meaning from, and acting on, feedback
  • There needs to be a social and interpersonal and relational aspect to feedback, which is even more pertinent at the moment.
  • There also needs to be opportunities for acting upon feedback. Proof of feedback pudding is in the eating! Timing of feedback needs to allow for opportunities – think about peer feedback and internal self-evaluation.

Suggested practices for doing this:

  • Audio and video peer feedback;
    • enables students to make academic judgements and they can compare their own work with peers. In this climate, it can also help develop a sense of community (Filius et al, 2019). In research done with peer assessment in MOOCs, it was shown that multiple peer reviews aligned with self-evaluation of own work were most effective. It can be a really rich process in the composing of peer review.
  • Collaborative writing, e.g. Google Docs – multiple sources of feedback and action works in process.
  • Online quizzes with automated feedback
  • Teaching screencast or give video feedback to students via online conferencing tools. Allows us to build rapport, nuance, trust and builds social presence. Also encourages students to take action and helps develop shared responsibilities.

Workload needs to be wisely deployed – we need to reduce teacher commentary at times when it cannot be taken up.

To summarise:

  • Pedagogy drives technology use
  • We need students to have active involvement in assessment and feedback
  • Social presence, care and trust is of upmost importance
  • Support and coaching for feedback literacy should be available.

Jess Moody then went on to deliver her short presentation on inclusion and online assessment in the Covid-19 pandemic.

She identified the key aspects of the challenge:

  • Decisions about assessment must ensure that all students are equally enabled to demonstrate their learning.
  • The key factors in decision making are changing or unknown (both delivery and health concerns, economic distress).
  • The danger of compounding existing structural inequities – award, progression, grants and careers.

Jess then went on to discuss some priority issues:

  • Digital equity – students do not have equal access to home to both learning materials and access to feedback. Things like internet at home, space and a place to work, privacy at home, access to resources. We need to enable software and hardware for students at home they would normally use on campus.

Safeguarding – Not all of our students are safe at home, need to think of stress of that on top of assessment. Also the online spaces present different challenges (gendered/ racialised issues) for our students.

  • Temporal equity – students are craving normalcy but time is not available equally to students. There are issues around caring responsibilities, health religious observance and access. Students need option to disengage where they can not prioritise assessments. Not all days are equal – students may have part time jobs etc that means they need extra time to complete tasks. We also need to consider how we check in with students wellbeing during this time.

This is not a binary switch from assessment ‘A’ to assessment ‘B’. We need to understand the diversity and uncertainty of individual needs and we have to support their informed choices about things like delaying exams, taking assessments in a different format, etc. We need to give agency and sense of control to student who may otherwise be feeling powerless.

Policy, procedure and impact analysis – we should embed equality impact analysis in decisions about change. Priorities are changing and we need to ensure we have more streamlined extenuating circumstances, resits, progressions rules and deferral and interruption procedures. Certain groups are likely to apply for these more than others, so be prepared for this. Built into all of this needs to be a commitment to reviewing the impact of decision on different groups.

Key principles

  • No one should be left behind – 0identiy our most vulnerable groups
  • Do no more harm – don’t compound existing inequalities in the crisis
  • Be transparent and flexible
  • Support should be first
  • Make sure you understand the impact of your decisions.

Geoff Stoakes – special advisor in advance HE and close involvement in TEF

At this point in the webinar my neighbour came round to drop off some shopping he had picked up for us so I missed the first part of Geoff’s presentation. When I rejoined the webinar, Geoff almost immediately lost his connection to the internet so all I can do is post the slide we were on! Please speak to the AQPO about any quality questions you have.

We then started the discussion element of the webinar while Geoff sorted out his internet connection.

(Geoff did then go on to finish his presentation but at this point I had been listening and writing notes for 50 minutes and was finding it very difficult to concentrate. There was a great deal of text on his slides and he was going through them too fast for me to take good notes. You can see all of his slides on the AdvanceHE website, which provide a good enough summary of what he was saying.)

I walked away from my computer for a couple of minutes to get a drink and have a quick conversation with my husband. This seemed to reset my concentration ready for the final discussion/ questions part of the presentation.

Discussion following the talks covered:

How lecturers could minimise their own bias when marking online – Jess talked about how bias impacts our decisions more when we are stressed, tired, hungry, etc – which is more evident now at the moment. Institutionally how do we support out staff, deadline for markings could be extended, as well as when and how anonymisation is helpful, how you design assessment mitigates bias and continuous monitoring to ensure that we minimise bias where possible. David discussed evaluative judgements and what we can learn from art and design communities and make professional judgements, it is part of their subject to discuss this and so we need to bring it into other discipline conversations.

How to make it easier to record video feedback – David says that sometimes hard-working staff do too much with feedback (and students can find it overwhelming!) – less is more. We need to train students to self-evaluate and make use of peer feedback.

Resources for students for peer feedback – David has covered this is his previous writings (Carless and Winstone, 2019 – ‘meaty’ chapter on it) – we need to train and coach students in how to do it, model our own experiences, sell the benefits, negotiate with them how to tackle the challenges.

Increase in student anxiety with the flexibility offered in assessments – students are worried they might make wrong choice – how do we mitigate this? David has seen this in his research and encountered this – the more choice, the more confusing for students! We need to negotiate choices with them and asking them to think it through. Jess discusses informed choices and how we communicate in different ways – how can we make things as clear as possible? And consider – are there certain choices that may impact on certain groups more than others?  We also need a space where people can come and have that conversation. Why and how are people making certain choices in these times too?

Issues with internet connections – can’t give feedback online – is responsibility of HEI to provide internet access or they need to provide alternative feedback and resources? Jess starts the conversation and says there are legal requirements here that need to be considered depending on where you are in the world. There are moral questions – who are we leaving behind? Other institutions are making funds available for students but internet access is a really difficult one – there are things around proportionality in implementing the Equality Act. Geoff adds that some universities are partnering with a company to ensure students have laptops. We also need to consider alternative forms of assessments that allow for students that do not have internet access.

Recording of this webinar is in the Advance HE Connect membership benefit series, also in Teaching and Learning forum. Advance HE Connect is available as an app on iOS and Android.

If you’re thinking of a doing a webinar, make it shorter than an hour unless you build in long enough breaks for people to have a concentration reset!

Amy Palmer

computer keyboard
Teaching Stories

Learning from the experience of higher education in China

Chinese higher education institutions are ahead of the rest of the world in adapting to the Covid-19 pandemic, testing new approaches and technologies to identify the best ways to help their students. AdvanceHE welcomed staff and students to a live webinar earlier this week to share these experiences and lessons learned. This “Lessons from China” webinar was broadcast to over 400 hundred global participants using the Zoom platform.

The conversation started with some tips on motivation: helping others and sharing how to succeed can be a source of encouragement; along with staying patient and focused on the work.

The range of technologies employed was next on the agenda. The experience was described as “a technology experiment” that employs a broad swathe of learning resources and digital tools. Social media is a useful backup facility if core teaching systems are having difficulty. Each platform has its own strengths, but students can become confused when switching between platforms. So it’s best to stick to one or two core platforms. At Bristol, we may consider how this advice plays out within Schools and especially within programmes that run across Schools, such as our Innovation degrees.

The experts highlighted the need to ensure that students understand how to use the selected online platforms by providing dedicated tutorials. This extends to individual support that falls outside normal timetabled hours. It was not clear how these additional hours were balanced against academics’ overall workload, so this is something that should be highlighted with one’s line manager and School for review.

The panel noted that getting continuous feedback from students ensured that this new way of learning was effective, and any issues could be remedied quickly.

Students were observed to communicate together effectively on social media platforms. They also used these platforms to collaborate on social good projects, such as a celebration video in support of those in Wuhan. While outside the formal curriculum, this activity gave them a sense of belonging within the cohort.

Learning materials are accessed differently in different regions of the world. In China, there was a need to open up access to online learning resources, using VPNs (virtual private networks), direct provision of e-text books and PowerPoints, and additional source materials. Some academics even mailed books to students’ homes, especially when those students had limited internet.

Teacher-student relationships changed as a result of this situation. Student panellists noted that when the academic’s camera was turned off, they felt more disconnected. However, this was typically done to reduce bandwidth issues for those with limited internet. Students also noted that they were aware of the stress the academics were under and were reticent to ask questions or make additional demands on their teachers. One reflected how he felt the experience made him a better, more independent learner, more able to study by himself. However, students also noted that at the start of a new term two areas were negatively affected: (1) they were unable to make new friends easily and (2) newly-assigned teachers were unable to forge a bond with their students. While making friends became more challenging, the change in the way students interacted also led some to develop more intimate relationships.

For subjects that require practical labs, a number of new techniques were employed. At one institution, a pre-existing three-year VR platform project allowed basic experiments to be completed online covering almost every discipline that uses practical labs. These labs are not recordable however, and difficult to include in student reports. Another approach required students to use their mobile phones to take photos and video. The panellists recommended conducting a survey in the first instance to see what tools students have ready access to, what physical space they have (such as a garden), and that can help inform what tasks they can complete. For some subjects, such as Chemistry, those students may need to return to campus earlier than others and complete more paper reading and writing assessments in the meantime.

The question of how to track or understand engagement on online platforms was addressed by using interactive activities, such as yes/no questions during live sessions. Most platforms also provide analytical tools that can help inform participation queries.

Patience was identified as “our most powerful weapon” wherein both staff and students should aim to do their best to participate, and use this as an opportunity to explore existing online learning opportunities like free online courses. At Bristol, several such courses are available via the FutureLearn platform, see https://www.bristol.ac.uk/bristol-futures/open-online-courses/.

The panellists spoke of their hopes and expectations for the future. In China, the focus is on employment, distribution channels, and how the government can provide more opportunities. The student panellists reflected that they cherish the opportunity to study even more and look ahead to when they graduate and can better serve the community. Career planning sessions and counselling services were expanded by their institutions and were gratefully received by students.

Finally, the session looked to assessment. The biggest challenge is how to conduct exams. For some disciplines this was considered straightforward, such as live face to face oral exams for language programmes. However, for physics, medicine and other subjects, decisions are still in flux as to how to assess. A major concern is about equality. When staff and students do not have reliable fast internet, and not every home has a laptop, the panelists reflected that a “no detriment” approach should be taken, and no student should feel left behind. Assessment regulations were freed up to allow a greater degree of assessment flexibility. For example, exams could be cancelled, assessments based on weekly assignments and quizzes could be used for grading, and individual professors would decide what is best for their cohort. Bristol has taken a similar approach by introducing a “no detriment” policy to our students, creating a safety net that will ensure no student is disadvantaged by the current crisis.

Staff can find further support on assessment here: https://www.bristol.ac.uk/digital-education/guides/coronavirus/assessment/.

Further guidance on online teaching is available here: https://www.bristol.ac.uk/digital-education/guides/coronavirus/.

As always, we welcome questions and requests for support, and we encourage you to share any good practice with us!

Dr. Ash Tierney, BILT Lecturer.

digital education office logo

Getting the most out of the Blackboard Grade Centre: advanced administration skills for electronic management of assessment (EMA)

Increasing numbers of staff are supporting electronic management of assessment (EMA) in Blackboard. This workshop is aimed at colleagues who wish to deepen their knowledge and expertise of administering the grade centre effectively, in particular when dealing with large cohorts of students. During the workshop participants will be able to explore examples and try out a range of advanced grade centre tools. They will also be given access to a set of resources to take away.
By the end of the workshop participants will:
· be able to apply advanced features of the grade centre to their Courses eg use grading periods to organise assignments by teaching block.
· be able to make the grade centre easier to work with eg for markers and moderators
· have strategies to administer the grade centre effectively including filtered views and managing large cohorts of students
· be able to import and export data from the grade centre from/to other systems.
· be able to ensure controlled student access to relevant grade centre data

Pre-requisites

Participants should already have a basic knowledge of setting up and managing assignments in Blackboard.

Contact information

To book, please see our booking form on Eventbrite. Getting the most out of the Blackboard Grade Centre.
digital education office logo

Managing Student Access to Online Submission Points in Blackboard: Understanding Pre-Submission Quizzes, Groups and Adaptive Release [DEO Webinar]

Whether you just need a refresher or are new to managing submission points in Blackboard, this short webinar will explain how pre-submission quizzes and groups can be used with adaptive release to manage students’ access to submission points and how you can check if a student can see a submission point or not.

This webinar is suitable for all levels though some basic knowledge of Blackboard would be helpful, but not necessary. This webinar would be most suitable for administrative staff. 

The room will be available from around 15 minutes before the webinar starts.

This webinar will be presented by David Perkins de Oliveira and Naomi Beckett from the Digital Education Office.