News

Moving Assessment Online: Key Principles for Inclusion, Pedagogy and Practice

This AdvanceHE webinar was chaired by Patrick Baughan with presenters David Carless, Jess Moody and Jess Stokes discussing different aspects. The format of the webinar was that each presenter gave a 10-15 minute presentation (some followed these guidelines more closely than others) and at the end questions were taken and the panel had a discussion.

Screengrab of the three speakers and panel chair on Zoom.

David Carless was the first to speak, covering assessment and feedback in online learning environments. His recent tweets (@CarlessDavid) cover a lot of the material discussed below, but I’ve summarised the main points he addressed below.

Assessment principles:

  • Flexibility and choice to enable – we want to give students opportunity to show best knowledge and performance.
  • Assessment needs to be a partnership with students, rather than something that is done to students.
  • Assessment during this period should be of ‘no detriment’ to our students. We need to provide alternative assessments that can meet the learning outcomes we are looking for – David offered examples of these but you can see a similar list on this DEO page.

Feedback principles:

  • Pedagogy even more than technology should guide planning feedback.
  • Students need to be active in the feedback processes, making meaning from, and acting on, feedback
  • There needs to be a social and interpersonal and relational aspect to feedback, which is even more pertinent at the moment.
  • There also needs to be opportunities for acting upon feedback. Proof of feedback pudding is in the eating! Timing of feedback needs to allow for opportunities – think about peer feedback and internal self-evaluation.

Suggested practices for doing this:

  • Audio and video peer feedback;
    • enables students to make academic judgements and they can compare their own work with peers. In this climate, it can also help develop a sense of community (Filius et al, 2019). In research done with peer assessment in MOOCs, it was shown that multiple peer reviews aligned with self-evaluation of own work were most effective. It can be a really rich process in the composing of peer review.
  • Collaborative writing, e.g. Google Docs – multiple sources of feedback and action works in process.
  • Online quizzes with automated feedback
  • Teaching screencast or give video feedback to students via online conferencing tools. Allows us to build rapport, nuance, trust and builds social presence. Also encourages students to take action and helps develop shared responsibilities.

Workload needs to be wisely deployed – we need to reduce teacher commentary at times when it cannot be taken up.

To summarise:

  • Pedagogy drives technology use
  • We need students to have active involvement in assessment and feedback
  • Social presence, care and trust is of upmost importance
  • Support and coaching for feedback literacy should be available.

Jess Moody then went on to deliver her short presentation on inclusion and online assessment in the Covid-19 pandemic.

She identified the key aspects of the challenge:

  • Decisions about assessment must ensure that all students are equally enabled to demonstrate their learning.
  • The key factors in decision making are changing or unknown (both delivery and health concerns, economic distress).
  • The danger of compounding existing structural inequities – award, progression, grants and careers.

Jess then went on to discuss some priority issues:

  • Digital equity – students do not have equal access to home to both learning materials and access to feedback. Things like internet at home, space and a place to work, privacy at home, access to resources. We need to enable software and hardware for students at home they would normally use on campus.

Safeguarding – Not all of our students are safe at home, need to think of stress of that on top of assessment. Also the online spaces present different challenges (gendered/ racialised issues) for our students.

  • Temporal equity – students are craving normalcy but time is not available equally to students. There are issues around caring responsibilities, health religious observance and access. Students need option to disengage where they can not prioritise assessments. Not all days are equal – students may have part time jobs etc that means they need extra time to complete tasks. We also need to consider how we check in with students wellbeing during this time.

This is not a binary switch from assessment ‘A’ to assessment ‘B’. We need to understand the diversity and uncertainty of individual needs and we have to support their informed choices about things like delaying exams, taking assessments in a different format, etc. We need to give agency and sense of control to student who may otherwise be feeling powerless.

Policy, procedure and impact analysis – we should embed equality impact analysis in decisions about change. Priorities are changing and we need to ensure we have more streamlined extenuating circumstances, resits, progressions rules and deferral and interruption procedures. Certain groups are likely to apply for these more than others, so be prepared for this. Built into all of this needs to be a commitment to reviewing the impact of decision on different groups.

Key principles

  • No one should be left behind – 0identiy our most vulnerable groups
  • Do no more harm – don’t compound existing inequalities in the crisis
  • Be transparent and flexible
  • Support should be first
  • Make sure you understand the impact of your decisions.

Geoff Stoakes – special advisor in advance HE and close involvement in TEF

At this point in the webinar my neighbour came round to drop off some shopping he had picked up for us so I missed the first part of Geoff’s presentation. When I rejoined the webinar, Geoff almost immediately lost his connection to the internet so all I can do is post the slide we were on! Please speak to the AQPO about any quality questions you have.

We then started the discussion element of the webinar while Geoff sorted out his internet connection.

(Geoff did then go on to finish his presentation but at this point I had been listening and writing notes for 50 minutes and was finding it very difficult to concentrate. There was a great deal of text on his slides and he was going through them too fast for me to take good notes. You can see all of his slides on the AdvanceHE website, which provide a good enough summary of what he was saying.)

I walked away from my computer for a couple of minutes to get a drink and have a quick conversation with my husband. This seemed to reset my concentration ready for the final discussion/ questions part of the presentation.

Discussion following the talks covered:

How lecturers could minimise their own bias when marking online – Jess talked about how bias impacts our decisions more when we are stressed, tired, hungry, etc – which is more evident now at the moment. Institutionally how do we support out staff, deadline for markings could be extended, as well as when and how anonymisation is helpful, how you design assessment mitigates bias and continuous monitoring to ensure that we minimise bias where possible. David discussed evaluative judgements and what we can learn from art and design communities and make professional judgements, it is part of their subject to discuss this and so we need to bring it into other discipline conversations.

How to make it easier to record video feedback – David says that sometimes hard-working staff do too much with feedback (and students can find it overwhelming!) – less is more. We need to train students to self-evaluate and make use of peer feedback.

Resources for students for peer feedback – David has covered this is his previous writings (Carless and Winstone, 2019 – ‘meaty’ chapter on it) – we need to train and coach students in how to do it, model our own experiences, sell the benefits, negotiate with them how to tackle the challenges.

Increase in student anxiety with the flexibility offered in assessments – students are worried they might make wrong choice – how do we mitigate this? David has seen this in his research and encountered this – the more choice, the more confusing for students! We need to negotiate choices with them and asking them to think it through. Jess discusses informed choices and how we communicate in different ways – how can we make things as clear as possible? And consider – are there certain choices that may impact on certain groups more than others?  We also need a space where people can come and have that conversation. Why and how are people making certain choices in these times too?

Issues with internet connections – can’t give feedback online – is responsibility of HEI to provide internet access or they need to provide alternative feedback and resources? Jess starts the conversation and says there are legal requirements here that need to be considered depending on where you are in the world. There are moral questions – who are we leaving behind? Other institutions are making funds available for students but internet access is a really difficult one – there are things around proportionality in implementing the Equality Act. Geoff adds that some universities are partnering with a company to ensure students have laptops. We also need to consider alternative forms of assessments that allow for students that do not have internet access.

Recording of this webinar is in the Advance HE Connect membership benefit series, also in Teaching and Learning forum. Advance HE Connect is available as an app on iOS and Android.

If you’re thinking of a doing a webinar, make it shorter than an hour unless you build in long enough breaks for people to have a concentration reset!

Amy Palmer

News

Event Summary of ‘Making IT* Happen: from strategy to action’ at the University of Leicester

*Inclusive teaching

The following post was written by Emilie Poletto-Lawson, who has been a BILT Fellow since January 2018. 

I am a BILT fellow (based in Academic Staff Development), working on the BILT theme of inclusive assessment. On the 14th of November 2018 I attended a two-part workshop lead by Pete Quinn (Pete Quinn Consulting) and Dr Mike Wray (inclusininhe.com) that explored inclusion from theory to practice at University of Leicester.

The first part of the day focused on inclusivity from a disability perspective and the session looked at the current situation in universities regarding inclusivity and reflected on where we are at 8 years after the Equality Act was introduced.  Even though all stakeholders agree on inclusivity in principle, “making it happen” can prove somewhat difficult as new initiatives can meet resistance. Lecture Capture (new in some institutions) would be a very good example, it is a very important step for a great number of students and, in particular, students with a disability and yet this practice is still being greatly challenged by lecturers within universities and in the news. The main criticism is that students no longer attend lectures but the USS pensions-strike in 2018 also highlighted issues regarding who owns the rights to the recordings.

While universities must assure legal compliance, it is important to define what it is “reasonable” when it comes to reasonable adjustments we make to support our students learning. Pete Quinn highlighted the risk, stated by the Inclusive Teaching and Learning in Higher Education as a Route to Resilience publication, of “insufficient institutional oversight” which could lead to “unduly influenced [decisions] which are made by what individual members of staff perceive as reasonable” without a proper understanding of what the Equality Act requires or with appropriate emphasis being placed on relevant considerations”. It is therefore essential to break silos of practice within an institution and ensure all the relevant stakeholders work together to assure consistent and appropriate adjustments are in place.

Pete Quinn also presented the overview by Abi James (Assistive Learning Ltd) of public sector website and application regulations that all websites will need to comply with to ensure accessibility for all. Websites and content shared on intranets and extranets (this includes Blackboard) created after 23.09.18 will need to be compliant by September 2019, with anything created before 23.09.18 has September 2020 as a deadline. Finally, the deadline for mobile apps will be September 2021. You can find more information on this on the government’s website.

After looking at the work of the Office for Students, the Office of the Independent Adjudicator and the equality and human rights commission we reflected on our institutions’ situation regarding the following themes:

  • Non-medical help provision
  • Assistive technology
  • Lecture capture
  • Inclusive teaching, learning and assessment policy
  • Inclusive teaching, learning and assessment in practice
  • Placement and internships

 

In the afternoon, Mike Wray presented Inclusive Learning and Teaching and Assessment Framework (ILTAF), an audit framework to help universities improve their level of inclusivity. The framework contains four sections:

  • Quality assurance
  • Before teaching
  • During teaching
  • Assessment

We were also given time to discuss inclusive learning and teaching in our institutions with a view to share good practice, agree goals and take stock in the future.  I am very grateful to colleagues from the Universities of Leeds, Durham, Bath and Edinburgh for very interesting exchanges.

What next?

A number of questions arose from the day and left me wondering how best to initiate change and champion inclusivity within Bristol and what can we do in our practice to support this?  I will be working on developing a self-assessment document that would support unit/programme leads in reviewing inclusive assessment (and teaching) practices across their units and programmes.

If you are interested in finding out more on inclusive practice, you might want to look into the following MOOC, from the University of Southampton, recommended by the speakers: Inclusive Learning Teaching.

Here are a few reading suggestions from the event:

On key actors and texts regarding inclusivity mentioned by the speakers:

http://www.legislation.gov.uk/ukpga/2010/15/pdfs/ukpga_20100015_en.pdf

https://www.officeforstudents.org.uk/

http://www.oiahe.org.uk/

https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/587221/Inclusive_Teaching_and_Learning_in_Higher_Education_as_a_route_to-excellence.pdf

https://www.equalityhumanrights.com/en

 

 

News

Should all assessments be inclusive?

The following post was written by Emilie Poletto-Lawson, an Educational Developer and BILT Fellow. 

I am a BILT fellow based in Academic Staff Development where I work as an Educational Developer. I have been working on the BILT theme of assessment – focusing on inclusive assessment since February 2018. I am undertaking a literature review with a view to making recommendations around inclusive assessment principles that we can embed into our units and programs at the University of Bristol to work alongside our institutional principles on Assessment and Feedback.

From my reading to date the  main take away is that inclusivity is predominantly discussed as a means for supporting students with disabilities. It is very much viewed as a deficit approach to considering assessment, however, I strongly believe it is far more than that, we want to be inclusive of all learners and for inclusive assessment to actually be more inclusive.

As part of my BILT fellow role I recently attended an event at the University of Leicester called “Making IT* Happen: from strategy to action (*Inclusive Teaching)’, led by Pete Quinn and Mike Wray (blog available here). The focus was very much on supporting disabled students in our institutions and ensuring universities are legally compliant with the Equality Act. In preparation for the event, the experts highlighted good practice in the work we do at Bristol, for example we received positive feedback on our institution website regarding inclusivity (http://www.bristol.ac.uk/disability-services/study-support/reasonable-adjustments/) and in particular videos created by Louise Howson from Academic Staff Development (http://www.bristol.ac.uk/staffdevelopment/academic/resources/learning-and-teaching-resources/learning-and-teaching-videos/ ).

Regarding the literature review I am working on, when researching the key words “inclusive” “assessment” in “higher education”, I obtained 9596 results in ERIC and yet, going through the abstracts not that many articles encompass all three parameters. It appears there might be a gap in the literature here despite inclusivity being key to university strategies in the UK and beyond for a number of years now. So far, the key emerging themes from my searching can be seen below.

Inclusive Assessment in Higher Education map created by Emilie Poletto-Lawson
Created with Mindmeister 21.09.2018

In the US literature the Inclusive aspects of articles relates to the idea of an inclusive campus and looks at inclusivity from the selection process (access) to the students completing their degree (success). In the UK, the literature shows there is an acknowledged need for policies, strategies and processes as well as professional development to bring about inclusive practices.

Initial readings suggest there is a rhetoric of inclusivity as a given good, but it is difficult to identify concrete examples, especially when it comes to assessment. The literature review is the first step to articulate a clear definition before focusing on what inclusive assessment means for us at the University of Bristol.

If you are interested in this topic why not read “Against being Inclusive” by Jeffrey Carlson, interim provost and Vice President for Academic Affairs at Dominican University? I appreciate it might be an odd recommendation since this post advocates that all assessments should be inclusive, but I think this article, published in 2016, does offer food for thought and reinforces the need to clearly define what we mean by “inclusivity” before we move to making recommendations at Bristol.