Tag: assessment design

News

Moving Assessment Online: Key Principles for Inclusion, Pedagogy and Practice

BILT

This AdvanceHE webinar was chaired by Patrick Baughan with presenters David Carless, Jess Moody and Jess Stokes discussing different aspects. The format of the webinar was that each presenter gave a 10-15 minute presentation (some followed these guidelines more closely than others) and at the end questions were taken and the panel had a discussion.

Screengrab of the three speakers and panel chair on Zoom.

David Carless was the first to speak, covering assessment and feedback in online learning environments. His recent tweets (@CarlessDavid) cover a lot of the material discussed below, but I’ve summarised the main points he addressed below.

Assessment principles:

  • Flexibility and choice to enable – we want to give students opportunity to show best knowledge and performance.
  • Assessment needs to be a partnership with students, rather than something that is done to students.
  • Assessment during this period should be of ‘no detriment’ to our students. We need to provide alternative assessments that can meet the learning outcomes we are looking for – David offered examples of these but you can see a similar list on this DEO page.

Feedback principles:

  • Pedagogy even more than technology should guide planning feedback.
  • Students need to be active in the feedback processes, making meaning from, and acting on, feedback
  • There needs to be a social and interpersonal and relational aspect to feedback, which is even more pertinent at the moment.
  • There also needs to be opportunities for acting upon feedback. Proof of feedback pudding is in the eating! Timing of feedback needs to allow for opportunities – think about peer feedback and internal self-evaluation.

Suggested practices for doing this:

  • Audio and video peer feedback;
    • enables students to make academic judgements and they can compare their own work with peers. In this climate, it can also help develop a sense of community (Filius et al, 2019). In research done with peer assessment in MOOCs, it was shown that multiple peer reviews aligned with self-evaluation of own work were most effective. It can be a really rich process in the composing of peer review.
  • Collaborative writing, e.g. Google Docs – multiple sources of feedback and action works in process.
  • Online quizzes with automated feedback
  • Teaching screencast or give video feedback to students via online conferencing tools. Allows us to build rapport, nuance, trust and builds social presence. Also encourages students to take action and helps develop shared responsibilities.

Workload needs to be wisely deployed – we need to reduce teacher commentary at times when it cannot be taken up.

To summarise:

  • Pedagogy drives technology use
  • We need students to have active involvement in assessment and feedback
  • Social presence, care and trust is of upmost importance
  • Support and coaching for feedback literacy should be available.

Jess Moody then went on to deliver her short presentation on inclusion and online assessment in the Covid-19 pandemic.

She identified the key aspects of the challenge:

  • Decisions about assessment must ensure that all students are equally enabled to demonstrate their learning.
  • The key factors in decision making are changing or unknown (both delivery and health concerns, economic distress).
  • The danger of compounding existing structural inequities – award, progression, grants and careers.

Jess then went on to discuss some priority issues:

  • Digital equity – students do not have equal access to home to both learning materials and access to feedback. Things like internet at home, space and a place to work, privacy at home, access to resources. We need to enable software and hardware for students at home they would normally use on campus.

Safeguarding – Not all of our students are safe at home, need to think of stress of that on top of assessment. Also the online spaces present different challenges (gendered/ racialised issues) for our students.

  • Temporal equity – students are craving normalcy but time is not available equally to students. There are issues around caring responsibilities, health religious observance and access. Students need option to disengage where they can not prioritise assessments. Not all days are equal – students may have part time jobs etc that means they need extra time to complete tasks. We also need to consider how we check in with students wellbeing during this time.

This is not a binary switch from assessment ‘A’ to assessment ‘B’. We need to understand the diversity and uncertainty of individual needs and we have to support their informed choices about things like delaying exams, taking assessments in a different format, etc. We need to give agency and sense of control to student who may otherwise be feeling powerless.

Policy, procedure and impact analysis – we should embed equality impact analysis in decisions about change. Priorities are changing and we need to ensure we have more streamlined extenuating circumstances, resits, progressions rules and deferral and interruption procedures. Certain groups are likely to apply for these more than others, so be prepared for this. Built into all of this needs to be a commitment to reviewing the impact of decision on different groups.

Key principles

  • No one should be left behind – 0identiy our most vulnerable groups
  • Do no more harm – don’t compound existing inequalities in the crisis
  • Be transparent and flexible
  • Support should be first
  • Make sure you understand the impact of your decisions.

Geoff Stoakes – special advisor in advance HE and close involvement in TEF

At this point in the webinar my neighbour came round to drop off some shopping he had picked up for us so I missed the first part of Geoff’s presentation. When I rejoined the webinar, Geoff almost immediately lost his connection to the internet so all I can do is post the slide we were on! Please speak to the AQPO about any quality questions you have.

We then started the discussion element of the webinar while Geoff sorted out his internet connection.

(Geoff did then go on to finish his presentation but at this point I had been listening and writing notes for 50 minutes and was finding it very difficult to concentrate. There was a great deal of text on his slides and he was going through them too fast for me to take good notes. You can see all of his slides on the AdvanceHE website, which provide a good enough summary of what he was saying.)

I walked away from my computer for a couple of minutes to get a drink and have a quick conversation with my husband. This seemed to reset my concentration ready for the final discussion/ questions part of the presentation.

Discussion following the talks covered:

How lecturers could minimise their own bias when marking online – Jess talked about how bias impacts our decisions more when we are stressed, tired, hungry, etc – which is more evident now at the moment. Institutionally how do we support out staff, deadline for markings could be extended, as well as when and how anonymisation is helpful, how you design assessment mitigates bias and continuous monitoring to ensure that we minimise bias where possible. David discussed evaluative judgements and what we can learn from art and design communities and make professional judgements, it is part of their subject to discuss this and so we need to bring it into other discipline conversations.

How to make it easier to record video feedback – David says that sometimes hard-working staff do too much with feedback (and students can find it overwhelming!) – less is more. We need to train students to self-evaluate and make use of peer feedback.

Resources for students for peer feedback – David has covered this is his previous writings (Carless and Winstone, 2019 – ‘meaty’ chapter on it) – we need to train and coach students in how to do it, model our own experiences, sell the benefits, negotiate with them how to tackle the challenges.

Increase in student anxiety with the flexibility offered in assessments – students are worried they might make wrong choice – how do we mitigate this? David has seen this in his research and encountered this – the more choice, the more confusing for students! We need to negotiate choices with them and asking them to think it through. Jess discusses informed choices and how we communicate in different ways – how can we make things as clear as possible? And consider – are there certain choices that may impact on certain groups more than others?  We also need a space where people can come and have that conversation. Why and how are people making certain choices in these times too?

Issues with internet connections – can’t give feedback online – is responsibility of HEI to provide internet access or they need to provide alternative feedback and resources? Jess starts the conversation and says there are legal requirements here that need to be considered depending on where you are in the world. There are moral questions – who are we leaving behind? Other institutions are making funds available for students but internet access is a really difficult one – there are things around proportionality in implementing the Equality Act. Geoff adds that some universities are partnering with a company to ensure students have laptops. We also need to consider alternative forms of assessments that allow for students that do not have internet access.

Recording of this webinar is in the Advance HE Connect membership benefit series, also in Teaching and Learning forum. Advance HE Connect is available as an app on iOS and Android.

If you’re thinking of a doing a webinar, make it shorter than an hour unless you build in long enough breaks for people to have a concentration reset!

Amy Palmer

News

The Big Scary Word Beginning with C (not that one)

robytoberts

This might not seem like a time of opportunity. Everything is cancelled or postponed, and it seems like our worlds are shrinking (both metaphorically, and physically – something I’m acutely aware of as I’m currently working out of my dad’s bike shed). But there’s a chance here to take a huge stride towards something the university has been inching towards for some time. And it’s more than just a chance – I think in these extraordinary circumstances, there’s a serious need for it.

Co-creation is using students’ feedback, opinions and skills to develop learning and teaching. Like I say, it’s nothing extremely new to the university, but it’s usually on a much smaller scale. These aren’t business as usual times. I’m asking any heads of year, heads of school or anyone else involved in decision making around assessment and teaching to co-create like you’ve never co-created before. I’m mostly talking about final year students, as I think this is the most pressing concern, but this applies to end of year assessment for all students, and the transition to digital teaching for the last few weeks of term.

There’s a mountainous task ahead. Re-designing the in-person, timed, high-pressure, exam-based assessment that a lot of subjects use as a heavy proportion of a student’s final degree classification seems almost impossible. Or finding a way to account for a lack of support and teaching for students who don’t have exams. And not to mention it’s during a time of incredible mental stress on academics and students alike. 

There’s plenty of literature out there about moving to online assessment – what works, what doesn’t, how to mitigate against plagiarism, loads of fun stuff. And as academics, it might be tempting to look through research and case studies, talk to other academics in other universities, and come up with a robust plan, backed up by literature and experts alike. But there’s a huge human element here that is never going to get captured without getting students heavily involved in the decision making process from the start. 

So please, as soon as possible, start thinking about how you and your students are going to face this together. There’s an endless list of tools you can use to find out what your students want, and generate ideas, without anyone having to leave their house. You can send out polls, you can run q&as, you can use padlet to collect ideas and comments, you can use discord or skype to organise small group discussions. You can even use tools like Blackboard Collaborate to run online workshops. There’s a community of students who are scared, nervous, uncertain about their future, and feeling like huge decisions are about to be made that they have no control over but will have a massive effect on them, in the short and long term. You’ve got the tools to turn that anxiety into real solutions that work for staff and students and might even be able to set a precedent for student-led decision making in the future. 

I assume the idea of workshops doesn’t fill you with glee. I know how hard it is to get students to engage with them, and you often only hear from the same sort of students. But I promise you, if you advertise through as many channels as you can, and are honest with students, and tell them you aren’t sure what to do, but want to work with them to figure it out, you will be inundated with students wanting to be involved (and not just the annoying ones who write blogs..).

Everything feels out of control at the minute, but this is our future, and if there’s a chance to have a say in it, we won’t miss it. We want our degree classification to be fair. We want everyone to have a chance to get what they are aiming for. We want it to represent everything we’ve done over the last three years. Every lecture, seminar, lab, late night in the library, hours spent searching for just the right paper; every moment when something finally clicked into place, every lecture watched five times on replay at half speed until we got it; the experiments that went right, the experiments that went wrong; the 9am monday lectures which we really wish we’d gone to more of, the 9am lectures we dragged ourselves into with a lounge stamp on our arm and a pounding headache, and all the other parts of the three or more years that we’ve given to this degree.


Please listen to your students. Trying to manage this chaotic situation must be a nightmare, and just like us, you’re only human and you can only do so much. In your shoes, I can’t even imagine what I’d do. But I know where I’d start.

Toby Roberts, BILT Student Fellow

Teaching Stories

An (a)typical day in The Office

BILT

Amy Palmer, Toby Roberts and Marnie Woodmeade all visited James Norman’s ‘The Office’ project and have shared their experiences below.

Amy’s reflection:

I’d been looking forward to visiting James’ ‘Just Timber’ office since before the concept had even come to fruition. Last year, while catching up over coffee, James told me about his plan to turn on one his classrooms into an office, and I was instantly excited. The project was a combination of all my favourite pedagogies: active learning, blended learning, challenge-led learning, authentic learning, group work… all coming together to form the ‘Just Timber’ office in 1.32 Queens Building.

‘Just Timber’ is a fictional engineering company James has created. His students (employees) were all set timber engineering design projects at the beginning of the unit and are working together to create real designs with real equations in an almost real-world environment. If you want to read more about the project so far, I highly recommend reading ‘The Office’ blog series .

A date was set in the diary to visit, and I invited two of our Student Fellows to come along and join me. And so, last Thursday, we went along and waited for James to show up fifteen minutes late to our visit after almost forgetting we were coming – helping further imitate the real-world, authentic environment (just kidding) and heighten our anticipation further.

And, when we arrived, we were not disappointed. We were welcomed by an offer of tea or coffee (served in a ‘Just Timber’ mug, of course) and then proceeded to look around the office and take it all in.

The first thing I was struck by was the buzz in the room. Not a noisy, can’t-do-any-work’ buzz, but the natural up-and-down of a genuine office environment, with students in their groups switching between their sketching, calculations and discussions with each other over how best to proceed in their projects. Barely a glance was thrown in our direction when we bumbled into the room, the students so engaged in their projects that they weren’t looking for distraction.

We had a quick look around the office, admiring the various projects pinned on the wall, and browsing the elegant engineering magazines by the break-out space. We then proceeded to interrupt students by asking them questions about how they were finding the unit and how it compared to others they were taking.

As you can imagine, they loved it. Of course, they are students, and so the conversations were not void of the odd grumble (nothing you wouldn’t expect from a unit being run for the first time), but the overwhelming response was that they looked forward to Thursdays – regardless of the fact they were spending eight hours in the office – and that the learning they were doing there was both enjoyable, challenging and reflective of an authentic engineering office environment. Some of the students even ended the day with a traditional post-work trip to the pub, further preparing them for life out working in the ‘real world’.

If you’ve been keeping up with The Office blog series, you’ll know that students prepare for their day’s work by watching videos James has created on their VLE, and then come to work to study their projects. This means that time in class/ office is dedicated purely to student-centred work with no didactic teaching. Students highlighted that their favourite aspects of the unit were the group work element, the room layout (large groupings of desks together), the project-centred work and the fact that they had a day dedicated to the unit. Student wouldn’t want all of their modules to run as full-day units, however, but would have found a unit like this in their previous years of study valuable and enjoyable and a great chance to get to know others in their cohort.

James’ Just Timber office is a product of a great idea, hard work and dedication to a new way of learning, and there are many lessons we can take away from designing a unit in this way. Please get in touch with BILT if you’d like to learn more about setting up a project similar to this in your unit.   

Marnie’s reflection:

I had an extremely positive outlook on the Office before I even entered the building; the concept reflected the challenge-led work that I only ever dreamed about during my undergrad. My positivity was only enhanced by the almost immediate offer of cake and tea (an important part of daily office life).  

However, two pieces of feedback from students struck me as unexpected. The first was that students said they genuinely enjoyed being in the Office. Not merely that it was a great educational experience, but they actually looked forward to coming in each day. Having an open-plan office where interaction is encouraged clearly enabled students to really enjoy their time there. 

The other piece of feedback was perhaps more sobering. One student pointed out that although they enjoyed the way the office replicates an engineering company, a ‘real boss’ is unlikely to give mass amounts of work on the same day that five of your other supervisors have given you a deadline. In order for more projects like the Office to succeed, students felt that communication between units is key. This not only has the benefit of reducing their stress but treating students as valued workers positively impacts their outlook on university.  

Toby’s reflection:

One of the things that really struck me about the atmosphere in the Just Timber office (other than the delicious smell of cake courtesy of James’ wife and son) was how much it reminded me of a classroom. Not a school classroom – there wasn’t any paper being thrown around and James hadn’t sent me out to think about how my behaviour affects the rest of the class – but a calmer and more focused 6th Form classroom. And to me that’s a real positive. All of the ‘employees’ were clearly getting work done and you could tell there was a strong sense of purpose. But at the same time, they were relaxed and there was friendliness between them, and the noise of conversation was a world away from the awkward silence of lectures or the hyperactive buzz of a library in exam time. 

However, I wasn’t there to drink in the atmosphere and reminisce about college. The students seemed more than happy to talk to us about the unit, and I think that was in part due to how much they enjoyed it. It was clear that the effort James had put into it had had an effect on them and their attitudes, but it had paid off in producing a rewarding learning experience. 

One area that students weren’t unanimous on was the intense one-day-a-week schedule. Some felt more productive, others exhausted (there was a suggestion that maybe in the future it could not be the day after sports night). However, one interesting element to me was the effect on wellbeing. With so many units and assignments to contend with at once it’s very easy to get overwhelmed as a student. Containing the work within a single 9-5 day helps to compartmentalise and means there’s one less thing to worry about for the rest of the week. Instead, something to look forward to every Thursday.

Teaching Stories

The Primary Experience: What Can We Learn about Cross-Institutional Changes?

BILT

The following post was written by Dr. Isabel Hopwood-Stephens, a TESTA Researcher.

As one of the TESTA researchers attached to BILT, I’m going to be involved in collecting and analysing data about Bristol undergraduates’ experience of assessment. The aim of TESTA is to provide an evidence-based starting point for discussions among Programme Teams about how students’ experience of assessment might be improved, thereby increasing their engagement with their study and satisfaction with the course.

This is done by sharing any issues identified in the analysis and providing ideas which are likely to involve teaching staff making changes to aspects of the assessment experience; for example, offering detailed verbal feedback on a draft of an essay, which the student can use to improve it, before the essay is submitted for grading, or explicitly discussing and exemplifying the marking criteria with students to help them internalise standards.

Having a good idea about how to improve students’ experience of assessment is one thing, though; making the required modifications to working habits to enact those ideas is another. My recent research into the factors that enable or inhibit changes to assessment practice among primary school teachers has provided some interesting pointers.

As part of my study into primary teachers changing their assessment practice, I looked at the main vehicle for teachers’ professional development in primary schools: the staff meeting. I was expecting to find that staff meetings with particular characteristics – where teachers could discuss how they worked, were encouraged to raise questions, and where the focus on learning was clear – would be significantly linked to subsequent reports of school-wide changes to their assessment practice.

Instead, I found out that the characteristics of the wider workplace seemed more influential. Teachers who felt that their workplaces encouraged collaborative, cross-departmental working and innovation were more likely to also report school-wide changes to how they carried out assessment.

This made me think that the kind of professional learning that helps primary teachers to change the ways that they do their job takes place during the wider working day, through ongoing conversation with colleagues, rather than within the confines of a staff meeting. When I looked at communication style between teaching colleagues, I also found that the activities which school-wide changes to teaching practice seemed to entail – negotiation and agreement of shared goals; reflection upon and review of progress; sharing of best practice; questioning and clarification of aims – were underpinned by an open and dynamic communication style that facilitated the involvement of all in discussion and decision-making. This research was conducted with primary teachers in state-maintained primary schools, a working environment which we might consider somewhat removed from the more selective and purposeful atmosphere of a university. However, it will be interesting to see whether the characteristics of the working environment and the interpersonal communication style experienced by academic staff plays a role in enabling programme-wide changes to aspects of practice as a result of participating in TESTA.

500 Words, News

My Retirement from Competitive Baking

BILT

Yesterday, after an excruciating three-week wait, it was the Education Services Charity Bake Off Final. I had made it through to the final after winning my heat (cheese and rosemary scones, if you must know) and I had been practising for my chance at winning the title ever since.

I was as happy with my cake as a novice baker could be, having opted for a chocolate and passionfruit cake, and eagerly awaited the results as the morning went on. By the time it came to 1pm, when colleagues from across the office gathered around waiting our Director to announce the winner, I was actually nervous.

I didn’t win. I didn’t expect to win – there were some amazing cakes on offer from some equally amazing bakers – but no one likes to lose do they? I spend the afternoon texting my husband about how I was never going to bake again and fanaticising about throwing my rolling pin away when I got home.

And I don’t plan on entering another baking competition; I didn’t like the waiting around for weeks not knowing what the result is going to be – yet this is exactly what so many 17 and 18-year olds are going through today.

Having sat their exams months ago, they have spent their summer nervously awaiting the results that will determine their future. Whether they go to university or not, and whether, if they do choose on university, that university is their ‘first choice’, or whether they have to go though ‘clearing’ (an awful process and even more awful word to use for it – surely there is a better way it can be done?*).

But there is no option for a university student to ‘never bake again’ – doing a degree is like a three-year baking competition. For the few students who do well in all of their assessments this is fine (read: smash the soufflé), but for the majority of students who struggle though at least some of their degree, the process of endlessly awaiting the next result is hugely detrimental for their wellbeing – and yet we continue to assess in this way.

As an adult, we don’t experience this same kind of stress. The wait to hear if you’ve been accepted for a mortgage, or if your latest paper has been accepted in to journal, is about as close as we come. But these are annual occurrences at best and, as adults, we have the experience of know we can always resubmit a paper or apply for a different mortgage. I wonder if we experienced the continual insecurity and nerves that students face around assessment that we would still choose to assess in this way?

One way to reduce this insecurity could be a move towards more formative assessments and less summative assessment may be one approach, or a move away from numerical grading may be another, but it is difficult to know what balance could be reached between keeping students motivated while still removing the carrot of a grade they are happy with.  

So, while I’ll be hanging up my apron for the foreseeable future, I’ll be thinking of all the students starting in September (and coming back) who will be facing another year of blind bakes and wondering what we can do to help reduce the anxiety around results and assessments this causes.  

*If this area interests you, I highly recommend this WonkHE piece on making university admissions truly inclusive – including two very viable recommendations.  

Amy Palmer

Teaching Stories

Strategic Students and Question Spotting

BILT

The following piece was written by Helen Heath, a BILT Fellow, Reader in Physics and (soon to be!) University Education Director (Quality).

Why do we think that students being strategic in their learning is a bad thing? Is this an example of emotive conjugation as brilliantly illustrated by Anthony Jay and Johnathan Lynn in the “Yes Minister” series, “I give confidential security briefings. You leak. He has been charged under section 2a of the Official Secrets Act.” ?

“I only have time for important things, you have concentrated on the wrong things, students are question spotting rather than learning.”

Academics are very strategic in the tasks they decide to undertake. They pick tasks that will result in promotion, they tune their lectures to give students what they want to get those good questionnaire responses and they leave jobs undone that they have decided are not worth the time and effort. Yet we seem to criticise students for the same behaviour. We decide not to read the majority of the 200 papers in the Senate pack. Quickly reviewing the headings and deciding what matters to us. This is sensible use of precious time. A student decides they don’t have time to read and understand the whole textbook so they will look at previous examinations and see what topics are more likely to come up and this is “question spotting”.

But is “question spotting” such a bad idea? There is some sense academically. If a question (or a variation of a question) about the same topic appears every year then the examiner is giving a message that this is a topic they regard as important. We might hope that students had realised what were the key topics in other ways. We might stress these key topics in our lectures. We might like to think our students were able to just “get” what is key but that’s a high-level skill and the key topics may only be obvious when they have reappeared in subsequent years. When students are struggling with the nuts and bolts of a subject it’s not surprising that they can’t manage to see the wood for the trees.

Many weaker students are known to find difficulty with scaffolding their learning and identifying the key elements that will enable them to succeed later. They use every piece of information they can to work out what these key topics are and that includes judging what we regard as important by what we assess them on. The topics we choose to place an emphasis on in our final assessment must be import so question spotting is a way of understanding what it is that academics regard as important.

I’d suggest that this strategic planning is not only useful for passing examinations but it’s a useful life skill. The difficulty arises where students question spot and learn by rote with no understanding. The symptom of this in Physics is often a good response to a question that looked like the one that was asked but was slightly different.

The HEA training materials used in the programme focussed assessment training for the pilot project encouraged academics to consider what are the threshold topics in their area. There is much written about threshold topics in physics a recent paper even suggests that there are too many threshold concepts in physics to count them (“Identifying Threshold Concepts in Physics: too many to count” R. Serbanescu 2017). If this is the case, we need to guide the students by deciding what we think is key. If we fail to do that then we shouldn’t blame the students for looking at what we indicated was key by our assessment. Assessment does drive learning and if we are assessing the same topic repeatedly then it is driving the students to learn that topic.

One mechanism we have tried in physics which has some advantages is giving the students a list of questions of which a subset will be a people be guaranteed to appear on the paper and make up ~40% of the material. These direct students towards the bare bones of the course. If they can answer this set of questions they should at least be able to reproduce the basic information in the course Looking our definition of what constitutes a third class performance in assessment (“some grasp of the issues and concepts underlying the techniques and material taught” UoB 21 point scale 40-50 descriptor) the ability to simply regurgitate with reasonable accuracy some basic concepts could be seen to meet these. Ideally students would want to go further but, in some cases, they haven’t had the time to absorb that particular piece of knowledge and digest it in the depth we would expect. While there are still time constraints on the acquisition of knowledge in a Higher Education programme inevitably almost everyone will come up against a concept that they are unable to grasp before the assessment.

And is learning by rote so bad? I do not set out to prove Pythagoras’ theorem every time I need to use it for a question.

Forms of assessment should have a range of tasks that test both use of tools and deeper concepts, but students should not be criticised for directing their learning towards topics they think are likely to come up in an examination. By putting these topics on the examination regularly we have declared them to be important.

500 Words, News

Should we go ‘The Whole Hog’ with programme-level assessment?

BILT

The following post was written by Amy Palmer, BILT Digital Resources Officer.

Since the launch of BILT in 2017, the implementation of programme-level assessment across the University has been a widely-discussed topic. But what do we really mean by programme-level assessment?

Tansy Jessop, while delivering her TESTA workshop in January, outlined her ‘Five Hogs of Programme-Level Assessment’, breaking down the term into five different ways this assessment framework could be implemented.

The first, ‘The Whole Hog’, advocates an integrated and connected assessment plan, running though entire programmes, using capstone and cornerstone assessments to bring together learning from different modules. Teaching is separated from the [summative] assessment, allowing students to make their own connections between content in different modules. This approach is the most widespread understanding of what ‘programme-level assessment’ is and is arguably the simplest implement and there is a clear split between teaching and summative assessment.

The next, ‘Half the Hog’, still has an assessment piece that runs throughout the entire programme, separate from individual modules, but it doesn’t require all assessments to be disconnected from teaching. This connective assessment could be a research project that runs from first to third (or fourth) year and draws on concepts from all of the individual modules. A benefit of this ‘Hog’ is that there is an overall reduction in summative assessments across the degree to make room for the programmatic assessment piece.

The ‘Other half of the Hog’ employs synoptic assessment from across a number of modules (i.e. 50% of the degree modules are assessment via a synoptic assessment while the other 50% have assessments that are directly related to their module’s content). Each module has a combination of formative and one summative assessment, and the synoptic assessment integrates concepts, makes connections between the modules and is challenging for students.

The next pig- or pigs- ‘Both the Hogs together’ (originally named ‘Eat the Hogs Together’, but we didn’t think that was appropriate for our plant-based friends 😊) is when both the curriculum and assessment design is done as a team, using TESTA (programme and student evidence to inform the assessments). Summative assessment is reduced across the entire degree so that students engage more with formative assessments. Teams are encouraged to integrate assessment in the shared process so that everyone has a shared understanding and practice.

The final hog, ‘The Warthog’, is the most radical of approaches. Instead of running parallel modules, students take one module at a time in blocks (for example, one module runs week 1-4, second module runs week 5 – 8, etc.). Assessments are joined up though shared units that weave across the programme. This method has been adopted to some extent at Plymouth University through their immersive induction module in first year.

Some of these ‘hogs’ would be easier to achieve than others, but we don’t know yet which one would create the best outcomes for students. With the amount of modular choice available across most degree programmes, a singular approach would have to be taken at least within a faculty, and potentially across the entire university – it wouldn’t be possible for one programme to undertake a ‘Warthog’ approach while another employed ‘Half the Hog’. But how do we decide which approach to take? And how would this one approach be implemented across the hundreds of programmes we have on offer with limited time for programme teams to sit down and redesign their assessments?

 There are examples of institutions where programme-level assessment has been successfully put into practice (Brunel’s IPA and Bradford’s PASS are two good examples), but we need to understand the impact it has had on student learning, outcomes, wellbeing (both staff and students) before deciding whether going the ‘Whole Hog’ is the right approach for Bristol.

Student Voice

In conversation with a fourth year Liberal Arts student

BILT

Check out this snippet of conversation our Student Fellow Zoe Backhouse recorded with a fellow fourth year Liberal Arts student on the topic of assessment.  Want to know why Europe’s doing HE better than the UK and why playing Donald Trump in class may not be a bad thing? Read on…

Z: How was your assessment on your year abroad?

A: Well, when I was in Amsterdam it was broken down so much into different areas. It wasn’t all reduced down to an essay because that isn’t the one mode of intelligence in the world.

One of my assessments was I became Federica Mogherini who’s the Foreign Minister for the EU and we played out a simulation of the Middle East. Everybody was a different country – someone was Donald Trump! – and literally I learned so much about applying the theory and the logic and actually putting in a practical sense. I think that’s just so important because university should be about teaching skills that can be transferred to employability.

I also loved how we did presentations abroad. At Utrecht you had to lead a seminar for 45 minutes after a 20 minute presentation. In your presentation you couldn’t just read from a piece of paper like everyone does at Bristol. You would stand and deliver a lesson, not looking down at notes, you’d talk to people and have eye contact. And then you had to lead a discussion amongst your peers.

I found it pretty nerve-wracking and I’m quite a confident public speaker. But that’s because the way we’ve always been indoctrinated here is… it’s just very insular. I don’t know, I just think there is a lack of discussion in general in all forms. Discussion only happens as an internal monologue that gets reproduced in an essay. People can’t have conversations in seminars because they get nervous, because they feel like they’d look stupid. I think you should take that away.

We used to be marked on class participation at Utrecht which was like 20% of the mark. I actually do think that’s really important? In the UK people are so scared of saying something because they think there’s only one right answer. In our education system we’re taught that there’s only one right answer and it’s at the back of the book and don’t look and don’t copy and don’t speak to anyone else about it. But it’s not that. Art is about taking things and reinterpreting them and making them better. So I think discussion has been lost from education.

I did another module called Digital Citizens. And literally, we were just coming in to talk about what was going on in the news that day, we’d all just sit around and have a discussion. One of the requirements of that course was to write a journalistic article which was liberating. And it wasn’t just GCSE journalism, it was like, can you write a legitimate article? So I wrote about how data analytics is perpetuating gender stereotypes.

You did have essays as well because that’s important. It’s just about diversifying assessment, and making people feel more comfortable and able in their abilities as opposed to constantly critiquing people and telling them they’re wrong all the time because they don’t fit one style of system.

500 Words, Uncategorized

Informal exploratory writing: three activities you can try with your students

BILT

The following post was written by Amy Palmer, BILT Digital Resources Officer. 

Studies have shown there is a strong correlation between the amount of writing a learner completes and their attainment (Arum and Roksa, 2011). John Bean, in his book ‘Engaging Ideas’ (2011), outlines a number of methods to increase the amount of informal writing your students undertake. He groups these under the theme of ‘thinking pieces’, and he highlights a number of benefits. He believes thinking pieces:

  • Promote critical thinking.
  • Change the way students approach reading – with an increase in writing down their thoughts it forces them to consider alternative and opposite arguments to the piece they are reading.
  • Produce higher levels of class preparation and richer discussions in class. Similar to the point above, if informal exploratory writing is done at the point of reading, students are more prepared with arguments and counter-points in discussion classes.
  • Are enjoyable to read, and make a nice change for markers from the normality of essays
  • Help to get to know your students better as you can see how their arguments are formed and where their beliefs lie.
  • Help assess learning problems along the way. Like any increase in formative work, the teacher can see any gaps in learning at an earlier point and assess whether this is the case for others in the cohort.

Bean describes 22 different exploratory writing tasks, which you can find in his ‘Engaging Ideas’ book; we have selected three to share in this blog.

Bio-poems

This task is easier to apply to some disciplines rather than others (philosophers, historians and politicians come to mind first) and is designed to make students think about the personal dimensions of a subject being studied in a course. A bio-poem is semi-structured and goes as follows:

  • Line 1: First name of the character
  • Line 2: Four traits that describe the character
  • Line 3: Relative of (brother of, sister of, etc)
  • Line 4: Lover of (list three things or people)
  • Line 5: Who feels (three items)
  • Line 6: Who needs (three items)
  • Line 7: Who fears (three items)
  • Line 8: Who gives (three items)
  • Line 9: Who would like to (three items)
  • Line 10: Resident of
  • Line 11: Last name

(Gere, 1985:222)

Not only does this make the subject more human and therefore more memorable, but it also provides a great revision tool when it comes to exams. If this is done as a task before the class, each person’s poem can be discussed to see differences they have found in their perception of the subject.

Writing dialogues between two different theorists/ arguments

This task asks students to write an ‘meeting of the minds’ piece (Bean, 2011:136), where they conjure a script between two theorists arguing different sides (e.g. Hobbes and Locke arguing over the responsibility in a state). This encourages the students to truly consider each side of the argument and also prepares them for discussion in class. This can be done as an individual task or in small groups, and suits many disciplines.

Writing during class to ask questions or express concerns.

Less creative than our other two suggestions, this piece asks students to ‘freewrite’ during a break in the class. You could ask students to summarise the lecture so far, or write down any puzzlements or questions they have. At the end of the freewriting time (which should be a maximum of five minutes), ask a couple of students to feedback. Not only do student practice writing, but it also means you can get real time feedback and allows students to ask questions part way through the lecture.

References

Bean, J., 2011. Engaging Ideas: The Professor’s Guide to Integrating Writing, Critical Thinking and Active Learning in the Classroom. Jossey Bass, United States of America.

Gere, A. R. (ed.), 1985. Roots in the Sawdust: Writing to Learn Across the Disciplines. Urbana, Ill.: National Council of Teachers of English.

Arum, R. and Roksa, J., 2011. Academically Adrift: Limited Learning on College Campuses, University of Chicago Press, Chicago.

News

Should all assessments be inclusive?

BILT

The following post was written by Emilie Poletto-Lawson, an Educational Developer and BILT Fellow. 

I am a BILT fellow based in Academic Staff Development where I work as an Educational Developer. I have been working on the BILT theme of assessment – focusing on inclusive assessment since February 2018. I am undertaking a literature review with a view to making recommendations around inclusive assessment principles that we can embed into our units and programs at the University of Bristol to work alongside our institutional principles on Assessment and Feedback.

From my reading to date the  main take away is that inclusivity is predominantly discussed as a means for supporting students with disabilities. It is very much viewed as a deficit approach to considering assessment, however, I strongly believe it is far more than that, we want to be inclusive of all learners and for inclusive assessment to actually be more inclusive.

As part of my BILT fellow role I recently attended an event at the University of Leicester called “Making IT* Happen: from strategy to action (*Inclusive Teaching)’, led by Pete Quinn and Mike Wray (blog available here). The focus was very much on supporting disabled students in our institutions and ensuring universities are legally compliant with the Equality Act. In preparation for the event, the experts highlighted good practice in the work we do at Bristol, for example we received positive feedback on our institution website regarding inclusivity (http://www.bristol.ac.uk/disability-services/study-support/reasonable-adjustments/) and in particular videos created by Louise Howson from Academic Staff Development (http://www.bristol.ac.uk/staffdevelopment/academic/resources/learning-and-teaching-resources/learning-and-teaching-videos/ ).

Regarding the literature review I am working on, when researching the key words “inclusive” “assessment” in “higher education”, I obtained 9596 results in ERIC and yet, going through the abstracts not that many articles encompass all three parameters. It appears there might be a gap in the literature here despite inclusivity being key to university strategies in the UK and beyond for a number of years now. So far, the key emerging themes from my searching can be seen below.

Inclusive Assessment in Higher Education map created by Emilie Poletto-Lawson
Created with Mindmeister 21.09.2018

In the US literature the Inclusive aspects of articles relates to the idea of an inclusive campus and looks at inclusivity from the selection process (access) to the students completing their degree (success). In the UK, the literature shows there is an acknowledged need for policies, strategies and processes as well as professional development to bring about inclusive practices.

Initial readings suggest there is a rhetoric of inclusivity as a given good, but it is difficult to identify concrete examples, especially when it comes to assessment. The literature review is the first step to articulate a clear definition before focusing on what inclusive assessment means for us at the University of Bristol.

If you are interested in this topic why not read “Against being Inclusive” by Jeffrey Carlson, interim provost and Vice President for Academic Affairs at Dominican University? I appreciate it might be an odd recommendation since this post advocates that all assessments should be inclusive, but I think this article, published in 2016, does offer food for thought and reinforces the need to clearly define what we mean by “inclusivity” before we move to making recommendations at Bristol.