Assessment and Grade Setting: An Overview

We design all of our question papers and assessment tasks to follow the specification. This means that they are at the correct level for the qualification they assess, they are valid for the specification, they assess the content reliably over time and they differentiate between learners of different abilities. At the same time they allow learners of all abilities to demonstrate what they have learnt. We also ensure that our assessment materials are
unbiased, accessible, are not overly predictable and where there is a choice of questions or tasks they will be of equal difficulty. If we find evidence during the assessment process that a particular assessment item has not performed as expected we will take that into account at the grade boundary setting stage. We review the performance of our assessment materials annually.

We modify question papers for learners with specific needs, for example, we produce papers with enlarged fonts, or papers in Braille. We modify question papers on request within a specified lead time. The available formats and procedure for requesting modifications can be found in the JCQ document Access Arrangements and Reasonable Adjustments

Our authors of assessment materials write the mark schemes in tandem with the questions, question papers or tasks. We also check the mark schemes in tandem with the assessment materials and we keep them secure until our assessors need them for marking.

We engage specialist assessors to write questions, question papers and assessment tasks, and to construct question papers and mark schemes for us. These assessors are subject specialists in the subjects for which they write. In the majority of cases they are also practising or retired teachers who have taught at the level of the qualification for which they are writing.

Before they write any assessment materials we train all our authors of assessment materials in how to write good questions and how to construct good question papers or tasks. We alsofeed back to them how their question papers or tasks performed in the previous examination, so they can take that into account in future years.

We keep all our assessment materials securely until we despatch them to centres.

We require centres to keep them securely according to the JCQ document Instructions for conducting examinations

We check all our assessment materials to make sure that spelling, punctuation, grammar, graphs, formulae, maps, and subject specific content and terminology are correct. We also check that the layout and presentation of the assessments make them easy to follow for learners. In the rare circumstances where an error occurs systems are in place to identify those affected and to ensure that this does not have an adverse impact.

We mark learners’ scripts in three different ways:

• 1. We scan scripts when they come back to us from centres after the examination and then assessors mark them online with marking software. We mark more and more scripts in this way.
• 2. We mark some scripts on the paper on which the learners wrote their answers.
• 3. We use optical scanning software to mark multiple-choice scripts.
  On the very rare occasions that scripts are received after we have issued results, we will mark them provided we are satisfied that their integrity has been maintained.

We engage examiners to mark scripts online or on paper. They are usually practising or retired teachers who have experience of teaching in the subjects and at the levels at which they mark. Examiners may work in teams, led by team leaders. Team leaders monitor and supervise groups of assessors to ensure they are marking consistently. There is a lead examiner for each question paper and for each assessment task. The lead examiner is responsible for setting the marking standard and for the final mark scheme. The lead examiner supervises all other examiners, including the team leaders, to ensure that everybody marks consistently. For all online marking we allocate scripts to examiners, so they will mark scripts from a wide variety of centres.

We train and standardise all our examiners before they can start marking. The training includes how to mark online and how to annotate scripts in general and in specific subjects. During standardisation examiners learn and practise how to apply the mark scheme, how to mark consistently to the lead examiner’s standard and how to deal with unexpected answers from learners. The lead examiner’s decisions on the marking standard are final.

Standardisation can happen in face-to-face situations or remotely. We ensure that all examiners can apply the standard correctly before we allow them to start marking.

We monitor all our examiners and all our marking – for all of our marking methods that are described above. Depending on the marking method we use different types of checks in different combinations.

We use scripts that have been marked by the lead examiner and mix them in with all the other examiners’ scripts and we then check whether the other examiners are marking consistently with the lead examiner. These scripts are called ‘seed scripts’ and we mix them into all examiners’ allocations at regular intervals.

We compare each examiner’s marking patterns against all other examiners’ patterns on the same question paper at regular intervals.

We ask team leaders and lead examiners to check scripts from their marking teams at pre- determined intervals.

We apply checks to all our examiners, including the team leaders and the lead examiners. Checks of lead examiners and team leaders vary depending on the marking method.

We keep records of our marking checks, so we get a complete view on the performance of our examiners.

When we find examiners who have not marked to the required standard we stop them from marking and we mark all their previously marked scripts anew. Very occasionally we may consider other options to align marking that is not completely consistent, for example by re- marking parts of scripts in an examiner’s allocation, because we have evidence to show that he or she is inconsistent on specific questions only, or by applying an adjustment.

We monitor the receipt of scripts by our examiners and scanning bureaux and may liaise with centres as part of our investigations into scripts reported as missing, where internal checks do not resolve the issue. If scripts are confirmed as lost, or if a learner is absent from the assessment for acceptable reasons and meets the minimum criteria for special consideration, which can be found in the JCQ document A guide to the special consideration process, we will, wherever possible, and at centres’ requests, mathematically estimate a mark for the learner for that part of the assessment.

On the very rare occasions that samples are received after we have issued results, we will moderate them provided we are satisfied that their integrity has been maintained.

We moderate all non-exam assessment, controlled assessment or coursework that is initially marked by centres. We expect centres to complete internal moderation before they send samples of non-examined assessment, controlled assessment or coursework and their marks to us.

We sample centres’ marking either remotely (e.g. by post) or by visiting moderation.

We engage moderators to moderate non-exam assessment or coursework that has been marked in centres initially. Moderators are usually practising or retired teachers who have experience of teaching in the subjects and at the levels they moderate. Moderators usually work in teams, led by team leaders. Team leaders monitor and supervise groups of moderators to ensure they moderate consistently. For each non-exam assessment, controlled assessment or coursework component there is a lead moderator. The lead moderator is responsible for setting the moderation standard. The lead moderator supervises all other moderators, including the team leaders, to ensure that everybody moderates consistently. Non-exam assessment, controlled assessment and coursework is allocated to moderators by centre because moderators need to check whether the centre has marked consistently at the right standard.

We train and standardise all our moderators before they can start moderation. The training includes what to do when the rank order of learners in the centre is incorrect, and when to ask for an additional sample. During standardisation moderators learn and practise how to apply the mark scheme, how to moderate consistently to the lead moderator’s standard and how to deal with unexpected submissions from learners and centres. The lead moderator’s decisions on the standard are final. Standardisation can happen in face-to-face situations or remotely. We ensure that all moderators can apply the standard correctly before we allow them to start moderation.

We monitor all of our moderators throughout the moderation period. Depending on the type of moderation, we use different types of checks in different situations:

For remote moderation we ask team leaders and lead moderators to check batches of learners work from their moderation teams at pre-determined intervals.

For visiting moderation team leaders and lead moderators accompany moderators on visits to observe and advise them, or we arrange visits for teams of moderators.

We apply these checks to all our moderators.

When we find moderators who have not moderated to the required standard we stop them from moderating and we moderate all their previously moderated work anew.

When we find that the initial centre marking is not at the correct standard, we adjust marks for the entire centre cohort.

If work is lost, or if a learner is absent from a timetabled internal assessment for acceptable reasons and meets the minimum criteria for special consideration, which can be found in the JCQ document A guide to the special consideration process, we will, wherever possible, mathematically estimate a mark for the learner for that part of the assessment.

Grade boundaries may be different in different examinations series to account for differences in the difficulty of question papers or non-exam assessments. We set grade boundaries based on both quantitative and qualitative evidence. When we set grade boundaries we also follow Ofqual’s instructions which, for general qualifications, can be specific to a particular examination and year, or can be more general. More information can be found on Ofqual’s website

Before we set grade boundaries we make sure we have marked enough scripts to make the award statistically reliable.

Our assessment specialists use their professional judgement, within the parameters set by Ofqual, when deciding where to set grade boundaries. They can take into account the following evidence:

• The question papers or tasks and final mark schemes
• Comments from the lead assessors and lead moderators on the performance of the assessments
• Samples of learners’ work and scripts
• Archive scripts and learners’ work
• Material from other qualifications that are of equivalent standard
• Views from senior assessors on whether scripts or tasks at certain mark points reflect expected standards
• Subject and cohort level predictions based on prior achievement where available and as mandated by Ofqual
• Information on learners’ performance in previous equivalent assessment periods
• Details of change in entry patterns, choices of options and prior attainment
• Information about the relationships between component/unit level data and whole subject performance
• Technical information, including mark distributions relating to the question papers or tasks and individual questions for the current and previous series
• Question performance data, i.e. item-level statistics
• Any other relevant information

For general qualifications in particular, we need to ensure that our grade boundaries are set so that the overall outcome for the specification meets cohort and subject specific predictions within tolerances prescribed by Ofqual. Senior assessors and moderators then confirm that learners’ scripts and work fit within those parameters and meet the expected standard. We only award outcomes outside of these parameters when there is strong and convincing evidence to do so.

For the maintenance of standards in general qualifications we use the comparable outcomes approach as prescribed by Ofqual Regulating GCSE, AS and A levels a guide for schools and colleges This ensures that standards of outcomes are maintained year on year without grade inflation and also that standards between different awarding bodies are fair and comparable. Ofqual monitor the outcomes of all awards through their published ‘data exchange’ procedure and through a series of regular ‘maintenance of standards’ meetings which all awarding bodies and all regulators attend.

Senior examiners, supported by technical, qualifications and operational staff from the awarding bodies are all involved with the awarding process.

Before we submit our awarding outcomes to the ‘maintenance of standards’ meetings our most senior technical and assessment staff review and endorse them. If, rarely, they do not agree with an outcome they will ask the assessment specialists to review or change it.

Ultimately, the awarding body’s Responsible Officer or Chief Executive Officer endorses all award outcomes.