Every year, over 18,000 question papers, each including many questions and stimulus materials are written for GCSE and A level alone. When vocational qualifications are added, the number increases significantly. Each question on each paper is subject to high level scrutiny.
The aim of all awarding bodies is to create a question paper that can test how well a student has demonstrated their skills and knowledge against the examination specification they have been taught. The paper must include questions aimed at the whole ability range of candidates taking it, so it is possible to differentiate between those that perform well and those that don’t. The type of assessment system we have chosen to have in the UK requires significant and valuable human input to create the papers, which comes with its own risks.
Awarding bodies follow very similar procedures to create a question paper, although there are some differences in approach, timescales and titles. At the root of all the awarding bodies’ procedures is the focus on accuracy.
- 1. Contracting and training staff involved in creating question papers
At the very start of the process, awarding bodies appoint subject experts to author either whole question papers or individual questions and the respective marking schemes. In addition to this, other subject experts are appointed to carefully consider each question and to ensure that it is a valid assessment that can be reliably marked. These subject experts are trained by the awarding body to ensure they operate to a very high standard.
Awarding bodies each have dedicated internal teams who produce and quality assure each question paper. - 2. Drafting of questions and question papers
Prior to the submission of questions or a question paper for review, a high level of planning is required to ensure that each assessment fulfils all of a qualification’s stated requirements. When designing questions, the subject experts carefully consider the content of the specification, the assessment objectives and questions set in previous series and how candidates have responded to them. - 3. A series of edits, reviews and revisions
When the questions/question paper has been drafted, it is then reviewed by a subject expert. The Reviewer considers the questions and mark schemes carefully with reference to the specification, ensuring they are of comparable demand to previous series and that there are no errors or omissions. The Reviewer may recommend changes and provides feedback to the author of the question/question paper. - 4. Evaluation stage
Once a draft of questions/a question paper has been submitted, the materials will then be reviewed by a group of subject experts. They will consider the validity of the proposed questions, the readability of the questions and any other issues that may impact a candidate’s ability to complete the assessment. Once the group of experts are satisfied with the questions/question paper, it is passed to the awarding body’s production team to create the final paper. - 5. Paper production and type-setting
Question papers are typeset and proof read to ensure accurate spelling, punctuation, grammar and layout. Subject experts review the paper prior to its moving on to the next stage. - 6. Working of questions and final quality checks
Once first quality assurance checks are completed, questions are submitted to an independent subject expert whose role is to work through the paper from a candidate’s perspective. This expert will not have been involved in any of the earlier stages in the process. The questions are scrutinised to ensure that they are clear and accessible and that there are no errors or omissions. Questions are reviewed to ensure that they can be answered by the candidate in the time allowed and to ensure that the questions are at the right standard. - 7. Formal sign-off
Each awarding body has a multi-staged approach to the formal sign off. Quality assurance checks are rigorous and are carried out by a number of internal and external experts including:- dedicated proof reading teams
- question authors and reviewers
- internal subject and assessment experts.
- 8. Printing
Following sign-off, papers are securely printed using the awarding body’s print supplier. Checking of the printed products is rigorous to ensure high quality final print. - 9. Post print review
In addition to print checks that ensure high quality final print copies, a final review of the finished paper is undertaken by subject experts and the papers confirmed as ready to progress to the final stage in the process. - 10. Despatch to centres
This is a huge logistical undertaking that involves tens of thousands of parcels of question papers being delivered to centres across the country.
Spotting and managing errors to minimise impact on candidates
Despite the process containing many checks, very rarely, errors do appear on question papers. And although the numbers are small (e.g. in 2017 40 papers out of 3233 contained an error) even one can affect how a student answers that question and possibly the whole paper.
From the moment an error is identified, the focus is on how to minimise the effect on candidates to the extent that is possible.
There are a range of options that can be taken by the awarding body, however, the most suitable course of action to manage an error cannot always be determined immediately. There are, however, a number of principles the awarding bodies work to after establishing whether the affected item can still usefully contribute to the assessment. These include:
- ensuring the examiners receive specific guidance on how to mark responses to any affected questions
- establishing the fairest and most appropriate mitigation for candidates affected by the error
- delivering fair, accurate and timely results to all candidates
- making sure that the post-results review of marking procedure is not compromised
- keeping stakeholders well informed throughout the process
Crucially, awarding bodies will consider the effect at three key levels – for candidates, for centres and for the whole cohort and any decision making will at all times follow the principle of addressing all three. Awarding bodies do this because it is fair and right. It is also in line with Condition A7.1 (a) of the General Conditions of Recognition:
“Where any incident occurs which could have an Adverse Effect, an awarding organisation must promptly take all reasonable steps to […] prevent the Adverse Effect and, where any Adverse Effect occurs, mitigate it as far as possible and correct it.”
What action can awarding bodies take to mitigate the impact on candidates?
When an error on a question paper is identified during or after an examination, the awarding body determines the effect of the error by carrying out statistical analyses. The “toolkit” available is wide ranging and allows for a tailored approach to be taken to each incident. For example, they may involve comparing candidates’ performances on the erroneous question with those on other similar questions in the same paper, or with performance on a similar question on previous examination papers.
The awarding body will then consider the fairest way to deal with the error to minimise the effect on candidates. This may mean discounting the question or making amendments to the mark scheme.
In managing the effect of errors, there is a high level of scrutiny that the awarding bodies apply themselves. They are also required to notify the regulators, outlining the various options that may be available, any substantiating statistical analyses, and the final approach taken to deal with the problem.
10 May 2018
