Study Zone

The life of an AAT exam paper

Rosemary Evans, chief assessor of unit 5 (FRA), explains how exams evolve from concept through to marking

Rosemary Evans started her relationship with AAT as a marker. She is now the chief assessor for FRA. She explained that in March 2009 she started work, with a blank sheet of paper, on the June 2010 exam paper. She began by reviewing the standards and previous papers. Evans explained that FRA is a big unit, so it can’t all be tested in one three-hour paper. She also warned question-spotters that there is not a pattern to how she covers all the key areas: “The papers have to sit together so you can’t spot the gaps. That means students should be revising properly.”
Evans then created the first draft of the exam paper, with answers and competency levels. This is when she thought about how much information she needed to provide students with. The first draft then went to the paper moderator, who looked at coverage of standards and sat the exam and sees how long it took. The paper is also looked over for ‘plain English’ and that the answers Evans has provided are right!
A scrutineer also looked at the paper to ensure it was at the same level as the other papers. He then provided feedback.
This dialogue took place during March and April and a redrafted paper was produced in the light of the moderator’s comments. Following another redraft the paper was then sent to the AAT. In fact, the June 2010 paper went to the AAT on 15 April this year.

Good and fair papers
In May, an exam panel sat for several days looking at the June papers. Each writer and moderator presented their exam for 90 minutes. Evans explained that the panel is there for the best interest of students, not to score points off each other. Their role is to ensure the paper is ‘fair and good’. The panel included a plain English specialist and a marking scheme consultant, who looked for consistency of marking across the exams. Among the other panel members is a technical education specialist.
The panel comments were then considered and recommendations made for changes to the paper. After the panel the changes were made and the first proof of the paper produced (this will be around September/October time). Any changes will take into account any layout problems or page turns to make the exam better for sitters. A second draft is followed by a third and final draft.

Students’ feedback
After the exam is sat comes feedback on the students’ forums. “Apparently, after one exam I was accused of coming from Mars,” Evan revealed.
The scripts are then sent to the markers in post-safe envelopes, either recorded or by special delivery. Evans said she would encourage everyone to send by special delivery, because if they are lost then the Post Office will be able to track the papers. Last December some 14 scripts went missing. Her exam is sat on the Wednesday afternoon and papers arrive with the markers on the Friday. The marker team receive their packs and the marking scheme and start to create a list of questions they want the examiner to answer. There are 30 markers for this paper and between them they look at 200 scripts over the weekend. Evans chooses five nasty papers – those ones that have half right and half wrong and uses them for the Monday markers meeting. This is a very intensive and full-on day. Everyone at that meeting marks those five papers together and by the fifth they have achieved standardisation. She explained any new marker has fewer papers and is ‘more controlled’. Markers also have a checker who makes sure the markers’ marks are right. Each marker now has three weeks to mark their 425-450 papers. So that’s 30 markers, three weeks and 12,000 scripts. The marks are then put online and checked again. Feedback is then sought; Evans said she takes no feedback as a good sign! All this feedback is looked at before the Exam Review Panel meeting, which meets to consider the results and comments.

Special circumstances
If there are any special circumstances (such as a mistake in the paper) and students have been put at a disadvantage then the ERP will act. Interestingly, Evan said that unit 5 has had the best results ever, but there is no pressure from the ERP to lower pass rates. The ERP then makes its recommendations to the education and training monitoring panel.
Evans said she understood why students wanted their results as early as possible but it is six days after the exams before a marker can finally put a red pen in their hand to start marking. She also revealed that the chief assessor and moderator look at every borderline script. She has a quick flick through these paper and admitted she is not disposed to pass those scripts where students have left whole sections blank. “I have a big worry about competence,” she said. So the message to students is, please try every question!
Now the student receives the ‘outcomes’ and the review process is opened. Evans tries to give an indication in the response to script reviews how far the student was from a pass. Guess which one of these got the bad fail:
“You need to do a lot more work in...”
“With focused revision you...”
Just 13 out of the 7,000 who sat the last exam asked for a review.
So that’s the 18-month life cycle of an exam paper. Evans has, of course, three or four exams on the go at once, so really couldn’t tell you what is coming up in the next exam, unless she had the exam paper in front of her!

[«all Studies]