Archive for October, 2011

…what, exactly, is the problem?

 Monday, October 17th, 2011

Education policy-making is in a very strange place at present, with politicisation very much to the fore and reform proposals, though often successful in winning headlines for ministers, sometimes having a superficial quality. This means they often do not bear up well against detailed analysis.

The latest examples came in a speech last week by Michael Gove to Ofqual, the exams regulator. While I have no problem with Mr Gove looking closely at the exam system and proposing changes, this new foray was, to this observer, remarkably unfocused.

I was left unclear not only as to the detail of what exactly Mr Gove was proposing as suggested improvements on the current system, but also why he thinks they are needed.

The speech (which you can read here) captured headlines with some thoughts on possible ways forward for England’s A-level system. Or at least that was the way it was reported, for the speech talks about a range of suggested problems with both GCSEs and A-levels, and the reported proposals seem not to be labelled within the speech as relating specifically to the latter.

But anyway, let’s concentrate on A-levels, for simplicity’s sake, here.  Mr Gove appears to want to respond to a string of concerns about the way the current system works – including, he said, complaints from universities that they struggled to choose between the best-performing 18-year-olds – by making two suggestions for what would be radical changes to the existing structure.

First, he floated the idea that “only a fixed percentage of candidates” should get an A* in each subject. Second, he suggested that, in future, pupils and – presumably, although he was not explicit about this – universities and employers would be told not just the individual’s grade in each A-level, but their rank: how they fared in the national order of marks gained by all their peers.

Ok, I need to deal with each of these ideas in turn.

To take the first one, it is not clear to me from the speech exactly what problem this suggestion is meant to be addressing. The idea follows a section when Mr Gove cites concerns from employers and universities about the levels of “knowledge” their new recruits are arriving with, with the Education Secretary then discussing grade inflation within the GCSE and A-level system.

He also says: “Over the last 15 years, the proportion of pupils achieving at least one A at A-level has risen by approximately 11 percentage points. In 2010, more than 34,000 candidates achieved three As at A-level or equivalent, which allow them to progress to one of the best universities. That’s enough to fill half the places within the Russell Group.

“Universities are increasingly asking: ‘how can they choose between so many candidates who appear to be identically qualified?’”

A cap on the number of A*s awarded would appear to be, then, an attempt to deal with this problem of grade inflation and to ensure that employers and universities do not have too many of the very top achievers from which to choose. There would be no caps on other grades, however, because Mr Gove said he did not “want to go back to the situation where exams all were graded on the basis of norm referencing”, or fixed proportions of grades awarded for all exams at all levels.

But those figures he quoted, again, relate to the number of A grades. The obvious question to address, and which I certainly would expect to be addressed if it were, say, an academic looking at this issue rather than a politician, would be whether the introduction of the A* last year had helped to ease the often-claimed difficulties of university admissions tutors in particular, in choosing between high-flyers with strings of top marks. What has been the A*’s effect? This should be a question at least to be approached empirically. But no evidence was offered here.

And is Mr Gove unhappy that too many A*s are already being handed out, or that at some point in the future this might be the case? For reference, 8.2 per cent of all A-level candidates were awarded the top mark this year, compared to 8.1 per cent the previous year. It would have been interesting, then, to have known whether he was concerned that this picture might change, with the number of A*s rising dramatically in the future as has happened in relation to the A grade: 27 per cent achieved at least an A this year.

Answers to these kind of questions are extremely important when you come to the technical detail of how a capped A* system might work. If, as the speech would seem to imply, Mr Gove really is suggesting that, even as things stand, employers and universities are struggling to choose between lots of applicants with top grades, then the logic of what he is saying is that the proportion of A*s should be capped at levels lower than the present 8.2 per cent, so that only the very highest achievers are identified in this way.

This, though, would create a problem for the exam boards, who work on the principle that examining should be fair to students from one year to the next. In other words, if a change was made to reduce the number of A*s from one year to the next such that student x, in the year before the change, gets an A*, but student y, for the same level of performance the next year, gets only an A, student y would be entitled to feel aggrieved if he or she then comes up against student x in the chase for a job or a university place, since student y will appear to be not as well qualified, but this will be not their fault, but because of the change in the exams system.

The boards could change the standard in this way, but for fairness they’d probably have to make it very clear that grading decisions from two successive years were not strictly comparable; they might even have to go so far as to change the name of the exams, in my view, to make this clear to universities and employers.

If, on the other hand, Mr Gove means that the A* is working OK at the moment as a selection device, but that the numbers achieving it may need to be capped at some point in future (ie at the current or possibly a higher rate of pupils gaining A*), it is hard to know where this leaves his point about universities struggling – presumably as things stand, or why mention it in the speech? – to choose between candidates under the current system.

There would be other technical issues to look at with regard to a capped A* system, including whether it would apply as a uniform percentage across all subjects, which would seem the most obvious, or if it would be different for different subjects.

If it were the former, ie a uniform rate for every subject, this would imply a very large shift from the current system, which last year saw the proportion of candidates awarded A*s varying from 27.5 per cent in further maths to 1.1 per cent in media studies. There would be, I think, potential and perhaps to Mr Gove not-very-desirable potential knock-on effects for candidate numbers in a subject such as further maths if, say, around 8 per cent in each subject were guaranteed A*s.

The latter version of this scheme– a varying cap within different subjects – might seem more sensible, but might again raise questions from students as to why certain subjects were guaranteed to have a higher proportion of A*s, whatever the quality of the students that year and how they did in the particular exam.

The larger point is this, though: ideas were being floated in this speech which could potentially have a profound impact on hundreds of thousands of students each year, but without any meaningful analysis of the detailed nature of the current problem they were seeking to address. Mr Gove said in his speech that this was just “one question for debate, and I don’t mind if, in the end, people shoot me down”, but this belies the fact that, as the most influential actor within education in England, even hazily-sketched ideas have the potential for large influence. It is surprising, to say the least, with all the technical expertise available in our system (of which more later), that this proposal begs so many questions not just as to how it would work, but as to, actually, what its goal would be: why, in detail, do we need a cap?

However, the speech, I am afraid, got more surreal after this. Mr Gove then cited a visit he had made to one school, Burlington Danes Academy, during which he had been told about its system of ranking its students based on the exams they sit in every subject, every half term. This, he said, seemed to have many benefits, according to the headteacher when he had asked her. The gains included parents knowing exactly where their son stood in the class (because before this system was introduced, Mr Gove said, teachers had simply said “he’s a lovely boy”, ie by implication provided no information to parents at all on their child’s progress at school). The claimed benefits also included pupils being able to compare their performance against that of their contemporaries, and even looking at the results of teachers and deciding they wanted to be in the classes of those teachers who added the most value, and demanding those teachers who were not getting them up the rankings were “moved on”.

On the basis of what this school’s head said was going on in this one school in its in-school exams, then, he seemed to be proposing a new system in which pupils across the entire country were ranked at A-level, although, again, whether it was to be both GCSE and A-level or just the latter is not specified in the printed speech.

It is kind of hard to know where to begin with that anecdote, and I am not going to analyse the detail of what Mr Gove said about the ranking system in Burlington Danes, except to say that I remember, as a pupil, on occasion getting a good idea of where I was in each class and responding, when I was doing badly, not by seeing it as a reflection on the teacher but on the quality of my own work, but maybe that was just me. However, it should be enough to note that this is a bizarre way of going about policy, even if Mr Gove was self-aware enough to acknowledge the dangers of reading too much into anecdotes, or as he put it “data is not the plural of anecdote”.

As it happens, the idea of giving pupils a national ranking as well as a grade is not so very odd. I think it already goes on in Australia, and the fact that it is a serious proposition may have been acknowledged, implicitly, by Mr Gove when he said “some boards” here are already “debating the advisability of this”. But, again, the question has to be asked: if this is the solution, what exactly is the problem?

If we must take the Burlington Danes anecdote seriously, Mr Gove would seem to be implying that competition is a good thing and that, if pupils know they are going to be given an exact ranking based on their performance in the exam hall, this is going to spur them on to even greater efforts. I’m not convinced by that, though, in relation to the national A-level system: good grades would seem to be incentive enough for most students. Those at the top, in the A* band, who in some hypothetical world might want to compete more in the chase for a better ranking, probably do not need to be made more anxious about exam success. This is before one gets to possible technical problems, such as the degree of uncertainty around individual rankings – marking can never be reliable enough to produce with certainty a ranking list – and whether the Uniform Mark Scheme (UMS) of A-level, which I’m guessing would have to be the basis for the rankings – will produce patterns such as bunching at the top of the mark distribution, with many students awarded 100 per cent for individual papers under UMS.

The larger point is that, as I understand it, universities can already get access not only to a student’s overall grade, but to their grade in individual papers and also, I think the number of UMS marks scored. (There is a story about this in relation to Oxford and Cambridge in this morning’s Times) I think they can also find out if an applicant’s grades were achieved in the first sitting of individual papers, or through re-sits. At the top end, the A* identifies not only the highest achieving students, but insists that they must do well in the harder A2 papers typically taken at the end of the sixth form, rather than stockpiling marks in the easier AS papers, designed to be taken a year earlier. 

In other words, the system as it is already provides plenty of information for universities.

It could be argued that ranking might help employers, who might not have access to all of the above data. But even for them, a string of grades at GCSE and A-level already provides a great deal of information.

Providing yet more data certainly fits with a government agenda which wants to get seemingly ever more statistics out into the public domain, almost as an end in itself (England’s system already seems to be a near world-leader in terms of data production). The speech also clearly won some publicity which Mr Gove was probably happy with, in the form of headlines suggesting he was shaking up a system which needed reform. It may well have been a useful distraction, for the government, from claims that the coalition’s move to “benchmark” England’s A-level system against other exams from around the world (see TES story here) may not yield the easy policy wins that may once have been envisaged.

I don’t think all of Mr Gove’s moves on exams have been wrong-headed: something had to be done , in particular, on early entry for GCSEs following the Advisory Committee for Mathematical Education’s scathing report suggesting results pressures on schools were having some very unattractive outcomes. 

Yet, this latest speech, for all the confidence in Mr Gove’s phrasing, lacked the serious, evidence-based analysis – or, really, meaningful analysis of any kind – which one would expect in an area this technical. These suggestions, then, have a gimmicky feel to them, though I would guess that Ofqual and the boards are now having to take them very seriously.

As an almost-final point within this blog, I would also highlight a section in the speech headlined “the role of Ofqual”, in which Mr Gove said that “with the leadership that Ofqual has, there is a new requirement for Ofqual to do more.”

He continued that the watchdog should be asking itself the question as to how it was performing, and how our exams compared to those elsewhere, so that “Ofqual moves from being an organisation that perhaps in the past provided reassurance, to one that consistently provides challenge to politicians, to our education system overall and to exam boards and awarding bodies”.

I highlight this because Ofqual was originally set up by Mr Gove’s predecessor, Ed Balls, with at least the stated intention that it should function along the lines of the Bank of England, ie independently of ministers. Serious concerns would be raised if ministers were ever seen to suggest policy priorities for the Bank of England, and even to say that they agreed with an already-taken Bank of England decision would be frowned upon. How independent is Ofqual, then?

-I should, finally, say that the education policy field is the poorer for the loss of the blog and twitter feed provided by Chris Wheadon, head of scientific research and development at the AQA board.

Chris’s twitter feed and blog have been deleted, apparently following complaints from the schools minister, Nick Gibb. (See Telegraph news story here).

Throughout my years covering the exams system, I have benefited from insights into its more technical aspects from those working for the boards, and their regulators. For all the criticisms that many people, including myself, make of aspects of England’s system, these conversations have underscored to me the quality of technical expertise and understanding on which this hugely complex structure now rests.

Chris’s blog in particular was very much in that tradition. I think it served both a public cause, in helping people to understand some of the statistical issues behind examining and that it was a reminder of the research expertise which is present both at AQA and within other boards. I also found Chris’s twitter feed a useful source of information.

Technical expertise of this kind is vital if any reform of our exams system is to be achieved successfully, a view that much of the blog above hopefully illustrates. It is a great shame, then, that this contribution to public debate is now no longer available.

- Warwick Mansell

No Comments
posted on October 17th, 2011

Friday, October 7th

Exam boards are facing fines from the Government’s qualifications regulator after a string of errors in this summer’s GCSE and A-levels.

Ministers are to propose an immediate change to the law to allow Ofqual to impose a financial penalty – capped at a certain proportion of an exam board’s turnover – if they make mistakes.

But the move was questioned by a head teachers’ leader, who said any fines would simply be passed on by the boards to schools, adding to already large exams bills. The boards themselves believe the move, to be introduced in an amendment to the education bill currently in the House of Lords, pre-judges an inquiry by Ofqual into this year’s mistakes, which is due to report by the end of the year.

The way the move has been handled – with boards given only two weeks to respond to the proposed legal change when told about the plans at the end of last month by Nick Gibb, the schools minister – has also annoyed awarding bodies, who are concerned that it will not be given time for proper legislative scrutiny.

Ofqual launched its investigation in July, after this summer’s GCSE and A-level results season featured at least 11 mistakes, affecting tens of thousands of pupils.

Errors ranged from a printing mistake by the AQA board, leading to some schools receiving GCSE maths papers, taken by 32,000 pupils, which included questions from a previous version of the exam, to an OCR maths AS level paper with 6,790 candidates which featured an impossible question worth 11 per cent of marks on the paper.

In total four boards, serving schools in England and Northern Ireland, have apologised for errors.  Ofqual already has power to take strong sanctions against them, including ultimately to “withdrawal of recognition”, effectively banning a board from setting exams

However, two weeks ago (September 29th), Nick Gibb, schools minister, wrote to the boards to tell them that Ofqual’s current powers “inhibit swift action and do not serve as an adequate deterrent to problems such as we saw this summer”. He said the Government would change the law to give Ofqual the power to fine.

Last week, in a letter to the Conservative peer Lord Lingfield, the Government said that it would bring forward amendments at the next stage of the education bill, which begins on October 18th, to “give Ofqual the new power to fine”.

The Government believes that the watchdog’s current power to de-recognise a board is such a “nuclear” option – with potential to cause major disruption for pupils and schools – that Ofqual needs additional sanctions.

Brian Lightman, general secretary of the Association of School and College Leaders, said: “Ofqual’s review is not due to publish until December, and it seems strange to pre-empt the findings in this way.

“A fine on awarding bodies will simply turn into a fine on schools and colleges, since they pay for all the costs of examinations through exam fees. Institutions are already spending large sums on exam fees, and any further burden would be a perverse consequence. It would be completely counter-productive.”

An exam board source said: “We are very unhappy about the way in which this has been carried out. There is a question over whether the Government wants to get this right, through proper consultation, or whether this is just a massive rush to cobble something together to go into an education bill which is already nine tenths of its way through Parliament.”

- Warwick Mansell

1 Comment
posted on October 7th, 2011