Monday, October 17th, 2011

Education policy-making is in a very strange place at present, with politicisation very much to the fore and reform proposals, though often successful in winning headlines for ministers, sometimes having a superficial quality. This means they often do not bear up well against detailed analysis.

The latest examples came in a speech last week by Michael Gove to Ofqual, the exams regulator. While I have no problem with Mr Gove looking closely at the exam system and proposing changes, this new foray was, to this observer, remarkably unfocused.

I was left unclear not only as to the detail of what exactly Mr Gove was proposing as suggested improvements on the current system, but also why he thinks they are needed.

The speech (which you can read here) captured headlines with some thoughts on possible ways forward for England’s A-level system. Or at least that was the way it was reported, for the speech talks about a range of suggested problems with both GCSEs and A-levels, and the reported proposals seem not to be labelled within the speech as relating specifically to the latter.

But anyway, let’s concentrate on A-levels, for simplicity’s sake, here.  Mr Gove appears to want to respond to a string of concerns about the way the current system works – including, he said, complaints from universities that they struggled to choose between the best-performing 18-year-olds – by making two suggestions for what would be radical changes to the existing structure.

First, he floated the idea that “only a fixed percentage of candidates” should get an A* in each subject. Second, he suggested that, in future, pupils and – presumably, although he was not explicit about this – universities and employers would be told not just the individual’s grade in each A-level, but their rank: how they fared in the national order of marks gained by all their peers.

Ok, I need to deal with each of these ideas in turn.

To take the first one, it is not clear to me from the speech exactly what problem this suggestion is meant to be addressing. The idea follows a section when Mr Gove cites concerns from employers and universities about the levels of “knowledge” their new recruits are arriving with, with the Education Secretary then discussing grade inflation within the GCSE and A-level system.

He also says: “Over the last 15 years, the proportion of pupils achieving at least one A at A-level has risen by approximately 11 percentage points. In 2010, more than 34,000 candidates achieved three As at A-level or equivalent, which allow them to progress to one of the best universities. That’s enough to fill half the places within the Russell Group.

“Universities are increasingly asking: ‘how can they choose between so many candidates who appear to be identically qualified?’”

A cap on the number of A*s awarded would appear to be, then, an attempt to deal with this problem of grade inflation and to ensure that employers and universities do not have too many of the very top achievers from which to choose. There would be no caps on other grades, however, because Mr Gove said he did not “want to go back to the situation where exams all were graded on the basis of norm referencing”, or fixed proportions of grades awarded for all exams at all levels.

But those figures he quoted, again, relate to the number of A grades. The obvious question to address, and which I certainly would expect to be addressed if it were, say, an academic looking at this issue rather than a politician, would be whether the introduction of the A* last year had helped to ease the often-claimed difficulties of university admissions tutors in particular, in choosing between high-flyers with strings of top marks. What has been the A*’s effect? This should be a question at least to be approached empirically. But no evidence was offered here.

And is Mr Gove unhappy that too many A*s are already being handed out, or that at some point in the future this might be the case? For reference, 8.2 per cent of all A-level candidates were awarded the top mark this year, compared to 8.1 per cent the previous year. It would have been interesting, then, to have known whether he was concerned that this picture might change, with the number of A*s rising dramatically in the future as has happened in relation to the A grade: 27 per cent achieved at least an A this year.

Answers to these kind of questions are extremely important when you come to the technical detail of how a capped A* system might work. If, as the speech would seem to imply, Mr Gove really is suggesting that, even as things stand, employers and universities are struggling to choose between lots of applicants with top grades, then the logic of what he is saying is that the proportion of A*s should be capped at levels lower than the present 8.2 per cent, so that only the very highest achievers are identified in this way.

This, though, would create a problem for the exam boards, who work on the principle that examining should be fair to students from one year to the next. In other words, if a change was made to reduce the number of A*s from one year to the next such that student x, in the year before the change, gets an A*, but student y, for the same level of performance the next year, gets only an A, student y would be entitled to feel aggrieved if he or she then comes up against student x in the chase for a job or a university place, since student y will appear to be not as well qualified, but this will be not their fault, but because of the change in the exams system.

The boards could change the standard in this way, but for fairness they’d probably have to make it very clear that grading decisions from two successive years were not strictly comparable; they might even have to go so far as to change the name of the exams, in my view, to make this clear to universities and employers.

If, on the other hand, Mr Gove means that the A* is working OK at the moment as a selection device, but that the numbers achieving it may need to be capped at some point in future (ie at the current or possibly a higher rate of pupils gaining A*), it is hard to know where this leaves his point about universities struggling – presumably as things stand, or why mention it in the speech? – to choose between candidates under the current system.

There would be other technical issues to look at with regard to a capped A* system, including whether it would apply as a uniform percentage across all subjects, which would seem the most obvious, or if it would be different for different subjects.

If it were the former, ie a uniform rate for every subject, this would imply a very large shift from the current system, which last year saw the proportion of candidates awarded A*s varying from 27.5 per cent in further maths to 1.1 per cent in media studies. There would be, I think, potential and perhaps to Mr Gove not-very-desirable potential knock-on effects for candidate numbers in a subject such as further maths if, say, around 8 per cent in each subject were guaranteed A*s.

The latter version of this scheme– a varying cap within different subjects – might seem more sensible, but might again raise questions from students as to why certain subjects were guaranteed to have a higher proportion of A*s, whatever the quality of the students that year and how they did in the particular exam.

The larger point is this, though: ideas were being floated in this speech which could potentially have a profound impact on hundreds of thousands of students each year, but without any meaningful analysis of the detailed nature of the current problem they were seeking to address. Mr Gove said in his speech that this was just “one question for debate, and I don’t mind if, in the end, people shoot me down”, but this belies the fact that, as the most influential actor within education in England, even hazily-sketched ideas have the potential for large influence. It is surprising, to say the least, with all the technical expertise available in our system (of which more later), that this proposal begs so many questions not just as to how it would work, but as to, actually, what its goal would be: why, in detail, do we need a cap?

However, the speech, I am afraid, got more surreal after this. Mr Gove then cited a visit he had made to one school, Burlington Danes Academy, during which he had been told about its system of ranking its students based on the exams they sit in every subject, every half term. This, he said, seemed to have many benefits, according to the headteacher when he had asked her. The gains included parents knowing exactly where their son stood in the class (because before this system was introduced, Mr Gove said, teachers had simply said “he’s a lovely boy”, ie by implication provided no information to parents at all on their child’s progress at school). The claimed benefits also included pupils being able to compare their performance against that of their contemporaries, and even looking at the results of teachers and deciding they wanted to be in the classes of those teachers who added the most value, and demanding those teachers who were not getting them up the rankings were “moved on”.

On the basis of what this school’s head said was going on in this one school in its in-school exams, then, he seemed to be proposing a new system in which pupils across the entire country were ranked at A-level, although, again, whether it was to be both GCSE and A-level or just the latter is not specified in the printed speech.

It is kind of hard to know where to begin with that anecdote, and I am not going to analyse the detail of what Mr Gove said about the ranking system in Burlington Danes, except to say that I remember, as a pupil, on occasion getting a good idea of where I was in each class and responding, when I was doing badly, not by seeing it as a reflection on the teacher but on the quality of my own work, but maybe that was just me. However, it should be enough to note that this is a bizarre way of going about policy, even if Mr Gove was self-aware enough to acknowledge the dangers of reading too much into anecdotes, or as he put it “data is not the plural of anecdote”.

As it happens, the idea of giving pupils a national ranking as well as a grade is not so very odd. I think it already goes on in Australia, and the fact that it is a serious proposition may have been acknowledged, implicitly, by Mr Gove when he said “some boards” here are already “debating the advisability of this”. But, again, the question has to be asked: if this is the solution, what exactly is the problem?

If we must take the Burlington Danes anecdote seriously, Mr Gove would seem to be implying that competition is a good thing and that, if pupils know they are going to be given an exact ranking based on their performance in the exam hall, this is going to spur them on to even greater efforts. I’m not convinced by that, though, in relation to the national A-level system: good grades would seem to be incentive enough for most students. Those at the top, in the A* band, who in some hypothetical world might want to compete more in the chase for a better ranking, probably do not need to be made more anxious about exam success. This is before one gets to possible technical problems, such as the degree of uncertainty around individual rankings – marking can never be reliable enough to produce with certainty a ranking list – and whether the Uniform Mark Scheme (UMS) of A-level, which I’m guessing would have to be the basis for the rankings – will produce patterns such as bunching at the top of the mark distribution, with many students awarded 100 per cent for individual papers under UMS.

The larger point is that, as I understand it, universities can already get access not only to a student’s overall grade, but to their grade in individual papers and also, I think the number of UMS marks scored. (There is a story about this in relation to Oxford and Cambridge in this morning’s Times) I think they can also find out if an applicant’s grades were achieved in the first sitting of individual papers, or through re-sits. At the top end, the A* identifies not only the highest achieving students, but insists that they must do well in the harder A2 papers typically taken at the end of the sixth form, rather than stockpiling marks in the easier AS papers, designed to be taken a year earlier. 

In other words, the system as it is already provides plenty of information for universities.

It could be argued that ranking might help employers, who might not have access to all of the above data. But even for them, a string of grades at GCSE and A-level already provides a great deal of information.

Providing yet more data certainly fits with a government agenda which wants to get seemingly ever more statistics out into the public domain, almost as an end in itself (England’s system already seems to be a near world-leader in terms of data production). The speech also clearly won some publicity which Mr Gove was probably happy with, in the form of headlines suggesting he was shaking up a system which needed reform. It may well have been a useful distraction, for the government, from claims that the coalition’s move to “benchmark” England’s A-level system against other exams from around the world (see TES story here) may not yield the easy policy wins that may once have been envisaged.

I don’t think all of Mr Gove’s moves on exams have been wrong-headed: something had to be done , in particular, on early entry for GCSEs following the Advisory Committee for Mathematical Education’s scathing report suggesting results pressures on schools were having some very unattractive outcomes. 

Yet, this latest speech, for all the confidence in Mr Gove’s phrasing, lacked the serious, evidence-based analysis – or, really, meaningful analysis of any kind – which one would expect in an area this technical. These suggestions, then, have a gimmicky feel to them, though I would guess that Ofqual and the boards are now having to take them very seriously.

As an almost-final point within this blog, I would also highlight a section in the speech headlined “the role of Ofqual”, in which Mr Gove said that “with the leadership that Ofqual has, there is a new requirement for Ofqual to do more.”

He continued that the watchdog should be asking itself the question as to how it was performing, and how our exams compared to those elsewhere, so that “Ofqual moves from being an organisation that perhaps in the past provided reassurance, to one that consistently provides challenge to politicians, to our education system overall and to exam boards and awarding bodies”.

I highlight this because Ofqual was originally set up by Mr Gove’s predecessor, Ed Balls, with at least the stated intention that it should function along the lines of the Bank of England, ie independently of ministers. Serious concerns would be raised if ministers were ever seen to suggest policy priorities for the Bank of England, and even to say that they agreed with an already-taken Bank of England decision would be frowned upon. How independent is Ofqual, then?

-I should, finally, say that the education policy field is the poorer for the loss of the blog and twitter feed provided by Chris Wheadon, head of scientific research and development at the AQA board.

Chris’s twitter feed and blog have been deleted, apparently following complaints from the schools minister, Nick Gibb. (See Telegraph news story here).

Throughout my years covering the exams system, I have benefited from insights into its more technical aspects from those working for the boards, and their regulators. For all the criticisms that many people, including myself, make of aspects of England’s system, these conversations have underscored to me the quality of technical expertise and understanding on which this hugely complex structure now rests.

Chris’s blog in particular was very much in that tradition. I think it served both a public cause, in helping people to understand some of the statistical issues behind examining and that it was a reminder of the research expertise which is present both at AQA and within other boards. I also found Chris’s twitter feed a useful source of information.

Technical expertise of this kind is vital if any reform of our exams system is to be achieved successfully, a view that much of the blog above hopefully illustrates. It is a great shame, then, that this contribution to public debate is now no longer available.

Exam boards are facing fines from the Government’s qualifications regulator after a string of errors in this summer’s GCSE and A-levels.

Ministers are to propose an immediate change to the law to allow Ofqual to impose a financial penalty – capped at a certain proportion of an exam board’s turnover – if they make mistakes.

But the move was questioned by a head teachers’ leader, who said any fines would simply be passed on by the boards to schools, adding to already large exams bills. The boards themselves believe the move, to be introduced in an amendment to the education bill currently in the House of Lords, pre-judges an inquiry by Ofqual into this year’s mistakes, which is due to report by the end of the year.

The way the move has been handled – with boards given only two weeks to respond to the proposed legal change when told about the plans at the end of last month by Nick Gibb, the schools minister – has also annoyed awarding bodies, who are concerned that it will not be given time for proper legislative scrutiny.

Ofqual launched its investigation in July, after this summer’s GCSE and A-level results season featured at least 11 mistakes, affecting tens of thousands of pupils.

Errors ranged from a printing mistake by the AQA board, leading to some schools receiving GCSE maths papers, taken by 32,000 pupils, which included questions from a previous version of the exam, to an OCR maths AS level paper with 6,790 candidates which featured an impossible question worth 11 per cent of marks on the paper.

In total four boards, serving schools in England and Northern Ireland, have apologised for errors.  Ofqual already has power to take strong sanctions against them, including ultimately to “withdrawal of recognition”, effectively banning a board from setting exams

However, two weeks ago (September 29^th), Nick Gibb, schools minister, wrote to the boards to tell them that Ofqual’s current powers “inhibit swift action and do not serve as an adequate deterrent to problems such as we saw this summer”. He said the Government would change the law to give Ofqual the power to fine.

Last week, in a letter to the Conservative peer Lord Lingfield, the Government said that it would bring forward amendments at the next stage of the education bill, which begins on October 18^th, to “give Ofqual the new power to fine”.

The Government believes that the watchdog’s current power to de-recognise a board is such a “nuclear” option – with potential to cause major disruption for pupils and schools – that Ofqual needs additional sanctions.

Brian Lightman, general secretary of the Association of School and College Leaders, said: “Ofqual’s review is not due to publish until December, and it seems strange to pre-empt the findings in this way.

“A fine on awarding bodies will simply turn into a fine on schools and colleges, since they pay for all the costs of examinations through exam fees. Institutions are already spending large sums on exam fees, and any further burden would be a perverse consequence. It would be completely counter-productive.”

An exam board source said: “We are very unhappy about the way in which this has been carried out. There is a question over whether the Government wants to get this right, through proper consultation, or whether this is just a massive rush to cobble something together to go into an education bill which is already nine tenths of its way through Parliament.”

The contradiction is, to this observer, breathtaking.

Last week, the Government said this: “Too many of our public services are still run according to the maxim ‘the man in Whitehall really does know best’…The idea behind this view of the world – that a small group of Whitehall ministers and officials have a monopoly on wisdom – has propagated a lowest common denominator approach to public services…”

“People should be in the driving seat, not politicians and bureaucrats,” said the Government, in its “open public services” white paper.

On Wednesday, it announced new decisions on what is to count in school league tables which clearly embody a view that, yes indeed, ‘the man in Whitehall really does know best’: certain qualifications are to be seen as valuable – no matter what pupils, teachers and parents think of them – and certain others are not.

These ‘others’ will continue to be funded by the Government, so that state schools will receive cash to offer them. But the results they generate are not to be published at the school level, because to do so would be to encourage schools to offer them. And the Government wouldn’t want to do that.

Confused enough yet? Well, I must admit that the latest developments on league tables have even this perhaps obsessive chronicler of their many twists and turns scratching my head.

Now, come back to that public services white paper. Now I am very sceptical about this document, having blogged about it here. However, it is useful in one sense as a reference point for a world-view being put forward by the coalition which certainly, I thought, had the benefit of clarity.

An idea which is central to the paper, and much other policy which has emerged from Government in the past year, is this concept of “transparency”. The argument runs as follows.

Whitehall collects huge amounts of data, across all public services. Ministers want to release as much of it as possible.

This would have two benefits, the theory went, I thought at least until Wednesday.

First, “transparency” is a good in itself. The public have a right to know as much about the public services they fund as is possible to provide, so releasing more and more stats on the different qualities of institutions must be a good thing.

Second, one of the key aims is to promote choice and competition. In education, by providing more and more data, the idea is that people get a more and more detailed idea of the quality of each school. In doing so, they get the chance to make more effective choices. And, is the implication, this forces schools to have a more and more tightly-defined regard for what the “consumer” – the parent or pupil – wants, and so, it is argued, the quality of education provided must rise.

An intriguing twist on this argument is that, third, in releasing huge amounts of data in different categories, effectively the Government democratises the use of these statistics for accountability purposes, the argument runs. In the old days, it is claimed, schools were judged by just one or two results formulae, laid down to very tight specifications by civil servants and ministers, meaning that they worried most about performing to goals which had been set for them not by the public, but by bureaucrats.

Now, with schools being able to be judged in any number of ways with the user of the service choosing what matters most to them, the entire process has been devolved, with accountability resting where it should: between institution and user, rather than between institution and policy-makers.

Much of this, I think, is actually very contentious and I hope I have questioned much of the above at some point or another. But it does at least have the virtue of being reasonably internally consistent; indeed, some would say that it is too simple, and too ideological. But, as I say, it is a view.

So, the Department for Education press release yesterday began: “The Department for Education today announced that only the highest quality qualifications will be included in new, transparent school league tables.”

GCSEs and iGCSEs would be included, but other qualifications would have to pass some kind of quality check in order to be released for publication, even though these latter qualifications would continue to be taken in schools and colleges, and funded by the state.

Um, so that would seem to violate the first principle that I thought the coalition’s reforms in this area were based on: complete transparency. If the government had data on something going on within a school, I thought the idea was that it would release it to the public.

Yet here we have a government which says it is committed to transparency seemingly, mind-blowingly perhaps given what I thought its philosophy was, proposing that pupils take a set of qualifications whose results will then be… kept secret.

Nick Gibb, the schools minister, is quoted as saying:  “Parents want more information so they can judge schools’ performance. The changes we have made mean that parents will have a complete picture of their local schools so they can choose the right school for their child.”

Eh? No they won’t have a complete picture. It’s only “complete” if you believe that the non-GCSE courses which are now longer featuring in the rankings, but pupils will continue to take, are not any part at all of what counts in a school. That’s a value judgement made by a minister, rather than coming from decisions at the school or family level. (I also wonder exactly what evidence there is for the at-face-value plausible assertion that “parents want  more information so they can judge schools’ performance”, but that’s another matter…)

And of course, just as strikingly, this move violates the third seeming principle, that publishing more data allows the public to judge what it values within what an institution provides, rather than the state. Yet here, the state is laying down exactly which qualifications are to be seen as high quality, and which are not. It is the officials and the minister – Nick Gibb is the one quoted in this release – who are acting as if they have the “monopoly of wisdom” here. For, if the individual chooses to work for a qualification which does not feature in the league tables, pupils, advised by their parents, will demonstrate that they believe it has some value. Mr Gibb and his advisers appear to be keen on telling them that they are wrong. It is a very un- free market approach, and very un-Tory.

So, as I say, my head is spinning with all of this. I can’t quite understand why a policy has come about which is so at odds with what I thought was the over-arching philosophy. However, I thought I would venture a couple of possible reasons.

The first is reasonably simple: the juxtaposition of this policy and the transparency/democratisation of accountability philosophy might not make much sense, but both of them potentially play well with the media, so, in the policy-maker’s mind, why not go for it? The “transparency” argument above will be accepted by many people, while the belief that this will provide headlines suggesting that ministers are getting tough with  “dodgy” vocational qualifications also appears to have paid off in some newspapers. It’s a win-win, and while this might make for contradictory policy-making, who will notice?

I think that’s only part of the answer, though. The second explanation is a clear implication of the press release: the Government simply is going along with the finding – in the report on vocational qualifications by Alison Wolf which lays the groundwork for these changes – that pupils have been incentivised to go for certain vocational courses not because of the worth of the course to the individual, but because of their high rating in league tables for the school.

That’s an argument I’ve been making since at least the time my book came out, of course. And yes, this is indeed a side-effect of the current league tables. (The press release amusingly says “the Wolf Report demonstrated that the current performance table system creates perverse incentives,” as if this had been in doubt beforehand, or as if any performance table system would not create some kind of side-effect).

The TES rightly points out, on its front page today, that this is the final confirmation that the contribution of non-GCSEs to headline “GCSE” measures will be capped, which must be a correct decision, given the way “GCSE” league table findings are interpreted by the public and given the perverse incentives which have existed up to now.

But otherwise the remedy to this problem is bizarre. This particular perverse incentive was created not because of the mere existence of vocational qualifications in any league table ranking (though all league tables will create perverse incentives), but because some of them were – seemingly, to this observer – so over-weighted in the central indicators that there was a huge incentive for schools to push pupils towards them, with the need of the school to raise its scores at least a large part of that calculation in many cases.

If the Government changed this so the results for particular individual non-GCSEs were simply published separately alongside each GCSE in the tables, this particular perverse incentive, I think, would have gone. Although schools would still have to think about success rates for every type of individual qualification they entered – which can be a perverse incentive, I think, in that I don’t think a pupil wanting to take a course should be pushed away from it just because they are unlikely to get a C grade, though I guess some teachers would dispute this -  at least schools would be relatively free to opt for courses they, and the parent and pupil, potentially valued. A course could still have its results published, but if parents and pupils did not value it, the “market” would kill it off in a way that might not have happened under the old system, when schools were incentivised to push pupils towards particular courses with high league table weighting.

As we are, the new league tables will not neutralise the incentives on schools to push pupils towards particular qualifications because of the benefit to the school, rather than to the pupil. It simply changes the type of qualifications which might be favoured, based on the  “wisdom of Whitehall ministers and policy-makers” as to which type of courses should be favoured. To put it another way: ministers don’t seem to like qualifications assessed entirely through coursework, and I would agree that it is tough for these courses to co-exist and have credibility with a high-stakes accountability system in which teachers are being held to account for the results.

But simply removing such courses from high-stakes accountability, in the sense that the results of these qualifications are not published themselves, does not remove them from the effects of league tables. Ministers are incentivising schools to move away from them. Is this the best move for the child? Again, schools are not able to that decision from a neutral perspective, because of the mechanism of high-stakes accountability.

To put it another way, the press release says: “Teachers will still be able to use their professional judgement to offer the qualifications which they believe are right for their pupils.” But this will, still, clearly be influenced by league table considerations, as, mind-blowingly, even the DfE knows, since it also says in the press release that its league table changes will “ensure that schools focus on valued qualifications”. (Sorry, I made the mistake of looking at the press release again; it’s not good for my head).

So, ahem, clearly I still think there are fundamental problems with league tables and results pressures at lots of levels. But I’m especially surprised the government did not come up with a solution which at least is a better fit with its own logic. Not to have done so either smacks of the thinking behind the rankings getting so complex that everyone gets confused, or of a belief that the incentives within league tables can be harnessed for the greater good, even despite the clear inconsistency. Some will also claim that there is a vindictiveness in ministers coming out against non-GCSE exams and those which are assessed by the teacher, although I am not sure about this verdict myself: some of the arguments within the Wolf report about needing to look properly at the quality of courses towards which “non-academic” pupils are being pushed are powerful. But then again, if these are not good courses, why should the Government continue to allow them to be funded in state schools?

I wonder if this is not also another example of that mad pendulum swinging in education policy: Labour worried that league table pressures would push schools away from vocational courses unless they were given an incentive not to do so – so it over-incentivised them – and the Tories respond by using one of the easiest levers they have to pull – how schools are held to account – to wipe many of them off the official map of what counts.

The Government’s move may have demonstrated something useful, though. Sometimes, to hear supporters of league tables talk, publishing data is both a largely “neutral” act of transparency, and almost inevitable. To open up the statistics is simply a matter of letting in some sunlight into previously obscure areas of school practice, one would think sometimes from listening to the advocates of this movement.

In reality, the choices the Government makes as to what is measured, how, and what data is released, are hugely important. It remains, in this sense, a very centralised system, which I suspect is why civil servants and ministers like it so much. It clearly drives how schools act. But in this sense, it is not a “neutral” act, to be judged on the criterion that you either like the idea of releasing more data, or you don’t. Politicians should be held to account for the effects of their moves on data, not just the existence of them. To the extent that this move will stop pupils being pushed towards non-GCSE courses because of the high-equivalence factor, the politicians should be praised. But believe me, the problems and injustices are not going to go away, and the inconsistencies here are now glaring.

One final point. The  press release also says: “The 2011 school league tables will highlight the performance and progress of disadvantaged groups compared with other pupils. This will create a powerful incentive to narrow the gap in achievement between richer and poorer pupils.”

I know what the thinking is behind this, and on the surface it seems commendable. But I can’t help cringeing when I read it. There’s a sense of a teacher reading this and thinking: “Oh yeah, helping disadvantaged kids do well. I’d never thought of that. I just wasn’t bothered before. Now, thanks to your wisdom, Mr Gibb, in putting another column on the spreadsheet by which I’m judged, I realise the error of my ways. I’ll try harder now. Thanks.”

I know one of the most persistent debates within education is about many teachers not having high enough expectations of pupils from tougher backgrounds. But it strikes me that if the main way of tackling it is for ever-closer monitoring through the statistics generated at the end of this process, we are in danger of missing an awful lot of tricks.

If the Government isn’t, through the way it trains teachers, develops them in the classroom, the messages it sends them in its rhetoric and the support it provides to improve the quality of the educational experience for all pupils and particularly those from disadvantaged families, trying to promote this, I don’t know what it is doing. If it had done that, and then effectively argued that teachers still need an “incentive” at the end of this process, you have to ask what has gone wrong along the way.

-There are other things to comment on in the league table announcements – including the fact that pupils taking the EBacc appear now to have to take seven GCSEs, raising question marks over how  much time they will have to study much else, and the contentious, for me, move from contextual value added to value added and unadjusted progress measures as the main “fair” ways of judging schools – but I seem to have run out of space and time today….keeping up with developments in the rankings is a full-time job, it seems….

Well, I said at the end of the last blog that I’d be writing something imminently on the relationship between the Bew assessment review and the government’s ongoing national curriculum review. Here, slightly earlier than planned, is what I had in mind.

This week’s Government response to the Bew review into primary assessment could be redundant within just over two years.

That is the implication of comments made by a leading figure within the test regulator Ofqual at a conference on the national curriculum I attended on Friday.

Stephen Anwyll, Ofqual’s head of 3-14 assessment, said the long-term future of testing would be “up in the air” until after the outcome of the current national curriculum review was known.

Assessment arrangements in primary and secondary schools would have to be “completely revised” if the review led to a fundamental rethink of what schools teach.

The standards pupils achieved in any national assessments created as a result of the national curriculum review might also not be comparable with current performance, he said, since measurement would need to be “recalibrated” as assessments changed.

Mr Anwyll also suggested there was a contradiction between the Government’s suggestion, in its remit for the curriculum investigation, that it should not cover assessment and the detail of what it was being asked to look at. “You cannot separate the curriculum from assessment,” he said.

English, maths, science and physical education aspects of the curriculum for 5- to 16-year-olds are due to be revised for first teaching from September 2013 following the curriculum review, which is expected to produce first recommendations by early next year.

Speaking at a Keele University Centre for Successful Schools conference last Friday, Mr Anwyll talked about Ofqual’s detailed work to be carried out in response to the Bew review.

He then added: “Sitting beyond all of this, in the slightly longer term…all of this is up in the air depending on the outcome of the national curriculum review.

“If we are talking about, actually, a new programme of study, in the first instance for English, maths and science, which we are expecting to see some examples of this year, that could change the entire picture.

“If you reform standards as part of the national curriculum review, it’s ground zero again; you calibrate  the standards from there- you cannot start comparing to previous standards.”

He added: “National curriculum assessments are excluded from the remit of the national curriculum review.

“But if you look at what’s included in the remit, it includes whether the national curriculum should be set out on a year-by-year basis, what should replace existing attainment targets and level descriptors to define better children’s standards of attainment, and what’s needed to provide expectations for progression to support the least able and stretch the most able.”

“All of these are absolutely fundamental to assessment, so you cannot separate curriculum from assessment.”

That comment appears to echo a statement by Sir Jim Rose, leader of England’s last curriculum inquiry, carried out in the dying days of the last Labour government. He was barred from considering assessment but said this was the “elephant in the room” when he visited primary schools.

Mr Anwyll added: “Much of what we do currently will have to be completely revised if we get a new national curriculum, new standards defined, and new ways of measuring them defined.”

That’s the newsy bit; the below is comment from me:

This notion of a contradiction between a national curriculum review which is supposed not to be looking at curriculum matters, and in practice it being impossible for a review of this type not to have serious implications for assessment was underlined this week in the Government’s response to the Bew report.

The remit for the national curriculum review says: “The review itself will not provide advice on how statutory testing and assessment arrangements should operate”.

Yet this week’s Government response to Bew says: “The national curriculum review will consider the suggestion from Lord Bew and the panel for statutory assessment to be divided into two parts….”*

It also says: “The National Curriculum Review will consider how we report statutory assessment in the long term.”

Hmm.

The full sentence of that quote above about statutory assessment being ‘divided into two parts’ is: “The National Curriculum Review will consider the suggestion from Lord Bew and the panel for statutory assessment to be divided into two parts in the future, with a ‘core’ of essential knowledge that pupils should have learnt by the end of Key Stage 2.”

This looks to be a suggestion that some “basic” skills literacy and numeracy tests be introduced at KS2. It is building on a somewhat mysterious idea flagged up near the end of the Bew report, which I blogged about here. One to watch, I think, and not only by people who wonder at the polarising language of “knowledge that pupils should have learnt”….

*I was reminded of this nugget of info via Helen Ward of the TES on twitter.

This is just a quick reflection on union reaction to the Government’s proposals on the future of assessment at Key Stage 2.

Ministers published today their response to last month’s final report by the Bew inquiry into this subject, the review which itself was triggered by last year’s Sats boycott by the National Association of Head Teachers and National Union of Teachers.

The unions’ reaction is interesting: four different associations produced arguably, three or four different positions in response.

This could be viewed as surprising, given that, for all the changes put forward in Bew, the fundamentals of the high-stakes testing regime remain in place, despite widespread concerns within the profession. Or it may simply reflect a beneath-the-surface belief that, whatever the problems with current structures, essentially the basics of the system are in the end unchallengeable, and therefore the argument must be confined to the detail as to how it works.

In terms of their reaction to the Government response to Bew, the heads’ associations were more upbeat. Perhaps unsurprisingly, the National Association of Head Teachers, which called off the possibility of a repeat of last year’s boycott in 2011 in return for the Bew inquiry, and which was allowed to recommend head teachers who would sit on the Bew committee, was broadly positive about its outcome.

Its press release was headlined: “Bew recommendations are a significant step forward towards fairer accountability system, say school leaders.”

But the NAHT said the Bew recommendations – every one of which has been accepted by the Government (always an interesting development for any inquiry which is billed as independent from ministers, I feel) – were only a “first, positive step on a long journey towards a system which reflects the achievements of all pupils and the contribution of all schools”.

Longer-term goals included a far greater role for teacher assessment and more trust in the profession, and the NAHT said it would be on the look-out for ministers breaking with the spirit  of the Bew recommendations.

The Association of School and College Leaders was also accentuating the positive, headlining its release: “KS2 assessment moving in the right direction.” I will come back to this.

Both the National Union of Teachers and the Association of Teachers and Lecturers were less optimistic.

The NUT argued: “The positive steps in this review will be undermined by keeping in place school performance tables, despite the fact that the majority of those who gave evidence called for their abolition.

“While league tables exist, teaching to the test and a narrowing of the curriculum will remain…The Review and the Government should have been bolder.”

The ATL said: “There is some good news in the government’s changes to key stage 2 testing, but so much more could be achieved if the government was not insisting on remaining judge, jury and executioner of schools by setting targets, closing schools, and forcing through its naïve free market policies on academies.”

I haven’t received a press release from the National Association of Schoolmasters Union of Women Teachers, but we know that that union has long favoured tests over teacher assessment, amid concerns about the effects of TA on teachers’ workloads.

For me, having looked at – and written about – the changes proposed by Bew (blog here), this feels like a very muted end to what has been years of pressure building on ministers over testing: the NAHT itself conducted a review into the architecture and effects of the current system, to which I contributed, and which dates back to 2007. Part of that pressure was exerted, amazingly perhaps as it appears now, by Michael Gove when he seemed to accept, in 2009, that test-driven teaching can be bad for children’s education. It also built through the testimony of the unions, subject associations and reports from organisations such as the Children, Schools and Families select committee, the Cambridge Primary Review and the Children’s Society’s Good Childhood inquiry.

One could look at the positive reaction with which many teachers are likely to greet the move, recommended by Bew and accepted by ministers, that the current writing composition test in KS2 English is replaced by teacher assessment, and take a different response to the quick verdict I’ve offered above, of course.

Or, for critics of the high-stakes regime, there is the fact that, since 2008, the following Sats tests have bitten the dust: English, maths and science at KS3; science at KS2; and creative writing at KS2. This might be considered a good outcome of all that pressure.

But, on the negative side, Bew put forward, shockingly, I think, the unbalanced assertion that “strong evidence shows that external school-level [presumably statistics-based] accountability is important in driving up standards”. And the essentials of our system – that test and exam results will remain the main mechanism by which both secondary and primary schools are held to account, with high stakes including closure to follow for “underperformers” – remain unchanged.

The underlying argument must be that this high-stakes system has been good for English education, and that it is a key to continuing progress in the future. If this were not the underlying assumption behind Bew, we would not be proceeding on the current basis, for it provides no fundamental attempt to re-engineer assessment and accountability so that the system gets the accountability it needs without the knock-on washback effects on teaching and learning.

As ever, the basic architecture of test- and exam-based accountability seems to be the unalterable fact of education in England, to which everything – including, I’m afraid, a fair-minded and rigorous consideration of its overall effects on children’s education – must come second. More than 15 years after the introduction of national testing in England,  there has still been no detailed Government inquiry into the nature, extent and effects of test-driven teaching in this country: how many schools go in for it, the detail of how children’s learning is affected and what pupils alongside teachers think about it. This is astonishing, really, if you believe that the quality of the child’s educational experience is to be looked after above all else.

Just finally, I want to return to ASCL’s position, which I think is the most curious.

Brian Lightman, ASCL general secretary, is quoted in its press release as saying: “There must be a robust but fair process of assessment for pupils as they move from primary to secondary school. This is important not only for pupils and their parents, but also so that their new schools have accurate and reliable information about their level of progress.”

I find this statement, which reflects what has been ASCL policy for a while, strange because of the contrast with the somewhat ambivalent relationship secondary schools have with KS2 assessment data, as documented in the Bew report (and elsewhere).

The final Bew report says: “We have heard widespread concern that secondary schools make limited use of the information they receive about their new intake. Many secondary school respondents have expressed concern that national curriculum test results or primary schools’ teacher assessment are not always a suitable proxy for the attainment of pupils on entry to Year 7.”

If many secondary schools don’t trust pupils’ Sats results (or test-influenced TA judgements), why does ASCL want them retained as “robust but fair” measures?

I’ve not put this to ASCL, but I believe the answer is that the union, while it doesn’t particularly trust Sats results as measures of pupils’ underlying understanding, doesn’t want them replaced with teacher assessment because secondary heads worry that primary schools would inflate TA judgements. This would leave secondaries’ results looking less good, because it would mean pupils would appear to be making less progress at secondary school.

So while secondary heads might have reservations about the value of the data provided by Sats, the implications in terms of the accountability system for them mean they back them. As usual, the demands of the accountability system, then, seem to trump other concerns.

This may be a scandalous explanation for ASCL’s position on this issue, but I am struggling to think of another one.

All of which leaves me slightly saddened. There is still an awful lot of evidence that this system is not serving at least a large proportion of children’s needs well. It is a shame that the unions have not seemed able, in the end, to come together to continue pressing home that point.

- Is this really the end of the story for assessment at KS2, though? The current national curriculum review may throw things up in the air again. I expect to write more about that in the next few days.

PS: It is interesting to play “spot the difference” between the stated purposes of national assessment, as laid down by the Bew report, and the previous attempt at this, by the Labour government’s “Expert Group” on assessment, which took in the fall-out to the 2008 Sats marking crisis and reported to the former schools secretary Ed Balls in 2009.

Bew lays down three main purposes of statutory end of Key Stage 2 assessment data as follows:

a Holding schools accountable for the attainment and progress made by their pupils and groups of pupils.

b Informing parents and secondary schools about the performance of individual pupils

c Enabling benchmarking between schools, as well as monitoring performance locally and nationally.

The “Expert Group” report came up with the following definition of the purposes of “assessment”, up to the end of Key Stage 3. It came up with four:

-To optimise the effectiveness of pupils’ learning and teachers’ teaching.

-To hold individual schools accountable for their performance.

-To provide parents with information about their child’s progress.

-To provide reliable information about standards over time.

As you can see, Bew’s top two purposes are extremely similar to the 2008 report’s purposes two and three. The 2008 report’s purpose four is a subset of Bew’s purpose three. The largest difference between the two is that the first purpose mentioned in the 2008, which in my view is correctly placed at the top of the list, does not feature in Bew’s list. Otherwise little, it seems, changes.

I should begin this blog post with a note of slight regret. It gives me no pleasure to be writing something which is critical of the Bew report, especially given the courtesy with which Lord Bew treated me in giving evidence to the review. He invited me to do so, and even wrote me a handwritten note to thank me afterwards.  The review’s interim report, published in April, was, I thought, a largely impressive synthesis of evidence on this subject which gave me hope that, whatever the outcome and whatever the constraints of the remit, the issues would be given a thorough and fair weighing in the final report.

Yet, I am afraid, despite some impressive passages, the report really does not do justice to this, I think, incredibly important subject.

I say this mainly for three reasons.

First, the report fails to follow through on what is said, at least in the foreword to the report, to be the first priority for the assessment and accountability system: ensuring that such a system supports children’s learning. Second, it misrepresents the evidential position on the effects of test-based accountability in a fundamental way. And third:  it does not address in any meaningful sense a central criticism of test-based accountability: that test results are being used for too many purposes and that key purposes can be at odds with one another (my italics, since this was the bit that was not meaningfully considered).

To deal with the first problem, Lord Bew says in the report’s foreword:

“We would like to be quite clear that throughout this process we have always focused on how best to support the learning of each individual child.”

If this had been the overall goal of the review, I would say “fantastic”. The trouble is, having set this up as an aim in the foreword, this approach is completely absent in the report, where the quality of the learning experience resulting from accountability – what, if anything, is happening in lessons as a result of test-driven accountability? – really gets only glancing consideration.

This becomes clearer when we look at the report’s consideration of evidence.

The report says: “Strong evidence shows that external school-level accountability is important in driving up standards and pupils’ attainment and progress. The OECD has concluded that a ‘high stakes’ accountability system can raise pupil achievement in general and not just in those areas under scrutiny.”

Well, I wrote in detail here about the OECD evidence on which Bew drew for this statement.

I do not think anyone reading that report in full could believe that it provides a ringing endorsement of an “English”-style accountability system. Consider, as I mentioned in that blog, the fact that that OECD report says: “Across school systems, there is no measurable relationship between [the] variable uses of assessment for accountability purposes and the performance of school systems.”

Moreover, although Bew says “the OECD has concluded that a ‘high stakes’ accountability system can raise pupil achievement”, with “high stakes” in quotation marks, in fact the phrase “high stakes” only occurs once in the main text of the 308-page OECD report which Bew references here, and its use does not back up the claim made here. (“High stakes” in the one instance referenced in this report refers to any qualification which is high stakes for a pupil, by which criterion the A-levels I took in the 1980s – which were low stakes for my school – would count but today’s Sats would not).

As I wrote in an article for the TES based on research for the NAHT, in fact there are many education systems which are not doing a demonstrably worse job than England and which do not have “high-stakes” accountability of the English kind.

If Bew’s claim is that this type of accountability “is important in driving up standards and pupils’ attainment and progress” is to be understood as meaning that it improves education in a more general sense than simply improving test scores, which must at least be considered if the quality of pupils’ learning is really what matters, then the report needs to consider more evidence.

Yet this section of the report, entitled “the impact of school accountability”, includes no studies raising concerns on the issue of test-driven schooling. It highlights only research which supports it.

This section then simply ends: “We believe the evidence that external school-level accountability drives up pupils’ attainment and progress is compelling.”

This is an absolute travesty of the evidential position. I would say that, given that I wrote my book on this subject from 2005 to 2007 seeking to put together all the evidence I could find on the effects of this system. Negative effects were not hard to come across: detailed concerns about the side-effects were coming to me naturally virtually every week around that time in my work at the Times Educational Supplement. To repeat, none of this evidence gets a mention in the section of the report where Bew is deciding whether or not high-stakes accountability is a good thing.

That is a shocking indictment of this final report. For all the evidence commented on in the interim report, it undermines any claim that this subject has been considered in a truly open-minded way.

If the evidence had been considered, weighed and a conclusion reached that the claimed advantages of hyper-accountability outweighed the claimed negatives (taken seriously and considered in detail); or if a conclusion had been reached that the current system, though imperfect should be retained because changing it in a fundamental way would present too many difficulties, well at least that would have been more honest. To try to claim that the evidence points entirely in this single direction is simply wrong.

Other inquiries to have raised deep concerns about test-driven schooling in recent years have been the Children, Schools and Families assessment investigation of 2007-8, plus its subsequent probe into the national curriculum; the Children’s Society’s Good Childhood Inquiry; and the exhaustive Cambridge Primary Review.  Sir Jim Rose, in conducting his own national curriculum inquiry for Labour which was barred from considering assessment, described it as the “elephant in the room”, in terms of the impact on the curriculum.

Consider some of the claims made in evidence to these various reviews.

The Mathematical Association told the select committee inquiry: “Coaching for the test, now occupying inflated teaching time and effort in almost all schools for which we have information at each Key Stage, is not constructive: short term ‘teaching how to’ is no substitute for long-term teaching of understanding and relationship within and beyond mathematics as part of a broad and balanced curriculum.”

The Cambridge Primary Review reported one witness to the review as, citing her experience as an English teacher, primary head and English examiner, as condemning “the ‘abject state of affairs’” where reading for pleasure in schools “has disappeared under the pressure to pass tests”.

The Independent Schools Council told the select committee’s curriculum inquiry: “National curriculum assessment should not entail excessive testing. Universally, a focus on testing was found to narrow children’s learning, teachers’ autonomy and children’s engagement in learning.”

Ofsted also told the select committee that “In some schools an emphasis on tests in English, mathematics and science limits the range of work in these subjects in particular year groups.” An Ofsted report on primary geography from January 2008, found that “pupils in many schools study little geography until the statutory tests are finished”, while an Ofsted report on music said “A major concern was the amount of time given to music. There were examples of music ceasing during Year 6 to provide more time for English and mathematics.”

The OECD itself said, in the education section of its report on the UK in March this year that: “Transparent and accurate benchmarking procedures are crucial for measuring student and school performance, but “high–stake” tests can produce perverse incentives. The extensive reliance on National Curriculum Tests and General Certificate of Secondary Education (GCSE) scores for evaluating the performance of students, schools and the school system raises several concerns. Evidence suggests that improvement in exam grades is out of line with independent indicators of performance, suggesting grade inflation could be a significant factor. Furthermore, the focus on test scores incentivises “teaching to tests” and strategic behaviour and could lead to negligence of non-cognitive skill formation”

 Either Bew has, then, defined “attainment and progress” in such a narrow sense – ie it means “there is compelling evidence that test-driven accountability drives up test scores” – that its claim to be interested in the learning of each child more generally cannot bear scrutiny (since it is only interested in the evidence of test scores).

Or improving “attainment and progress” is meant to stand for the quality of education as a whole improving as a result of “high-stakes” test-based accountability, in which case Bew has simply chosen to ignore that section of the research on this subject which conflicts with the way the review was framed by the government.

The report does, then, move on to “concerns over the school accountability system”, including “teaching to the test”. But it offers no detail of what the evidence says as to what this might mean for the pupil. The only substantial concern acknowledged here is the unfairness of the way results indicators are used for schools, which it says its recommendations will go on to tackle. This is an important argument, of course, but it is not the same as the claim, widely made, that the system of test-based accountability damages the learning experience of at least a proportion of pupils.

The only acknowledgement of this claim here is when the report says that many heads feel they “‘need’ to concentrate much of Year 6 teaching on preparation for National Curriculum Tests in order to prevent results dropping”. Bew then acknowledges that “the accountability system to date may appear to have encouraged this behaviour [my incredulous italics at the weakness of ‘may’, when heads face losing their jobs if results fall]”.

The report reacts by simply arguing that this need not happen: schools can get good results without narrowing the curriculum. That is exactly the conclusion of the last major report to look at this subject: the 2008 “expert group” report on assessment for Ed Balls as schools secretary.  That report suggested running a campaign to persuade teachers not to teach to the test, since there was simply no need.

Although teachers have argued with me that a good professional does not need to teach to the test, I’m afraid I think of this, when I read it in official reports, as the ostrich, or head-in-the-sand position. It is unscientific, I believe: the fact that some teachers and schools buck the trend does not negate the existence of the trend. The National Strategies, in the past have encouraged teaching to the test, so presumably they thought there was some value in it for schools, in terms of improving results. I suspect local authorities have also promoted a great focus on the content of the tests in schools where the data just has to improve. Overall, the incentives of the accountability system certainly push at least a proportion of schools towards test-driven teaching and thus, if one truly wanted to change this, it would be a good idea to look at changing the way accountability works, rather than effectively simply telling teachers not to follow what for many of them will be its logic.

Then the report closes down the debate, saying simply: “Given the importance of external school-level accountability, we believe publishing data and being transparent about school performance is the right approach.”

In other words, because the review team had already decided that the evidence of the beneficial effects of external accountability was “compelling” – ie without presenting any research on negative impacts – that was the end of the matter. There was no consideration of the actual impact on children’s learning during test preparation, and the nature of it.

Incidentally, because the review team believes that “high-stakes” accountability – ie making results high stakes for schools – works, it must then also believe that assessment should drive what goes on in schools, since the philosophy must be that making assessment results “high-stakes” for schools forces them to improve the quality of education they provide.  

The third problem of the report is related to this, and I don’t want to use too much space going into it in detail here. But in essence it runs as follows. Bew really ducks another criticism of test-based accountability: that test results are used for too many purposes, and that because of this, testing as currently constituted serves many of these purposes less than well.

I’ve put the second bit in italics, because Bew really doesn’t consider this implication. Essentially, Bew accepts the widespread claim that assessment data are put to very many purposes, but reacts to this mainly by listing the “principal” purposes to which they are already put, and then saying other uses should be considered as “secondary”.

It is, I suppose, at least an attempt to consider this issue. But the problem is that the purposes suggested as central by Bew include both that data should be used to hold schools to account, and to provide good information on the progress being made by individual pupils, for the benefit of those pupils and their parents. Bew’s claim, in the foreword, that test-based accountability should also support children’s learning should also be borne in mind here, for that must be another guiding principle if taken at face value.

The problem with the report is that arguably the argument at the heart of this debate is that the use of data to provide information on a school – and on teachers’ – performance can conflict with its use both to support pupils’ learning and to provide the best possible information on the quality of that learning.

This is a big part of what the many people who, Bew acknowledges, submitted evidence to the review mean when they say that the problem is not the tests, it is the league tables which are constructed on the back of them. Because teachers are worried about their school’s results, they take actions which, while right in terms of boosting results, may not be supporting the best learning experience for the child, or their long-term educational interests. And the very act of teachers directing so much attention at the tests and results indicators may also, paradoxically perhaps, make them less good measures of underlying education quality, an argument implicitly acknowledged in the report in a section where it says many secondary teachers do not trust KS2 Sats results because of the extent to which pupils have been prepared for the tests.

In other words, the purposes – and even these “principal” purposes – are in conflict. A report which took seriously the washback effects on learning, from the child’s point of view, of the accountability system, would look much more closely at each of these aims to try to ensure that the requirements of accountability do not conflict with the aim of providing the best possible education experience for pupils.

Some alternative proposals, not backed by Bew, have tried to look at re-engineering aspects of the system to stop some of the purposes conflicting in ways which look either harmful for pupils, or which give us less good data than we might want.

For example, the suggestion put forward by many that national education standards could better be monitored through a system of assessing a sample of pupils rather than through testing every child comes because the purposes to which the current testing system is put are felt to be in conflict. A sampling system, with a relatively small number of pupils being assessed and each on differing parts of the curriculum, would allow information to be collected, potentially, across a much wider and deeper spread of aspects of the curriculum than is possible through a system where all pupils must take every test produced. And its information on whether standards were improving or falling would be more robust because, as the results would be “low-stakes” for schools, test questions could be retained from year to year to allow direct comparisons of pupil performance to be made.

These kind of improvements on the quality of information provided are not possible in the current system because other purposes to which current national test data is put – to provide information on individual schools and on all pupils’ performance, meaning that every pupil must be tested, and papers must change from year to year to guard against schools “cheating” – make them unfeasible.

A more serious look at this subject would also have considered in detail the problems of seeking simultaneously to use test results as “objective” measures of pupil performance;  to support learning; and also to hold schools to account. In 2006, a proposal put forward  by Cambridge Assessment and the Institute for Public Policy Research acknowledged the problem that the purposes were in conflict: the need for schools to generate good results could lead to test-driven teaching and a narrowed curriculum, which was not an ideal form of learning. It therefore proposed a change whereby teacher assessment would become the main judgement on both pupils’ and schools’ performance, but then children in each school assessed through a “testlet”, measuring for each child just a small area of the curriculum. The testlet results would be used as an assurance that the accountability function now placed on teacher assessment was not leading schools to inflate their results. In other words, it retained accountability but, in trying to change the relationship with tests in a small number of subjects, attempted to stop it conflicting with the goal of supporting good learning. This idea was not considered in detail by the report.*

Another alternative, mentioned as my favourite in my book, would be to make inspection judgements the central focus of school-by-school accountability (with inspections offering a rounded look at the quality of education provided, to guard against curriculum narrowing), and to run sample tests to help provide national education quality information.

Instead of trying to look at the relationship between the purposes, Bew has simply left the mechanics of the system in place, in that assessment data is still to be used for all the main purposes it is now including : holding schools to account, producing data on individual pupils’ performance for the benefit of them and their parents, and generating national and regional achievement data.

The report says that through its proposals “we believe we can address the imbalances and perverse incentives in the school accountability system”.

Because the review has not addressed the issue of the conflict of purposes this idea of countering perverse incentives is, I think, a forlorn hope. Its proposals represent no significant change to the system’s fundamentals, but rather a restating of the basis of the system – (which the report must implicitly believe, in its essentials, to be a good thing) – and then an attempt to manage the detail.

Ok, so now, finally to turn to the concrete stuff in terms of those detailed changes recommended by the report, some of which, I think, are important.

-          The report proposes moving to a system of publishing schools’ results averaged over a three year period, to address concerns that judging institutions on single years is unfair, given the way pupil cohorts can change. Small schools, where the introduction of a few high- or low-achieving pupils can have a proportionally very large effect on results from year to year are particularly hard hit by the current system, and their concerns would seem to have influenced this change. However, three-year averages are not recommended to replace single year statistics, but to sit alongside them in league tables. A key consideration could be what weight they are given elsewhere in the accountability regime, including Ofsted reports and floor targets; the report does not, I think, stipulate that they should be given priority.

-          Additional measures are to be introduced recording schools’ achievements counting only those pupils who completed the whole of years 5 and 6 at the school, in response to concerns that schools with lots of children arriving from elsewhere feel an effect on their results. Again, it seems these results will be published alongside the existing measures, rather than replacing them.

-          The report talks about placing a greater emphasis on progress measures, alongside “raw” attainment. However, progress measures already feature in league tables, are central to Ofsted’s new systems and are included in the government’s new floor targets for primaries. So call me a cynic but it is hard to see that the report has added much here. (Overall, my hunch is that there is very little in the report as a whole with which the government would disagree – and you have to wonder after reading this report if this was always likely to be the outcome – but one test (pardon the pun) of that will have to await ministers’ reaction to the report).

-          Teachers will submit teacher assessment judgements before pupils’ test results are known. This seems sensible to me, as it negates the risk of the test judgement influencing the teacher assessment verdict. As the report correctly states, they are measuring different things, so the judgements reached through each assessment method should be kept separate.

-          Finally, the most significant change relates to writing. Bew proposes, first, the introduction of a new test of spelling, punctuation, grammar and vocabulary. I guess teachers will have views on that; I would not comment except to say that the comment in the report that these aspects of English have “right” and “wrong” answers was something some people were querying last week.

The recommendation, however, to replace the writing test with teacher assessment is substantial. It has always seemed to me strange, as someone who went through secondary and university assessments in the 1980s and 1990s and was never assessed on creative writing in the exam hall, it has always seemed to me to be strange that 11-year-olds were asked to be creative under the time pressure of Sats. I think a move to teacher assessment, then, would undoubtedly be a good thing. It could be argued that this change alone, in promoting a better assessment experience for many children, will mean the Bew review will have been worthwhile, despite some of its more fundamental findings being so flawed.

The report does, however, mention that the teacher assessment results are to be subject to external moderation. This is unavoidable, in a system which is using the scores generated to hold schools to account. Ministers, I am guessing, will want to ensure that the moderation is robust, as clearly there will be an incentive for schools to push up scores if they were under pressure over pupil achievement through, for example, the floor standards. The great danger, again, would be that the government decided that the need to use the results to judge schools is seen to be more important than providing the right assessment experience for pupils – that conflict of purposes again –  and therefore moved not to accept this recommendation to move towards teacher assessment. I have, though, no evidence that this is going to happen and hope it will find favour.

Summing up, Bew’s detailed changes do stand to make some difference. But I would suggest that the arguments over the system’s underlying dysfunctionality – or not – are not going to go away. It is a shame that this report did not take more seriously, in reaching its verdict in this final report, the detail and nature of some of the concerns.

*The report does briefly the merits of using tests to moderate a mainly teacher assessment system, concluding that this would not be feasible as tests and teacher assessment are not the same and thus, I think is the implication, it would be wrong to view the test as providing “true” validation of each teacher assessment verdict. I would not disagree with that as an argument, but I do not think it invalidates the Cambridge Assessment/IPPR model, since the “testlets” in this case are not meant to provide a judgement on the accuracy of teacher assessment in the case of every pupil, but merely to provide a more general check that a school has not inflated its teacher assessment judgements.

Friday, June 17th, 2011

A study apparently demonstrating the benefits of academy status seems to have been highly influential in recent weeks.

The research, by academics at the London School of Economics, was published in April. It has been picked up not only by Blairite commentators who backed the original academies policy, but now by the Department for Education in its push to encourage all schools to become academies.

I would also hazard a guess that it was in the mind of the Today programme presenter Sarah Montague when she asked a sceptical head teacher yesterday morning to accept the statement that academies improve schools’ results.

The research, by Stephen Machin and James Vernoit of the London School of Economics, produced some conclusions which look very positive for academies. As the Financial Times reported when the research was published, the study found that “turning a school into an academy improves its performance – and that of neighbouring schools”. The study was based on an analysis of pupil-by-pupil results of schools turned into academies under Labour, in the years 2002-9, when most of the institutions converting had low GCSE results. It includes a caveat that it does not relate to academies which have converted since the coalition came to power.

Having looked at this research in detail now, I am very impressed with a number of aspects of its methodology. Specifically, it performs statistical checks on institutional results which seem far more robust than similar exercises which have been carried out in widely-cited analyses of the academies policy in the past.

However, there is a gap in this research: any qualitative investigation into how academies opened under Labour have managed to produce their apparently impressive statistics.

This is an obvious question to ask: though academies’ benefits are often cited in broad-brush, quasi-ideological terms (such as allowing schools to break away from LA influence, encouraging innovation through a sponsor, or just simply promoting an often undefined quality called autonomy), why in detail would simply changing the structure of a school’s governance make a difference? What precisely have academies done to drive these results improvements? If they have greater independence, how have they used it and what has been the connection with results?

And once you look into that, as this blog and other research by the Civitas think tank has done, you start to have doubts over whether this policy is quite the panacea that is now widely being claimed.

OK, first the impressive bits, then. Well, for me if you want to know whether schools can improve their results by being turned into academies, and you want your research to have any claim on credibility, you have to do at least two things, neither of which seem to have loomed large in claims made about academy results in the past.

First, you have to compare like with like. Over the past few years, governments have looked at the GCSE (or equivalent, of which more below) results of academies, and compared them to those of the schools these academies replaced. On average, they have tended to find academy results improving, compared to those achieved in their predecessor schools, at least on the headline published figures, at a faster rate than those of the predecessor schools. Therefore, the argument goes, here is evidence that the academies policy is a success.

There are a couple of serious objections to any conclusions based on these calculations, though, including the following. What if the pupil clientele changed from the time before the school was an academy to now? The schools converting to academy status under Labour generally tended to have relatively large numbers of disadvantaged pupils. If the replacement of such schools by academies tended to draw in pupils from slightly less disadvantaged backgrounds – drawn, perhaps by the huge extra investment in new buildings that went with academies under Labour – with better results from their primary schools, this would be to the advantage of the academy but might mean that, when results rose, it was more to do with changing pupil intakes than anything the academy had done itself.

The Machin and Vernoit research tackles this issue by looking at the results achieved in key stage 2 tests by pupils who went on to attend schools which were to go on to become academies during the period under study, and those of children who joined the schools after they had become academies.

And the study finds that the pupil intake of academies did indeed “improve”. In other words, the academies under study were taking in pupils with better key stage 2 results than had been achieved by pupils entering the schools the academies replaced.

But here is the impressive bit: the researchers found that even after taking this pupil intake factor into account, the results achieved in the academies were better than achieved by a control group of schools.

The second impressive aspect about the study was that it sought to take into account the effect on neighbouring schools. This has always seemed to me to be important, since the success or failure of a policy should not be judged in terms only of the effect on an individual institution but in terms of its impact on an entire area: if an academy – which under Labour usually came with new buildings worth eight figure sums – succeeded only by drawing in more “educable” pupils from neighbouring schools, while those around it suffered and their results declined, this would raise questions about the policy.

But the Machin/Vernoit research looked at this issue, too. It found that neighbouring schools did suffer (to put it crudely) from the introduction of an academy nearby, in that the average achievement level of the pupils they recruited in year seven, as measured by their primary test results, fell.  In other words, some of the higher-achieving pupils moved, at the end of primary school, to the academy wheras in previous years they might have attended its neighbouring school. However, despite their intake getting “tougher” in this way, the results in these neighbouring schools at GCSE also improved. The paper suggests that this was probably the result of greater competition from an academy nearby spurring improvement, on the main results metrics, by the neighbouring schools.

Ok, that’s the good news. Here I come to my beef with this study. And I should say first that I am not trying to hit academies over the head for the sake of it with observations around strategies some schools might use to boost results. (The other day I met, as it happens, the principal of an academy with a very tough, non-selective intake in an area with grammar schools now under pressure from the Government’s new GCSE floor targets, and thought what a challenging, important job that must be.) But neither do I think that we should just abandon detailed scrutiny of whether academies are quite the answer to all educational problems that they are being made out to be, and what their results really tell us.

So back to the research. The trouble is, for all the statistical expertise and checking that has gone into this study, it is still based on the assumption that you can use a set of exam results formulae – on one or two performance indicators – to attempt to answer definitively the question as to whether these schools are actually providing a better education than their predecessors. In other words: the implicit view is that this question can be answered entirely statistically, without any reference to any qualitative understanding of what has actually happened to make these schools “better”.

Yet there are some fairly big alternative explanations. The obvious one is that academies have simply been more results-focused, in the main, than other types of school and thus they have sought to do whatever it takes to boost grades on the Government’s published indicators. That means that while the central indicators have improved, other indications – statistical or otherwise – might give cause for concern. So while the stats improved, actually if you tried to get a wider sense of what might be felt to matter in education, you would get a different picture. Academies might have, to put it more crudely as a hypothesis, paid more attention to gaming the results indicator system than other schools.

You could say it is unfair to single out academies in this way, and for newcomers to this blog, this might sound hyper-cynical. But, as I’ve written before, academies under Labour seem to me to have been under more pressure to raise results than other schools. Most of these schools were specifically created to address the claimed underperformance of a predecessor school. They came, often, with tens of millions of pounds of extra funding for new buildings. Their results were subject to extra scrutiny in the media, not just at the school level but at the level of the national politicians overseeing the academies policy, whose reputations were staked on headline scores improving. They might – though I am guessing here – also often come with a business mentality, reinforced by their sponsor, which incentivised senior leaders to get results up, come what may, through bonuses linked to GCSE exam performance. It would be surprising, then, if one or all of these factors did not produce a very strong focus on those headline measures.

So, how to check whether any other explanations lie behind those improvements cited in the study than just a general sense that education has improved in the academies under investigation?

Well, I have to confess here, that I have no killer line, or proof that this study is wrong in its conclusions. But I do think we should be wary about them. I want to come at this first statistically, and then anecdotally.

First, on the statistics, another impressive aspect of this research is that it does attempt to address, through the data of course, the most obvious way in which results could have been boosted artificially, if you like. This is through the use of non-GCSE qualifications.

Under the system in operation in recent years, other courses are counted as “equivalent” to GCSEs, for league table and results purposes. This is the case for the main measure used in this study: the proportion of pupils in each school achieving five A*-C grades at GCSE or vocational equivalent, including maths and English. Yet the fact that some of the GCSE-equivalent courses have been given high weightings in the results formulae – worth up to four GCSEs – and have high pass rates means that they can have a heavy influence on the overall published results. Schools encouraging high numbers of pupils to take these courses – whether they are doing so because of their own need to boost results, because of students’ needs or a bit of both – are therefore likely to get a results improvement out of doing so. Might not academies, then, under greater pressure to produce results gains, simply be turning to these courses to a greater degree than other schools?

So, back to the research. I was surprised to find that not only did Machin and Vernoit address this possible alternative explanation for the better results of academies, but that, when they did so, they found that it did not explain the results improvements academies seemed to show. In other words, the use of non-GCSE “equivalent” qualifications did not explain the relative success of academies, they suggest. The success, then, stood even after taking into account this possible alternative explanation.

The way they calculated this was fairly straightforward: simply to perform their calculations using GCSE qualifications alone as the measure of success in each school, rather than GCSEs “or equivalent”.

This, they say, represents their check on this idea – that I refer to above – “that the performance improvements [in academies] are largely driven by performance improvements in unconventional subjects”.

So, they conclude that putting pupils on “unconventional” GCSE-equivalent courses does not explain the academies’ results success. I should say, here, that I lack both the professional statistical expertise of these researchers or the time they no doubt spent on their study. But I would say that it is a slightly odd conclusion, given some other things we know about academy results, as revealed in more recent data sets.

First, I have performed a very crude version of a similar type of test to the one they used in their study, simply by looking at the latest published GCSE results of academies (all of them academies set up under Labour, and therefore the group from which the LSE study schools were taken) with “equivalents” and without. I have then compared these figures to those of non-academy schools.

I did this using Department for Education spreadsheets, adding up the number of pupils in academies in 2010 who achieved five A*-Cs including English and maths in GCSE or vocational equivalent, and comparing that to the total number of pupils in the academies they attended. The same calculation was performed to total up the number of pupils in academies achieving five or more GCSE A*-Cs when these were not allowed to include “equivalents”.

The figure for academy results – the proportion of pupils achieving five or more A*-Cs including English and maths with vocational equivalent, which was the main published measure used in league tables under Labour and is continuing to be the main target for schools under the coalition –  comes out at 43.3 per cent. Without them, it drops to 33.0 per cent, a drop of 10.3 percentage points.

Now, a similar comparison for non-academy schools reveals a far smaller gap. With equivalents, non-academies end up on a figure of 57.0 per cent. Without equivalents, they finish on 52.5 per cent. This is a gap of 4.5 percentage points.

So, on the 2010 figures, “GCSE-equivalent” courses have contributed far more to academies’ headline results than they have at non-academy schools.

Second, there is evidence from the Government’s much-debated new English Baccalaureate measure. This found, as I blogged about here, that nearly a third of academies with results to report had a score of zero per cent on the English Bacc, which records the proportion of pupils in each school with A*-Cs in English, maths, two sciences, a language and history/geography. Furthermore, the proportion of academies with that zero score on the EBacc was twice as high as it was with a comparison group of schools with similar intakes.

This data would suggest, then, that if academies were improving their results, they were not doing it exclusively on the narrowly “conventional” subjects that Michael Gove has chosen to highlight through the EBacc. Yet the LSE study says its figures do not show the improved results at academies are the product of gains in “unconventional” subjects. So, to repeat, it is strange how this evidence contrasts with the LSE research.

Other than the GCSE “equivalents” move, there are other strategies which can be used to boost school performance if schools of any kind are particularly desperate to see their statistics improve. These include entering pupils multiple times for GCSEs in English and maths in particular, with schools knowing that these are crucial to their published rates. The Advisory Committee on Mathematics Education documented this practice in relation to maths last month, pointing out that sometimes pupils would be removed from the subject by their school if they achieved a C grade earlier than the end of their course, to give them time to focus on other subjects important to the school’s results, even though the pupil might be chasing a grade higher than a C in maths (not important to the school’s published indicator). I have no evidence that this has happened to any greater degree in academies though, as I say, I think the pressures on most of them to improve results have been great. But any study should be aware that headline results indicators will often not present the whole picture of what has been going on in schools.

My final detailed response to the study is anecdotal. And here, I just want to refer back to my original blog on academies’ EBacc results, a couple of months ago, for evidence.

This made several points in relation to studies and anecdotes on the subject of history.

Academies were more likely to have fewer students studying history to GCSE than other types of school, according to research by the Historical Association. Academies were also more likely to have a two-year Key Stage 3, which gives pupils more time to prepare for GCSE but was a concern to the HA because it meant many were likely to lose one of the only three years they would study history at secondary school.

The report also quotes a teacher, from an academy, saying: “History is seen to be too academic! …Students who are predicted lower than a B are not allowed to study the course…We are also not allowed to run ‘entry level’ courses for students with specific needs, as that is not thought to be meeting the attainment targets for the academy.”

An Ofsted report on history teaching in primary and secondary schools, published earlier this year, also documented lower numbers taking history in academies. It found: “Entries for GCSE history from academies were significantly lower than for maintained schools overall.”

One online comment after a 2009 TES story documenting another academic report on the pressures facing history as schools sought to boost their results in league tables, ran as follows:

 “I used to work in an academy in London, and as I was leaving I had to rank every pupil in year 8 as an A, B or a C. A means that they could get an A or a B at GCSE. Therefore history appeared in their option forms. The B category were pupils who were borderline C/D. The C meant that they were predicted grades G to D. Neither categories B or C had history on their option forms! They were encouraged to take other less rigorous subjects.

“Even though I had known students previously predicted Ds and Es get outstanding results, who went on to do exceptionally well at A-level, and some even went on to do history at university.

“What was most upsetting was the case of one student, with a range of learning difficulties. He loved history, and orally he was phenomenal. He was put in category C, and was therefore being guided down a different pathway. He was devastated that he would not be able to take history in year 9-11. His mother rang the school, and explained that it was likely whatever course he was entered into, he would be unlikely to either pass or do very well in, so why couldn’t he at least take a subject he enjoyed?

“The plea fell on deaf ears and the boy was placed in some random BTEC or GNVQ course taught by some bland paper pushing academy drone who was being shipped in to ‘sort’ the school out of failing pupils and failing teachers.”

If you look back to my earlier blog, you will find reference to the parent of a pupil at a school taken over by the Harris chain of academies, who told me (and the local paper) that her daughter had been forced to take a BTEC sports course (worth two GCSEs to the school), at the expense of French GCSE, despite her daughter having no interest in sport. This was a clear case, said the parent, of the needs of the school to boost its published results taking precedence over those of her daughter.

So in response to this LSE study, I have put forward some statistics that run contrary to one of its more important findings, and also some anecdotes.

Not much, you might think. But there is a bigger point here: there should be more to the evaluation of a policy than simple results statistics, however clever the methodology and however robust the statistical cross-checks, especially in a complex system such as secondary schools results calculations which offer plenty of opportunities for schools to take tactical decisions to boost results. This runs the risk of following less-than-ideal behaviour, from a pupil’s point of view, within particular subjects.

And is all that matters the number that appears at the end of the educative process? Or do we care about what happens along the way, and how the numbers are generated? If particular subjects have been affected in the drive for higher results, should an influential study like this not be investigating and having something to say on this? Or should such a perspective just be ignored: the idea is that we lay down the statistical rules for success, check whether the statistics have been raised and that, apart from some clever checking of data, is pretty much it?

To sum up, how do we know that academies under Labour did not simply pursue a more relentlessly focused version of “Education by Numbers”?

I think if researchers are going to make claims which are going to be used, whatever the caveats in the original research, by others to say categorically that a policy “works” and by implication that the education on offer in academies is better in a general sense than in other schools, they are going to have to be prepared to dig a little deeper – and not just statistically – into what has been going on behind the figures. Economists who do not do this will never be able to see or pronounce on the whole picture, I believe. Their research will therefore always be incomplete.

So it is a shame that statistics are simply being held up as conclusive evidence, one way or the other. This really is not, I think, for all the complicated formulae and technical expertise on display in this paper, a very sophisticated way of understanding what has really been going on in our schools.

Right, I haven’t blogged for a while, but thought I’d just post here an extract from a speech I made just after Christmas about what can be read into English Sats results for 11-year-olds.

I’ve been prompted to do this after reading, over the last two days, the Evening Standard’s coverage of what it claims is a literacy crisis in London.

Yesterday, part of its front-page coverage talked about one in four children being “practically illiterate”, seemingly based on the proportion of pupils achieving level 3 or below in English Sats.

Today, it highlighted the number of pupils “with a reading age of seven”, based I think on the numbers achieving level two or below. (The normal level said by the Government to be the expected reading standard of a seven-year-old).

I don’t think test statistics can support the interpretation being put upon them. It may be that we have a literacy problem in the capital, or in the country as a whole. But the test data used as a good part of the news hook for the coverage don’t do a good job about telling us the nature of the problem. It’s probably not helped in that news coverage often fails to put the numbers in perspective. Ideally, it would give  us unsensationalised  information on whether the statistics are on an upwards, downwards or static trend, and what information we have about how this country compares to others, but this tends not to happen.

Anyway, here’s the extract of that speech, prompted in part by similar coverage on the Today programme before Christmas.

I want to talk about the over-interpretation of test results: they don’t tell us nearly as much as we might think they do. Perhaps just as importantly, we don’t use the data, in our public debate around education, really to understand what is going on in schools or with pupils’ learning, and in that sense we are letting children down because we should be using assessment information in a far more sophisticated way, I think. And bear with me, as I am going to have to go into a bit of detail here.

So, I’ll just start with a question: What is the definition of the level of understanding expected of an 11-year-old in reading? How is this defined by the government, by the media, and thus by people nationwide in the debate about this vitally important subject?

What does it mean, within the detail of what children have to achieve, for them to perform at that level?

Well, in 2010, it came down to this: the ability of a child to score 18 marks out of 50 in a one-off 45 minute test, taken by most pupils as they come to the end of their primary school years.

That is the number of marks needed to secure level four in reading, the Government expectation, and represents the entire official judgement on that pupil’s ability in reading over the past four years.

If a child scored 30 marks out of 50 in last year’s tests, they would have achieved a level five in reading, which statistically and according to the interpretation we are expected to put on these data, is the level of proficiency expected of a 14-year-old. If they scored between 11 and 17 marks, they would be at level 3.

That is it. Nothing else counts in official estimations of what it means to be able to read. Our entire primary education system –at least so far as reading is concerned, hinges around the proportion of pupils achieving these expectations and pass marks, which are very closely bunched, in a one-off test one day in May.

I highlight the case of reading because it came up in coverage shortly before Christmas by the Today programme. It led its broadcasts one morning with claims that “thousands of boys start secondary school only able to reach the reading standards of seven-year-olds or below”.

This was based on a technically accurate interpretation of figures generated by national test data, but which led me to question why people are putting such huge weight on figures which, if you step back from this for a second and think about the detail of what these data mean, cannot support this interpretation.

Today had obtained figures – released every year – which showed that in 2010, 10 per cent of pupils obtained below a level three in the reading test. This means that they either scored 10 marks out of 50 – 20 per cent – or below on the tests, or did not even take them.

The logic of Today’s argument was this. Pupils scoring below level three in the reading test have scored level two at best. Level two is the performance technically expected of a seven-year-old in the tests pupils take at this age. So the 10 per cent of boys failing to achieve level three are performing at the level expected of a seven-year-old.

This finding, which suggests a serious problem – implying, I would venture, to many listeners, that many boys are wasting years at school making no progress – is viewed as a national scandal; it, at least, is very serious for these boys.

Consider, though, more detail on how these data are generated. A child could fail to achieve a level three with 10 marks out of 50. But with another eight marks – 18 out of 50, or 36 per cent, these boys would have achieved a level four, in line with government expectations of an 11-year-old.

The difference between having the reading age of a seven-year-old, then, around which national debate centred, and that of an 11-year-old turns out to be eight marks on one 50-mark test. Put it another way, a seven-year-old who took this reading test could have scored 10 marks and be said to be performing in line with expectations for their age.

If they took a similar test four years later as an 11-year-old and scored 18 marks, then they would be deemed to be doing as well as expected for an 11-year-old. Thus, four years’ progress in reading could be said to come down to the ability to improve by two marks a year in a 50 mark test.

I went into some detail in this example to illustrate the difficulties we have in the way test data are being used. Believe me, I am not trying to minimise this problem: if a large number of boys really cannot read, it is a serious national issue.

The trouble is, I don’t think test data, and in addition to a certain extent the way they are reported, are helping us understand the nature of that problem and thus to do something about it.

Consider again the interpretation of the figures around which the Today programme that morning revolved, including an interview with Michael Gove, the Education Secretary.

The lead headline on the programme’s website read “Gove: 11-year-old illiteracy ‘unacceptable’”. John Humphrys, the presenter, also used the term “illiteracy”.

But, in fact, the test data actually tell us nothing about “illiteracy”. They don’t tell us whether the number of boys quoted in the programme actually are “illiterate” – can’t read or decode text – or whether their problems are different from that.

Strictly, they tell us only that a number of boys either couldn’t score a certain number of marks in a one-off reading comprehension test (further scrutiny of the government data shows 4 per cent were entered but didn’t achieve level 3), or their teacher did not enter them for such a test because they believed they would not pass (5 per cent), or that they simply missed the test(1 per cent).

We don’t know, then, from the test data, whether the problem with these children – is a) genuine inability to decode text – although the fact that nearly half of them scored some marks on this test would suggest this was not the issue for these children b)problems with reading for comprehension (ie they can actually read the words, but they don’t really understand either what they mean or what the question is asking) or c) a failure to cope with the format of being tested.

There is, of course, another explanation: that these children scored below their “true” level of understanding through having an “off-day” or just being unlucky: there will always be measurement uncertainty and inaccuracy in a one-off test.

If we don’t know what these figures actually mean, how can we do anything to help children to genuinely improve? Is what the nation needs a greater emphasis on helping children with decoding, as is suggested through the introduction of a new phonics test, or more work on comprehension, for example? The test data give us no answer.

It also was not reported – and generally isn’t – that we do know that substantial numbers of pupils in this category of failing to reach level three have special educational needs: by my calculations from government data, seven in 10 children who failed to reach level 3 in English in 2010 were classed as having a special need. Nearly 10 per cent of those failing to reach level three are classed as autistic; a further seven per cent have specific learning difficulties; 10 per cent have communications needs and a further 10 per cent have behavioural, emotional and social difficulties. None of these figures were presented in the Today programme reporting.

Neither, by the way, was any international context given: boys’ reading is a problem around the world, as last month’s Organisation for Economic Co-operation and Development PISA study showed. It included the following quote: “Across OECD countries, 24 per cent of boys perform below level 2[at the bottom of six levels of the PISA reading tests], compared to 12 per cent of girls. Policy makers in many countries are already concerned about the large percentage of boys who lack basic reading skills.”

The fact that these test data – and sometimes the reporting around them – allow us only a very superficial, decontextualised understanding means that we really are letting down pupils and the education system as a whole.

We could do so much better. For not only does the accountability system which centres on pushing schools to raise these test numbers – through league tables, targets, Ofsted inspections and the rest of the centrally-created performance apparatus – encourage schools to spend months trying to drill pupils to achieve, if you look in detail at what the figures mean, a few extra marks on one-off English and maths tests. We also lack the understanding – in terms of the national data that these test figures generate – both to help these pupils do better – ie to work out what it is they can and cannot do – and to help the system as a whole to improve.

Today presented the problem as a hugely serious issue for the nation. But we are not taking it seriously at all if this is the level of analysis being offered.

If Sats are the height of our ambition in assessment – and there are still signs, under the new government that this is what pupil progress will revolve around – we really have a problem, then. We need to look at the use of much more sophisticated and useful measures of children’s understanding, both from the point of view of helping the individual child improve, and from the point of view of getting a much better understanding of what is really happening nationally.

 The rest of this speech, to a meeting held in Parliament in January to launch a joint Association of Teachers and Lecturers/National Association of Head Teachers/National Union of Teachers pamphlet on assessment and accountability to which I contributed, went on to talk about problems of the washback effect on teaching of high-stakes test-based accountability, with which readers of this blog will be familiar.

Tuesday, 5^th April, 2011

This is just a brief blog to acknowledge the publication today of the interim report by Lord Bew’s inquiry into Key Stage 2 assessment.

I have to say, I have been impressed with the amount of evidence garnered by this review. More than 4,000 people responded to the online consultation, and the review also heard from 50 people in person. There is a lot of research referenced. I gave evidence myself, setting out concerns raised in Education by Numbers, and discussing with the panel the strengths and weaknesses of the current system.

On a snap judgement, Bew seems to me to be taking a more thorough look at this subject than any other government inquiry since I’ve been covering this ever-contentious field. There is, for example, more evidence on display in this report than that discussed in the last government’s “expert review” on assessment, which concluded in 2009 and led to the scrapping of Key Stage 2 science tests.

As I said to the latest inquiry, it is still amazing, I think, that more than 20 years into this system of national assessment there has been no comprehensive observational investigation into the extent and nature of test preparation and other side-effects, if extensive test preparation is to be seen as a side-effect, of results pressures in schools.

 In the absence of this detailed study, perhaps this latest investigation – brought about, don’t forget, following industrial action last year against the KS2 tests by the National Association of Head Teachers – will be the most thorough we are going to get.

That said, I worry that its remit – and perhaps the tight economic situation – will limit its scope to make radical change. OECD evidence stating, says this interim report in a reference to the remit handed to it by Michael Gove, that “external accountability is a key driver of improvement in education” needs to be scrutinised carefully, too. I’m not sure that the OECD does have quite the evidence that high-stakes test-based accountability of the English/American sort has driven improvement, as measured by OECD test data. It may be, of course, that the remit is not specifying that England sticks with test-based accountability similar to the current sort, but I wonder if that is the hint. Anyway, the OECD evidence does deserve closer study, which I will get on to in the coming days/weeks.

There is also evidence cited in Bew which is highly relevant to this debate which is new to me, and hopefully will form the basis for future blogs on this site. So watch this space.

 If you’ve been following me on twitter, you’ll also have noted a string of tweets from me when the report was published today. I’ve just cut and pasted them below, essentially because they serve as my instant judgement “edited highlights” of the report. They are meant to be read from the bottom up.

Thanks.

Warwick (@warwickmansell on twitter if you want to follow me and aren’t already doing so)

Russell Hobby, gen sec of NAHT: heads looking forward to a “radical shake-up” of assess system. Criticism of current system “overwhelming”.

Final report expected June at the earliest, I think.  

Phew! That’s enough on that, for now. Impressed with range of evidence being used.Recomms will be interesting, esp given tight remit,economy

That last tweet reflects what was always a quietly powerful criticism of current system, I think. (And was cited by M Gove in 2009)

Bew: “feedback suggests secondary schools make limited use of stat test data to support transition”.Many 2ndary heads concerned re over-prep  

Bew: some discussion among some assessment orgs of stimulating a “market” in testing.

Bew: Headteachers involved in the [now abandoned] pilot of single level tests said they had gone positively.

Bew: split views on abolition of science tests; some say science now more fun; but general view that science teaching had “lost impetus”

…but writing tests generated most concern. Some 43 per cent of online consultants said they were “inadequate”; 33pc “not v effective”.  

Bew: “Widespread recognition” that tests themselves are well-developed.

Bew: some junior school heads concerned they lose out under current system, as some “infant schools inflate their KS1 assessments”

…but some research suggests replacing tests with TA might disadvantage poorer pupils, some ethnic groups  

Blimey: Centre for Policy Studies says current tests are “biased towards families from middle class homes”.

Bew: testing regime can disadvantage both high- and low-attaining pupils.

 Bew: feedback suggests progress and achievements of children with special needs not “appropriately recognised, celebrated” by current system

Bew: Cambridge Primary Review team argue that current national test data provide little useful information on national education performance

Bew: 59 pc of respondents said more weight should be placed on teacher assessment

Bew: 2009 DCSF survey found 65pc of parents valued their children taking KS2 tests.  

Bew: 2008 survey by the National Confederation of Parent Teacher Associations found 78 pc of parents placed high/med value on external tests

Bew: 15 per cent of online respondents criticised impact of test data on Ofsted judgements, “some expressing deep concern”.

Lord Sutherland, who conducted review of test marking shambles in 2008, among those criticising current system.  

Bew: ASCL believes league tables are “driving the whole education system”, leading to assessment for own sake, rather than re pupils’ needs

Bew: many heads say they “have to” teach to the test, despite clear evidence it is a poor strategy. [Unsurprising, when jobs are on line]

Bew: most respondents said they supported testing; way the data get used is problem.  

Bew: 62 per cent of online consultation respondents had concerns about the way test data are used.  

Bew cites OECD reseach saying “high-stakes accountability”..But accountability systems work very differently in different countries, I think

 Bew: 50 per cent of respondents wanted league tables removed.

Interesting..Bew review says one of its key tasks will be to define purposes of statutory assessment;system then designed to fit these purps

Bew: almost all respondents have questioned the purposes of statutory assessment.

Bew: most of the evidence submissions (61 per cent) were from primary heads; 23 per cent from primary teachers; only 4pc from parents.  

Bew: many contributors recognise positive features of current system, eg “impact on driving up achievement, progress”, which we shd protect.

Bew: “significant concerns” about focus on children on borderline of national test levels.

Bew: 4,000 responses to online consultation. Many schools feeling that they must drill children for tests is “deeply worrying”.

Bew: It is “increasingly clear that there is not a single set of solutions which can command universal support.”  

Bew: “Change is clearly needed” but acknowledges “complexity in the challenge we face”.

 Monday, March 28^th, 2011

Last autumn, Michael Gove appeared on the BBC’s Question Time and launched a passionate attack on what he claimed was a glaring injustice within English education.

Children from disadvantaged backgrounds, he suggested, were being let down by a system which assumed they could not succeed in traditional academic subjects.

He said: “If you look at what happens in France, or in Holland, or in Canada, or in Singapore, or in Hong Kong, or in any of the countries which have got education systems many of which are much better than our own, they expect children at the age of 16 to have a rounded education.

“[This] means they are fluent in their own language…[and expected to] master the sciences, to study a humanities subject like history or geography, which build human sympathy. That’s the rounded education they expect.

“And the problem we have had in this country, as an historical problem, is we have automatically assumed an academic education is only for a minority: only 25-30 per cent of people can succeed. 

“Well, that is rubbish.”

“All of us are facing an educational challenge in this country,” he continued. “How can we ensure that we end the patronising twaddle of the last 30 years that assumes that just because kids come from working class backgrounds, they cannot succeed in academic subjects?

“With my background, I am determined to ensure that people have that chance. And when people say ‘oh, you are demoralising children because they cannot succeed’, what I hear is the next generation being written off because we do not have high aspirations for them.

“One of the reasons I am in politics is to make sure that we transform our education system so that kids who have been written off in the past at last have the chance to succeed.”

Well, all this was greeted enthusiastically by some commentators.

But is this passion real?

If so, you have to wonder why Mr Gove has not taken a much closer interest in what has been going on in his favourite type of school: academies. His lack of interest might suggest his emotion is synthetic. Or to be more charitable, when a seemingly heartfelt desire to do what he thinks is the best thing by working class pupils runs up against the demands of political ideology, ideology wins.

Before going into the detail on academy results, I should state something now.

It is this: I am a “what works” type of person.  I don’t like ideology, or the idea that something should be implemented because it fits a theoretical schema or model of how things ought to run best. This means I’m not one to dismiss any type of organisation of schooling out of hand. I am, as might be guessed from the length of some of the blogs on this site, a details person.

Any consideration of academies, then, should be carried out on the basis of as full as possible an understanding of the effects of these new schools across a local area. Academies are sometimes sold on the basis that their governance supports innovation, and that they have brought dynamism to England’s system. Evidence on this should be weighed against that relating to other arguments, including the financial implications of these new school arrangements, the impact of academy freedoms on equity, their effect on teacher recruitment and retention, their effect on local admissions and the interaction with accountability to local people. Above all, we should try to get an understanding of the detail of what has happened in academies and other schools in their localities.

The trouble is that we never get this fair reckoning, in my experience, because politicians of both this and the former government who shape the debate are so committed to the policy, as a structure of running schools which they prefer to the traditional model of state education, that they don’t present evidence even-handedly. GCSE results press releases have consistently highlighted academies’ results as better than those of non-academy schools, based on faster average improvements in academies than other schools on the main GCSE performance measures, when there are other ways of looking at what has been going on, and even though basic questions such as whether the pupil make-up of each academy has changed or not compared to its predecessor school(s) are not addressed in the statistics. Remarkably, this cheerleading presentation of academy results has continued even after the publication of results in Michael Gove’s new “English Baccalaureate” measure.

In January, league table results which saw Mr Gove introducing that new performance indicator – the English Baccalaureate – were revealing, although not surprising, in what they documented about the statistics of academies.

These schools, usually set up through a contract agreed between a sponsor and central government, had long been said by Mr Gove and his Labour predecessors to be improving their headline GCSE results at well above national average rates.

This was based on the main figures published under Labour: the proportion of their pupils achieving five A*-Cs including English and maths, in GCSEs or vocational equivalents.

But the “Baccalaureate” figures, which ranked schools on the proportion of their pupils achieving good GCSEs in not just English and maths, but also two sciences; history or geography; and a language, painted a very different picture. Many academies were right at the bottom of the English Baccalaureate league tables, with nearly a third of those with results to report recording zero per cent of their pupils achieving this new benchmark.

Some three quarters of the academies with results to publish had five per cent or fewer – that is one pupil in 20 – achieving the EBacc, compared to a national average figure that had 16 per cent of the cohort achieving the new benchmark. At all but 24 of the 187 academies with results, performance on the EBacc was below 10 per cent.

The press release put out by Mr Gove’s department – which would be very worried about the situation in academies, you would expect, if it shared his concern about pupils missing out on a broadly academic education as he defines it – said nothing about these statistics.

Instead, it only mentioned academies in reference to the old measure, proclaiming that: “Academies continue to show improvements in getting five good GCSEs (or iGCSEs or equivalents) including English and mathematics at a faster rate of 7.8 percentage points compared to other schools, which improved by 4.5 percentage points.” Government comment on schools’ results in the EBacc focused, then, on the 16 per cent figure for schools as a whole, which was seen as low and might underscore, in the public mind, a view that radical change – including academy status – was needed. In the press release, this seemed to be underlined by the inclusion of comments from two academy managers without, again, reference to academies’ EBacc results.

Many have said that, and might argue here, that the retrospective EBacc measure is unfair on schools. But it seems to me that someone introducing this measure simply with a desire to highlight the performance of schools in the subjects contained within it, without prejudice towards any particular type of school, would use the EBacc results to include a least a heavy element of caveat in what was being said about academy results overall. Yet the spin – or a desire to present academies as always better than other state schools – seemed to be taking over.

In recent weeks, the TES has been following these results up with stories, first, that only six per cent of pupils in schools run by the government’s three favourite academy companies achieved the English baccalaureate this year, compared to the national average of 16 per cent.

The government, or backers of academies, including those teaching in them, might respond here by saying that this comparison is unfair. Labour’s academies, which were the only ones open and able to provide the GCSE figures on which these statistics are based, were set up mostly in disadvantaged areas, with challenging intakes, so it would not be right to try to compare them to the national average, which will include schools with more middle-class  pupils.

But that, as Mr Gove’s comments to Question Time should make clear, is not a defence open to him if he wants to say what is going on in academies is not significant. For he has argued that we need to have high aspirations for good academic achievement for children from disadvantaged backgrounds. All schools should be getting good results according to his new benchmark, and the results of academies were especially concerning, you would expect him to say if he was being consistent.

Even more damningly, the TES also produced statistics claiming that academies actually fared worse, not just than the national average English Baccalaureate figure, but when compared to non-academy schools with similar intakes.

“No pupils gained the English Baccalaureate in 31 per cent of the academies that entered pupils for GCSEs and their ‘equivalents’ last year,” it said.

“But only 17 per cent of non-academy comprehensives and secondary moderns with the same proportions of pupils on free school meals with special educational needs completely failed to score on the EBac.” [My italics].

Going slightly further up the league tables, the TES found that 73 per cent of academies achieved less than five per cent on the EBacc measure, compared to only 55 per cent of non-academy comprehensives and secondary moderns with comparable numbers of special educational needs and free school meals.

Not once, though, as far as I am aware, has Mr Gove made any comment about this disparity. It begs the obvious question: is something peculiar going on in academies which is producing these numbers? Mr Gove doesn’t appear to have  looked very hard for an answer.

Indeed, in a letter to academy principals last month, after the EBacc results had been made public, Mr Gove began: “The Academy programme has already proved itself an exciting, powerful and dynamic force for higher standards in our schools.”

He added: “Sponsorship has been key to transforming some of our most challenging schools bringing added drive, vision, resources and expertise, to create a culture of higher aspiration.”

Although the letter talked about the importance of getting all schools above a new floor standard” of 35 per cent or more of pupils achieving five or more GCSE A*-Cs including English and maths, there was no mention anywhere within it about academies’ results in the EBacc.

Now, if Mr Gove were really concerned to look without prejudice at the effects of government policy in particular types of schools, he might also have wanted to consider a report produced in 2009 by the Historical Association, which contained some very interesting statistics, relevant to academies and other schools, on the exposure of pupils to one academic subject Mr Gove has been very concerned to emphasise.

This report, based on a survey of 644 schools including 23 academies, found that only 59 per cent of academies taught history as a discrete subject in year seven, which was the lowest of any of the four categories of schools. (The others were non-academy comprehensives, grammars and independents). Some nine per cent of academies had a two year key stage 3 curriculum, allowing pupils to drop history at 13, compared to six per cent in comprehensives, three per cent in grammar schools and one per cent in the private sector.

Nearly 48 per cent of academies reported that year seven pupils spent an hour a week or less on history, compared to 30 per cent in comprehensives, 12  per cent in grammars and seven per cent in independents. There was a greater spread of teaching time in comprehensives, however, with 38 per cent likely to devote more than 90 minutes to history a week, a higher figure than for grammar school and fee-charging schools. “The academies remain the least likely to give such generous allocations,” said the report. “Less than 20 per cent of them thought it worth investing more than 90 minutes a week in the subject.”

Academies also seemed to be reducing the time allocated to the subject faster than other types of schools. More than half reported that the time devoted to it in year seven had dropped since the previous year, compared to one third of comprehensives, while time reductions in year 8 and 9 were also most widely reported in academies (35 per cent, compared to 20 per cent for comprehensives).

In terms of GCSE history numbers, academies were the only type of institution where greater numbers reported a decrease in entries for the subject (33 per cent of academies, compared to 17 per cent of comprehensives) than an increase (19 per cent of academies, compared to 27 per cent of comprehensives).

The report also found extensive evidence, with no particular type of school mentioned, of history struggling for GCSE numbers in the face of competition from other subjects, with vocational qualifications “which in many cases lower-attaining students were being compelled to take” mentioned in a quarter of cases.

The report includes the following quotation, not mentioning what type of institution was involved. “Students have been deliberately denied an opportunity to study history by forcing them down vocational or academic pathways. GCSE students have also been taken off courses against their wishes to do BTEC qualifications in six months so that the school can boost its position in the league tables. This has happened to students who were otherwise on target for a C/B in history but were doing badly on their other optional subject.”

 It quotes a teacher, from an academy, saying: “History is seen to be too academic! Entrance to the course is based on Fischer Family Trust predictions, and students who are predicted lower than a B are not allowed to study the course…We are also not allowed to run ‘entry level’ GCSE courses for students with specific needs, as that is not thought to be meeting the attainment targets for the academy.”

Education by Numbers indeed, in both cases, and seemingly classic examples of the need of the institution to raise its statistics being put above individual student concerns, at least as these history teachers see it.

Ofsted’s report on history teaching in primary and secondary schools, published this month, also documented lower numbers taking history in academies. It found: “Entries for GCSE history from academies were significantly lower than for maintained schools overall,” at 20 per cent of students in academies compared to 30 per cent for non-academy state schools (and 48 per cent in fee-charging independent schools).

In 2009, the TES reported on a study by academics at the university of East Anglia and Southampton, which also found results pressures as a heavy influence on schools’ decisions over history.

The academics are quoted as saying, with no particular type of school identified: “Pupils’ interests were not necessarily put first. For the senior leadership team in some schools, the first priority was the school’s examination profile.”

Beneath the TES story, there was the following comment:

“I used to work in an academy in London, and as I was leaving I had to rank every pupil in year 8 as an A, B or a C. A means that they could get an A or a B at GCSE. Therefore history appeared in their option forms. The B category were pupils who were borderline C/D. The C meant that they were predicted grades G to D. Neither categories B or C had history on their option forms! They were encouraged to take other less rigorous subjects.

“Even though I had known students previously predicted Ds and Es get outstanding results, who went on to do exceptionally well at A-level, and some even went on to do history at university.

“What was most upsetting was the case of one student, with a range of learning difficulties. He loved history, and orally he was phenomenal. He was put in category C, and was therefore being guided down a different pathway. He was devastated that he would not be able to take history in year 9-11. His mother rang the school, and explained that it was likely whatever course he was entered into, he would be unlikely to either pass or do very well in, so why couldn’t he at least take a subject he enjoyed?

“The plea fell on deaf ears and the boy was placed in some random BTEC or GNVQ course taught by some bland paper pushing academy drone who was being shipped in to ‘sort’ the school out of failing pupils and failing teachers.”

The notion of pupils being forced into taking subjects for the good of the school’s statistics reminded me of a conversation I had in late 2009 with the parent of a child at a school which had just converted to be run by the Harris Federation of South London Schools.

I was following up on a story in the local paper on the anger of the mother, Moira Macdonald, that her daughter, studying at Harris Academy Purley, near Croydon, had been forced to take a sports BTEC worth two GCSEs.

The academy replaced Hailing Manor school, which was under pressure because its headline results were below the Labour government’s “floor targets”, at short notice in September 2009. In May of that year, Dan Moynihan, chief executive of the Harris Federation, wrote to parents saying that students would be “required” to do a BTEC in sport.

In the old school, French had been compulsory, but it became an option at the academy. The new academy’s options structure allowed pupils to take up to two optional GCSEs alongside English, maths, science, enterprise, religious studies and the BTEC sports course. The BTEC sports course (worth two GCSEs) would be taught in only three periods a week, said its options booklet, rather than the five the school was devoting to maths GCSE, which is worth one.

The parent, Moira Macdonald, told me her daughter had opted for geography and history and therefore had had to drop the French, even though she would have preferred not to rather than taking the sports BTEC because she had no interest in pursuing a career in sport.

Ms Macdonald said: “The academy is promising massively improved results and I am not surprised considering they are making soft subjects compulsory and dumping hard-earned GCSEs.

“The Harris Academy overrode the GCSE core subjects set for my daughter and her colleagues before the takeover, in order to improve their league table results.

“This is no way to educate kids – they need to be taught proper subjects and come away with proper qualifications.”

These quotations were featured in a report by the Civitas think tank in 2009, which tried to look behind the secrets of academies’ success. It asked the question which I think anyone should ask if confronted with statistics showing rapid improvements: how were they being achieved? It offers substantial evidence of what it says are some of the approaches within academies, of pupils being pushed towards “less challenging” [Civitas’s words] subjects and qualifications “to drive up headline results”. So this investigation, asking if there was a specific “academies effect” at play behind their generally improved headline results – ie searching for reasons behind it rather than the often-cited but too vague “sponsors’ ethos” claims - was available to Mr Gove but I know of no detailed reaction to it.

Now, I thought I’d say here what I think has been going on in academies.Perhaps it would be better to start with a question: if they have lower numbers of pupils taking academic subjects such as history, why is this? Well, I guess there are two responses.

The first is to say that, as mentioned above, the original academies set up under Labour tended to serve – though were not exclusively confined to – disadvantaged communities. All other things being equal, it could be argued that one might expect these schools to struggle to recruit pupils to the traditional academic subjects such as history and languages that Mr Gove now focuses on through the English Baccalaureate.

There is likely to be some truth in this. However, the TES figures suggest that not only do academies have lower results  on the EBacc measure when compared against the national average for all other schools  - driven partly, would be the assumption, by academies’ possibly lower take-up for EBacc subjects – but when compared to schools with similar intakes.

Further, as suggested above, this is not a defence open to Mr Gove if he is truly to be seen as a champion of academic education for the vast majority of pupils. If he really cared about that, he would be speaking out passionately against some of this practice.

To me, a second response suggests itself. It is this: the practice in academies could be seen as a kind of “Education by Numbers” squared, or “Education by Numbers” amplified. There are such large institutional forces on them to raise results on the published indicators that the kind of practices documented would be expected to have occurred, perhaps to a greater degree than elsewhere in the maintained sector, because of results pressures.

Of course, there is no evidence that these approaches, including  pushing pupils away from the academic subjects now included in the Ebacc towards those which have up to now carried high weight in league table calculations, were and are going on in all academies. But I do think there is likely to have been this general tendency, based on a few facts about academies, which run as follows.

Academies were usually set up, under Labour, in response to perceived problems of low attainment: the results of the schools they replaced were said not to be good enough. I have observed, for example, how raw exam statistics, in relation to some schools, were virtually the only evidence put forward as the rationale for the extensive restructuring and investment that comes with setting up an academy.

In this context, almost the raison d’etre of these new schools was to improve headline GCSE statistics. If they didn’t do so, one could ask why the change to academy status, which was often controversial and which had been backed with often tens of millions of pounds of investment in new buildings for individual schools, had come about. I suspect, also, though have never seen evidence other than a reference to the odd “performance related bonus” in a job adverts, that academy leaders have had performance pay tied to raising published exam numbers.

The published results are not just high-stakes locally for individual schools, of course. They are also important for academy chains, whose reputations – in a system which really does not look very hard for alternative evidence – rest on them.

And politically, at a national level, of course, the success or failure of the academies scheme was seen to be judged almost exclusively on whether one or at most two numbers rose: the central indicator of the proportion of pupils achieving five or more GCSEs or vocational equivalent at A*-C including English and maths, and arguably the old measure without the English and maths stipulation.

In this kind of atmosphere, academies will have been under even more pressure, I believe, to game the system in obvious ways – such as a very sharp focus on C/D borderline pupils and use of alternative qualifications – in order to deliver those results, against the context of them being demanded quickly from, often, very challenging pupil cohorts.*

But the impact on individual pupils in the chase for better results – in terms of denying them precisely the kind of curriculum Mr Gove claims now to want for all schools – can be large.

I should say here – and some will no doubt challenge this – that I don’t want to criticise non-GCSE qualifications. The idea that schools, pupils and parents should be free to choose the courses they think are right for the pupil, with a full understanding of the likely benefits to the individual in the long-run, is powerful. My concern is that our current system, including the lack of scrutiny of what has been going on beyond statistics which have placed a surely-too-high weight on some non-academic courses, has pushed schools to take decisions based on the worth of a course to the institution, rather than to the individual.

It was revealing, I think, that in a recent exchange I had on twitter on this subject with Sam Freedman, Mr Gove’s adviser, Sam predicted that academies’ results would improve quickly on the Ebacc indicator now that it has been introduced. But this only seemed to confirm, in my mind, that academies have been exceptionally focused on league table ranking metrics, ie on the results for the institution. Mr Freedman may suggest that this is OK, now, since this government has sorted things out so that the metrics are now better aligned with pupils’ interests. I think that is a very optimistic reading. It also implies a lack of interest in what has happened under the old system which, if you follow Mr Gove’s logic, has resulted in disadvantaged pupils being wrongly pushed towards courses which were not in their long-term interests.

In investigating school results and the impact of non-GCSE qualifications on league table rankings, I have been in contact for several years now with Roger Titcombe, the former head of a community comprehensive whose school eventually was turned into an academy.

Roger’s argument throughout was that he passionately believed in what he saw as an important strand of the comprehensive ideal. This was the right of all pupils – from whatever background – to pursue a broad liberal education, in which all would have access to a range of academic subjects.

He saw that as coming under threat from the academies movement, because these new schools were so desperate for better results, some would sacrifice that ideal by pushing children towards qualifications mainly because they would help the school’s data.

Not just through academies, but throughout the schools system, a new class divide was at risk of emerging, he thought, with those from better-off families concentrating on academic courses and the rest pushed towards non-GCSEs in their options. But he believed, and the evidence presented above would suggest, that there is an “academies effect”, which makes them particularly susceptible to the type of behaviour described here.

I think, actually, that Roger’s ideals in this respect are very similar to those put forward by Mr Gove. However, in the absence of any other explanation, it seems the Education Secretary’s desire never to be seen to criticise the actions of academies overwhelms his stated commitment to speaking out for the options of those from disadvantaged backgrounds. Ideology, then, in this new government, is king.

* Remember, also, that no school can have complete control over the results a child achieves after sitting at a desk to complete exam papers, so institutions under pressure seem to me to be particularly incentivised to go in for strategic approaches that afford them greater control.