Tuesday, November 2nd

My instinctive reaction, on learning of research published today by Bristol University’s Centre for Market and Public Organisation showing GCSE results have improved faster in England than in Wales in recent years was “surprise, surprise”. And what, exactly, does this prove?

Take two school systems.

Imagine that, in system A, huge effort and political attention is paid to improving schools’ results, on a few centrally-designed performance indicators. League tables are published which centre on those indicators, inspection judgements centre on them, schools which do badly are repeatedly threatened with closure, heads at such schools are very closely monitored by their local authorities, and numerous other effects, from performance related pay to rewards for heads of improving schools, accentuate the focus on these results numbers.

In system B, the mechanism for focusing on these indicators is not quite so developed. One change in recent years has been the decision by those responsible for overseeing this system not to publish this performance indicator information in school league tables, collated centrally. System B in general, though, also seems to be less overwhelmingly orientated towards pushing schools to improve on a particular metric of school performance, as measured by exam results.

Suppose, also, that academics come along and look at how the two systems have performed, as measured by results performance indicators, and find that results in system A have risen faster than those in system B, on these metrics. This, the research would seem to imply, shows that system A has “improved” faster, in a general sense, than system B. Moreover, the research also claims that one difference between the two systems, the lack of league tables in system B, explains that “performance” gap. It even goes further, arguing that the very fact of the removal of the league tables from system B has actually reduced pupils’ exam performance in this system, even though results had actually improved there, too, though not as fast as in system A.

Such an analysis would, of course, be highly problematic. It is hardly surprising that if you make one indicator of performance the be-all-and-end-all for schools, as has happened in the case of England in recent years with the proportion of pupils achieving five A*-C grades at GCSE (later changed to include a focus on English and maths because of concerns about schools “gaming” the system to focus on “easier” courses) that they will focus very closely on improving on that particular measure.

But, as many including the economist Charles Goodhart (of “Goodhart’s Law”) fame, have suggested, the key question is whether this signifies true underlying improvement, or just that, when you implicitly threaten people with losing their jobs if particular statistics do not rise, they are likely to rise. Research which shows only that the measures by which people are being judged have risen, then, really shows very little other than that people, when effectively forced to do so, can focus on what is being measured.

The Bristol study does indeed take improvements in the GCSE results statistics in England (system A), which has had league tables for many years, and compare them with those of Wales (system B) since the Principality dispensed with them for secondary schools in 2001. It argues that because results in England have risen faster than in Wales, that league tables work, saying: “We find systematic, significant and robust evidence that abolishing school league tables markedly reduced school effectiveness in Wales.” Further than this, the press release I have received today in association with the research says the study shows “naming and shaming schools works”.

But the Bristol research, which covers the period 2000 to 2008, was not quite so simplistic as to argue that only because the results on England’s published GCSE indicators have gone up faster in recent years than those in Wales, schools here (I am based in England) must have improved, in a general sense, at a faster rate than in Wales.

It does acknowledge the problem alluded to above: if you want to prove this in the way the academics want to, you must have an alternative set of indicators, against which schools were not being judged publicly over the period, or else all your research will show is an increased focus in England on the measurement mechanism itself.(Note 1)

When I read further into the paper to learn it was using a second set of measures to try to show this, some of my fears about it were assuaged. This would be a more sophisticated analysis, I thought.

Unfortunately, there are problems with this second indicator, too. It appears the research team have used for this second measure a “value-added” metric of school performance(Note 2), which looks at each pupil’s key stage 3 results, and then compares them to the pupil’s average GCSE points score two years later. By this measure, it is possible to arrive at figures for the performance of schools in England, and to compare them to those in Wales.

The paper implies that because this second measure is not “published” in league tables, schools will not focus on it to the same degree as they did for the five A*-C measure at GCSE, and so that it provides a cross-check as to whether the improvements of one system (England’s) have come simply through an obsessive focus on one particular indicator.

Although the team is right to look for this cross-check, there seem to me to be factual errors in what they say about the second indicator which they use. There are two problems, one less important and one major.

To take the less significant one first, it is not true to say, as the research team seems to be saying that average point score indicators have not been published under England’s system. As league table information I have looked at going back  to  the year 2000(or before) shows, average GCSE points scores for individual schools have been published for parents to look at if they want to since at least that time (see, for example, government information on what went in the tables in 2001 here, under “how the results are reported). The researchers might contend that value-added information of the precise sort they use (ie measuring KS3 to GCSE results in the way described above) has not been published, but a glance at information published on the education department’s performance tables website would suggest that it has been, at least for part of this period. In any case, if the main “outcome” or end measure of this indicator is which average points score each school ends up with, then that has been published, on a school-by-school basis, in English league tables for a long time. So this might have influenced schools’ actions in terms of trying to boost results on this indicator, making it not very good as an alternative measure of underlying education quality.

I say this is a minor point, though, because I would agree with the paper with the implication that the main statistical influence on schools’ actions in England over the period under consideration has been the five A*-C indicator. It is the published indicator that they have been focused on, as any reader of this site, or anyone familiar with secondary schools in England, will be aware. (Although, strangely, the research team does not acknowledge that the five A*-C indicator was itself superseded, in England’s league table system, by five A*-Cs including English and maths in 2007, a year before the final data which seems to have bee n used in this study).

The major issue is what looks to me like a factual howler on page 18 and 19 of the report. This considers whether England’s GCSE results could have been boosted by the use of non-GCSE “equivalent” qualifications in its official calculations. The paper concedes that non-GCSE courses have been included in its “GCSE” calculations (because these are based on official government figures which include them). It says that “In addition to the regular GCSE qualifications, schools in both England and Wales have used GCSE-equivalent qualifications, typically more vocational qualifications and more frequently used for less academically able children”. It adds though, that this is a relatively recent phenomenon, saying “the new equivalent qualifications were introduced in England in 2005 and Wales in 2007 as part of the restructuring of the curriculum between the ages of 14 and 19”.

It then uses this fact as one half of the explanation why GCSE-equivalent qualifications do not explain England’s more-improved performance, pointing out that there is no disjunction in performance trends before and after 2005, presumably as might be expected with the introduction of such qualifications into performance measures. 

The claim that “equivalent” qualifications were introduced in 2005 is wrong. “Equivalent” qualifications have been around far longer than that, in the form of General National Vocational Qualifications (GNVQs). I remember investigating how some English schools boosted their results, understandably, perhaps, given the pressures on them, through GCSE-equivalent Intermediate GNVQs, since at least 2003. Here is the link to an article a former colleague of mine at the TES wrote back in 2001. Moreover, a glance again at the Government’s rules for calculating schools’ average GCSE points scores (see, again, here) from, say, 2001, shows that GNVQs were being counted, for performance information purposes, as “worth” up to four GCSEs back then. That is, they appear to have featured in the indicators used for this research throughout the period, not just for part of it.

The reason the researchers suggest this phenomenon only started in 2005 can be traced to a footnote in their paper, giving a link in which the Qualifications and Curriculum Authority spoke of “turning the government’s vision for reform of 14-19 learning into a range of effective qualifications, programmes and initiatives”. It is true that the range of qualifications which counted in league tables was expanded in 2005, alongside these reforms. But the GNVQ route was already well-established by then as a well-known way some schools sought to transform their league table standings through an equivalent qualification, worth multiple GCSEs.

The other half of the paper’s explanation as to why any greater reliance of English schools on “alternative” of “GCSE-equivalent” courses to drive up their results is insignificant as a factor behind differential results patterns in the two countries is that, statistically, GCSE-equivalent courses do not amount to much in the figures over this period, accounting for only 9.5 per cent of total scores among the lowest-performing schools, and 4.3 per cent among median-performing schools in 2006. I have no way of checking that statistic, of course, though I do wonder how significant that 10 per cent effect on low-performing schools is, given that another conclusion of the paper is that schools with generally low-attaining pupils in Wales have been particularly hard hit by the decision there to abolish league tables (ie their performance has suffered, in a general sense, compared to England because league tables are no longer published).

Moveover, GCSE “equivalent” courses, as demonstrated several times in recent years, (see, for example, here) have certainly had a big impact on some schools’ published league table results, including those towards the bottom of the rankings which have been subjected to just the “naming and shaming” approach the paper ends up advocating.

The underlying point is that, while this second indicator is meant to provide a cross-check ruling out the contention that schools in England have been artificially (understandably, in my view, given the pressures on them) boosting results by concentrating on narrow performance metrics, in fact some tactics used by schools to do just this would seem to me to have had the effect of boosting performance on both measures. They are, then, not really independent variables at all.

Examples of tactics which could boost both measures would include the use of these non-GCSE qualifications, as discussed above – which would boost both the main five or more A*-C measure and average pupil points scores – and teaching geared very precisely to the contents of particular exams. A better alternative measure might be something which was much more divorced from the centrally published performance indicator, such as performance in other tests than GCSEs, although even this might reflect in part a greater stress on the importance of test performance – as opposed to other aspects of education – in one system’s schools as compared to those of another. But no other measure is used in the paper.

Pulling back from this somewhat now, there is the problem that the paper takes two countries and then notes that one policy – secondary league table publication – changed in one of those countries and then seeks to assign causation for changes in the exam results of these two countries to that change.

This is a bold attempt, and it is understandable that academics should be looking for the “natural experiment” that the paper suggests is created by parallel policy developments in England and Wales. They point out that previous attempts to investigate the effectiveness of school accountability have been hampered by the lack of a “control group”: a set of schools which have not had certain accountability mechanisms, to be considered alongside those that have.

The academics have looked at Wales, with the cessation of secondary league tables, compared this to what has happened in England and thought that this will make for a good experiment, with one aspect of school accountability present in one country, but not in another.

The trouble is that it’s a stretch too far. For while league tables did indeed change in Wales, they are only one element of what look like different approaches to school accountability. Many other aspects of policy have also changed between the two countries over the period.

The academics, of course, do consider, in some detail to be fair, whether more than one variable in this “experiment” might be changing. While noting that Wales’s schools have tended to be poorer funded than England’s, for example, they say they are able to control for this as a factor, effectively saying, if I have understood them correctly, that they can show that English schools have improved more, under their indicators, than those in Wales, even after taking into account possible effects of funding differentials. Fair enough.

They also consider whether inspection regimes might have changed in one country compared to the other over the period. But this is dismissed, the paper saying that “during the period of our study there were no major differences between the England and Wales inspection regimes”, seemingly based on a single study in 2008 by the academic David Reynolds.

My hypothesis is that English inspection systems, which have become very data-focused in recent years, might be encouraging schools here to obsess about their results to a greater degree than those in Wales. This, then, would be an alternative or additional explanation alongside league tables in explaining differences in results improvements between the two countries. To be fair, I cannot be sure, as my knowledge of the Welsh inspection system is sketchy, certainly when compared to what I know about England’s.

But, again, this is not discussed in detail in the paper as a possible alternative explanation. I find it strange that, in this section, there is no mention that the English inspection system actually changed dramatically during the period of the study, from one that was based on longer visits to schools in which classroom observation was important, to one, introduced in 2005, in which inspections focused relentlessly on test and result data. For references to this, see articles here and here, and chapter 16 of my book. Again, it is creditable of the researchers to acknowledge this as a potential extra factor, but why no detail?

The truth is that league tables are only one element – admittedly a very high profile and always controversial element – of the accountability system which has worked to push schools to prioritise results improvements in England in recent years.

Other changes could also have driven the results improvements in England – such as the increasing availability of exam board endorsed textbooks which facilitate very close teaching to the exam, and other support from the boards such as training for teachers in what examiners are looking for; the use of direct threats of intervention from Government and local authorities to schools which are underperforming (for an example of how this pressure was stepped up in the years covering the study, see here); and the advent of School Improvement Partners, who as far as I can work out have been going around English schools emphasising to school leaders, in recent years, just how important it is that they improve their pupils’ exam results.

Have such changes been taking place in Welsh schools to exactly the same degree, so that we can be confident that league tables can be isolated as the main reason for any improvement in results which follows? Well, I would like to see convincing evidence.(Note 3)

Pulling even further back from this, I suspect that it just is not true that all that has changed in a meaningful way between England and Wales over the intervening years –looking outside of accountability itself- has been the publication of secondary league tables. Changes such as the introduction of the national literacy strategy in English schools are considered in the paper, but discounted as an alternative explanatory factor, but policy change in England goes beyond this. It was completely hyper-active over the period, so isolating the effects of one particular policy is a perilous game.

Right, that’s my analysis done of this paper’s evidence and arguments. But probably my main concern with it is how such research – premised as I believe it is on ultimately thin grounds – is used. It will be used to promote a view that league tables “work”, on the basis of at best partial statistical analysis and, really, given the other evidence which is available, not much else – and even that tough policies such as “naming and shaming” schools are right.

It also, obviously, promotes a view, inadvertently or not, that all that matters in education is exam results. Do the public really share this opinion that exam results are all? Of course, many will think they are important. But if there has been a national debate about this, and the public have concluded that good exam grades are all that matters, I have missed it. It is a tremendous shame if those who view grades as the best, most “precise”, and most convenient way of measuring schools help, inadvertently or not, to encourage this opinion for that reason.

The paper does admit, in one paragraph on page 22, that “Of course, teachers and schools may have broader aims than GCSE exam results for their pupils. These are not measured in our data so we can say nothing about the potential impact on (ceasing) publishing performance information on these broader educational outcomes, nor on any potential impact on teacher and headteacher motivation and morale”.

I would submit – and it pains me to say this as I have had some contact in the past with two of the research team and have respected the expertise behind their work, but I am going to say it anyway – that this is just not good enough from serious researchers. This is especially the case, given the boldness of the claims being made on the basis of this statistical analysis: that league table reform in Wales has reduced the effectiveness of schools there, and that naming and shaming schools “works”. Not “reduced schools’ effectiveness according to particular measures of pupil exam performance”, or “works in terms of driving up performance on exam indicators”, but “reduced schools’ effectiveness” and “works” full stop.

(It could be argued, here, that “effectiveness”, when used by statisticians to describe school performance, has been defined and used in quite a precise way to mean “effectiveness as measured by particular results indicators”. But this is not how it is interpreted by those outside of the statistical community reading this report).

In essence, the position is: “there may be other things to consider than exam results, but we haven’t bothered doing so in any depth”, possibly because these qualities cannot easily be measured statistically, although this is not specified in the paper and would be contentious. It seems no further explanation is needed as to why such an investigation was not at least a part of the analysis. And I say that despite the researchers clearly having gone to the trouble of investigating potential caveats and limitations within their research.

For all the copious statistical expertise clearly involved in this analysis, research which essentially only considers performance statistically in this way cannot provide the whole picture of what has been going on in schools in the two countries.

It is providing at best a part of the picture of what has been going on. The other part could only be provided through detailed qualitative research, which you have to say is lacking here. If English schools have improved, how have they done it, for example? Do teachers simply not bother to teach effectively without league tables, do league tables make them work harder, or what? Have there been any side-effects, for pupils, parents and teachers? (I can point, of course, to evidence of many, included subtle effects such as claims that certain “academic” subjects such as languages can struggle to compete for pupils under accountability pressures facing schools to maximise published scores).There is no analysis here, with the numbers supposed, in effect, to serve as sufficient evidence in themselves of general improvement. Yet there are powerful examples around of a rich qualitative approach to finding out what has been really been going on in English education in recent years, the best of which, I think, in recent years has been the hugely detailed Cambridge Primary Review.

It seems to me there is a division, in education research, between academics who trained as economists, and therefore are trained to be fluent with numbers and multi-level modelling, and those focusing more on the qualitative.

Because this research is very much of the former variety, with lots of formulae, it will probably be taken seriously. Numbers, even without, I believe, their full context, are very powerful, and can be assumed to be showing the “truth” about a particular phenomenon. They are also, I believe, seductive, in offering relatively easy-to- collect evidence on school “effectiveness”, without necessitating hugely detailed qualitative study. They are not “fuzzy” in a qualitative sense. And many people whose eyes glaze over at the sight of a complicated results equation will simply take it on trust that the economist-education academic is reaching the right conclusions.

The truth, of course, is usually more complicated than can be demonstrated through formulae alone. My worry is that these findings will be used to push forward a particular view of education reform, implying in the minds of reformers that getting schools to focus relentlessly on exam success –however measured – is a good thing, that there are no negative impacts on what goes on in schools arising from centrally collated Government performance tables and that hard-hitting policies which many will argue end up ultimately undermining public education, such as “naming and shaming”, are the way to go.

This report does not demonstrate the effectiveness of these policies in any general sense. It certainly seems to me to be very far from proof that “league tables work”. But its findings will be taken in just such a way. I think education deserves better.

Note 1: This of course, has one obvious problem in that schools can focus on improvements on the published indicators through concentrating, for example, on pupils just on the borderline of achieving the particular indicator, at the expense of other learners. More generally, measures could be taken to “game” the system such that schools improve on the measured indicator, but not in any fundamental or wider sense.

Note 2: Actually, it is not completely clear to me that this is the alternative measure being used, after reading the paper, or whether the team did not use a value-added measure at all here, simply comparing pupil performance as measured by “raw” average GCSE points scores. But whichever measure is being used does not change the argument above, because average points scores did feature in published league tables during this period for individual schools.

Note 3: It could be argued, of course, that the differential approaches to improving performance in English and Welsh schools really do not matter; all that really matters is that England has somehow managed to generate faster-improving scores than Wales. These results are really important to young people, so essentially it does not matter how they have been generated, or what they signify.I don’t buy this argument, I’m afraid, since GCSE results are a relative, not an absolute, good to young people. Pupils in England – most of whom will be competing against other pupils from English schools in the employment and higher education market after they finish school – will only benefit from a national improvement in results if these truly signify underlying improvements in the quality of what they understand, as supposedly measured by the results, since pupil A does not benefit if he or she then finds that, though his or her results have improved, so have those of pupil B, against whom he or she is competing in the employment or higher education market, because of a national results improvement. That is, results are useful to a pupil in themselves in a relative sense – ie to the degree they confer advantage over competitors – rather than in an absolute sense: pupils do not gain, relative to others, if results for all pupils improve.

Wednesday, October 27th, 2010

I was disappointed to hear Rachel Wolf, of the New Schools Network, talking in what sounded like fairly ideological terms about “accountability” on the Today programme yesterday.

Ms Wolf was being put on the spot about the ability of free schools, which the New Schools Network promotes, to employ teachers who lack Qualified Teacher Status (QTS).

It was a brief interview, with frequent interruptions, but I think the thrust of her argument was fairly clear.

First, she said, schools needed the flexibility to decide whom to recruit. I’m not going to discuss that point directly here.

But second, she seemed to be arguing that if there were any problems with teachers recruited without having gained QTS, these would simply show up through “accountability”, which seemed to consist of inspections and test/exam results.

She said: “The way that parents decide about whether teachers are good is by whether the school is doing well, and one of the things about free schools is that they are going to be held strictly accountable through inspections”. Then, as the discussion headed towards a slightly frantic conclusion, she seemed to be talking about test and exam results being the main vehicle.

She said: “I think it’s absolutely key that groups [setting up schools] are able to innovate in the classroom, that they are able to try different things, and that they are held accountable for their results. I think we need to move to a system where people are held accountable for how well they do.”

James Naughtie, the interviewer, then intervened. He said: “If [the teachers] get bad results, it’s too late.”

Ms Wolf replied: “We have to allow people, as they have done in America, as they have done with charter schools, and this is why [President] Obama is pushing charter schools so heavily, to try something different for children who are not getting the results they should.”

I think the argument, then, is clear, and the thinking is entirely akin to the ideology behind the testing/charter school/corporate education model now being pushed very hard across the Atlantic. Charter schools are one of the school reform models being promoted by the New Schools Network and are, as Ms Wolf says, also backed by President Obama.

“Accountability” in the way it has developed through an influential strand of opinion in America, says at its crudest that it doesn’t matter what credentials a teacher might have because they can simply be held to account through their pupils’ test and exam results, with low performers fired. It is a huge debating point in the United States.

Some, including the author and journalist Malcolm Gladwell, have reportedly argued that “anyone with a pulse and a college degree” should be allowed into teaching, with performance data then used to root out underperformers.

Media attempts to put pressure on teachers with poor scores have included the Los Angeles Times publishing value-added test data on thousands of named  elementary school teachers.

The current edition of one of America’s most well-known education magazines, Phi Delta Kappan, for which I write regularly online and which fell through my letterbox this morning, has two lengthy articles debating the problems with this version of accountability, including the assumption that there are legions of great potential teachers waiting to take the place of anyone sacked because their pupils’ test scores are too low.

One* concludes: “A decade of accountability mandates has caused schools to respond in predictable and unproductive ways. One of its effects has been to obscure certain truths about education as actually practiced [sic: US spelling] in the classroom. If schools are to improve, they must abandon the business-orientated rhetoric of the accountability movement and concentrate on what we know will improve student achievement. That is, schools must focus on improving the quality of instruction in all classrooms.”

I have also just been sent a statement to sign, put together by 10 very prominent US educationists which argues against that “even the most sophisticated use of test scores, value added modelling (VAM) is a flawed and inaccurate way to judge whether teachers are effective or ineffective”.

Ranged against them in this debate are prominent media organisations, aggressive reformers – who argue, rightly or wrongly, that drastic change is needed in America’s schools – and corporate sponsors.

But accountability in the simplistic sense that reformers push it promotes, of course, dumbed-down education, narrowing teaching to what can be measured, while there is what seems to me persuasive evidence that it can actually inhibit innovation, in creating a culture of fear among professionals.

I would also wager that if anyone asked Ms Wolf who her own best teachers were, she would not be asking for data on their pupils’ exam results.

For these reasons, actually, I think Rachel Wolf is wrong: I’m sceptical whether accountability actually will take off in this fashion if free schools are really given the freedom that those trying to set them up will hope for. For the middle class parents who seem to have at least a say in the running of these schools (for an example, see this group at Toby Young’s proposed free school) strike me as too smart to let teacher quality be judged on such narrow measures as is being advocated in the States. Teacher quality, I think, if left to parents running schools, would be judged in a more nuanced and qualitative, and less reductive, way, if these parents really have a stake in ensuring the schools promote good teaching.  

Anyone doubting this should consider that the other independent sector over here – in particular, the successful, household name fee-charging schools – as far as I know do not go near anything like this kind of measurement system. I don’t think their reputations would stand it.

Alternatively, if a system of crude numbers-based accountability is going to be imposed on free schools, then they are likely to be “free” only to the extent that they can guarantee their pupils’ good performance on the tested metrics,  which presumably will be centralised indicators.

Just finally, watching the American debate develop into this incredibly polarised, bitter battle for the future of state education, with corporate donors seeming to have a very strong say in how it is framed and state schools regularly pilloried, I do wonder why anyone over here would look at it and say: “that sounds good to me, let’s have some of that”.

*The US article on accountability which I mention, for which unfortunately I’m unable to provide a link, is “Truths Hidden in Plain View” by Thomas M McCann, Alan C Jones and Gail Aronoff.

Wednesday, October 20^th

So the funding situation for schools is getting a little clearer, after George Osborne’s spending review announcements today. There is still some way to go, though.

Here are a few thoughts.

First, although the funding situation for schools appears at first glance better than many were predicting – and certainly appears better than that facing universities and further education, there are several caveats.

For, although Mr Osborne was able to say that the schools budget for 5- to 16-year-olds would rise in real terms every year for the next four years, this relates to only to a 0.1 per cent real terms increase per year. But more significantly, perhaps, the spending review documents make clear that “underlying per pupil funding will be maintained in cash terms”.

In other words, although the amount of money going into the overall pot, the Government says, will rise just a bit more than it needs to do to keep pace with inflation, funding per pupil is actually going to fall, after taking into account inflation, presumably because pupil numbers are increasing.

The Government might retort here that teachers’ pay – the major element of schools’ budgets, at around 80 per cent – is being frozen over the next two years, so a big part of costs will not increase over that time. But that still leaves the 20 per cent.

But returning to that overall pot, the fact that it will keep pace with inflation at all is in part because the pupil premium, at £2.5 billion, has been factored in to the total.

This, I think is very interesting, as it shows that the pupil premium is not just an addition to the schools budget, as it could have been.

In the Institute for Fiscal Studies policy paper from March, which I reported on in my first blog on the pupil premium, the first scenario modelled was one where the pupil premium was simply added on to existing schools budgets.

The IFS modelled what would happen under this scenario, which would mean that no school would lose out from the advent of the premium, and many schools would gain. This model, it said, was based on a 2009 policy paper from the Liberal Democrats which proposed a pupil premium which would “involve extra money for schools”.

I cannot see how this scenario survives under what has been proposed today, since it would seem to imply preserving baseline real terms funding to schools before the pupil premium is added. But the pupil premium is being added only to bring funding up to real terms parity.

Because the overall 5-16 schools budget is being held roughly equal, in real terms, then, even after taking into account the pupil premium, there are going to be winners and there are going to be losers from this settlement. In fact, given that per-pupil funding is only being held constant, in cash terms, it may be hard to find many schools whose budgets are going to go up, in real terms, as a result of this settlement.

And what will be the effect of the premium itself? Well, it seems to me there are only two scenarios, here.

Because the schools budget is to increase only roughly at the pace of inflation, this is a “standstill” budget, in real terms. If the effect of the pupil premium is to direct more money towards pupils from disadvantaged backgrounds, this must come at the expense of non-disadvantaged homes (ie they will lose out, in real terms, because the pot is not large enough to ensure that no-one loses).

On the other hand, as the Institute for Fiscal Studies argues in its March pamphlet, in reality in recent years there has already a large amount of money going to schools educating disadvantaged pupils (through an implicit pupil premium). Given this, it may be that the pupil premium does not end up directing more money to disadvantaged pupils, on average, than was already the case under Labour. In that case, non-disadvantaged pupils would not lose out but it would hard to see how the politicians could claim the pupil premium had really gone a long way towards tackling disadvantage.

There is still much detail to work out, including how direct grants to schools are allocated. These are significant because, according to the IFS, they accounted for 15 per cent of primary school funding, and 16 per cent of secondary school funding, in 2008-9.The Government has said it will take grants including the Schools Standards Grant, specialist schools funding and money for one-to-one tuition, into account when calculating schools’ budgets, implying that these grants are going to be maintained but with “ringfencing” removed so that heads can decide how they want to spend the money.

However, does this mean that each school receives the same amount of grant as they did in the past, or simply that the money goes into a general national pot to be redistributed?

The other thing that has not been factored into any of this, I believe, is the effect of any cuts in local authority support services, which will have an impact in schools.

Despite this lack of detail, if I were a betting man, I would wager that schools in relatively well-off areas with large numbers of disadvantaged pupils would emerge as the relative winners from this settlement, and those educating lots of disadvantaged pupils in poorer areas as relative losers.

I’ve got a couple of reasons for believing this. First, the Government’s consultation on the pupil premium, published over the summer and discussed in my last blog, does not envisage the pupil premium being a single figure per disadvantaged pupil for the whole country. Rather, schools educating disadvantaged pupils in relatively poorly funded areas (which tend to be outside the poorest inner cities) will get a higher pupil premium, and those in well-funded areas, a lower. The effect of this, I think, will be to redistribute funding towards schools in less disadvantaged, rural areas, as the IFS paper from earlier this month suggests.

The Liberal Democrats, in their pupil premium paper from last year, say that funding for disadvantage is not “targeted” effectively enough at the moment, with schools in different parts of the country funded very differently (up to £1,000 per pupil) even when they have “very similar levels of need”. It implies that Labour grants targeted at areas of deprivation, rather than particular pupils, miss out on supporting many disadvantaged children. Expect redistribution to reflect this, then.

The other reason to believe that the change could work this way is pure politics. Unsurprisingly, figures for children eligible for free school meals tend to show that more are concentrated in Labour-supporting areas. In a tight budget overall, the coalition would be brave to allow the pupil premium to move funds for deprivation further towards Labour-supporting areas, at the expense of schools in areas which have Liberal Democrat or Conservative MPs. Political realism would suggest the formula has not been engineered to make this happen. Perhaps the politicians will be bold enough to allow schools in “their own” areas to take the hit for this new funding system. But, as I say, I think it would be a brave person who bets on it.

Tuesday, October 19th

Since I wrote my article on the Pupil Premium for Education Journal back in May, there have been some developments which give clues to some of the questions posed in that piece.

Specifically, there is more information available in relation to a central dilemma referred to there: whether or not the pupil premium is to be paid for by cuts to Labour’s existing grant schemes, which went directly from government to schools and which have had the effect, according to the IFS’s paper which was published in May, of helping to provide much greater funding for children from deprived backgrounds.

On page 23 of the Government’s consultation on the pupil premium, published in late July, there is part of an answer. The paper says that the Government intends to “mainstream” relevant grants into its calculation of schools’ “core” budgets, provided through what is called the Dedicated Schools Grant. The relevant grants likely to be bound up into the Dedicated Schools Grant include the School Development Grant, the School Standards Grant and the School Standards Grant (Personalisation).

On first reading this, it seemed to me that this meant that the coalition was simply scrapping the money that goes with these grants. But that does not seem correct, on reflection. The recommendation would seem to imply, instead, that schools might continue to receive similar amounts as they did last year under these direct grants, only all bound together as part of their core budgets.

If true, this would go some way to alleviating worries in the IFS’s paper that the money for the pupil premium would come from scrapping existing direct funding which applied under Labour completely. However, we don’t know yet if all of the grants are to be transferred in full to schools’ core budgets. And just to make things more confusing, the paper stresses that the “mainstreaming” of these grants is subject to a final decision at the review.

The consultation paper is also interesting in setting out a methodology for the calculation of the pupil premium which is already proving controversial. The method proposed is not the obvious one that one might assume: that the Government sets a simple amount of funding per pupil – possibly adjusted in certain parts of the country to reflect a higher or lower cost of living – and adds that figure to schools’ core budgets for every child categorised as “disadvantaged”.

No, instead the method proposed is that the pupil premium should vary according to how well- or poorly-funded councils already are in terms of their mainstream budgets (ie the amount they receive through Dedicated Schools Grant, plus the “mainstreamed” grants). Those in poorly-funded areas would receive a higher pupil premium than those in better-funded areas, so that the total amount allocated per deprived pupil would be broadly similar across the country.

A second paper from the IFS, published only this month, underlines the fact that pupils from disadvantaged homes in relatively well-off areas will therefore gain a higher “pupil premium”, under this system, than disadvantaged pupils in relatively poor areas. The effect of this, I think, will be to equalise funding for disadvantaged pupils across different local authorities. Schools educating disadvantaged pupils in more well-off areas will gain more than those in poorer places.

It also looks, from the Government’s consultation, that the pupil premium will be phased in over four years, with increases to gradually bring it up to a roughly equivalent figure for disadvantaged pupils from all areas over that time. I’m reluctant to write more than this, however, at least until after we see the detail of the spending review tomorrow.

Monday, October 18th, 2010

With Wednesday’s Comprehensive Spending Review due to include an announcement on the new “Pupil Premium”, I thought I’d post here a piece I wrote on this shortly after the general election in May. It is based on an impressively detailed paper on the Pupil Premium by the Institute for Fiscal Studies. I originally wrote it for Education Journal, for whom I write a monthly article.

I intend to do another blog – possibly shorter! - by tomorrow updating the position on the pupil premium based on what has been announced since May.

My Education Journal piece follows below:

It was reportedly one of the key points of convergence between the Conservatives and the Liberal Democrats during their negotiations over the formation of the new government.

Accordingly, it was one of only five education policies mentioned in the “coalition agreement” signed between those parties and published during one of the most momentous weeks in British politics for many years.

But what exactly is the “pupil premium”? And will its introduction, advocated for at least two years now by both of the former opposition parties, really bring about the transformation in funding and achievement among poorer pupils that they hope?

For all the warm feelings and positive sentiment that this policy would appear to generate, I am not sure that its introduction will be quite as unproblematic as those backing it might hope.

The concept behind the pupil premium is relatively simple. During the calculation of schools’ budgets, children from poorer backgrounds are allocated extra funding, compared to those from better-off homes. This allocation then follows the child to whichever school they attend. So a school with many disadvantaged pupils will be better funded than one educating primarily better-off children.

The motivation for introducing it is also clear. All parties want to address the achievement gap between children from poorer homes and the rest. The presence of this chasm is constantly underlined through research, with children eligible for free school meals – to choose the most commonly-used measure of deprivation – well behind their peers from wealthier families, on average, on any metric of achievement.

Although both back it, the two coalition parties have come at the pupil premium from slightly different perspectives. This, at least, is the view of an impressively analytical recent report published in the spring by the Institute for Fiscal Studies.

For the Liberal Democrats, the idea is that targeting extra money at disadvantaged children will have a direct reward: with more investment, it would be expected that their education improves and, by extension, that their academic results also rise.

Perhaps unsurprisingly, the Conservatives see the premium more in terms of its effect on the “market” of school places. With poorer children attracting more cash, schools will have an incentive to chase their “business”, and thus any tendency in the current system to incentivise schools not to take on poor – or what could be seen as harder-to-educate – children could be negated.

So where are the difficulties? Well, I think there are potentially many.

Most fundamentally, to hear the way this policy is put across, one might expect that poorer children have actually been getting a very raw deal, in terms of funding, under the arrangements which have operated under Labour. But this really does not seem to be the case. The IFS paper makes clear that poorer pupils are already well-supported financially, relative to others within the state system, and that the money allocated to those eligible for free school meals (FSM) in particular has increased dramatically in recent years.

 The IFS statisticians looked at the data on funding for each school in England. School budgets consist of a core element, received from local authorities. This is then topped up by specific grants to schools from central government. The statisticians then looked at the number of pupils eligible for free school meals in each school. They found that schools with many FSM pupils were much better off, all other things being equal, than those without them.

There was, then, extra cash, or an implicit premium, for schools educating FSM pupils, even though this was not spelt out nationally through a particular formula. This was in part because of the increasing amount of direct funding targeted at schools serving disadvantaged areas.

The IFS paper then works out that, on average, in 2008-9 this premium amounted to £2,460 for each primary FSM pupil, and £3,370 for each secondary FSM pupil. In other words, in practice these children have been bringing a great deal of extra cash for their schools.

By comparison, average total per-pupil funding overall equates to £5,580 for 2010-11 in the IFS paper.

Moreover, the current implicit premium given to FSM pupils has been growing sharply in recent years, rising in real terms by 69 per cent in primary schools and 53 per cent in secondaries over only four years, the paper says, compared to only 17 per cent for the overall real terms growth in spending for all pupils since 2006.

The upshot, then, is that any new pupil premium giving “extra” money to children from disadvantaged homes does not start from a neutral position, in which the poorest children are now funded only on a par with the richest within the state system. There already is extra funding, and the new system will have to do better than this if it is to achieve its aim of further redistributing cash.

This, then, is the crux. Will the overall effect of the pupil premium be to add to the amount directed at disadvantaged pupils? Or will it leave things broadly unchanged, or even take money away?

The answer may depend on whether the new government adopts the Liberal Democrat proposal to fund the premium with an extra £2.5 billion taken from outside the schools budget, or goes with Conservative plans, which have not so far set out how the premium is to be funded. At the time of writing, all that was being made public about the government’s plans was a statement in the coalition agreement which said: “We will fund a significant premium for disadvantaged pupils from outside the schools budget by reductions in spending elsewhere”.

So extra money is to be found. But how much?

The Liberal Democrats were upfront before the election with their proposed £2.5 billion figure, funded in part by cutting tax credits to families with above-average incomes. The IFS paper models how injecting this amount of extra cash, on top of all current spending, might affect schools’ budgets if it were targeted at disadvantaged children. The effects could be dramatic, one in five secondary schools and one in three primaries receiving a rise in their budgets of at least 10 per cent as a result.

Crucially, because the cash would simply come on top of all existing spending programmes, no school would be worse off. However, this is not the only model set out by the paper. There would be two other options for an incoming government, it said.

First, it could fund the pupil premium partially by scrapping existing direct grants which the government now makes to schools, which include cash targeted specifically at those serving disadvantaged areas.

Second, it could overhaul the entire funding system, changing today’s arrangements whereby the cash for schools’ core budgets goes from central to local government, with councils free to supplement this before passing the money on to schools. Instead, a national funding formula would be introduced, with all schools receiving a set amount for each pupil they educate directly from the government, with an extra figure allocated for disadvantaged children.

Unsurprisingly, these latter two models produce less clear gains for schools educating disadvantaged pupils. Under the first option, which would scrap the government grants, up to 70 per cent of primary schools and 67 per cent of secondaries would either be worse off or broadly unchanged. The other option, the new national funding formula, would again produce winners and losers, with around half of schools gaining cash and half losing it.

And, amazingly in one version of this change modelled by the IFS, among the losers would be the 10 per cent of secondary schools which have the highest number of deprived pupils. For them, the introduction of the national funding formula, with its explicit premium for disadvantaged children, would not compensate for the loss of direct grants they currently receive from Whitehall for tackling disadvantage, all of which would go to pay for it.

The key in all of this, then, both politically and in terms of its effects on individual schools, will be how many of Labour’s existing programmes to tackle deprivation get retained, if any, and, of course, how much extra cash overall is devoted to the pupil premium.

There are reasons to fear for existing direct funding in particular.

First, it seems unlikely that the new government will want to keep all of Labour’s central funding programme for schools, simply because it will have its own priorities.

Second, a 2008 paper from Policy Exchange, the think tank with close links to the Conservartives, proposes paying for a national funding formula for schools by scrapping Labour schemes including the standards fund, education maintenance allowances and the National Challenge, which targets schools with low results.

As ever, the devil as to whether this policy actually does achieve its ambitious goal of improving the lot of disadvantaged pupils will be in the detail of how it works. And there is an awful lot of detail.

The IFS paper, “Pupil Premium: Assessing the Options”, can be viewed at: www.ifs.org.uk/publications/4775

 Warwick Mansell

Wednesday, October 6^th, 2010

What are the feelings in schools over the decision by the NAHT’s national council not to boycott Sats tests for a second year in 2011?

If the reception the move received at a meeting of heads which I addressed on Friday is anything to go by, there is a lot of anguish, and anger, out there.

The meeting, for heads in a London borough, was a regular chance to discuss issues of current interest. I was there to talk about the evidence on testing and accountability, but the heads were keener after I spoke to discuss the union’s move to go ahead with the tests in 2011.

As an aside, this is the second time in the last few months I have come away from a meeting struck by the strength and depth of feeling among heads against the current test-driven accountability regime – the first was at NAHT’s annual conference itself – and I would challenge any defender of hyper-accountability to spend some time at such events and still be of the opinion that there is nothing to these views beyond professional self-interest.

Anyway, as readers no doubt know, the NAHT’s council overwhelmingly voted 10 days ago not to repeat last year’s boycott, as Michael Gove, the Education secretary, announced details of a review of Sats testing. Next year’s tests will be unaffected by the review.

At the London meeting, one head, who boycotted the tests in May, spoke first. She said: “I am feeling totally let down by the NAHT. Last summer, I put my head on the line for something I felt very strongly about. Ofsted criticised me for not having done the Sats. This is the first opportunity we have had for decades to say what we feel strongly about, and now it’s gone.”

A second head said that more of his colleagues were committed locally to boycotting the tests this year, and expressed his feeling in strong terms that no attempt had been made to measure the feeling of heads towards supporting a further boycott before the decision was taken.

He said: “I think the commitment to take this further forward was there. I think it’s appalling that the NAHT did not measure that. We were looking at a 100 per cent boycott [in this borough] next year. I’m very disappointed with the NAHT, and considering my membership as a result.”

Another spoke of “great disappointment”. “Sometimes you have to do something because it is the right thing to do.”

Another spoke of the unfairness of a testing system which, she said, could put great pressure on schools to get children to come in to sit the assessments even if they were not in the right frame of mind to take the tests. The school had had a pupil who was unwell last year, while another had just experienced a bereavement. Yet if they had not taken the tests, they would be counted as failing them and the school’s results would suffer by nine percentage points. The often unjustified pressures, then, on schools as a result were “horrific”.

It was argued on several occasions that, where schools had boycotted the tests, there had been no protests from parents, while it seemed that heads had also not found their pay docked, despite warnings at the time. A couple had, though, had a tough time at the hands of their governing bodies.

The meeting also saw several speakers analysing the content of Mr Gove’s letter to Russell Hobby, the NAHT’s general secretary, in which he outlined the terms of the review. In my speech I had also analysed the letter, along similar lines as my blog a few days ago on this subject. The heads who spoke were not convinced the letter provided evidence the government was really about to change its position and listen to the profession on high-stakes testing.

There were also comments that the momentum behind the NAHT’s campaigning on this issue – the boycott having followed several years of building the intellectual case against the current test-based accountability apparatus – had now been lost. 

If there were any dissenters to the sense of bewilderment and frustration at the decision, they were not heard at this meeting, although one head seemed to argue that the deeper problem was that the union had never fully recognised that teacher assessment arrangements being proposed as an alternative to testing would still create problems for schools if the results it generated were then published in league tables.

As I understand it, there were several factors behind the NAHT’s decision. As Mr Hobby told the TES last week, elements within the coalition government are known to be strongly in favour of testing and league tables, and the worry in the union was that continuing with the action might make it more likely that the government’s position would harden. This way, it is clearly felt, there is a chance to influence change with the government.

There have also been concerns about a possible legal challenge. This did not happen in the spring, but the timing may have helped with that, with the politicians’ minds on the general election which took place just before Sats week. A Conservative-led government may have had even fewer qualms than Labour about trying to quash a boycott through the courts, it was thought.

Of course, the union’s decision on Sats cannot be considered in isolation from other issues. Part of the calculation will inevitably have been that a confrontational approach here would have removed its ability to wield influence over ministers over other aspects of the education agenda. Having been outside the “social partnership” arrangements between unions and the Labour government for a period, only to return, the union must be conscious of the dangers of being left out in the cold

These are reasonable points, and I am certainly no union strategist. But I do wonder if the power balance between the government and head teachers is quite as one-sided, in favour of the government, as these calculations might suggest.

I cannot see that many schools that boycotted the tests in May would not have done so again, given that frankly the world did not end for those which refused to run the tests in the conventional way, with pupils and parents still receiving information on their child’s overall level of performance.

Would more schools have joined in in 2011? Of course, it is impossible to know, but I’d hazard a guess that some would have observed what happened this year and jumped this time. Some will also say that the precedent set in May meant that heads would have had time to prepare for the boycott by not preparing for the tests with their children this time (one of the reasons widely cited in favour of doing the Sats even by those who hate the system), although one source who boycotted the tests this year was scathing of this view, arguing that it was clear throughout last academic year that the NAHT was likely to push for the boycott. Therefore, it was argued, heads had no reason not to boycott them this year.

This source argued firmly that those complaining about the NAHT’s decision now who did not boycott the tests this year really had no case: they should have supported the boycott this year and therefore put the union in a stronger position.

Nevertheless, I think a government faced with a boycott of more than the 26 per cent of schools which took part in the action this year – with, say, even a third to half of heads avoiding administering the tests – would have had a serious problem on its hands. Head teachers are hardly seen by the general public, I would venture, as the most militant of people.

But we will now never know how ministers would have reacted to this pressure. As I say, if the reaction in this room was representative of the country as a whole, there is a great deal of disappointment in many English primary schools now.

Thursday, September 30^th, 2010

If there is anything certain about the education white paper which is going to be published by the Government by the end of the year, it would seem that a long-standing loophole in league tables is about to be closed.

The system of “equivalences”, whereby non-GCSE qualifications are given weightings worth several GCSEs for the purposes of the tables and of all official statistics, is about to change radically, it would appear.

For years, Intermediate General National Vocational Qualifications were counted in the rankings as “worth” four GCSEs. When schools realised that getting pupils to take them could see the school’s published statistics improve dramatically, numbers taking GNVQs rocketed accordingly.

GNVQs were phased out a few years ago, and the loophole also seemed to have been partially closed by the stipulation that schools were to be judged on the proportion of their pupils gaining five or more GCSEs (or equivalent), including English and maths. The latter part of that formulation made it impossible for schools to do well in the table solely on account of “equivalent” qualifications.

However, in recent years, other courses have been growing in popularity, seemingly because schools observed that these courses had high pass rates and that, if they could get pupils entered and then passing one of them, say, they only needed to get these students to achieve C grades in English and maths and at most one other qualification and the school’s statistical goal would be reached.

As I reported for the TES last month, entries for BTEC First qualifications (worth two or four GCSEs, according to the course), and OCR Nationals (worth 1, 2, 3 or 4 GCSEs) have shot up.

Michael Gove, education secretary, told the Edge Foundation charity earlier this month: “Some of these qualifications badged as vocational enjoy a ranking in league tables worth two or more GCSEs, making them attractive to schools anxious to boost their league table rankings. That has to be changed.”

Having covered some of the peculiarities of the exam system for years now, I still find myself surprised at times. The latest example came a few days ago, in relation to these equivalencies.

Thanks to the efforts of Roger Titcombe, a former head teacher with whom I’ve worked before investigating school results, I obtained the full GCSE – and equivalent – results for 2009 of a school which has won national recognition for improving its grades.

And the patterns unveiled in the grades would be astonishing , I think, to anyone who came to this information afresh, knowing only that GCSEs and other qualifications were deemed to be equivalent for official purposes.

The pattern was as follows. For almost all of the 20 GCSE courses offered in the school, almost the full range of grades had been awarded. In other words, children had scored grades ranging from A*s (although only a few of these were achieved in this school) down to Fs, Gs and even the occasional U. For maths, for which there were 165 entries, every grade – A*, A, B, C, D, E, F, G and U – was awarded to at least two students, while for English, which had the same number of entries, the picture was similar.

For the vocational qualifications, though, the picture was very different. Amazingly, of the 303 entries, in eight different BTEC First or OCR National qualifications, not a single one failed. All achieved starred distinctions, distinctions, merits or pass grades. The school also had pupils entered for 15 different assorted vocational qualifications, categorised simply as “Vocationally Related Qualifications” or basic skills tests. Again, all entries passed. This high pass rate, and the multiple-GCSE equivalences of some of these courses, meant that the school, which has been lauded for its success in rising up the “GCSE” league tables, actually chalked up more A*-C passes through non-GCSE courses than through GCSEs. Confusing, isn’t it?

Now, this may not come as a surprise to anyone who has experience of how this works in schools. It is well known that BTEC Firsts have had high pass rates, and indeed, I reported on the national rates in that TES article. More than 99 per cent of grades awarded nationally in the four-GCSE equivalent BTEC First Diploma last year were at pass or better. For the two-GCSE BTEC First Certificate, the figure was 97 per cent. This is despite Jerry Jarvis, former managing director of Edexcel, which runs the qualification, describing BTEC Firsts as qualifications schools could offer “If you think a student will fail to realise their potential by doing GCSEs”. And, for the OCR Nationals, the line from the OCR board which I received when writing that piece seemed to be that they could not give pass/fail rates for the qualification partly because pupils were only entered for it if they were ready to pass it.

All these points may strike some readers as not new. My latest thought, though, on seeing the relative grade profiles in this one school is this: how were these “equivalent” courses ever given equivalence with GCSEs, when the way they use the grade profile is so different?

Since GCSEs were introduced in 1988, GCSE examiners have used the full range of grades down to G to indicate a pass. Thus, officially at least, D and E grades were seen as worth something for pupils in that they were not deemed failures. The idea was that most children would come away from the exam with at least some credit for their work. The concept of meaningful D-G grades survives to this day, at least officially, despite the huge emphasis now given by the government and schools to achieving a C or better, making it the unofficial pass grade. And a glance at the national grade profiles for GCSEs confirms this, because the full range of grades are awarded.

The vocational qualifications also, clearly, have a pass grade. But the equivalence sets this as being the same not as a grade G, which is the official pass grade for GCSE, but a grade C. And because the GCSE grading system still uses the full A*-G range to reward pupils’ efforts, pupils at GCSE can easily find themselves missing out on a C (the crucial grade for schools, for accountability purposes). Yet the vocational qualifications system is, unsurprisingly perhaps, reluctant to see pupils coming away from a course with nothing. But there is no option for the system, under the equivalences, to give them less than a grade C (actually, often it will not give them less than two Cs, or four Cs), without actually failing them, leaving them with nothing. So almost all are awarded at least a pass grade. To put it another way, and to use the OCR National argument cited above, it could be argued that a child would only be entered for certification for an OCR National when they had a chance of a pass grade. But the fact that this pass grade has been set at equivalent of a GCSE grade C means there is no chance for them to gain credit for their work by earning a GCSE grade at less than C. So the entry system may more or less guarantee that if they are entered, they are going to get at least the equivalent of a GCSE grade C. Yet that is not how GCSEs work.The grading systems are different, so the equivalence is weird.

The confusion also reflects, I think, the politicisation of this system, and a bodged attempt at compromise by politicians. The fact that the GCSE allowed pupils to pass with a G grade may well have rankled traditionalists, who compared it to the old O-level structure in which a D was a fail. But rather than either sticking to the official position, in school accountability, that all grades from G upwards constitute a pass, or changing it so that the C grade was actually made the official pass mark, those in charge kept the G pass officially, but unofficially made the old O-level pass grade (a C) the benchmark by making this the centrepiece of league tables and top-down results pressure on schools. This is confusing: accountability says a C grade is the pass threshold, but the exam grading system says a G grade marks the boundary.

Some will counter that employers also have long regarded the C as the cut-off for success, so it is right for the focus to be put at this boundary. But I have to say I don’t completely buy that argument: a detailed report from the CBI in 2006 provides evidence that D grades were being accepted by employers then, while clearly higher grades than a C are also often required. So the situation is a lot more nuanced than the focus on C/D borderline pupils in schools would suggest. Grades confer relative, rather than absolute, advantage to students: employers (and higher education) tend to want students with the best grades, not always simply those above a somewhat arbitrary cut-off point.

Returning to the equivalences issue, the system really is comparing apples with bananas. It has to go, because the inflated value of non-GCSE courses has skewed schools’ decisions and has certainly meant some pupils have been pushed towards qualifications which are not as valued, in the reality of the employment and post-16 education worlds, as GCSEs, mainly because of the worth of the qualification to the school. That does not mean that these qualifications have not had value to individual pupils achieving them, by the way, just that the setting of the equivalences has clearly influenced schools’ decisions over who should take them. If the equivalence goes, it will be interesting to see how take-up of these popular non-GCSE qualifications fares; I would expect it to fall.

The detail of exactly how the Government sorts out this problem, though, is going to be interesting.

Monday, 27^th September

Michael Gove, the Education Secretary, today gave more details on the content of the forthcoming review of Key Stage 2 Sats.

In a letter to Russell Hobby, general secretary of the National Association of Head Teachers, Mr Gove said the review would “consider a number of key issues”, including:

-“How best to ensure that schools are properly accountable to pupils, parents and the taxpayer for the achievement and progress of every child, on the basis of objective and accurate assessments.” 

-“How to avoid, as far as possible, the risk of perverse incentives, over-rehearsal and reduced focus on productive learning.”

-“How to ensure that tests are as valid and reliable as possible, within an overall system of assessment (including teacher assessment) which provides the best possible picture of every child’s progress.”

-“How to ensure that performance information is used and interpreted appropriately within the accountability system by other agencies, increasing transparency and preserving accountability to parents, pupils and the taxpayer, while avoiding the risk of crude and narrow judgements being made.”

Mr Gove writes of sharing the NAHT’s concern that “too many schools [are] spending too much time on test preparation in year 6 at the expense of…productive teaching and learning”. This, he writes, is “clearly undesirable”.

He says that “raising standards and narrowing gaps are the central goals of the Government’s education policy.” These goals are “best achieved through ensuring that schools and teachers are free to set their own direction, trusted to exercise their professional discretion and accountable for the progress of the children in their care”.

He adds: “Mindful that the OECD concludes that external accountability is a key driver of improvement in education and particularly important for the least advantaged, the Government continues to view a system of objectively measuring pupil progress and holding schools to account as vital.”

Some system of key stage 2 testing, league tables and the use of the data generated by the tests to hold schools to account for their performance, then, looks to be a non-negotiable part of Mr Gove’s agenda.

The key question in all of this is whether it is possible to have a high-stakes system of school accountability which avoids the problems which Mr Gove acknowledges and seeks to mitigate.

I wonder, in particular, about the notion that schools will be “free to set their own direction” and “trusted to exercise their professional discretion” and yet remain accountable for their pupils’ progress.

If high-stakes accountability is realised in the way it has been in the past, in this and other countries, then the result will be that schools are judged on pupils’ performance on a set of fairly narrow indicators. If large consequences follow, for schools, on the basis of what a narrow set of indicators reveal, then expect a tendency to narrow teaching towards what is measured in those indicators to continue, no matter how much the Government talks of giving teachers freedom over teaching methods and even curricula.

In other words, if ministers talk about handing freedom to the profession, but retain tight control over the statistical measures by which the success and failure of schools is judged, and punish and reward them accordingly, real freedom will be limited.

It may be that real pressure from Government on schools to improve their results will only be exerted on those who rank towards the bottom on the chosen performance indicators, as the letter says “the accountability system…must be able to identify and tackle cases of sustained under-performance”. But if this is the case, it risks setting up a gap between schools at the bottom on these rankings, where teaching is likely to be very narrowly focused on what is needed to improve the indicators, and the rest.

When Mr Gove talks about “raising standards and narrowing gaps” being his over-riding goals, I also take that as translating to raising test scores, and narrowing the performance gaps between particular groups of pupils, as measured by test scores. This, at least, is what the terms have meant under Labour, and I get no sense here that some of the problems inherent in this approach, around, again, the narrowing of focus on specific indicators, are being acknowledged.

Being hyper-sceptical, I also wonder whether holding schools to account for the “achievement and progress of every child”, which is a continuation of the policy under Labour, is not a step too far, in that it takes away the child’s own responsibility to try to do better. That is, if every time a child fails to make progress, the school is blamed, where is the child’s place in achieving a good result?

 Of the four bullet points, I think the fourth (“How to ensure that performance information is used and interpreted appropriately within the accountability system by other agencies, increasing transparency and preserving accountability to parents, pupils and the taxpayer, while avoiding the risk of crude and narrow judgements being made.”) represents the best hope of genuine improvements over the current system.

If the new government were genuinely committed to trying to educate parents and other users of test result data what can be read into the statistics, and what cannot, this would be a step forward.

Overall, though, it looks as if what is being proposed will still contain many of the ingredients of the current system, with weight being put on a limited range of statistical indicators, at least for some aspects of school accountability.

Mitigating some of the problems of high-stakes accountability is important. And it may be that Mr Gove, possibly working with the NAHT, will uncover new ways of trying to square the circle of having a system of high-consequence accountability without some of the downsides which have dogged it in the past. But there is no detail as of yet of how that can happen, so I remain to be convinced.

 While I understand the pressure on politicians over accountability, I still find it disappointing that some kind of statistics-based system seems to be non-negotiable, with the quality of the learning experience for children seemingly something which must be worked in around it.

Friday, September 17^th

An explosive – in education policy terms, at least – new book published today offers a series of highly newsworthy insights into the political process.

“Reinventing schools, reforming teaching” by John Bangs, who until this summer was head of education at the National Union of Teachers, and the Cambridge professors John MacBeath and Maurice Galton, comes full of quotations from some of the leading movers-and-shakers of the New Labour years.

Anyone reading these extracts will end up distinctly unimpressed with the way education and politics have intersected over the last 15 years.

Among the book’s revelations, some of which I comment on below, are:

-A claim by  Mick Waters, former director of curriculum at the Qualifications and Curriculum Authority, that the exams system is “diseased, almost corrupt.” Exam boards, competing in a market place for schools’ business, sometimes implied to their customers that their exam was easier than their rivals’, he said. He also criticised the boards for endorsing/publishing textbooks on their own exams.

Mr Waters says: “Before I went for this job, I used to think that all this criticism of exams that they were being dumbed-down was unfair. You know, the old argument, more people passed than ever before. Since I’ve been there, I think the system is diseased, almost corrupt. I don’t mean QCA or Ofqual or anybody. We’ve got a set of Awarding Bodies who are in a market place. In previous jobs, I had seen people from Awarding Bodies talk to headteachers implying that their examinations are easier. Not only that, we provide the textbooks to help you through it.”

I think whether you describe the system whereby competing exam boards seek the business of schools, who are working to the demands of politicians, all of whom have some interest in results rising or at least not falling as “corrupt” or not, it is not healthy. In my book, I quote the board Edexcel, which in 2006 advertised a range of new science GCSEs to teachers in the following way: “More chances to succeed. Our curriculum enables all students to perform to their best…students can be assessed at any time, allowing them to be tested on material when it’s fresh, and can take multiple tests before submitting their best performance”.

Last year, I reported how Edexcel, in a teachers’ guide to an engineering GCSE says: “Find out why Edexcel is your best choice for better results.”

While, as I say in the book, in many ways England’s exam system is admirably well administered, bringing enviable technical knowledge to the process, it is crazy that this has been allowed to happen. It is akin, perhaps, to a football referee saying “play in my games, and you’ll score lots of goals”. To understate, it should not be exam boards’ role.

Mr Bangs suggested at the press conference to launch the book yesterday that the solution was to move to a single exam board. I think this would also have downsides, however. The structural problem with the exams system is that those with influence both on the “demand” side (schools) and “supply” side (exam boards and the government) all have an interest in seeing grades rise (or in the case of competing boards, at least not to have their exams seen as harder than their rivals’).

For me, as has been said before, a better solution would be to ensure that voices outside of the chase for grades – a strong regulator, and those who will use the qualifications as checks on what young people understand, such as employers and universities – are given more influence. The Government should also be taken out of the process by using sample tests as the main mechanism by which national standards are checked, so that the chase for better grades – which does not benefit pupils when it happens nationwide – becomes less of an end in itself.

Right, back to the book…

-Sandy Adamson, former head of the standards division in Labour’s Standards and Effectiveness Unit, criticises primary test targets as “stupid”.

He says: “It was the delivery mechanism that was the problem and the stupidity of targets, unobtainable targets, simply pulled from the air and then applied to every school in the country. Twenty thousand primary schools and the majority of reaction was ‘this is unachievable’ and for a substantial minority it was never achieved.”

Mr Adamson also claims Labour lost the confidence of the teaching profession “in the first 12 to 18 months”. “Talked, talked, talked at them. Command and control. Told them what had to be done and the way it was going to be done.”

-Estelle Morris, the former education secretary, justifies Labour’s policy of “naming and shaming” struggling schools in political terms.

She says: “I think naming and shaming of schools gave two clear messages; in the eyes of the public politically it put it on the side of the users of the services, not the producers of the services; and secondly it gave the message to the teaching profession that we weren’t the same Labour party as last time we came into power but we would have a different focus.”

To this observer, she seems to be saying: schools were used to serve politicians’ needs. Am I alone in finding this, put in those terms, as outrageous?

It is, though, in line with a statement in the 2007 book by Sir Michael Barber, the former head of Tony Blair’s “Delivery Unit”. He cites the “name and shame” plan as evidence that the party would be “hard as nails” [with teachers].

- Former Downing Street speechwriter Peter Hyman said that Tony Blair believed that Alastair Campbell’s “bog standard comprehensive” remark had helped give the party “some definition”.

-Sir Mike Tomlinson, who led the two-year review of secondary qualifications whose central recommendation was rejected by Tony Blair on the eve of the 2005 general election, says there is “nothing rational about decision-making and policy-making at all”.

- Kevan Collins, former head of the Primary National Strategy, says the introduction of academies in the early 2000s “segmented the profession”, undermining an attempt to focus on “universal language of teaching and learning” as the focus moved towards “creating certain types of schools”.

- Stephen Byers, another former school standards minister, is also reported as having written to Professor Robin Alexander, the Cambridge academic, “expressing his welcome for the co-operation of academics but attended by the caveat – only to the degree that they supported government policies.”

- Mr Bangs also told the press conference yesterday that the book shows none of his interviewees, from across the political spectrum have confidence in Labour’s Every Child Matters/Children’s Services agenda.

He did praise aspects of New Labour’s record on education, including its school buildings programme and, possibly, its literacy and numeracy strategies. But…well, there is a lot to chew over here. I hope to post a review of the book here in the next few weeks.

Monday, July 26^th

Most journalists –I like to think, at least – try to probe beneath the surface a little bit in seeking to get some insight into what is going on. Sometimes, this process can be frustrating, especially when evidence comes in which does not fit a pre-conception, hypothesis…or strong news line.

Late last week, I had some experience of this when doing some number-crunching on this year’s national test results.

I wanted to find out if the thousands of schools boycotting this year’s tests – some 26 per cent of the total – would have any effect on the national data generated at the end of the process.

My thinking went along the following lines. I knew that strenuous efforts were made by the testing authorities – principally, the Qualifications and Curriculum Development Agency (QCDA) – to hold the standard of the tests constant every year.

That is: the QCDA oversees a complex process, including “pre-testing”, whereby it is supposed to ensure that it is just as easy – or difficult – for a child of a certain ability and understanding of their subject to achieve a certain level one year, as it would be the next.

When entire national cohorts of 11-year-olds take the tests every year, if the difficulty* of the tests is held constant, then the numbers passing them should give a good idea of the overall national standard of understanding of that particular cohort, in the tested parts of each subject at least, it is argued.

However, what if, in the case of a boycott, the average ability** levels of pupils taking the tests in one year changed? What if pupils from the boycotting schools tended to come overwhelmingly from those who would have been expected to achieve the Government’s target level very easily? In that case, assuming there was no major change in the overall national ability profile compared to the previous year, the sample of pupils taking the tests would have been skewed, because the number of high-achieving pupils taking the tests would have been reduced because many would have been part of the boycott.

If the tests were just as hard as in previous years, results would then fall because of the number of likely high achievers taken out of the test-taking “sample”. But this would say more about the effect of the boycott than it would reflect any fall in national standards overall.

Conversely, if the boycotting schools tended to have a pupil population of lower-than-average ability in the tested subjects, then results might rise as the proportion of high-ability youngsters taking the tests went up.

A couple of weeks ago, after getting hold of the list of schools which boycotted the tests, I began to wonder if the second scenario might hold true. I wrote an analysis piece which ranked the regions of England on take-up for the boycott.

Although finding precise patterns was tricky, it did seem that regions with higher proportions of pupils eligible for free school meals tended to have higher support for the boycott, while those with lower free school meal eligibility tended to have higher numbers of pupils taking Sats as normal.

With pupils eligible for free school meals established, by a great deal of research, as less likely to do well in tests, I thought this might translate to mean that the boycott disproportionately took out from the test-taking sample children who might have been expected, on average, not to do well in the tests.

If this were the case, I thought, even if the results of those taking the tests rose, this might say less about national standards rising, and more about the effects of the boycott.

But this was only a hypothesis. Fortunately, it was testable, if only in a rough way. I got hold of the national test results for 2009 of all primary-age schools in England. I then checked to see if the schools which boycotted the tests this year had higher-than-average results in 2009, or lower-than-average.

And the answer? Well, to my surprise their pupils’ performance, at least in terms of the numbers achieving the Government’s “expected” level four, was almost spot on the national average last year by my calculations.

Nationally, 80 per cent of pupils achieved level four in English, 79 per cent in maths and 88 per cent in science in 2009. Among the schools which boycotted the tests this year, the respective percentages were 79.6, 79.2 and 88.5. The average points score, which the Government calculates by simply adding up the percentage of level fours across all three subjects in each school, was 247 nationally last year and 247.4 in the 2010 boycott schools, I found.

So, there I was expecting to write about how the sample had been skewed massively by the effect of the boycott, and how we should be able to read even less into the national scores than ever this year. Yet the numbers appear not to support such a conclusion.

That said, this may just be luck on the part of the Government. I have checked with the QCDA, whose answer suggests to me that it did not investigate the characteristics of pupils in schools boycotting the tests this year.

It said that its job was to ensure the tests were kept at a constant difficulty level each year and that this process was not affected by the industrial action.

In other words, if for any reason only a proportion of the cohort took the tests in any year and this affected what could be read into the results as a guide to overall national standards, this was not a matter for the QCDA because it was concerned with the difficulty of the tests themselves, rather than the nature of the cohort taking them.

I find that answer strengthening my belief in the merits of national sample tests/assessments as measures of overall standards for England as a whole. Under this system, by design every year the testing authorities would select only a proportion of pupils to take tests, but they would do so in such a way as to ensure it was nationally representative. In other words, they would have to control the sample of pupils taking the tests each year to ensure the results could be read as standing for England as a whole.

By contrast, if the Government sticks with the current testing model, any future boycott action could have an effect on what could be read into the results as guides to national standards.

I think this is more evidence that the problem with the current system is that there are multiple purposes for these tests: they are not designed solely as a check on national standards, of course, but to check on schools, teachers and pupils, as well as providing evidence for many other aspects of education. So the QCDA seeks to ensure that the tests are comparable to previous years for pupils and schools, but sees it as not its job to investigate the possibility of any effect on national standards caused by the sample of pupils taking the tests changing.

The Department for Education told me: “Results from the 2010 Key Stage 2 National Curriculum Tests will be published on 3 August. This publication will include any relevant commentary about the impact of industrial action on the results, including issues of representativeness.”

That looks to me as if there will be some disclaimer offered by the Government next week about being cautious about how much can be read into this year’s results as indicators of national standards, because of the boycott. My number-crunching suggests one should be cautious, but perhaps not as cautious as I thought before looking a bit more closely.

*By “difficulty”, I mean not just how hard the questions are, but how easy or hard it is for a pupil of a given ability to achieve a certain level on a paper each year.

** I am using the word “ability” loosely here, to mean the overall likely capability of an 11-year-old to do well in the national tests. I realise that there are debates over whether ability is fixed from a young age, or, of course, whether test-taking ability reflects overall understanding of a subject.