Friday, June 25th

How long did Department for Education officials and ministers spend cobbling together the official document they have published assessing the likely impact of their new academies policy?

Not nearly long enough, to judge from what looks like the thrown-together character of this paper, and some truly heroic assumptions on which some of the calculations it comes up with appear to be based.

The paper is an “impact assessment” of the academies bill, which is currently going through Parliament. The idea behind this sounds sensible: when policies come out, the government is supposed to show that it has thought through the implications of them, and considered alternatives.

So far, so good, then. But problems begin to appear for this document in the first paragraph. It opens with the questions: “What is the problem under consideration? Why is government intervention necessary?”

The answer begins: “As part of their [sic] pre-election commitments, the Government said that outstanding maintained schools would be allowed to adopt Academy status by September 2010.”

Erm…this wouldn’t be both parties within the coalition government, would it? In fact, the commitment appears to be a reference to the Conservative manifesto, which said “all existing schools will have the chance to achieve academy status, with outstanding schools pre-approved”. There is no mention, in the Conservative manifesto, however, of the September 2010 date, so the document seems to have got that factually wrong. Leaving that aspect of detail aside, clearly, the Liberal Democrat manifesto is not relevant here, as it says conventional academies should be replaced by “sponsor-managed schools”, accountable to local authorities. The equation of coalition government policy with what appear to be Conservative manifesto commitments occurs several times in the document, which states later on page 1 that the policy “is a Government manifesto pledge”.

Later on page 1, the question is posed: “When will the policy be reviewed to establish its impact and the extent to which the policy objectives have been achieved?” The answer? “It will be reviewed.” So that’s that sorted out, then. An annex at the back of the paper leaves space for the government to fill in a “Post Implementation Review Plan”, with information to be added under seven categories. All have been left blank.

There are other strange aspects of this 11-page paper. Its date is listed as effective from 01/01/2010, rather than the period on or after the coalition government’s formation, which is bizarre. (The document actually bears a signature which could be that of Michael Gove, education secretary, which itself is dated 26/5/2010, which was the day after the Queen’s Speech, when the Academies Bill began its passage through Parliament.)

Against a box in which the department is asked to sum up the policy’s impact on equalities  – a note says that “race, disability and gender impact assessments are statutory requirements for relevant policies” -  the document says there are no implications. Again, it seems bizarrely, and worryingly, to this observer, counter-intuitive to have reached this position with no evidence offered: what if “outstanding” schools being offered academy status first disproportionately educated children from one ethnic group, for example? To take this document at face value, the government deems this as not worthy of consideration. The assessment also ticks “no” in a section asking the government to identify nine other possible categories of effects, including on small firms, greenhouse gas emissions and human rights.

Under “evidence base”, the document says that academies are freer from regulation than conventional state schools, are independent of local authorities and thus have “far greater autonomy than maintained schools”. They are able to take decisions on priorities “to suit the pupils in the Academy”; are not constrained by national teacher pay structures; and can form partnerships that work for them and their pupils “without constraint”.

“These as part of a broader package of freedom flowing from autonomy, mean that Academies are freer than other schools to focus their efforts on teaching and learning and push up attainment as a consequence.”

Hmm, whatever the details of how particular freedoms may benefit academies, I don’t think it’s true in any general sense that academies have more freedom to “concentrate on teaching and learning” than other schools.

But it is the figures on the likely results improvements at academies, and the highly convoluted arguments about economic benefits that could follow that I want most to focus on here.

The paper attempts to come up with a cost-benefit analysis for the policy. On the debit side, it reckons that “total annual economic costs” flowing from setting up and running another 200 academies a year in 2010-14 will be £462 million. “Total economic costs” (I confess after reading the document I cannot tell the difference between these two terms) are put at £530 million.

But on the credit side, the document estimates that the academies will produce economic benefits  – to those individuals educated within these institutions, I think – of  £1.072 billion.

The chain of reasoning seems to go as follows.

First, academies have been shown to improve GCSE results, the document says. The paper assumes that a school changing its status from mainstream to academy will generate a 1.5 percentage point improvement in the proportion of its pupils achieving five or more GCSEs, including English and maths.

I cannot quite see, from the paper itself, where this figure has come from. Earlier in the document, it states that 63 academies which had results for both 2008 and 2009 had been found to have improved their five-or-more GCSE including E and M rate by five percentage points, which was twice the national average.

Explaining why the 1.5 percentage point figure had been chosen, the paper says that the figure had been “down-rated by half” to allow for the fact that new academies would be different from those covered by the 2008 and 2009 figures. 1.5 per cent, of course, is less than a third of the five percentage point increase shown by the academies in 2008 and 2009, so this is confusing-or have they taken the 2.5 percentage points that academies improved above the national average, and then more-or-less halved that? Having read the document several times, I don’t know.

The paper then attempts to assign an economic effect to an individual achieving five or more good GCSEs (there is no mention of this having to include English and maths in this section), as opposed, presumably, to not achieving this benchmark. It puts a figure on this as an extra £100,000 in “lifetime productivity” (earnings, I guess), for men, and £85,000 for women.

This, says a note, is based on an “internal DfE analysis” of three existing studies, which seem to focus mainly on the returns for individuals of gaining vocational qualifications.

It appears that this figure is then multiplied by the number of extra pupils that might be expected to gain five good GCSEs, including English and maths, as a result of the academies policy (the document assumes there will be 200 new academies each year to 2014), to produce an overall economic benefit arising from academies of £282 million per year, or £1.072 billion over four years, after taking into account inflation.

It is difficult to know where to start with all of this, without denouncing it as pie-in-the-sky rubbish.

The previous government’s own, largely sympathetic, evaluation of the academies policy, by PricewaterhouseCoopers, found, in November 2008, that “there is insufficient evidence to make a definitive judgement about academies as a model of school improvement”. How the new government can therefore assume that changing schools’ status to academies, in itself, would bring about a 1.5 percentage point improvement above what would have been achieved without it is puzzling.

And trying to predict improvements in academic results in the “new” academies – which seem likely to be drawn from among the more successful schools with, probably, better results as a starting point – on the basis of what predecessors which tended to come from the lower end of league tables achieved, seems especially perilous.

To be fair to the document, it does say towards the end that the figures “may be an overestimate” of the likely results gain, since the new academies “will have less scope for improvement than existing Academies” (presumably because their results are already higher) and “will receive much less start-up funding”.

Fair enough, except that on page 2, it stresses that: “Future benefits are assessed as only half those that have been achieved in past academies. Benefits are therefore likely to be underestimated.” (my italics)

Really, reading this document, you wonder why they even bother doing these impact assessments. I wonder if it has been done in a terrible rush in the early days of the new government, to try to get some political momentum behind the new policy. Whatever, if this is the level of analysis that has gone into this policy, there could be trouble ahead.

You can read the impact assessment in all its glory at: http://www.education.gov.uk/~/media/Files/lacuna/academiesbillimpactassessment.ashx

- Warwick Mansell

Friday, June 18th

The “free schools” policy discussed on the radio this morning, and being launched in detail today by Michael Gove, is very interesting.

It does pose legitimate questions about the influence of the state over what goes on in classrooms, and about the balance between national and local government power over what goes on in schools, professional freedom over decision-making, and parental decision-making. The most immediate questions centre on how the schools will be funded.

I want to blog more on this subject in the coming weeks.

However, I thought now I would just post on one particular aspect of the free schools policy which Mr Gove alluded to again on the Today programme this morning: its inspiration, at least in part, from the American charter schools policy, and particularly one set of charter schools, the Knowledge is Power Program (KIPP), set up by two teachers.

KIPP schools are seen as among the most successful charter schools in the US, and have been referred to enthusiastically by Mr Gove. He may be right; they certainly have a good record. But there are other debates around these schools. Having read Diane Ravitch’s fascinating book on the US education system (see my last blog), I thought I would simply quote at length her section on KIPP schools. I have not checked her facts myself; but it is another perspective that is worthy of consideration, I think, in this debate. Ravitch is a former assistant secretary of state for education  in the US, and now a research professor of education at New York University and a columnist for Education Week newspaper.

Ravitch writes (page 135): “The charter schools with the most impressive record of success are the KIPP (Knowledge Is Power Program) schools, which have been called culture-changing schools, because they aim to teach students not just academics but also self-discipline and good behavior. KIPP was launched in 1994 by two teachers, David Levin and Michael Feinberg, after they completed their two-year assignment in the Teach for America program in Houston. Feinberg opened a KIPP school in Houston, and Levin opened one in the South Bronx in New York City. Both schools achieved exceptional results. Generously funded by foundations, Levin and Feinberg opened dozens more KIPP schools across the nation, specifically to prepare poor minority students for college. Fifteen years after the organization was founded, there were eighty-two KIPP schools with approximately 20,000 students.

Almost every KIPP school is a charter school, and most are middle schools (grades five through eight). In contrast to regular public schools, KIPP schools have longer days (nine and a half hours), some Saturday classes, and three weeks of summer school; typically, a KIPP school provides 60 percent more time in school than a regular public school. Students, parents, and teachers sign a contract agreeing to fulfill specific responsibilities. The central organization does not define KIPP’s pedagogy and curriculum; it leaves these decisions to individual school leaders.

In the demands they make on students, teachers, and parents, the KIPP schools are reminiscent of the American public schools of the 1940s, or even the 1920s, before the onset of class-action lawsuits and union contracts. In those days, it was not unusual to encounter schools with strict disciplinary codes and long working hours (though not nine-and-a-half-hour days).

Despite its successes, KIPP has its detractors. Critics question the applicability of the KIPP model to public education in general. One persistent question is whether KIPP enrolls all kinds of students, as regular public schools must. Like other successful charter schools, KIPP admits students by lottery; by definition, only the most motivated families apply for a slot. Charters with lotteries tend to attract the best students in poor neighborhoods, leaving the public schools in the same neighborhood worse off because they have lost some of their top-performing students. They also tend to enrol fewer of the students with high needs—English-language learners and those needing special education.

 The students who remain in KIPP schools for four or more years tend to achieve large test score gains. Most KIPP schools consistently outperform traditional public schools in the same neighborhood. But KIPP schools often have a high attrition rate. Apparently many students and their parents are unable or unwilling to comply with KIPP’s stringent demands. A 2008 study of KIPP schools in San Francisco’s Bay Area found that 60 percent of the students who started in fifth grade were gone by the end of eighth grade. The students who quit tended to be lower-performing students. The exit of such a large proportion of low-performing students—for whatever reasons—makes it difficult to analyze the performance of KIPP students in higher grades. In addition, teacher turnover is high at KIPP schools, as well as other charter schools, no doubt because of the unusually long hours. Thus, while the KIPP schools obtain impressive results for the students who remain enrolled for four years, the high levels of student attrition and teacher turnover raise questions about the applicability of the KIPP model to the regular public schools.

KIPP has demonstrated that youngsters from some of the toughest neighborhoods in the nation can succeed in a safe and structured environment, if they have supportive parents and are willing to work hard, spend long days in school, and comply with the school’s expectations. Thus far, public schools have not copied their methods. Regular public schools must accept everyone who applies, including the students who leave KIPP schools. They can’t throw out the kids who do not work hard or the kids who have many absences or the kids who are disrespectful or the kids whose parents are absent or inattentive. They have to find ways to educate even those students who don’t want to be there. That’s the dilemma of public education.

The theory of the charter movement is that competition with the regular public schools will lead to improvements in both sectors, and that choice is a rising tide that lifts all boats. But in reality, the regular public schools are at a huge disadvantage in competition with charter schools. It is not only because charter schools may attract the most motivated students, may discharge laggards, and may enforce a tough disciplinary code, but also because the charters often get additional financial resources from their corporate sponsors, enabling them to offer smaller classes, after-school and enrichment activities, and laptop computers for every student. Many charter schools enforce discipline codes that would likely be challenged in court if they were adopted in regular public schools; and because charter schools are schools of choice, they find it easier to avoid, eliminate, or counsel out low-performing and disruptive students.

Yet, even with their advantages, charter schools—like all new schools—face daunting challenges. Reformers declare their intention to open new schools as though this would solve the problems of low performing schools. But new schools cannot be mass-produced or turned out with a cookie-cutter design. Opening a new school is difficult. It involves starting with or recruiting a strong leader and a capable faculty, obtaining facilities, developing a program, assembling a student body, creating an effective administrative structure, and building a culture. Getting a new school up and running may take as many as five years. Some will succeed, some will be no different from the schools they replaced, and others will fail.”

Ravitch was initially an enthusiast for a concept of charter schools which saw them as giving teachers the chance to take more professional control over what happened in the classroom. However, she now also has reservations about how this has turned out in practice in the American context. This is another aspect of the coalition’s free schools policy which was heavily discussed this morning, which I want to write about in the coming days.

- Warwick Mansell