Mr Balls told MPs on December 16th: “There was a systemic failure by NAA and QCA executives to assess risks properly and then to pass these risks to the board and to ministers.”

He added: “The delivery failure was by the QCA and the NAA, and Lord Sutherland is very clear on that point. Ken Boston said in his evidence that the information that was being made available to him was also being made available to Ministers and our Department. The problem was-as the report makes clear-that he did not have a management system in place that meant that concerns in the NAA were escalated up to him. In fact, months earlier, NAA had a red risk rating and put scores of its own staff in to run the ETS contract, while telling Ken Boston that things were okay. But when he knew about the red risk ratings he did not act, and he assured us that things were okay. We did not have all the same information, but in any case it was his responsibility to act and he did not do so.”

At the select committee hearing, Dr Boston said ministers had “sexed up” the evidence against him.

 In a letter to Barry Sheerman, the committee’s chairman, he cited Mr Knight’s recollection of the meeting of June 17^th, as quoted in the Sutherland report.

In his evidence to Sutherland, Mr Knight said: “We had that meeting on the 17 June with Ken [Boston] and David Gee. I may well have met David Gee before that point but it was the first time I had had a substantial discussion with him about things in great detail. He basically answered all the questions; Ken [Boston] referred everything to him. We went through everything with David [Gee].”

 In his letter, Dr Boston said that the account of this meeting was a “fiction”. He had not attended it. Furthermore, he told the committee, DCSF officials who had attended it should have corrected the contents of the Sutherland report to make this clear after they had seen a draft of the report, and before it was published.

In his letter, Dr Boston went on to cite another paragraph from the Sutherland report, which said:

“In practice what happened in 2008 was that DCSF observers escalated their own assessment of risks [in relation to test marking] to the DCSF ministers on a number of occasions. On this basis, ministers usually pressed QCA’s Chief Executive for answers. At this point, because information was not being escalated within QCA effectively, ministers were given strong reassurances, by QCA that all was on track. As late as 17 June when the Schools Minister met QCA’s Chief Executive and NAA’s Managing Director, they provided reassurances.”

 Dr Boston said, in his letter: “This too is fiction. Not only was I not present at the June 17 meeting, but until the delivery failure at the end of June I had had only two meetings with the DCSF ministers in 2008…Nor was I being ‘pressed’ by Ministers for answers on the telephone or by email.”

 He went on: “The flawed evidence…has been used to portray me as complacent, disengaged and constantly beleaguered by ministers with questions I was unable to answer. This is far from the truth; it was not corrected by ministers or DCSF officials at draft [stage of the Sutherland report]; and it has been used by Ministers to my serious disadvantage”.

 After an initial 10-minute statement to the MPs, of which more below, Dr Boston was told by the committee chairman, Barry Sheerman, that Mr Knight had written to the committee in February (two months after the Sutherland report’s publication) to say that he had made an error in  his evidence. Dr Boston was also informed of a response in which Lord Sutherland had said, said Mr Sheerman, that “he did not think this made much difference to his report”.

 Dr Boston appeared to have been taken by surprise by Mr Knight’s letter. He said: “I think it was disrespectful that I was misrepresented by a minister in a public report and I have not been contacted by the minister. I have not been given any information on this at all. I think that’s outrageous.”

 The Sutherland Inquiry website now carries the notice: “At paragraph 4.92, the Schools Minister has subsequently clarified that on 17 June 2008, he met with David Gee alone. His recollection of Dr Boston’s role related to a separate meeting on 2 July 2008.”

 Dr Boston added that ministers knew on the 26^th of June “that there was a failure”, and not the 30^th of June that was reported in the Sutherland report. He added that he had advised the minister to stick with the July 8^th date for issuing the results even though this would have meant some schools would not have received them on that date. Ministers, though, had overruled him.

 Summing up, under questioning from the MPs, Dr Boston said: “What I resent is evidence against me being sexed up in a report by Lord Sutherland on the basis of falsified evidence, given by ministers, to characterise me as something I am not. I failed to deliver, and I resigned. That’s it.”

 I will leave readers of this blog to make up their own minds about this exchange.

 One thing I would add, however, is that Mr Balls did have some detailed knowledge of what was going on as early as June 2^nd.

 A marker, who I am going to have to keep anonymous, wrote to the Secretary of State in April about problems with the organisation of training events for markers, the number of scripts they were being asked to mark and the fact that they were having to enter every mark, for every pupil,  individually at the computer screen.

On June 2^nd, having had only an automated response from Mr Balls’s office, he sent the schools secretary a further five-page letter with a list of more difficulties, including markers’ inability to get through to ETS on the telephone, what he saw as inadequate checks on markers’ quality and the fact that some examiners were still waiting for pupils’ scripts, four weeks after the tests were taken. He informed Mr Balls he had resigned as a marker a result.

Mr Balls wrote back, on June 6^th, to say, in a letter that I have in front of me: “I am sorry that you have experienced problems with test delivery this year and that you have resigned as a marker. These matters are the responsibility of the Qualifications and Curriculum Authority and I have personally asked Ken Boston, chief executive, QCA, to deal with your current issue and write directly to you.”

 Mr Balls went on: “I am aware that there have been problems surrounding this year’s exercise and that the National Assessment Agency (NAA) and their contractor ETS Europe have had to make a number of changes.

 The helpline set up by ETS Europe has received four times the envisaged number of calls and as a result many callers were unable to get through. This was acknowledged by both the NAA and ETS Europe and as a result more lines and advisors have been made available.

 ”I am sorry to hear you are not happy with the introduction of the online capture of marks which has been introduced this year. This change has been made so that more accurate data will be produced for schools and the system will be easier for markers. In acknowledgement of the time needed to input the data the payment to markers for this task has been increased from 10 pence per script to 25 pence.

Delivery of the NC tests is always challenging in view of the scale and timing of the task and it is most unfortunate that this year’s exercise has encountered problems. The NAA and ETS Europe are working hard to make sure that the remainder of the exercise runs smoothly.

 ”Thank you, once again for bringing your concerns to my attention.”

 This letter suggests that ministers were fully aware of the detail of what was going wrong as of early June. However, it also suggests that if Mr Balls did indeed ask Dr Boston to look into the letter-writer’s complaints, the Schools Secretary’s contention that he had been “pressing” Dr Boston – or at least that some within his department were pressing the QCA or NAA – for answers might have been correct.

 There are several other comments made by Dr Boston which, if they are taken at face value, are extremely revealing.

 Dr Boston said:

• - That Lord Sutherland’s inquiry was too “narrow”, because it was restricted to assessing the involvement of the QCA/NAA and its contractor, ETS. Yet, he said, the Government had great influence in many decisions which affect testing, from the nature of the tests themselves to whether they are marked manually or on-screen, the amount of money to be spent on the tests, which results data are collected and the interval between the tests being taken and the results being issued. Dr Boston suggested that the DCSF had taken the decision that markers would input individual question-level data at the computer screen – resulting in the “several thousand [mouse] clicks” which so frustrated the marker who wrote to Mr Balls. He also said that the then QCA chairman, Sir Anthony Greener, had only signed the contract approving the appointment of the contractor (ETS) which failed in 2008 “after he had the then Secretary of State’s [Ruth Kelly] approval for it to be signed”.The implication, then, was that Lord Sutherland had being prevented from doing his job properly because he had been unable properly to consider any culpability of the DCSF for what went wrong.

 

• - That “decisions taken by ministers and officials in 2006 had a major impact in the failure of 2008″. Specifically, he said, this involved the rejection by the Government of a recommendation by the QCA in 2005 that the tests be marked on-screen by examiners, rather than on paper. This was a recommendation of the Sutherland report and Dr Boston said it would have helped avert the problems of 2008. He later said that the 2008 results would have been issued on time if there had been on-screen marking last year.

 

• - That ministers had rejected the recommendation to go to on-screen marking because they were concerned that this would have an unpredictable effect on national test results.

 

• - That the fact that Department for Children, Schools and Families officials sit in on QCA board meetings, and those of other quangos, “has undercut the authority of the QCA, and will undercut the authority of Ofqual if we are not careful”, by “reducing the formality of relations” between the QCA and the Government, “leading to negotiated compromises from time to time which erode public accountability”. In board meetings, he said, officials greeted proposals from the board with statements such as “ministers would be minded to or not minded to agree with this or that proposal, or that they would be content to or not be content to agree to that particular recommendation. Or even: ‘If I had put that proposal to a minister, it would be laughed out of court’”. Dr Boston added: “Too often, boards like the QCA are put in a position where they are asked to negotiate the advice [they give to ministers] in advance. I think that is a pernicious process which compromises independence.”

 

• - That ministers would want to avoid any independent inquiry into the purpose and use of the key stage tests, but that Ofqual should take that on.

 

• - That David Bell, the permanent secretary at the DCSF, had “sought to negotiate my early resignation” as early as September last year, with a prediction of what the Sutherland report would contain. Dr Boston said: “On the 24^th of September, three weeks before Balls and Knight gave evidence [to Sutherland] and nearly three months before the Sutherland report was published, and certainly before Sutherland had ever put pen to paper, the permanent secretary sought to negotiate my early resignation on the grounds that the Sutherland report would be bad for me. Ministers had not even given their evidence. He [Mr Bell] said to me that he wanted me out of that job before March.” Dr Boston then said that he had engaged an employment lawyer, and then told the Government that he intended to see out his contract and that he would only leave on the basis that the contract was paid up. The correspondence ended after that. Dr Boston added: “I was very clear that by the time the two ministers gave their evidence [on September 14^th and 15^th], the skids were under me, or it was proposed to put the skids under me at departmental level. I think that goes back to the meeting of the 18^th March [2008] and the issue of the future of the QCA.”

 

• - That David Bell had also sought his suspension from the QCA, in December. Dr Boston said he had resigned on December 12, a day after he first became aware of the “error” in the Sutherland report surrounding the details of the meeting of June 17th. Dr Boston then added: “I was then, on the following Monday, the 15^th, the day before the Sutherland report, suspended by the QCA board on the proposal of the DCSF permanent secretary. I contested that, or reserved the right to contest it, because I should have been put on gardening leave, rather than being put on some ill-defined suspension.”

 

• - That there were tensions between the QCA and the Government, with ministers concerned about Dr Boston speaking out. Dr Boston said there had been an intention among ministers of taking the “strategic” role away from the QCA. Now, the “authority” part of its name is to be removed in favour of the word “agency”, as it becomes the Qualifications and Curriculum Development Agency. Dr Boston said: “This became clear in a meeting I had with Jim Knight and Ed Balls on the 13^th March [2008]. One of the great sources of agitation [for ministers] had been the way in which I had played my role, and allowed other senior people to play their roles, including particularly, perhaps, Mick Waters.” Mr Waters, the QCA’s former head of curriculum, won the favour of many teachers through a series of roadshows around the country in 2005 to 2007 in which he sold the virtues of a new secondary curriculum which sought to give schools more freedom over what they teach, although he was not popular with some subject specialists and, it seems, the Government. Mr Boston added: “I have seen the QCA as a statutory authority under the 1997 [education] Act, which is not to be a critic of Government, but has to be a constructive participant in education discussion nationally, to lead part of that discussion, but not to be bound by a set of speeches that are basically constructed around the Government press release of the day, never publically criticising Government but being prepared to shoot the breeze about other ways of doing things. And I, and Mick, in particular sought to do that in relation to the curriculum and, indeed, on occasion in relation to assessment and modernisation. The QCDA will not have that capacity, it will be virtually an arm of the department and my personal view is why not incorporate it within the department? What it is going to be is a body which is at arms’ length in one sense, but not able to do anything in terms of having a public role.”

 

 

There is a huge amount here on which some will want to comment. I would just say that in general the evidence, if it accepted at face value, paints a picture of a system in which ministers and their officials appear to wield huge power. But because this power is usually hidden from public view, it is largely without ultimate responsibility or accountability. Some might say the ultimate accountability lies in the ballot box, in the power to vote out elected politicians. But are voters really clear what they are voting about, when the basis on which decisions are taken is not being made clear to them? (See my earlier blog post, based on an article written after the Sutherland report, here).

 Although Dr Boston is undoubtedly right to have resigned over last summer’s fiasco, the wider issue of the role of independent voices within the policy-making process is important. It is a shame if, as his evidence suggests, experienced, knowledgeable professionals are indeed frozen out of that system if they dare to depart from a Government script. English education will be the poorer for it. That said, notional but, perhaps,  independent-minded “insiders” such as Dr Boston are hardly alone in their treatment, as the Government’s recent dismissive reaction to Robin Alexander’s painstaking Primary Review illustrates.

 As I understand it, the role of “observers” in the meetings of quangos has been well-established since at least the Thatcher era, and that it is routine for them to intervene in meetings with guidance on what would or would not be acceptable to ministers. They therefore have great influence on the deliberations of these bodies, before the bodies even get to make formal “independent” recommendations to Government. This advice, though, never seems to be minuted, so the public will have been unaware that it goes on.

 I think many teachers might find this picture of what has gone on hard to stomach, given the intrusive nature of the accountability regime they are saddled with. Accountability seems a very fuzzy concept in a system where the Government in reality has power and influence behind the scenes, but is largely absent when things go wrong. One rule for our rulers, it seems, and another rule for everyone else.

 The video of Ken Boston’s evidence session is here:

 

 

 

 

 

 

- Warwick Mansell

Bearing in mind the evidence from Ken Boston at the Children Schools and Families select committee, about which I hope to make a posting later in the week, I thought I’d post the article I wrote on the Sutherland inquiry at the end of December (for publication in January) for Education Journal.

The text follows below:

There was something deeply unsatisfying about last month’s report on the biggest administrative shambles I covered in my nine years as a reporter with the TES.

The Sutherland Report, ordered by ministers and their new testing regulator, Ofqual, largely cleared the Government of any responsibility for the catalogue of mistakes which led to the late publication of 1.2 million children’s national test results this summer.

Was that a fair verdict? Well, a close reading of the report, and an understanding of the way the Government, the Qualifications and Curriculum Authority and its test marking contractor ETS have operated, suggests not.

ETS and the QCA, and particularly the former, clearly must bear the bulk of the blame. But the Government was hardly uninvolved in decisions which helped to shape this summer’s events.

The report rightly heaped opprobrium on the American testing firm, which was in the first year of a five-year contract to mark the key stage 2 and 3 tests. Its conduct, it said, had been “wholly unacceptable and lacked professionalism”.

This, perhaps, was the only possible verdict when one considers the events up to and including the summer’s marking delays.

From the botched trial of a new system of online training for markers in late 2007 to many examiners receiving incorrect contracts and wrong information on training events, and schools complaining of faulty on-line registration systems, administrative bungles characterised the build-up to the marking itself.

This was then blighted by papers being sent to the wrong markers’ homes; a helpline staffed by people who appeared to know little about the marking process and which was overwhelmed by the sheer number of calls; and software systems for checking markers’ work which broke down.

“The primary responsibility must therefore rest with the American organisation, ETS Global BV, which won the public contract to deliver the tests and failed its customers,” said the report.

But the QCA had also failed to deliver its remit, it added. This was to “select a first-rate delivery contractor, oversee its work and ensure that the Department for Children, Schools and Families [sic] policy objectives were met”.

Moreover, Sutherland said the QCA, and its testing division, the National Assessment Agency, should have realised sooner that the results would not be released on time, only informing the DCSF of this a week before they were due, on June 30^th.

This “it’ll be all right on the night” sense of complacency at the NAA may be well-founded.

Yet, among the report’s 46 recommendations, only two suggested failings by Ed Balls’s schools department: one proposing that the role of DCSF observers at QCA board meetings should be clarified; and the second stating that Government officials “may not have challenged QCA sufficiently on its project and risk management”.

Yet a further consideration of the evidence might prompt a different view. For example, the DCSF had oversight of the process of appointing ETS as a contractor.

At least one of its senior officials was present at a meeting when the procurement was discussed, and the Office of Government Commerce, an arm of the Treasury, approved the process.

Ken Boston, the QCA’s chief executive, told the inquiry that officials were closely involved at all stages leading up to the marking of the tests.

He said: “Throughout the process of procuring the contract and delivering the tests according to DCSF specification, ministers and officials had access to exactly the same data and information as the NAA and the QCA; they were active participants in the process.”

The report does not comment on this observation. However, it is widely known that DCSF officials shadow the work of their counterparts at the QCA. The detail of what ETS was setting out to do had to have been known to the Government, as part of the procurement process. And so, too, must have been several major changes to the system, introduced all at once this year. These  included alterations to the arrangements for delivering scripts to markers; the removal of double-checks on the marking of papers where a pupil’s result left them just short of achieving a certain national curriculum level on first marking; and the requirement for examiners to enter each individual mark at the computer screen.

All of these would appear, to this observer, to have added to the risk of failure this summer. But the report offers little sense that this danger should have been on ministers’ radars.

Sutherland also concludes that the ambiguous nature of the NAA – it is a division of the QCA, but the report said that it was viewed by ETS and Ofqual as a separate organisation – got in the way of proper risk management.

But the report fails to point out that one of Mr Balls’s predecessors, Charles Clarke, approved the NAA’s creation.

Making these points might seem, to some, like quibbling. Yet they are, I believe, important. The central point is that, in its relationship with the QCA, it is the DCSF that often wields the power.

It sets very tightly defined remits for the authority. It sets the QCA’s budget. And the QCA chairman and chief executive can expect a dressing down from ministers if they step out of line.

Given all this power, where is the responsibility when things go wrong?

- Warwick Mansell

Ok, I mentioned a while back on this blog, I think, that I was keen on setting up a section on this site called “Data watch: Ofsted”, which is intended to feature observations on the use of exam statistics in Ofsted inspections. This is something I’ve written about before, notably in a front page story in the TES last year about the connection between schools’ Ofsted gradings for their test and exam results and their overall inspection verdicts.

I’ve recently been trying to do some more work on whether patterns can be found in the details of schools’ exam results and the outcomes of their inspections. Specifically, I wanted to investigate claims that schools’ contextual value added (CVA) scores play a large part in how effective they are adjudged to be by inspectors.

I thought there might be a strong link here, as CVA could be seen as the most sophisticated (it is certainly the most complicated…) statistical measure of school performance there is. It compares a pupil’s overall test or exam performance at a certain point with what they might have been expected to achieve, given their previous achievements and a bewildering array of other characteristics about each child which might be thought to have an effect on performance. Results for each child are then added up to give a total figure for the school, which might be seen, if the system were infallible, as the last word on whether or not it “adds value” to its pupils’ education.

As part of my investigation, which centred on secondary schools, I got hold of a spreadsheet which Ofsted now produces of all schools’ inspection judgements over a given time, this one relating to the 2007-08 academic year. I then plotted these judgements against league table results for 2007 for each of the inspected schools. This allowed me to compare each school’s overall inspection verdict with its CVA score for 2007, which is likely to have been the set of CVA results inspectors would have had access to when visiting the school.

And the results were…well, to be honest I was surprised that there was not a closer association between CVA and overall verdict. Although there was a clear trend that schools with higher CVAs tended to have better Ofsted verdicts, the relationship was not as strong as I thought it might have been. (I would like to show you an Excel graph of the relationship, but alas I think this has defeated my computing abilities at this stage). I did find schools which were adjudged to be “inadequate” by Ofsted but which had above-average CVA figures (the highest I found, among “inadequate” schools, was a figure of 1013, in a school which also, puzzlingly, had figures for the percentage of pupils achieving five GCSEs at A*-C, and achieving five GCSEs at A*-C including English and maths, which were both above the national average). And I found that 18 out of the 174 schools which were said by inspectors to be outstanding had below average CVA scores. There was the same sort of relationship, overall when I compared schools’ figures for 5-plus A*-Cs with their inspection verdicts, and for 5-plus A*-C including English and maths with inspection verdicts.

This doesn’t seem to me to be proof that CVA is not important in the inspection process: it may be tricky, for example, for inspectors to reach a verdict which is strongly at odds with what CVA says. But the relationship is not as clear as perhaps I thought it might be.

If anyone out there would like more on this, or a look at the spreadsheets and graphs on which this posting is based, please email me at [email protected]

- Warwick Mansell

I thought this article in the Guardian today, a first person piece attacking teaching to “Assessment Objectives” was pretty insightful. I thought the comments, too, were interesting. Some criticised the author for focusing too much on some ill-defined notion of creativity. But I didn’t read any properl- reasoned defence, by those commenting, of the approach which she criticised. I’m sure it’s very prevalent in schools; my book carries has quite a bit of evidence on this.

Another article on the effects of targets, this time in terms of their effects on further education, was published in the TES’s FE Focus section last week.

I should have provided a reference to this at the time, too, but here is also a link to an article in Guardian Education in which Shirley Williams, the former Labour education secretary, talks about her reservations about the current testing/accountability regime.

- Warwick Mansell

We were round to dinner on Saturday with a teacher friend and her partner. She told us that her school, which is in an Ofsted “category” is under such pressure to improve its pupils’ results that seven-year-olds are being set tests of up to an hour and a half in length. Is this in the pupils’ best interests? Well, it’s safe to say this teacher thinks not. One disaffected tot apparently scribbled “I hate tests” all over a recent task. She couldn’t blame him. What a bonkers system this is.

- Warwick Mansell