An unhealthy set of ratings

What’s the best way to measure a hospital’s performance? For starters, don’t rely on what it tells you…

By Nigel Hawkes

Monitoring the performance of NHS hospitals has proved a tough challenge. Since Labour came to power in 1997, healthcare inspectorates have succeeded one another at disconcertingly frequent intervals, each promising to do better than the last. Baroness Young, the chair of the latest of these, the Care Quality Commission (CQC), recently announced her resignation after barely nine months in charge.

Why is she going? A fierce row with Health Secretary Andy Burnham, not normally the knuckleduster type, is one explanation. According to this version of events, Mr Burnham did not welcome Lady Young’s wish to chuck out the existing regulatory regime and replace it with something tougher. A rival account, deriving from the Department of Health, portrays Lady Young as “volatile and hot-headed” and prone to pick arguments. There is no special reason to believe either of these narratives.

What is plain, however, is that the CQC (like its predecessor the Healthcare Commission) has stumbled into trouble by giving “good” or “excellent” ratings to hospitals that it was subsequently forced to admit were neither. Risk-based inspections, devised by the Healthcare Commission, were meant to use intelligence from a wide range of sources to identify hospitals that needed a close watch, thereby focusing limited resources where they could do the most good.

Among these sources of intelligence are measures of how good hospitals are at keeping people alive. Mortality may be a crude measure, but it is an unambiguous one. The public is prone to think that all doctors are equally good, and often have a soft spot for their local hospital because it looked after Auntie Edna so well. Lacking any basis for comparison, they can go on experiencing bad care for years without realising it.

The answer is to compare hospital mortality – either for specific conditions or treatments, or across the board. The tools for doing this were developed by Professor Brian Jarman of Imperial College in the 1990s, and exploited by two journalists, Tim Kelsey of the Sunday Times and Roger Taylor of the Financial Times, who in 2001 set up the healthcare analysis company Dr Foster.

Hospitals were reluctant, to start with. Such crude comparisons, they argued, could not take adequate account of case-mix, social deprivation, co-morbidity, age - or a host of other variables that meant some were bound to show worse results than others. Professor Jarman’s Hospital Standardised Mortality Ratios (HSMRs) do indeed have to make many statistical corrections to account for these variables, and there are some statisticians who question whether the outcome is terribly meaningful.

The issue came to a head over Mid-Staffordshire NHS Foundation Trust and other hospitals in the West Midlands, which ranked low in HSMRs tables produced by Dr Foster in May 2007. The local strategic health authority commissioned a report from the University of Birmingham to probe the reasons for the poor scores. It identified several reasons, including poor “coding” (recording of data); a lack of local hospices, which increased the chance that terminally-ill patients would die in hospital, rather than elsewhere; and the validity of the method itself.

Thus reassured, the Strategic Health Authority brushed aside Dr Foster’s evidence that Mid-Staffs was a poor performer, only to find later what a grievous mistake it had made. Persistent complaints from patients had been ignored, death rates were unacceptably high, and the hospital had focused so hard on gaining Foundation Trust status that it had let clinical standards slip. This does not prove that Dr Foster was right and the Birmingham team wrong, but that’s certainly the way you would bet.

Professor Jarman also published a point-by-point rebuttal of the Birmingham findings that pointed out that the factors the team had identified might affect the scores, but not in a hugely significant way. This incident left the Healthcare Commission (still in office at the time) looking very foolish, and it was perhaps unfortunate that Lady Young had meanwhile recruited Cynthia Bower as her chief executive of the successor organisation, the CQC. Ms Bower came from the West Midlands SHA, where she had been responsible for commissioning the Birmingham report.

Whether this had any influence on how the CQC dealt with inconvenient HSMRs is unclear. But it continued to use intelligence from Professor Jarman’s team at Imperial College to inform its judgment, so he was not the only listener to be taken aback when Lady Young chose to denigrate the Dr Foster ratings in interviews. If they were so questionable, why pay money for them? And hadn’t they identified Mid-Staffs as a poor performer, even as the official ratings classed it as good?

The difficulty the regulator faces is that it is bound to form judgments on the basis of what hospitals tell it. They are often the last to realise that they are performing badly, even though many of them subscribe to Dr Foster’s services. (The paradox here is that information gathered at public expense is passed free to Dr Foster, which then sells it back at a profit to the very hospitals who provided it.)

This year, some hospitals were shocked when the tables emerged. Their regular intelligence had indicated that they were doing well, but suddenly they found that, by comparison with others, they weren’t. This year, HSMRs across the board fell by 7 per cent, and the results are automatically rebased each year so that the average across England is 100. This process meant that hospitals that had cut their HSMRs, but by less than 7 per cent, found themselves above average.

Hospitals genuinely find it hard to believe the evidence of the HSMRs. Take Basildon and Thurrock University Hospitals, which in the 2008 tables scored an HSMR of 132.2, the highest figure in that year’s guide. This came just a month after the hospitals had been awarded an “excellent” rating in the official Annual Health Check. The trust that runs them issued a lengthy statement quoting extensively from the Birmingham report and saying it was confident the root of the problem lay in the way it “coded” patients’ diagnoses, and not in the care it provided.

This year the CQC rated Basildon Hospital as “good” just weeks before sending in investigators amid fears that dozens of patients might have died after receiving sub-standard care. So, once again, HSMRs seem to have provided a better, and earlier, guide than the inspectorate. In retrospect it would clearly have been wiser for Basildon and Thurrock to have taken the 2008 Dr Foster rating seriously rather than trying to explain it away.

At a deeper level, the row is between believers in “outcome” measures (Dr Foster) and those who prefer “process” measures (the CQC and the Birmingham team). Hospitals and regulators tend to believe that if the right processes are in place, the right results will be achieved. Processes are much easier to check, and confirm, than outcomes. They lend themselves to the ticking of boxes, at which the NHS is world-class.

The evidence, however, suggests that outcome measures do a better job in identifying outliers, and in several high-profile cases should have been taken more seriously than they were. By trying to have the best of both worlds, the inspectorates have achieved the best of neither, constantly looking foolish when their process-based judgments have proved wide of the mark.

The NHS is now consulting on a range of new outcome measures, but is asking hospitals to choose their own rather than laying down a national list. This could prove a serious mistake because it will make harder the very comparisons that enable such measures to identify poor performers. The lesson Brian Jarman is trying to teach has yet to be learned.

To find out more about this article, visit: http://www.drfosterhealth.co.uk/

8 December 2009

Nigel Hawkes. Director of Straight Statistics and former Health Editor of The Times,