Only telling half of the story

The Mail headline is pretty damning:

Things are equally grim in the story itself:

Up to two in three patients are unhappy with the care they are given on the NHS, an alarming report has found.

Many of those treated by the Health Service complain that they were dehumanised, humiliated and embarrassed by the doctors and nurses who were supposed to be looking after them.

The survey of almost 12,000 patients found that staff were criticised for being rude, arrogant and lazy – too often refusing to treat their patients with dignity or compassion
And only a third of those polled said they were content with the standard of care that they had received on an NHS ward or at a surgery.

Dig a little deeper, however, and it quickly becomes clear that this is just the latest Daily Mail attack on the NHS – an institution the paper has previously criticised both for spending too much on supplies and not spending enough on supplies.

The first port of call is the “alarming report” on which the story is based. It is published by Patient Opinion, an independent website that collates individuals’ experiences of the NHS and passes on their comments to politicians and health service managers. Today’s report is a compilation and analysis of the various comments received by Patient Opinion over the past decade.

Despite what the Mail says, it is not a “survey” and nobody was “polled”. Patient Opinion merely asks patients to come to them with their own experiences – and as even the Mail admits (tucked away at the bottom of the story):

“…since many of its contributors are likely to log on because they are unhappy with their care, it is difficult to quantify exactly what proportion of all NHS patients are similarly disgruntled.”

In other words, people who comment at Patient Opinion are entirely self-selecting. There’s no way of knowing how representative of all patients the site’s visitors are – it’s a bit like trying to get a statistically valid handle on society’s views based solely on the comments left on the Daily Mail website. It’s certainly interesting to see what so many people have to say, but to use it as the basis of a claim that “two in three patients are unhappy with the care they are given” is somewhat more dubious – the claim should be that “two in three patients who use a certain website” are unhappy.

The Mail is also factually wrong when it claims that “almost 12,000 people” were surveyed for the report. As we’ve already seen, the report is not a survey. And, as anyone who bothers to read the methodology of the report will see, its findings were not based on all of the comments received since it launched in 2005:

“The analysis used a sample of 537 accounts from patients and carers. The analysed data was sampled by initially collating: All public postings made on Patient Opinion since February 2008 that had a criticality score of three, four or five; all concerns received prior to February 2008 that were over 500 characters in length, to give the fullest example of patient feedback in this period. This gave us 1,790 accounts, of which 30 per cent were randomly selected, creating a final sample total of 537 accounts.”

The “Our Reasoning” section adds:

“Since February 2008, Patient Opinion has categorised all posts made on a scale of zero to five, with five being the most severe. For this report, the research team selected stories with a severity threshold of three or more, to ensure that the most serious concerns were prioritised. For concerns submitted prior to February 2008, they included entries longer than 500 characters. This gave a dataset of 1,790 stories, with a 30 per cent sample size of 537 stories (from the overall total of 15,605 posted on Patient Opinion in this time period).”

In other words, the researchers deliberately only chose comments from people with more extreme views, with the result being that the study was based on just 537 comments – roughly 3.4 per cent of the 15,605 “stories” left during this period. And it’s entirely possible that the same person commented more than once.

But even if you don’t mind the issues with the report itself, the Mail’s quoting of it is selective in the extreme. Yes, it identifies many problems with the NHS, and calls on health service managers to improve the situation. However, other findings that the Mail couldn’t find room for include:

Out of 11,982 stories posted, 33.1 per cent were positive, 45.6 per cent were neutral and just 21.1 per cent were negative (p9)

79 per cent of the stories on Patient Opinion talk about a positive experience of the health service “indicating that the NHS continues to provide excellent care to patients and their families all over the country.” (p8)

Staff attitudes are the most common aspect of the very best care experienced by patients and their carers (p3)

To Patient Opinon’s credit, they’re open about the fact that the report is not statistically ideal as it based on subjective accounts of individuals’ experiences rather than objective quantifiable data. The researchers are also happy to make clear, on page eight, that that they have deliberately focussed on negative aspects at expense of the positives:

“This report is intended to prompt change and drive improvements. We have therefore focused on examples where improvements can be made within the NHS … There will always be room for improvement though, and this analysis exposes these areas.”

However, all such caveats are missing from the Mail’s coverage. The Mail is also happy to reproduce harrowing accounts from patients who felt they were treated badly. In the report itself, each comment is followed by a response from the NHS, usually expressing horror, apologising profusely and explaining what steps have been taken to remedy the situation so it doesn’t happen again. But of course the Mail can’t find room for these, either.

As usual, if the facts don’t fit the paper’s narrative, they are conveniently ignored.