interviews

The link between trust I.T. maturity and patient mortality

A new study has claimed to find a correlation between the maturity of electronic patient record systems within NHS hospitals and improvements in patient outcomes. But Dr Justin Whatling, chair of BCS Health, the health informatics division of The Chartered Institute for IT, has some doubts about the study’s methodology.

A new study using public and proprietary data claims a link between more advanced health IT systems and lower mortality. 

The investigation by HIMSS (the Health Information and Management Systems Society, a multinational not-for-profit organisation) assessed NHS hospitals’ IT maturity using its EMRAM data (Electronic Medical Record Adoption Model), a cumulative eight-stage model that classifies hospital IT from stage 0 to stage 7. 

No UK hospitals have reached stage 7 (though 200 hospitals in the USA have, plus a handful in Europe), and the UK’s only stage 6 site, Croydon, was not included in the HIMSS study, whose cohort was made up of 8% at stage 0; 15% at stage 1; 16% at stage 2; 9% at stage 3; 4% at stage 4; and 48% at stage 5. 

HIMSS’ analysis showed that those hospitals with an EMRAM ‘score’ of stage 4 or higher have a lower Summary Hospital-Level Mortality Indicator (SHMI). It said these findings “will come as no surprise to many of those working in healthcare IT”. 

John Rayner, director of professional development at HIMSS UK, said: “The indicators that have been used as evidence are the foundations of a more compelling case which we will build on, but the report provides some real and long-awaited assurances that IT investment has absolute links to improving patient care and quality.” 

Shortfalls 

BCS Health chair Dr Justin Whatling was sceptical about aspects of the study, however. He told NHE: “HIMSS are really trying to help improve care delivery and patient safety, and attempting to do a report like this signifies what the opportunity is, and it’s a good thing to be looking at. I give them credit for doing that. However, there are a lot of shortfalls in the report.”

He had specific concerns with the headline finding, the apparent link with mortality statistics, because in essence the figures “get worse before they get better” – the group at EMRAM stage 2 shows a higher average variance from the SHMI target than the group at stage 1, and stage 3 is worse again than stage 2. Only at stage 4 is there a pronounced improvement. 

Dr Whatling said: “My concern is that there’s a lot of deviation that’s negative before it gets positive. It’s not a consistent, uniform improvement: it’s all over the place. It’d be good to understand why. 

“Part of the challenge is that it’s being compared to a national average, which might be creating some confounding aspects.” 

Dr Whatling was disappointed at the lack of 95% confidence intervals plotted on a key graph in the report – ‘EMRAM Score vs SHMI Index’. “I’d like more rigour in seeing r values [correlation coefficients], and in later cases p values [probability], to explain the probability that the observed pattern is the true pattern.” 

Higher-order decision support 

Dr Whatling has “a lot of confidence” in the EMRAM model itself – though he added: “You can use the power of the EMR to change the way you deliver your healthcare processes, but a lot of that isn’t really observed until stages 6 and 7. 

“A lot of stages 1 to 5 are the building blocks; once you get to the decision support, that’s when you’re really leveraging the system to change the way healthcare is delivered.” 

HIMSS should be congratulated for trying to show positive outcomes and correlations at lower stages of maturity, he said, to complement existing studies showing positive return on investment at those higher stages.

Patient experience

Another issue is sample bias, which the report does flag up as “unavoidable” – namely that any hospital proactively putting themselves forward for EMRAM evaluation may be unusually proactive in other areas too, for example patient experience. “I wonder if there are confounding factors that might actually explain a correlation that haven’t been discussed in the report,” Dr Whatling said. “They can’t really set up a randomised control trial, but they can at least explore what confounding factors there might be.” 

The patient experience tables, which show that providers tending towards a higher

EMRAM score performed consistently better than their lower-rated counterparts for treatment-related patient satisfaction questions, have face validity, Dr Whatling said. “It would require more explaining if that was not the case. But they ought to describe how they are drawing the distinction between ‘high’ and ‘low’ for the EMRAM scores. Without visibility of [that], I wasn’t quite sure how to interpret the rest of the findings – that methodology ought to be disclosed.” 

Robust 

Dr Whatling raised a number of other technical concerns with the methodology and findings, and said he may address these directly to HIMSS in writing. 

“I don’t want people to think the study is not valid just because it’s not robust [in the disclosing of its methodology],” he said. “I’d like to see more rigour.”

Tell us what you think – have your say below or email [email protected]