Improved tools to fairly compare the performance of critical care services

Please note that this summary was posted more than 5 years ago. More recent research findings may have been published.

This is a plain English summary of an original research article. The views expressed are those of the author(s) and reviewer(s) at the time of publication.

This NIHR-funded study refined the risk prediction models which are used to compare performance of critical care services and developed new risk prediction tools for heart attack sufferers in hospital.

Critical care is highly specialised, intensive and costly, so audit of service performance is important for providers and commissioners. Risk prediction and audit are already quite well established in general critical care, but this study has improved and added new tools for assessing outcomes for specialist cardiac critical care and general critical care services. The improved prediction models are being included in existing audits, the next stage is to use the results of these audits to improve services.

Why was this study needed?

Audits help assess the performance of hospitals against national standards for emergency and intensive care. In order to make valid comparisons between different hospitals, it is necessary to have accurate risk prediction models which take into account patients’ condition on admission (particularly the severity of their acute illness and chronic health conditions) and also baseline population characteristics for the catchment area (such as average age, prevalence of chronic and acute health conditions and socio demographics).

ICNARC, a research charity, currently carries out two such audits: the Case Mix Programme Audit (for adult critical care in England, Wales and Northern Ireland) and the National Cardiac Arrest Audit (for UK patients older than 28 days who have a heart attack in hospital and are resuscitated). Knowing where the performance of a service stands nationally and in relation to other hospitals is vital for improving service quality.

The current study aimed to improve these models for their use with specialist, as well as general services for critically ill adults.

What did this study do?

An existing risk prediction model for critically ill adults (ICNARC) was validated against APACHE II using external data from over 23,000 admissions to 24 Scottish general critical care units between 2007 and 2009. This part of the study also tested approaches to handling missing data and modelled reasons for admission to critical care units.

The second aim was to develop and validate new risk prediction tools for cardiothoracic intensive care, general intensive care and for in-hospital cardiac arrest. Researchers drew on data from over 300,000 patients from the Case Mix Programme and over 22,000 patients from the National Cardiac Arrest Audits between 2010 and 2014.

What did it find?

The existing ICNARC model was validated successfully against external outcome data. This was a useful improvement on the existing and widely used APACHE II model.
A risk prediction model was developed for patients admitted to cardiothoracic intensive care units. After further refining, statistical likelihood of concordance and predictive accuracy of the model were very favourable.
New models were developed to predict mortality after admission to adult general and specialist critical care units. When tested against later data from the same hospitals, these models performed very well with good statistical likelihood of concordance and predictive accuracy.
Two new models were developed to predict immediate and more long-term survival after cardiac arrest occurring in hospital. These models were validated against existing outcome data and showed acceptable performance.

What does current guidance say on this issue?

The audits covered by this study complement the NHS England’s Health Quality Improvement Programme. The Care Quality Commission is responsible for inspecting critical care, drawing on the standards stated in its 2015 framework.

What are the implications?

Critical care is now one of the core services inspected by the Care Quality Commission, who draw on the Case Mix Programme audit and the National Cardiac Assessment Audit for data about services. The new and improved risk prediction models may be used in Trusts’ public Annual Quality Reports and internal Quarterly Quality Reports, and the research team are promoting awareness of them within the field of critical care. This should improve providers and commissioners’ ability to compare and assess performance of hospital services. The implication of changing audit procedures is the need for staff training, time and cost of data collection and processing. However, these may be offset by increased efficiency of services, which could be measured in an economic study.

Further research may include linking the audit data with death registrations, in order to assess mortality over different lengths of time, and linking data across different national audits. The researchers also suggest investigating whether specific information should be collected for national audits, rather than relying on data that is gathered routinely.