NIHR Signal Improved tools to fairly compare the performance of critical care services

Published on 2 February 2016

This NIHR-funded study refined the risk prediction models which are used to compare performance of critical care services and developed new risk prediction tools for heart attack sufferers in hospital.

Critical care is highly specialised, intensive and costly, so audit of service performance is important for providers and commissioners. Risk prediction and audit are already quite well established in general critical care, but this study has improved and added new tools for assessing outcomes for specialist cardiac critical care and general critical care services. The improved prediction models are being included in existing audits, the next stage is to use the results of these audits to improve services.

Improved tools to fairly compare the performance of critical care services

Share your views on the research.

Why was this study needed?

Audits help assess the performance of hospitals against national standards for emergency and intensive care. In order to make valid comparisons between different hospitals, it is necessary to have accurate risk prediction models which take into account patients’ condition on admission (particularly the severity of their acute illness and chronic health conditions) and also baseline population characteristics for the catchment area (such as average age, prevalence of chronic and acute health conditions and socio demographics).

ICNARC, a research charity, currently carries out two such audits: the Case Mix Programme Audit (for adult critical care in England, Wales and Northern Ireland) and the National Cardiac Arrest Audit (for UK patients older than 28 days who have a heart attack in hospital and are resuscitated). Knowing where the performance of a service stands nationally and in relation to other hospitals is vital for improving service quality.

The current study aimed to improve these models for their use with specialist, as well as general services for critically ill adults.

What did this study do?

An existing risk prediction model for critically ill adults (ICNARC) was validated against APACHE II using external data from over 23,000 admissions to 24 Scottish general critical care units between 2007 and 2009. This part of the study also tested approaches to handling missing data and modelled reasons for admission to critical care units.

The second aim was to develop and validate new risk prediction tools for cardiothoracic intensive care, general intensive care and for in-hospital cardiac arrest. Researchers drew on data from over 300,000 patients from the Case Mix Programme and over 22,000 patients from the National Cardiac Arrest Audits between 2010 and 2014.

What did it find?

  • The existing ICNARC model was validated successfully against external outcome data. This was a useful improvement on the existing and widely used APACHE II model.
  • A risk prediction model was developed for patients admitted to cardiothoracic intensive care units. After further refining, statistical likelihood of concordance and predictive accuracy of the model were very favourable.
  • New models were developed to predict mortality after admission to adult general and specialist critical care units. When tested against later data from the same hospitals, these models performed very well with good statistical likelihood of concordance and predictive accuracy.
  • Two new models were developed to predict immediate and more long-term survival after cardiac arrest occurring in hospital. These models were validated against existing outcome data and showed acceptable performance.

What does current guidance say on this issue?

The audits covered by this study complement the NHS England’s Health Quality Improvement Programme. The Care Quality Commission is responsible for inspecting critical care, drawing on the standards stated in its 2015 framework.

What are the implications?

Critical care is now one of the core services inspected by the Care Quality Commission, who draw on the Case Mix Programme audit and the National Cardiac Assessment Audit for data about services. The new and improved risk prediction models may be used in Trusts’ public Annual Quality Reports and internal Quarterly Quality Reports, and the research team are promoting awareness of them within the field of critical care. This should improve providers and commissioners’ ability to compare and assess performance of hospital services. The implication of changing audit procedures is the need for staff training, time and cost of data collection and processing. However, these may be offset by increased efficiency of services, which could be measured in an economic study.

Further research may include linking the audit data with death registrations, in order to assess mortality over different lengths of time, and linking data across different national audits. The researchers also suggest investigating whether specific information should be collected for national audits, rather than relying on data that is gathered routinely.

Citation

Harrison DA, Ferrando-Vivas P, Shahin J, Rowan KM. Ensuring comparisons of health-care providers are fair: development and validation of risk prediction models for critically ill patients. NIHR Journals Library; 2015.

This project was funded by the National Institute for Health Research Health Services and Delivery Research programme (project number 09/2000/65)

Bibliography

Care Quality Commission. Inspection framework: NHS acute hospitals. Core service: critical care. London: Care Quality Commission; 2015.

ICNARC. The Intensive Care National Audit and Research Centre [internet]. London: ICNARC; 2015.

Data from the Case Mix Programme and the National Cardiac Arrest Audit can be obtained from ICNARC (see www.icnarc.org/Our-Audit/Audits/Cmp/Reports/Access-Our-Data  and www.icnarc.org/Our-Audit/Audits/Ncaa/Reports/Access-Our-Data).

Harrison DA, Lone NI, Haddow C, et al. External validation of the Intensive Care National Audit & Research Centre (ICNARC) risk prediction model in critical care units in Scotland. BMC Anesthesiol. 2014;14:116.

Harrison DA, Patel K, Nixon E, et al. Development and validation of risk models to predict outcomes following in-hospital cardiac arrest attended by a hospital-based resuscitation team. Resuscitation 2014;85:993–1000.

NHS England. Clinical audit [internet]. London; NHS England.

Why was this study needed?

Audits help assess the performance of hospitals against national standards for emergency and intensive care. In order to make valid comparisons between different hospitals, it is necessary to have accurate risk prediction models which take into account patients’ condition on admission (particularly the severity of their acute illness and chronic health conditions) and also baseline population characteristics for the catchment area (such as average age, prevalence of chronic and acute health conditions and socio demographics).

ICNARC, a research charity, currently carries out two such audits: the Case Mix Programme Audit (for adult critical care in England, Wales and Northern Ireland) and the National Cardiac Arrest Audit (for UK patients older than 28 days who have a heart attack in hospital and are resuscitated). Knowing where the performance of a service stands nationally and in relation to other hospitals is vital for improving service quality.

The current study aimed to improve these models for their use with specialist, as well as general services for critically ill adults.

What did this study do?

An existing risk prediction model for critically ill adults (ICNARC) was validated against APACHE II using external data from over 23,000 admissions to 24 Scottish general critical care units between 2007 and 2009. This part of the study also tested approaches to handling missing data and modelled reasons for admission to critical care units.

The second aim was to develop and validate new risk prediction tools for cardiothoracic intensive care, general intensive care and for in-hospital cardiac arrest. Researchers drew on data from over 300,000 patients from the Case Mix Programme and over 22,000 patients from the National Cardiac Arrest Audits between 2010 and 2014.

What did it find?

  • The existing ICNARC model was validated successfully against external outcome data. This was a useful improvement on the existing and widely used APACHE II model.
  • A risk prediction model was developed for patients admitted to cardiothoracic intensive care units. After further refining, statistical likelihood of concordance and predictive accuracy of the model were very favourable.
  • New models were developed to predict mortality after admission to adult general and specialist critical care units. When tested against later data from the same hospitals, these models performed very well with good statistical likelihood of concordance and predictive accuracy.
  • Two new models were developed to predict immediate and more long-term survival after cardiac arrest occurring in hospital. These models were validated against existing outcome data and showed acceptable performance.

What does current guidance say on this issue?

The audits covered by this study complement the NHS England’s Health Quality Improvement Programme. The Care Quality Commission is responsible for inspecting critical care, drawing on the standards stated in its 2015 framework.

What are the implications?

Critical care is now one of the core services inspected by the Care Quality Commission, who draw on the Case Mix Programme audit and the National Cardiac Assessment Audit for data about services. The new and improved risk prediction models may be used in Trusts’ public Annual Quality Reports and internal Quarterly Quality Reports, and the research team are promoting awareness of them within the field of critical care. This should improve providers and commissioners’ ability to compare and assess performance of hospital services. The implication of changing audit procedures is the need for staff training, time and cost of data collection and processing. However, these may be offset by increased efficiency of services, which could be measured in an economic study.

Further research may include linking the audit data with death registrations, in order to assess mortality over different lengths of time, and linking data across different national audits. The researchers also suggest investigating whether specific information should be collected for national audits, rather than relying on data that is gathered routinely.

Citation

Harrison DA, Ferrando-Vivas P, Shahin J, Rowan KM. Ensuring comparisons of health-care providers are fair: development and validation of risk prediction models for critically ill patients. NIHR Journals Library; 2015.

This project was funded by the National Institute for Health Research Health Services and Delivery Research programme (project number 09/2000/65)

Bibliography

Care Quality Commission. Inspection framework: NHS acute hospitals. Core service: critical care. London: Care Quality Commission; 2015.

ICNARC. The Intensive Care National Audit and Research Centre [internet]. London: ICNARC; 2015.

Data from the Case Mix Programme and the National Cardiac Arrest Audit can be obtained from ICNARC (see www.icnarc.org/Our-Audit/Audits/Cmp/Reports/Access-Our-Data  and www.icnarc.org/Our-Audit/Audits/Ncaa/Reports/Access-Our-Data).

Harrison DA, Lone NI, Haddow C, et al. External validation of the Intensive Care National Audit & Research Centre (ICNARC) risk prediction model in critical care units in Scotland. BMC Anesthesiol. 2014;14:116.

Harrison DA, Patel K, Nixon E, et al. Development and validation of risk models to predict outcomes following in-hospital cardiac arrest attended by a hospital-based resuscitation team. Resuscitation 2014;85:993–1000.

NHS England. Clinical audit [internet]. London; NHS England.

Ensuring comparisons of health-care providers are fair: development and validation of risk prediction models for critically ill patients

Published on 19 October 2015

Harrison DA, Ferrando-vivas P, Shahin J, Rowan KM.

Health Services and Delivery Research Volume 3 Issue 41 , 2015

Background National clinical audit has a key role in ensuring quality in health care. When comparing outcomes between providers, it is essential to take the differing case mix of patients into account to make fair comparisons. Accurate risk prediction models are therefore required. Objectives To improve risk prediction models to underpin quality improvement programmes for the critically ill (i.e. patients receiving general or specialist adult critical care or experiencing an in-hospital cardiac arrest). Design Risk modelling study nested within prospective data collection. Setting Adult (general/specialist) critical care units and acute hospitals in the UK. Participants Patients admitted to an adult critical care unit and patients experiencing an in-hospital cardiac arrest attended by the hospital-based resuscitation team. Interventions None. Main outcome measures Acute hospital mortality (adult critical care); return of spontaneous circulation (ROSC) greater than 20 minutes and survival to hospital discharge (in-hospital cardiac arrest). Data sources The Case Mix Programme (adult critical care) and National Cardiac Arrest Audit (in-hospital cardiac arrest). Results The current Intensive Care National Audit & Research Centre (ICNARC) model was externally validated using data for 29,626 admissions to critical care units in Scotland (2007–9) and outperformed the Acute Physiology And Chronic Health Evaluation (APACHE) II model in terms of discrimination (c-index 0.848 vs. 0.806) and accuracy (Brier score 0.140 vs. 0.157). A risk prediction model for cardiothoracic critical care was developed using data from 17,002 admissions to five units (2010–12) and validated using data from 10,238 admissions to six units (2013–14). The model included prior location/urgency, blood lactate concentration, Glasgow Coma Scale (GCS) score, age, pH, platelet count, dependency, mean arterial pressure, white blood cell (WBC) count, creatinine level, admission following cardiac surgery and interaction terms, and it had excellent discrimination (c-index 0.904) and accuracy (Brier score 0.055). A risk prediction model for admissions to all (general/specialist) adult critical care units was developed using data from 155,239 admissions to 232 units (2012) and validated using data from 90,017 admissions to 216 units (2013). The model included systolic blood pressure, temperature, heart rate, respiratory rate, partial pressure of oxygen in arterial blood/fraction of inspired oxygen, pH, partial pressure of carbon dioxide in arterial blood, blood lactate concentration, urine output, creatinine level, urea level, sodium level, WBC count, platelet count, GCS score, age, dependency, past medical history, cardiopulmonary resuscitation, prior location/urgency, reason for admission and interaction terms, and it outperformed the current ICNARC model for discrimination and accuracy overall (c-index 0.885 vs. 0.869; Brier score 0.108 vs. 0.115) and across unit types. Risk prediction models for in-hospital cardiac arrest were developed using data from 14,688 arrests in 122 hospitals (2011–12) and validated using data from 7791 arrests in 143 hospitals (2012–13). The models included age, sex (for ROSC > 20 minutes), prior length of stay in hospital, reason for attendance, location of arrest, presenting rhythm, and interactions between rhythm and location. Discrimination for hospital survival exceeded that for ROSC > 20 minutes (c-index 0.811 vs. 0.720). Limitations The risk prediction models developed were limited by the data available within the current national clinical audit data sets. Conclusions We have developed and validated risk prediction models for cardiothoracic and adult (general and specialist) critical care units and for in-hospital cardiac arrest. Future work Future development should include linkage with other routinely collected data to enhance available predictors and outcomes. Funding details The National Institute for Health Research Health Services and Delivery Research programme.

The audits and data sources included in this study were:

  • Scottish Intensive Care Society Audit Group database
  • Case Mix Programme
  • National Cardiac Arrest Audit

The models included were:

  • Intensive Care National Audit and Research Centre (ICNARC) model
  • Acute Physiology and Chronic Health Evaluation (APACHE) II model
  • Admissions to cardiac care units prediction model
  • New Intensive Care National Audit and Research Centre model for prediction of acute hospital mortality for admissions to adult critical care units
  • New risk prediction models to predict outcomes following heart attack in hospital

Expert commentary

Patients who become critically ill need rapid treatment to give them the highest chance of surviving. This rigorously conducted study funded by the NIHR has improved the ‘black-box’ statistical models that allow hospitals that treat critically ill patients to be fairly benchmarked. The researchers have also developed much needed new models to allow fairer benchmarking of outcomes for cardiothoracic intensive care units and for patients who suffer a cardiac arrest in hospital. These models have already been, or will shortly be, implemented into everyday practice in the NHS. Whilst no risk prediction model will ever be perfect, such robust, transparent benchmarking of care allows patients, clinicians and those commissioning health care to have greater confidence in the high quality of care provided in hospitals across the NHS for critically ill patients.

Dr Nazir I. Lone, Senior Clinical Lecturer in Critical Care, University of Edinburgh