NIHR DC Discover

NIHR Signal Benchmarking study helps hospitals improve measurement of adverse events

Published on 20 June 2017

doi: 10.3310/signal-000428

One in ten inpatients in Welsh hospitals experienced an adverse event of some sort, about half of which were potentially preventable. This is very similar to the rate in other countries. The figures come from a review of 4,388 patient records between October 2010 and March 2013.

Welsh hospitals were using a screening tool at the time called the Global Trigger Tool to estimate rates of adverse effects. The tool surveys case notes of 20 inpatient records each month. Alongside the trigger tool process, researchers reviewed case notes using a more comprehensive two-step process – nurse screening followed by physician review – to compare how different the estimates were and to better understand how many events were preventable and the degree of harm. The two-step process found 10.3% of patients had an adverse event, while the tool found 8% adverse events.

Hospitals which carried out more surgery had higher rates of adverse events. Use of this trigger tool has since been dropped in Wales and a new “hybrid” system of adverse event identification is being developed.

Share your views on the research.

Why was this study needed?

Global estimates of rates of adverse events range from about 8% to 12% of inpatient admissions – meaning that about 1 in 10 patients will have something happen to them in hospital that will harm them. However, measuring rates of adverse events is not easy. Methods mainly rely on ‘post-hoc’ analysis of patient notes after an adverse incident, although not all incidents are recognised or recorded at the time.

Up until recently, Welsh hospitals used the tool to measure adverse events. Twenty randomly selected inpatient case notes records per month per organisation are reviewed to estimate rates of adverse effects. The aim of this study was to compare the rate estimated by the tool to the rate estimated by a more comprehensive, two-stage review of case notes performed over four years.

What did this study do?

The rate of adverse events was estimated in two hospitals from each of the six Welsh health boards using two methods. One method used the Global Trigger Tool and the second used a comprehensive, two-stage review of case notes.

The Trigger Tool system: Hospitals reviewed case notes from 20 randomly-selected adult patients each month that had spent more than 24 hours in hospital. This involved a short, 20 minute review of notes using a checklist of criteria to identify actions that could have or did cause patient harm.

The two-stage review: Researchers subjected case notes, covering 4,388 inpatient episodes, to a two-stage check. Nurses did an initial checklist screen, passing notes with potential adverse events to a physician for a second review. The physicians also looked at whether adverse events could have been prevented.

What did it find?

  • The two-stage process found more adverse events than the trigger tool system – 10.3% (95% confidence interval (CI) 9.4% to 11.2%) versus 9% (95% CI 8.82% to 9.18%)..
  • The percentage varied between hospitals, with results ranging from 7.9% to 16.1%. Individual hospitals were not identified in the report.
  • Factors linked to higher levels of adverse events included longer hospital stay and being admitted for planned surgery. Hospitals which carried out more surgery had higher rates of adverse events.
  • Just over half (51.5%) of all adverse events were thought to be preventable (95% CI 46.88% to 56.12%) using the two-stage process.
  • An extension of this study looked at problems in care linked to the preventable adverse events. These included failure in clinical monitoring (33.5%), failure in infection control (29.4%), problems from surgical or invasive procedures (21.2%) and problems from administering drugs or fluids (18.7%).

What does current guidance say on this issue?

The Global Trigger Tool is no longer used in Wales. NHS England has a Serious Incidence Framework for the reporting and management of adverse events, but this does not include routine surveillance for and monitoring of adverse events.

What are the implications?

The study provides reassurance that rates of adverse events are in line with those reported in other parts of the UK and internationally. Any potentially preventable adverse event is of concern and use of real-world data (data collected as part of routine care) is becoming easier as hospitals use electronic patient records more, potentially this could lead to earlier identification of risks and prevention of harm.

The report identifies areas of care which most often led to an adverse event. This may provide useful lessons for hospitals to focus on.

The resources for implementing any safety system also need considering. While the tool is no longer used in Wales, the two-stage process was resource-intensive. The team who carried out this research is developing a replacement system for ongoing adverse event monitoring in Wales using the concept of risk signatures and using incident reports.

Citation and Funding

Mayor S, Baines E, Vincent C, et al. Measuring harm and informing quality improvement in the Welsh NHS: the longitudinal Welsh national adverse events study. Health Serv Deliv Res. 2017;5(9).

This project was funded by the National Institute for Health Research Health Services and Delivery Research programme.

Bibliography

NHS England. Serious Incident Framework. London: NHS England; 2015.

Why was this study needed?

Global estimates of rates of adverse events range from about 8% to 12% of inpatient admissions – meaning that about 1 in 10 patients will have something happen to them in hospital that will harm them. However, measuring rates of adverse events is not easy. Methods mainly rely on ‘post-hoc’ analysis of patient notes after an adverse incident, although not all incidents are recognised or recorded at the time.

Up until recently, Welsh hospitals used the tool to measure adverse events. Twenty randomly selected inpatient case notes records per month per organisation are reviewed to estimate rates of adverse effects. The aim of this study was to compare the rate estimated by the tool to the rate estimated by a more comprehensive, two-stage review of case notes performed over four years.

What did this study do?

The rate of adverse events was estimated in two hospitals from each of the six Welsh health boards using two methods. One method used the Global Trigger Tool and the second used a comprehensive, two-stage review of case notes.

The Trigger Tool system: Hospitals reviewed case notes from 20 randomly-selected adult patients each month that had spent more than 24 hours in hospital. This involved a short, 20 minute review of notes using a checklist of criteria to identify actions that could have or did cause patient harm.

The two-stage review: Researchers subjected case notes, covering 4,388 inpatient episodes, to a two-stage check. Nurses did an initial checklist screen, passing notes with potential adverse events to a physician for a second review. The physicians also looked at whether adverse events could have been prevented.

What did it find?

  • The two-stage process found more adverse events than the trigger tool system – 10.3% (95% confidence interval (CI) 9.4% to 11.2%) versus 9% (95% CI 8.82% to 9.18%)..
  • The percentage varied between hospitals, with results ranging from 7.9% to 16.1%. Individual hospitals were not identified in the report.
  • Factors linked to higher levels of adverse events included longer hospital stay and being admitted for planned surgery. Hospitals which carried out more surgery had higher rates of adverse events.
  • Just over half (51.5%) of all adverse events were thought to be preventable (95% CI 46.88% to 56.12%) using the two-stage process.
  • An extension of this study looked at problems in care linked to the preventable adverse events. These included failure in clinical monitoring (33.5%), failure in infection control (29.4%), problems from surgical or invasive procedures (21.2%) and problems from administering drugs or fluids (18.7%).

What does current guidance say on this issue?

The Global Trigger Tool is no longer used in Wales. NHS England has a Serious Incidence Framework for the reporting and management of adverse events, but this does not include routine surveillance for and monitoring of adverse events.

What are the implications?

The study provides reassurance that rates of adverse events are in line with those reported in other parts of the UK and internationally. Any potentially preventable adverse event is of concern and use of real-world data (data collected as part of routine care) is becoming easier as hospitals use electronic patient records more, potentially this could lead to earlier identification of risks and prevention of harm.

The report identifies areas of care which most often led to an adverse event. This may provide useful lessons for hospitals to focus on.

The resources for implementing any safety system also need considering. While the tool is no longer used in Wales, the two-stage process was resource-intensive. The team who carried out this research is developing a replacement system for ongoing adverse event monitoring in Wales using the concept of risk signatures and using incident reports.

Citation and Funding

Mayor S, Baines E, Vincent C, et al. Measuring harm and informing quality improvement in the Welsh NHS: the longitudinal Welsh national adverse events study. Health Serv Deliv Res. 2017;5(9).

This project was funded by the National Institute for Health Research Health Services and Delivery Research programme.

Bibliography

NHS England. Serious Incident Framework. London: NHS England; 2015.

Measuring harm and informing quality improvement in the Welsh NHS: the longitudinal Welsh national adverse events study

Published on 27 February 2017

Mayor S, Baines E, Vincent C, Lankshear A, Edwards A, Aylward M, Hogan H, Harper P, Davies J, Mamtora A, Brockbank E & Gray J.

Health Services & Delivery Research Volume 5 Issue 9 , 2017

Background, objectives and setting Despite global activity over the past 15 years to improve patient safety, the measurement of adverse events (AEs) remains challenging. Objectives We aimed to obtain definitive longitudinal data on harm across NHS Wales and to compare the performance of the Global Trigger Tool (GTT) with the two-stage retrospective review process, using our findings to consolidate an approach to the ongoing surveillance of harm in Wales. Data sources Eleven of the 13 major Welsh NHS hospitals. Review methods The two-stage retrospective review methodology was used to quantify harm across NHS Wales. In total, 4536 inpatient episodes were screened for AEs by research nurses. Records that were highly suggestive of AEs were further assessed by physicians. NHS-led teams undertook GTT reviews on the same case notes. Results At least one AE was determined in 10.3% of episodes of care [95% confidence interval (CI) 9.4% to 11.2%] and 51.5% were preventable (95% CI 46.9% to 56.1%). The percentage of patients identified with AEs using the GTT methodology was lower, at 9.0% (95% CI 8.82% to 9.18%). Differences in AEs were evident across study sites. Methods were developed to profile the risk of AEs in individual organisations by producing signatures of harm for each NHS site. Analysis indicated that neither the GTT nor the two-stage process was a candidate tool for routine surveillance, and a hybrid one-stage tool (Harm2), based on phase 1 findings, was developed for ongoing AE monitoring. Using the Harm2 tool, AEs were identified in 371 out of 3352 randomly selected discharge reviews (11.3%, 95% CI 10.2% to 12.4%), and 59.6% (95% CI 55.3 to 63.9) of these were preventable. In a cohort of randomly selected deceased patient reviews, at least one AE was determined in 315 out of 1018 admissions (30.1%, 95% CI 28.1% to 33.8%), and 61.7% (95% CI 57.5% to 65.9%) of these were preventable. Factors associated with AEs in the randomly selected discharge reviews included having peripheral vascular disease [odds ratio (OR) 2.52], hemiplegia (OR 2.27) or dementia (OR 2.27). No association with chronic disease was identified in the deceased episodes of care. Limitations The dependence on our health service partners in identifying notes to be reviewed, along with the small sample examined each month, limits the generalisability of these findings and rates were not standardised for hospital and size and level of services provided. We cannot rule out the possibility that the rates we report may be underestimated. Conclusion The extent of harm detected across NHS Wales using both the two-stage retrospective review process and the new Harm2 tool conforms to the findings in the literature, but this is the first longitudinal study using these methods. With training and using a structured review process, non-physician reviewers can undertake case note review efficiently and effectively, and the rates of AEs and of the preventability and the breakdown of problems in care conform to those reported in studies in which physicians undertake these classifications. Whether the patient died or was discharged alive significantly influences the rate and composition of AEs. The Harm2 tool performed with moderate reliability in the determination of AEs. Future work Future large-scale studies should attempt to specify types of AEs, such as hospital-acquired infections and surgical complications, to enable the surveillance of the specific types of harm as well as the overall level of AEs. In the longer term, we need to automate harm surveillance and set measures of harm alongside measures of the beneficial effects of health care. Funding The National Institute for Health Research Health Services and Delivery Research programme.

These characteristics and definitions varied depending on the tools that were used.

Adverse event:  An injury to a patient related to medical management, in contrast to complications of disease. Medical management includes all aspects of care, including diagnosis and treatment, failure to diagnose or treat, and the systems and equipment used to deliver care. Adverse events may be preventable or non-preventable.

Harm: (two stage process) An unintended injury or complication that results in disability at discharge, death or prolonged hospital stay and is caused by health-care management rather than the patient’s underlying disease. Includes errors of commission and omission

Harm: (global trigger tool) Unintended injury resulting from or contributed to by medical care that requires additional monitoring, treatment or hospitalisation, or that results in death. Includes errors of commission only.

Expert commentary

A ‘Trigger Tool’ is used around the world to identify case-notes that warrant further search for adverse events. This paper shows we can get rid of Trigger Tools and just review random notes – we will find just as many adverse events. If we want a ‘canary in the mineshaft’ then we should examine adverse events in case-notes of patients who have died – they have a much higher than random adverse event rate. This is important because England/Wales have mandated reviews of hospital deaths, and the case-notes can tell us more than just whether or not the death was preventable.

Richard Lilford, Honorary Professor of Clinical Epidemiology, Institute of Applied Health Research, University of Birmingham