NIHR Signal Providing regular feedback on patient outcomes improves anaesthetists’ performance

Published on 23 December 2015

This NIHR-funded study found that enhanced continuous monitoring and feedback was associated with an improvement in the performance of anaesthetists. A detailed monthly report contained data about outcomes, such as nausea, pain and discomfort, for individual patients. Anaesthetists received data by subspecialty and this included details of unusual cases, comparative data across patients and anaesthetists and long-term data on trends, where available.

The study also gathered information about the impact on quality of this continuous feedback and the anaesthetists’ experiences of its implementation. Data about the effectiveness informs whether to implement continuous feedback or not. The information about people’s experiences helps maximise the chances of successful implementation. This study did not compare the outcomes or cost of continuous monitoring and feedback against the current approach of intermittent focussed audits, so it is unclear whether continuous monitoring and feedback is superior to other methods of quality improvement.

Share your views on the research.

Why was this study needed?

Monitoring the quality of care is important for NHS Trusts as they try to maintain and improve the quality of care delivered. The Care Quality Commission assesses them on this. Publishing performance data helps the public make informed decisions about which healthcare provider they want to choose. Anaesthetists also need to understand their performance in detail to make improvements, especially when undergoing professional revalidation.

Anaesthetists interact with patients before their operations, but generally have little follow-up with them afterwards. Immediately after surgery, patients may experience nausea, pain and discomfort, all of which may be affected by the quality of care provided by the anaesthetist. Currently anaesthetists receive little routine feedback on patient outcomes; instead, feedback usually comes from larger audits undertaken from time to time. The NIHR funded this study to evaluate the impact of continuous monitoring and feedback on patient outcomes and productivity, and on staff experiences of implementing the programme.

What did this study do?

The IMPAQT study evaluated two programmes of continuous monitoring and feedback in the post-anaesthetic care unit (recovery room) in 50,235 cases performed by 44 anaesthetists. The basic feedback included monthly personal data summaries with some long-term follow-up information. The enhanced feedback included individual case data, data across specialties and long-term follow-up information. Interviews with 24 consultant anaesthetists, six surgical nursing leads and five perioperative service leads provided views on how the feedback programme was implemented. Surveys were run at the start of the study, after 12 months and around 27 months into the study to collect perceptions of current quality indicators, the value of feedback in quality improvement and attitudes to quality improvement.

What did it find?

  • The basic feedback programme had little impact on patient outcomes. The enhanced feedback programme had a positive impact on patient temperature, quality of recovery, postoperative pain and postoperative nausea. These changes were the same regardless of the patients’ age, gender or disease severity.
  • Only three of the eight wards studied had a significant improvement in ward waiting time - the time between the ward being informed that the patient was ready for discharge from the recovery room and them arriving on the ward. The interviews suggested that broader organisational and infrastructure changes would be required to impact on ward waiting time.
  • Clinicians felt that it was important to collect indicators that are locally relevant and possible to interpret clinically. For example, considering the different characteristics of patients being put to sleep was seen as important when comparing anaesthetists’ performance side-by-side.
  • There was also a need to translate quality performance data into usable, actionable information at departmental and individual level.
  • Anaesthetists clearly identified a role for this quality improvement initiative in their revalidation. Connecting the quality improvement initiative with their appraisal and revalidation appeared to increase anaesthetists’ levels of engagement.
  • Following the implementation of the feedback programme, perceptions of the effectiveness of quality monitoring and quality indicators improved.

What does current guidance say on this issue?

The Royal College of Anaesthetists has produced a guide to undertaking audits for continuous quality improvement. The publication covers different aspects of quality improvement, including how to choose and measure your outcomes, ensuring that audits lead to changes in practice and disseminating your findings. It also contains a number of “recipes” for different types of audits that can be undertaken.

What are the implications?

The IMPAQT study indicates that enhanced continuous monitoring and feedback can improve the quality of care delivered by anaesthetists. The use of mixed methods provides data on the effectiveness of the programme and on the experience of implementing it. This will enable people to learn from that experience and plan the implementation to maximise the chance of successful implementation. Monthly feedback of personal data summarised in table form was not effective.

It is not clear whether continuous feedback delivers a greater improvement in outcomes compared with the traditional “discontinuous” audit process. The implementation and on-going cost of each approach was not evaluated. Without such cost-benefit analysis it is not possible to say whether continuous monitoring and feedback should be implemented more widely.

Citation

Benn J, Arnold G, D’Lima D, et al. Evaluation of a continuous monitoring and feedback initiative to improve quality of anaesthetic care: a mixed-methods quasi-experimental study. Health Serv Deliv Res.2015;3(32).

This project was funded by the National Institute for Health Research Health Services and Delivery Research programme.

Bibliography

RCOA. Raising the standard: a compendium of audit recipes for continuous quality improvement in anaesthesia. London: Royal College of Anaesthetists; 2012.

Why was this study needed?

Monitoring the quality of care is important for NHS Trusts as they try to maintain and improve the quality of care delivered. The Care Quality Commission assesses them on this. Publishing performance data helps the public make informed decisions about which healthcare provider they want to choose. Anaesthetists also need to understand their performance in detail to make improvements, especially when undergoing professional revalidation.

Anaesthetists interact with patients before their operations, but generally have little follow-up with them afterwards. Immediately after surgery, patients may experience nausea, pain and discomfort, all of which may be affected by the quality of care provided by the anaesthetist. Currently anaesthetists receive little routine feedback on patient outcomes; instead, feedback usually comes from larger audits undertaken from time to time. The NIHR funded this study to evaluate the impact of continuous monitoring and feedback on patient outcomes and productivity, and on staff experiences of implementing the programme.

What did this study do?

The IMPAQT study evaluated two programmes of continuous monitoring and feedback in the post-anaesthetic care unit (recovery room) in 50,235 cases performed by 44 anaesthetists. The basic feedback included monthly personal data summaries with some long-term follow-up information. The enhanced feedback included individual case data, data across specialties and long-term follow-up information. Interviews with 24 consultant anaesthetists, six surgical nursing leads and five perioperative service leads provided views on how the feedback programme was implemented. Surveys were run at the start of the study, after 12 months and around 27 months into the study to collect perceptions of current quality indicators, the value of feedback in quality improvement and attitudes to quality improvement.

What did it find?

  • The basic feedback programme had little impact on patient outcomes. The enhanced feedback programme had a positive impact on patient temperature, quality of recovery, postoperative pain and postoperative nausea. These changes were the same regardless of the patients’ age, gender or disease severity.
  • Only three of the eight wards studied had a significant improvement in ward waiting time - the time between the ward being informed that the patient was ready for discharge from the recovery room and them arriving on the ward. The interviews suggested that broader organisational and infrastructure changes would be required to impact on ward waiting time.
  • Clinicians felt that it was important to collect indicators that are locally relevant and possible to interpret clinically. For example, considering the different characteristics of patients being put to sleep was seen as important when comparing anaesthetists’ performance side-by-side.
  • There was also a need to translate quality performance data into usable, actionable information at departmental and individual level.
  • Anaesthetists clearly identified a role for this quality improvement initiative in their revalidation. Connecting the quality improvement initiative with their appraisal and revalidation appeared to increase anaesthetists’ levels of engagement.
  • Following the implementation of the feedback programme, perceptions of the effectiveness of quality monitoring and quality indicators improved.

What does current guidance say on this issue?

The Royal College of Anaesthetists has produced a guide to undertaking audits for continuous quality improvement. The publication covers different aspects of quality improvement, including how to choose and measure your outcomes, ensuring that audits lead to changes in practice and disseminating your findings. It also contains a number of “recipes” for different types of audits that can be undertaken.

What are the implications?

The IMPAQT study indicates that enhanced continuous monitoring and feedback can improve the quality of care delivered by anaesthetists. The use of mixed methods provides data on the effectiveness of the programme and on the experience of implementing it. This will enable people to learn from that experience and plan the implementation to maximise the chance of successful implementation. Monthly feedback of personal data summarised in table form was not effective.

It is not clear whether continuous feedback delivers a greater improvement in outcomes compared with the traditional “discontinuous” audit process. The implementation and on-going cost of each approach was not evaluated. Without such cost-benefit analysis it is not possible to say whether continuous monitoring and feedback should be implemented more widely.

Citation

Benn J, Arnold G, D’Lima D, et al. Evaluation of a continuous monitoring and feedback initiative to improve quality of anaesthetic care: a mixed-methods quasi-experimental study. Health Serv Deliv Res.2015;3(32).

This project was funded by the National Institute for Health Research Health Services and Delivery Research programme.

Bibliography

RCOA. Raising the standard: a compendium of audit recipes for continuous quality improvement in anaesthesia. London: Royal College of Anaesthetists; 2012.

Evaluation of a continuous monitoring and multi-level feedback initiative to improve quality of anaesthetic care and perioperative workflow efficiency

Published on 4 August 2015

Benn J, Arnold G, D'Lima D, Wei I, Moore J, Aleva F, Smith A, Bottle R, Brett S.

Health Services and Delivery Research Volume 3 Issue 32 , 2015

Objectives To conduct a quasi-experimental evaluation of the feedback initiative and its effect on quality of anaesthetic care and perioperative efficiency. To understand the longitudinal effects of passive and active feedback and investigate the mechanisms and interactions underpinning those effects. Design Mixed-methods evaluation with analysis and synthesis of data from longitudinal qualitative interviews, longitudinal evaluative surveys and an interrupted time series study. Intervention Continuous measurement of a range of anaesthetic quality indicators was undertaken in a London teaching hospital alongside monthly personal feedback from case summary data to a cohort of anaesthetists, with follow-up roll-out to the whole NHS trust. Basic feedback consisted of the provision of passive monthly personalised feedback reports containing summary case data. In the enhanced phase, data feedback consisted of more sophisticated statistical breakdown of data, comparative and longitudinal views, and was paired with an active programme of dissemination and professional engagement. Methods Baseline data collection began in March 2010. Implementation of basic feedback took place in October 2010, followed by implementation of the enhanced feedback protocol in July 2012. Weekly aggregated quality indicator data, coupled with surgical site infection and mortality rates, was modelled using interrupted time series analyses. The study anaesthetist cohort comprised 50,235 cases, performed by 44 anaesthetists over the course of the study, with 22,670 cases performed at the primary site. Anaesthetist responses to the surveys were collected pre and post implementation of feedback at all three sites in parallel with qualitative investigation. Seventy anaesthetists completed the survey at one or more time points and 35 health-care professionals, including 24 anaesthetists, were interviewed across two time points. Results Results from the time series analysis of longitudinal variation in perioperative indicators did not support the hypothesis that implementation of basic feedback improved quality of anaesthetic care. The implementation of enhanced feedback was found to have a significant positive impact on two postoperative pain measures, nurse-recorded freedom from nausea, mean patient temperature on arrival in recovery and Quality of Recovery Scale scores. Analysis of survey data demonstrated that anaesthetists value perceived credibility of data and local relevance of quality indicators above other criteria when assessing utility of feedback. A significant improvement in the perceived value of quality indicators, feedback, data use and overall effectiveness was observed between baseline and implementation of feedback at the primary site, a finding replicated at the two secondary sites. Findings from the qualitative research elucidated processes of interaction between context, intervention and user, demonstrating a positive response by clinicians to this type of initiative and willingness to interact with a sustained and comprehensive feedback protocol to understand variations in care. Conclusions The results support the potential of quality monitoring and feedback interventions as quality improvement mechanisms and provide insight into the positive response of clinicians to this type of initiative, including documentation of the experiences of anaesthetists that participated as users and codesigners of the feedback. Future work in this area might usefully investigate how this type of intervention may be transferred to other areas of clinical practice and further explore interactions between local context and the successful implementation of quality monitoring and feedback systems.

Expert commentary

Reducing variation in care and improving patient outcomes after surgery are shared aims of the Royal College of Anaesthetist’s Perioperative Quality Improvement Programme and Perioperative Medicine Programme. This study highlights the importance of monitoring patient outcomes combined with effective feedback. If implemented, clinical teams and individuals are better able to understand variation in care and pro-actively respond to opportunities to improve standards and evaluate impact of changes to services.

A challenge is how to actually use the data collected for the purposes of improvement. Broader professional engagement activities including regular presentation of local results and topic-based education meetings can assist with this, as can multi-professional team meetings. These can focus on how to respond to evidence of variation in care and opportunities for improvement.

Sharon Drake, Director of Education and Research, Royal College of Anaesthetists