Safer Surgery part 4 doc

Safer Surgery
4
Millman, M. (1976) The Unkindest Cut. Life in the Backrooms of Medicine. New
York: Morrow Quill.
Ruhlman, M. (2003) Walk on Water. Inside an Elite Paediatric Unit. New York:
Viking.
Weston, G. (2009) Direct Red. A Surgeon’s Story. London: Jonathan Cape.
Wilpert, B. (1996) Psychology in high hazard systems: Contribution to safety
and reliability. In J. Georgas, M. Manthouli, E. Besevegis and A. Kokkevi
(eds) Contemporary Psychology in Europe. Proceedings of the IVth European
Congress of Psychology. Seattle, WA: Hogrefe and Huber.
PART I
Tools for Measuring Behaviour in
the Operating Theatre
This page has been left blank intentionally
Chapter 2
Development and Evaluation of the NOTSS
Behaviour Rating System for Intraoperative
Surgery (2003–2008)
Steven Yule, Rhona Flin, Nikki Maran, David Rowley, George Youngson,
John Duncan and Simon Paterson-Brown
Introduction
In 2002, a number of surgeons in Scotland were intrigued by the development of
the ANTS (Anaesthetists’ Non-Technical Skills) system and the use of behaviour
rating checklists in other industries such as nuclear power and civil aviation.
There was a realization in the healthcare and medical literature that adverse events
occurred in the operating theatre. Surgeons and their patients have also had to
come to terms with the uncovering and analysis of the true nature and extent of
surgical misadventure and failure. Ten years ago, it was not generally realized that
a signicant number of surgical patients were harmed not as a result of underlying
illness or disease but as a result of their treatment (Vincent et al. 2001). Further

analysis of this problem revealed that non-technical aspects of performance play a
contributory role in the multifaceted nature of surgical adverse events; failures in
decision-making, teamwork, coordination and leadership have all emerged from
case reviews and studies of behaviour in the operating theatre (Gawande et al.
2003, Studdert et al. 2006, Christian et al. 2006). Non-technical skills are dened as
the critical cognitive and interpersonal skills that complement surgeons’ technical
ability (Yule et al. 2006a). Despite the fact that the behavioural (Baldwin et al.
1999) and cognitive demands of surgery have been recognized as a critical part
of surgical performance (Hall et al. 2002, Jacklin et al. 2008), and that effective
leadership has been shown to improve team performance (Edmondson 2003),
non-technical skills are often referred to as ‘non-operative’ and deemed not as
important as clinical science in the surgical literature. Scant attention has been paid
to the cognitive and social processes that underpin intra-operative performance in
training as well: training and assessment in these skills are only conducted in a
rather tacit and discretionary basis, and the surgical curriculum in the UK does not
yet extend to non-technical skills.
Safer Surgery
8
Amid this backdrop, the surgical profession has also been rapidly changing
to cope with internal and external pressures such as the European Working Time
Directive, which restricts the working week to 48 hours; the challenges of new
professional roles such as nurse practitioners (Kneebone and Darzi 2005), the
modernization of training and education including Medical Training Application
Service (MTAS) and new technology. These changes mean that trainees have fewer
training opportunities than their trainers had, before reaching consultant level, so
there is now a greater need to maximize the available learning opportunities.
Changes to the conguration of surgical training and education are currently
underway in the UK to attempt to streamline development of doctors and ensure
that they are skilled at communicating and working as effective members of a
team. This approach recommends that progress through and completion of surgical

training be based on competence; it has moved the emphasis of assessment away
from set-piece examinations of knowledge towards learning and assessment of
skills in the workplace (see Pitts and Rowley, Chapter 3 of this volume). Selection
of trainees into surgical specialties has also been radically altered and provides an
opportunity to formalize the role of non-technical skills in surgical education and
assessment.
The main methods of workplace-based assessment of surgical trainees in
the UK are observational tools which cover skills such as ability to work in a
multi-professional team (Mini-PAT: Peer Assessment Tool) and communication
(Mini-CEX: Clinical Evaluation Exercise). However, these tools are for the
assessment of perioperative skills, often using interactions with patients as a basis
for assessment. This is to be encouraged, but the skills assessed do not necessarily
relate to those required for working with other professionals during a surgical
procedure, commonly with an anaesthetized patient. The systems that are used
to assess trainees’ intra-operative competence, such as surgical DOPS (Direct
Observation of Procedural Skills) and Procedure Based Assessment (PBA) are
focused almost entirely on technical ability. However PBAs, which are written for
specic index procedures, sometimes integrate non-technical aspects (see in this
volume Chapter 3 by Pitts and Rowley, Chapter 4 by Marriott et al.).
The cognitive and social skills which underpin clinical and technical prociency
are recognized as requirements for surgical competency and rank highly as core
competencies within organizations such as CanMeds (Frank 2005), the General
Medical Council (GMC 2001), and the Royal Colleges of Surgeons in the UK
(Youngson 2000, Giddings and Williamson 2007) but until recently there were no
tools to reliably assess these skills in the workplace. To begin to address this, a
behavioural observation and rating system called NOTSS (Non-Technical Skills
for Surgeons) was developed and tested under funding from the Royal College
of Surgeons of Edinburgh and NHS Education for Scotland, from 2003 to 2007.
This chapter outlines the development and evaluation of the NOTSS system. Like
similar systems in civil aviation and anaesthesia, the behaviour rating system was

based on a skills taxonomy which was developed with subject matter experts.
Development and Evaluation of the NOTSS Behaviour Rating System
9
NOTSS Project Design
The project was run by the University of Aberdeen, with a multidisciplinary
steering group of surgeons, psychologists and an anaesthetist. The research drew
on previous work in Scotland on surgical competence, professionalism and the
skills surgeons required to operate safely

and followed on from a similar project
which developed a behaviour rating system for anaesthetists – the ANTS system

(Fletcher et al. 2004, see Chapter 11 in this volume by Glavin and Patey). The aim
of the NOTSS project was to develop and test an educational system for assessment
and training based on observed skills in the intra-operative phase of surgery. The
system was developed from the bottom up with subject matter experts (consultant
surgeons), instead of adapting existing frameworks used in other industries.
It was considered important to recognize and understand the unique aspects of
non-technical skills in surgery, and not to assume that those non-technical skills
identied for pilots, nuclear power controllers or anaesthetists would be exactly
mirrored in, or be relevant to, surgery. The NOTSS system is in surgical language
for suitably trained surgeons to observe, rate and provide feedback on non-technical
skills in a structured manner. An adaptation of Gordon’s (1993) model of systems
design

was used to guide the iterative development of NOTSS.
This three-phase model maps the process from task analysis through system
design to evaluation. The phases relate to the three objectives set by the NOTSS
steering group in 2003: to identify the relevant non-technical skills required by
surgeons, to develop a system to allow surgeons to rate these skills, and to test the

system for reliability and usability. A fourth phase was added to cover a trial in the
operating theatre using NOTSS to debrief surgical trainees over the course of an
attachment (see Figure 2.1).
Phase 1: Task Analysis
In Phase 1 we used three main methods to collect data on individual surgeons’
intra-operative non-technical skills, as follows:
Literature review on surgeons’ non-technical skills (Yule et al. 2006a).
Survey of theatre personnel attitudes to teamwork, error and safety (Flin
et al. 2006a).
Critical incident interviews with subject matter experts (Yule et al.
2006b).
These methods were supported by eld notes taken during observation sessions
in the operating theatre during operative surgery and a review of surgical adverse
event and mortality reports.
1.
2.
3.
Safer Surgery
10
Literature Review
The aims of the literature review (Yule et al. 2006a) were to examine the surgical
and psychological literature on surgeons’ intra-operative non-technical skills in
order to (i) identify the non-technical skills required by surgeons in the operating
theatre, and (ii) assess the behavioural marker systems that have been developed
for rating surgeons’ non-technical skills. In order to achieve this, we searched
the literature using dened search terms and a set of inclusion criteria. Databases
searched included BioMed Central, Medline, Web-of-Knowledge, PsychLit, and
ScienceDirect. Relevant studies were organized according to the source material
used. This yielded published research from observational studies, questionnaires
and interviews, adverse event analyses and papers on surgical education (including

curricula and standards of competence). Within these, the review highlighted
the main non-technical skill categories to be: anticipation, decision-making,
teamworking, leadership and communication. At the time of the review (August
2005), there were three research tools in the literature which could be used to
measure surgeons’ non-technical skills. On closer examination, these existing
frameworks were found to be decient either in terms of their psychometric
Figure 2.1 Developing the NOTSS system
Phase 2: Design and development (Yule et al., 2006b)
Iterative development (n=4 panels of consultant surgeons)
Write and agree behaviour markers (n=16 consultant surgeons)

Phase 4: Debriefing on non-technical skills (in progress)

Phase 3: System evaluation (Yule et al., 2008a, 2008b)
Reliability (standardized scenarios, n=44 consultant surgeons)
Usability: 2 studies, n=27 surgeon-trainee dyads in total

Phase 1: Task analysis (Yule et al., 2006a; Flin et al., 2006a)
Literature review, cognitive interviews (n=27), attitude survey, Adverse
event report analysis

Developing the NOTSS system
(based on Gordon, 1993)
Development and Evaluation of the NOTSS Behaviour Rating System
11
properties or suitability for assessing individual surgeons rather than a surgical
team in theatre. On the basis of this review, we concluded that further research
was required to develop a taxonomy of individual surgeons’ non-technical skills
for training and feedback.
Attitude Survey (ORMAQ)

The literature review highlighted the lack of basic data on cognitive and social
skills in surgeons, and little was known about prevailing attitudes to teamwork
and safety in the operating theatre. Attitude surveys of theatre personnel had
been conducted in other countries and can provide useful diagnostic information
relating to behaviour and safety in surgical units. There were no such data
available in Scotland, so as part of our initial task analysis, we ran a baseline
survey (Flin et al. 2006a) using a version of the Operating Room Management
Attitudes Questionnaire (ORMAQ), initially developed by Helmreich et al.
(1997) to assess surgical team members’ attitudes to safety and teamwork in
operating theatres. The ORMAQ was adapted from an instrument measuring
pilots’ safety attitudes in aviation. At the time (late 2005), it was the most
extensively used attitudes questionnaire with operating theatre personnel with
data collected from Israel, USA, Germany, Switzerland and Italy (Helmreich and
Schaefer 1994, Helmreich and Davies 1996, Sexton et al. 2000). It was not clear
to what extent these earlier ndings would generalize to a British sample but the
questionnaire topics of leadership, teamwork, stress and fatigue and error were
shown to be relevant from our literature review. The ORMAQ was modied for
language only by a panel of consultant surgeons and was distributed to surgical
teams in 17 hospitals in Scotland. A total of 352 responses were analysed, 138
from consultant surgeons (response rate: 47 per cent), 93 from trainee surgeons
(27 per cent) and 121 from theatre nurses (19 per cent). Respondents generally
demonstrated positive attitudes to behaviours associated with effective teamwork
and safety. Attitudes indicating a belief in personal invulnerability to stress and
fatigue were evident in both nurses and surgeons. Consultant surgeons had
more positive views on the quality of surgical leadership and communication in
theatre than trainees and theatre nurses. While the ubiquity of human error was
well recognized, attitudes to error management strategies (incident reporting,
procedural compliance) suggest that they may not be fully functioning across
hospitals. While theatre staff placed a clear priority on patient safety, against
other business objectives (e.g., waiting lists, cost cutting), not all of them felt

that this was endorsed by their hospital management. Discrepancies were found
between the views of consultants compared to trainees and nurses, in relation
to leadership and teamwork. While attitudes to safety were generally positive,
there were several areas where theatre staff did not seem to appreciate the impact
of psychological factors on technical performance. These results were taken into
consideration in the design of the NOTSS system.
Safer Surgery
12
Observations
To provide context and meaning for the literature review and interviews, a
psychology researcher conducted observations of surgical cases. Observations
were made at three hospitals in a variety of specialisms: general, orthopaedic and
cardiac surgery. No formal method was used for structuring observations because
we did not want to narrow the observer’s data collection at this stage. Field notes
were taken. The observer also shadowed surgeons in the perioperative environment
to understand how this stage impacted on operative performance. During this
phase of the project, detailed eld notes revealed that surgeons displayed a range
of non-technical skills, communication was variable and there often seemed to
be conicting priorities between training and service delivery. There was no
standard method of conducting a given operation, the atmosphere or climate in
the operating theatre would change depending on which surgeon was operating
that day, and the number of people in the operating theatre ranged from four to
eighteen. In comparison with other industries, the formal work procedures, if they
existed, were not explicit. Team members seemed to start critical tasks such as
commencing the anaesthetic, positioning the patient and making the rst incision
to start the operation without speaking to other members. Often operations would
start without critical team members in the operating theatre and without all the
information being present. Distractions seemed to be commonplace and normal;
on several occasions the operating surgeon had to answer questions about another
ongoing operation or speak to someone on the telephone while in the middle of

what appeared to be a complex part of the operation for which he or she was
scrubbed. Despite all this, the observers were struck by how well the surgeon and
the team performed under those circumstances. As with any observation study, it
was not possible to plan to see surgeons perform under stress or to analyse surgical
adverse events in a systematic manner during live cases. For this, we selected
other methods, as will be discussed below.
Adverse Event and Mortality Reviews
The systematic analysis of near misses, incidents and accidents is an essential
diagnostic process for safety management in industry (Reason 1997) and we thought
that these data sets in surgery could provide us with a rich source of information
on error and surgical failings that would credibly t into the skills analysis for
NOTSS. Surgical colleagues indicated that data were not usually collected on
non-technical skills, so this would be a short task. In the end, we reviewed the
Scottish Audit of Surgical Mortality (SASM) reports from 2001 (SASM 2003)
and commented on them in the literature review. The nature of data fed back to
individual hospitals and in case assessments highlights that SASM is strong on
providing technical feedback and on reporting the proximal causes of error but
provides relatively little in the way of human factors information and therefore
offered limited insight into non-technical skills in surgery. There are two likely
Development and Evaluation of the NOTSS Behaviour Rating System
13
causes of this: (i) the forms used to collect data do not adequately capture human
factors or non-technical contributions to incidents, and (ii) the coding framework
used to analyse the incident reports does not adequately deal with non-technical
skills. A similar situation emerged in the analyses of anaesthetic adverse event
reports for the ANTS project (Fletcher et al. 2004). These conditions explain the
current technical (e.g., what happened) bias in published audit reports in favour of
non-technical (e.g., why it happened) causes of adverse events. The SASM forms
since 2007 include non-technical skills categories.
Critical Incident Interviews

The critical incident technique (CIT) is a type of cognitive interview (Crandall et
al. 2006, Flanagan 1954, Hoffmann et al. 1998) used to identify tacit knowledge
about the way an expert manages a stressful or non-routine situation at work. CITs
were conducted with 27 consultant surgeons in order to identify non-technical
skills used by surgeons in the intra-operative environment. By focusing on a
specic memorable incident, the interviews provided insight into the surgeon’s
use of information, strategies, meta-cognition, resources and interpersonal skills
during an operative case (Yule et al. 2006b; see also Fletcher et al. (2004) who
used this technique with anaesthetists). To summarize the method, surgeons
were asked to recall events in theatre during a challenging, non-routine case and
were probed about the course of events a further two times. After the surgeon
described the case, the interviewer recounted the sequence of events back to the
surgeon and asked for clarication and more explanation of the course of events.
This second sweep of the case allowed for more detail to be gleaned. The case
was then discussed for a third time with the addition of cognitive cues which
recreate aspects of the case to elicit deeper-held tacit knowledge about the non-
technical skills that were or were not being used. Examples of the cognitive cues
used include: ‘what cues were you using to help understand the situation’ and
‘how did you re-establish goals?’ The interview questions were developed by a
multidisciplinary group, based on work in other domains including anaesthesia
and piloted with three consultant surgeons. The sample of surgeons interviewed
were consultant surgeons (n=27) from 11 hospitals in Scotland in general
surgery (n=13), orthopaedic surgery (n=10) and cardiac surgery (n=4). One of
the participants was female. A variety of cases were discussed in the interviews
which lasted around one hour each, including emergencies with duodenal ulcers,
difculties in hip and knee replacements, problems in transplant operations and
difculties with cardiac bypass. The interview transcripts were analysed using the
line-by-line coding technique from grounded theory (Glaser and Strauss 1967)
in order to explore the data and aid system development. Coders were asked to
identify when non-technical skills were discussed in the interview and to interpret

those specic skills. Three pairs of psychologists who were experienced at coding
interview transcripts each coded six transcripts independently to an acceptable
level of inter-rater reliability before the remaining transcripts were then coded. This