Safer Surgery part 14 ppsx

Safer Surgery
104
threats and increasing the likelihood of further error, creating a cascade that leads
to more serious surgical problems, and subsequently to harm or adverse event.
Threats either predispose errors that cause minor failures in process, or directly
cause minor failures themselves. These minor failures either lead to more threats,
and so to more errors, or lead directly to more serious or potentially dangerous
major failures. Major failures may expose more threats, create more errors and can
lead directly to an adverse outcome (Catchpole et al. 2005).
Though we did not directly measure surgical outcomes or observe any death,
in more than 40 cases the Great Ormond Street team observed over 500 minor
problems and 8 major problems that represented considerable lapses in the quality
of care given, and a serious threat to the safety of the patient. The reader is
directed to our accompanying chapter (Catchpole, Chapter 19) in this volume that
describes the results in orthopaedic surgery. The multidisciplinary, cross-industry
team wondered why some operations went more smoothly than others, and why in
some operations even a large number of small problems did not result in serious
problems. In part, the research team observed that the escalation from small,
Figure 7.1 Escalation model of surgical error
Rating Operating Theatre Teams
105
seemingly innocuous problems to these sometimes life-threatening situations was
dependent upon the type of operation performed, and the risk (or complexity) of
the operation. Some operations had more critical stages, and so the coincidence
of a minor problem at a critical time might also be more likely, and also certain
operations would be more demanding and thus might be more likely to result in
human errors, or were more sensitive to the overloading of individual mental and
physical capacity. The Great Ormond Street team therefore concluded that non-
technical skills, which are specically trained for in aviation to address these types
of situations, might also have an inuence in surgical care, and thus sought at rst
to derive a simple scale for evaluating this hypothesis.

NOTECHS was designed through a pan-European collaboration between
airlines and academics to provide a generic structure for non-technical skills in
aviation training and to allow consistent assessment across different national and
organisational cultures (Avermaete and van Kruijsen 1998). It was this broad
applicability, demonstrated utility (Avermaete and van Kruijsen 1998, Flin et al. 2003),
operational validation (O’Connor et al. 2002, Lodge et al. 2001) and the success with
which the method of developing the scale had been adapted to specialties in medicine
previously (Fletcher et al. 2003, 2004) that this behavioural marker methodology
was selected for adaptation to the surgical environment. The NOTECHS system
consists of elements of behaviour grouped into four categories, which can be further
grouped into two social skill categories (teamwork and cooperation, leadership and
management) and two cognitive skills categories (situation awareness and problem-
solving and decision-making), and was adapted for use with surgical teams following
consultation with two cardiac surgeons, one vascular surgeon, one orthopaedic
surgeon, one human factors researcher and two aviation non-technical skills trainers
who regularly used NOTECHS in civil airline simulation. In this rst iteration of the
Surgical NOTECHS system, the changes from aviation to surgery largely related to
alterations in language used, in the types of behaviour that were dened as markers,
and the structure of the situation awareness dimension. The Surgical NOTECHS
scale was applied to the whole team in each operation by assigning a score from 1 to
4 to each of the four dimensions (Table 7.1).
This was conducted once each for three pre-dened periods of the operation.
The rst phase, described as the access phase, started with the rst incision, and
lasted until the site of the surgical treatment had been exposed. The second phase,
known as the treatment phase, followed directly afterwards, and lasted until the
completion of the surgical treatment. The nal phase, known as the closure phase,
lasted from the completion of the surgical treatment, to the moment that the nal
suture in the closure of the incision was tied off. The human factors (non-clinically
trained) observer (KC) recorded his observations on the scoring sheet, using ticks,
crosses and notes, in order to promote consistency and balance of judgement intra-

operatively and inter-operatively, and then made a global estimate of his overall
impression of performance. For paediatric cardiac surgery, the Surgical NOTECHS
evaluation was conducted entirely from the videotaped operation. In orthopaedic
surgery, it was conducted by the observers in situ. Operations were then ranked
Safer Surgery
106
from best to worst according to the number of ‘below standard’ Surgical NOTECHS
scores obtained in each operation, then the number of ‘basic standard’ scores, then
‘standard’ and nally ‘exceed’ scores. This gave an overall order to each operation
purely in terms of the positive and negative non-technical behaviour observed.
In paediatric cardiac surgery this simple approach proved effective, useful and
informative, and helped to establish a methodology and a substantive link between
non-technical skills, process and, by implication with previous work (de Leval et al.
2000), outcome. There was a moderately close relationship between the number of
minor process problems and the ranked Surgical NOTECHS score (Figure 7.2), since
this type of surgery requires considerable use of these types of skills, particularly in
the management of blood circulation between anaesthetist, perfusionist and surgeon.
Moreover, because this type of surgery involved large teams, and is amongst the most
complex, technically demanding, and potentially risky of any surgery, the surgical
process can quickly fall apart if the operating team do not work together effectively.
Combined with previous observations regarding operative risk and operative type,
the Great Ormond Street team found that escalation from small problems to bigger
Table 7.1 Summary of rst iteration of the surgical NOTECHS scoring
system
LEADERSHIP and MANAGEMENT
Leadership, planning and preparation, workload management, authority and
assertiveness
TEAMWO
RK and COOPERATION
Team building/maintaining, support of others, understanding team needs, conict

solving
PROB
LEM-SOLVING and DECISION-MAKING
Denition and diagnosis, option generation, risk assessment, outcome review
SIT
UATION AWARENESS
Notice (patient, procedure, people), understand (patient, procedure, people), think
ahead (patient, procedure, people)
Below standard = 1 Behaviour directly compromises patient safety and effective
teamwork.
Basic standard = 2 Behaviour in other conditions could directly compromise patient
safety and effective teamwork.
Standard = 3 Behaviour maintains an effective level of patient safety and
teamwork.
Excellent = 4 Behaviour enhances patient safety and teamwork. A model for
all other teams.
Source:
Catchpole et al. 2005
Rating Operating Theatre Teams
107
problems was partially mitigated in teams with higher non-technical skills ratings
(Catchpole et al. 2007). Thus, it was possible to describe a mechanism for surgical
failure, which suggested direct methods for improvements in surgical performance
and safety, one of which was the explicit training of non-technical skills.
Though the model also tted orthopaedic surgery, it was clear that this
simple method for scoring non-technical skills as a team did not t as well when
an operation had widely differing demands on individuals. Knee replacement
operations are demanding of some members of the team, with success relying
heavily on the relationship between the scrub nurse and the surgeon, while the
anaesthetist was rarely involved directly with the surgical procedure, and had little

interaction with the team, even though, for example, patients can have rare but
extreme reactions to the surgical treatment. Thus, with nurses and surgeons who
worked well together, the course of the operation might be smooth, even though
the anaesthetist might not be as situationally aware – or as safe – as might be
desirable. The escalation from small problems to bigger ones, and the inuence of
risk, operative type and non-technical performance is illustrated in Figure 7.3.
To provide greater quality to the evaluation of non-technical skills in a wider
range of operations, and to evaluate the success of an aviation-style non-technical
skills training programme in more common types of surgery, the original method for
non-technical skills measurement needed renement. The key limitation was that
the rst iteration of the Surgical NOTECHS system could not account for different
contributions of individuals or sub-teams to the success of the surgery. This might
be particularly important where certain types of surgery did not usually demand the
constant interaction with another team member or sub-team, encouraging a lack
of awareness that could prove deleterious during the development of more acute
events in the process of surgery. Moreover, it might also help to better understand
Figure 7.2 Relationship between minor failures and ranked non-technical
skills performance in paediatric cardiac surgery
Source: Catchpole et al. 2005
Safer Surgery
108
the team dynamics, and the contribution of sub-teams to the overall success of
an operation. In the remainder of this chapter, we describe the development and
validation of this rened non-technical performance measurement methodology.
The Oxford NOTECHS System
The tool was developed by a consultant upper gastro-intestinal surgeon, a
consultant vascular surgeon, a clinical research fellow, who was also a trainee
Figure 7.3 Mechanisms of surgical failure
Rating Operating Theatre Teams
109

surgeon, and the human factors researcher and aviation trainers involved in the
rst iteration. The most obvious change from the rst iteration was the scoring of
performance of each of the three sub-teams (surgeons, anaesthetists and nurses) in
each dimension for every operation. Overall sub-team performance was taken as
the sum of the dimension performances (out of 16). The overall team non-technical
performance was calculated from the sum of the overall sub-team performance
scores (out of 48). Each overall team dimension performance was scored as the
sum of all the sub-team performances in that dimension (out of 12). Thus, a non-
technical score was obtained in each dimension for the theatre team, and for each
sub-team of surgeons, anaesthetists and nurses. Repeat scoring for three phases of
the operation was abandoned to ease the burden on the observers, and because the
operations were generally of shorter duration and without such clearly delineated
phases as cardiac surgery.
As part of a larger study that sought to examine the value of multidisciplinary
aviation-style teamwork training on performance in the operating theatre
(McCulloch et al. 2009), we applied the Oxford NOTECHS scale to examine
behavioural change. This provided the opportunity to examine a number of
properties of the scale for validity and reliability purposes. We observed 65
laparoscopic cholecystectomies (LC) and 45 carotid endarterectomies (CEA) in
the pre- and post-intervention phases of the study. For each case, a non-technical
performance score was given using the Oxford NOTECHS system. We further
evaluated relationships between technical errors, minor process problems and the
sub-team non-technical scores, as well as the four dimensions. This allowed a
more detailed analysis of the aspects of teamwork most closely associated with
changes in these outcome measures. Technical errors were identied at the same
time using the Observational Clinical Human Reliability Assessment (OCHRA)
system (Tang et al. 2004), while minor process problems, were identied using
the previous framework (Catchpole et al. 2006), and operative duration was also
recorded. In 36 cases, the teams were assessed independently by the study’s two
principal observers, and in 14 other cases by a third independent observer. Five

cases were also studied with one observer using Oxford NOTECHS, and the other
using the Observational Teamwork Assessment for Surgery (OTAS) scale (Undre
et al. 2007).
Reliability of the Oxford NOTECHS Tool
Inter-rater reliability was evaluated using the Rwg statistic for overall Oxford
NOTECHS scores and component dimensions by parallel independent scoring of
cases by two observers. Including all pre- and post-intervention cases, 24 LCs and
12 CEAs were independently dual observed. Reliability of Oxford NOTECHS
scoring is reported for all 36 cases in Table 7.2, and in CEA operations only in Table
7.3. Results for the 24 LC operations on their own have been reported elsewhere
(Mishra et al. 2009). In all cases, and for most sub-teams and dimensions, overall
Safer Surgery
110
reliability is high. In CEA reliability on the problem-solving and decision-making
dimension was lower than might be desired for some teams, as were the ratings
on the surgeons’ cognitive dimensions. This reects difculties in scoring where
observable behaviours may be limited.
For ten of the LCs and four of the CEAs, a third observer was invited to
independently score the theatre teams on their non-technical performance. Overall
reliability was good (Table 7.4) but again lowest reliability was noted in scoring
of situation awareness (SA). This is perhaps because the third observer, from
an aviation background, had a very basic understanding of the workings of an
operating theatre.
α LM TC PD SA Total
Surgeons 0.87 0.90 0.83 0.83 0.96
Anaesthetists 0.90 0.97
0.83 0.93 0.97
Nurses 0.87 0.90
0.93 0.83 0.95
Total 0.91 0.94 0.96 0.93 0.98

Table 7.2 Reliability (R
wg
) of Oxford NOTECHS tool for 36 dual observed
LCs and CEAs
α LM TC PD SA Total
Surgeons 0.89 0.89 0.91 0.86 0.96
Anaesthetists 0.94 0.91
0.93 0.93 0.97
Nurses 0.87 0.89
0.87 0.90 0.96
Total 0.94 0.93 0.96 0.96 0.98
Table 7.3 Reliability (R
wg
) of Oxford NOTECHS for 12 dual observed
CEAs
α LM TC PD SA Total
Surgeons 1.00 0.89 0.91 0.83 0.98
Anaesthetists 0.94 0.94
0.94 0.97 0.98
Nurses 0.94 0.83
0.97 0.86 0.94
Total 0.98 0.91 0.96 0.93 0.98
Table 7.4 Reliability of Oxford NOTECHS in 14 cases observed
independently with third observer
Rating Operating Theatre Teams
111
We also examined the reliability of the non-technical skills ratings over
time by dividing data in both pre- and post-intervention conditions into three,
and comparing the performance in these one-third cohorts. In laparoscopic
cholecystectomy, differences between the thirds of the cohorts were not signicant

using ANOVA either before (F=1.34, p=0.28) or after (F=1.03, p=0.34) the
training intervention. Similarly in carotid endarterectomy, differences between
the thirds of the cohorts were not signicant before (F=1.93, p=0.17) or after
(F=1.01, p=0.38). Though clearly, reliability over time needs to be studied in
more detail, this at least suggested that when comparing between pre- and post-
intervention data to examine the effect of the training programme, we could be
more condent that the effect was not brought about by changes in scores over
time.
Convergent Validity
Overall agreement between OTAS and Oxford NOTECHS was excellent (r= 0.88,
n=5, p=0.04). The mean OTAS score for the ve cases compared was 18.8 (range
14–22 out of a possible maximum of 30), and the mean Oxford NOTECHS score
was 37.8 (range 33–45, out of a possible maximum of 48), also suggesting that
data on both scales covered a similar range in relation to the overall scale maxima
and minima (see Figure 7.4).
Figure 7.4 Oxford NOTECHS scores against OTAS scores for 5 LCs
Safer Surgery
112
Relationship between Oxford NOTECHS, Technical Errors and Minor
Failures
We exploited our developing model of the relationship between non-technical skills
and intra-operative performance to examine the relationship between the Oxford
NOTECHS scores and technical errors and minor intra-operative problems. We
observed 65 LCs and 45 CEAs, in which non-technical performance was evaluated
using the Oxford NOTECHS scale, and technical errors noted using the OCHRA
system. There were no associations between non-technical performances and
technical errors of statistical signicance in CEA, but in LC there was a strong
association with surgeons’ SA (ρ = –0.54, p<0.001). Linear regression analysis
suggests that surgeons’ SA and nurses’ problem-solving and decision-making (PD)
combined are responsible for 40.5 per cent of the variation in technical errors seen.

For carotid endarterectomy, signicant relationships are between nurses’
teamwork and cooperation (TC) and minor problems (ρ=–0.30, p=0.04) and
between nurses’ sub-team Oxford NOTECHS and minor problems, but there were
no associations between non-technical skills performance and technical errors of
statistical signicance. As with LC, there is a suggestion of the number of technical
errors being negatively related to the overall non-technical performance, but in
this group the associations were not signicant.
Examining operative duration, for LC, signicant relationships were seen
between anaesthetists’ non-technical skills and operative duration (ρ=–0.25,
p=0.041) and between team leadership and management (LM) and duration
(ρ=–0.25, p=0.046). For CEA, correlations between surgeons’ LM and non-
technical skills (ρ=–0.31, p=0.037), and surgeons’ SA and non-technical skills
(ρ=–0.31, p=0.040) were signicant.
Discussion
The Oxford NOTECHS tool has been found to be reliable and to relate in different
ways, depending upon operative type, to other intra-operative performance
measures. One nding was that greater surgical situation awareness may result
in fewer technical errors for LC and CEA operations. Surgical SA is assessed
by gauging the surgical team’s awareness of the state of the patient, stage of the
procedure and availability of theatre staff. It is unsurprising therefore that excellence
in this dimension translates into technical outcome, and so perhaps disappointing
that no relationship was found in the other type of operation (CEA). Laparoscopy
is known to be cognitively demanding, which may explain the closer relationship
with cognitive skills scores. Very few technical errors were recorded in CEA, and
we did observe a trend toward the results in LC, so it may be that simply a greater
sample of cases would be necessary to demonstrate the relationship. Clearly, we
also need to be cautious with these results since SA can be difcult to observe.
Rating Operating Theatre Teams
113
Minor failures are a reection of errors produced mainly outside the operating eld,

and it is therefore logical that better coordination amongst the nursing staff results in
fewer problems. For example, where higher nursing teamwork scores were recorded,
fewer psychomotor general errors (such as dropping instruments) were noted. Clearly,
one criticism is that the Oxford NOTECHS and failure assessments were scored by the
same observer and thus may be contaminating one another. However, this relationship
was found to be consistent after analysis of scores performed by both observers, and
the Oxford NOTECHS scores were further validated by comparison with scoring by
an independent observer uninvolved in the recording of minor failures. So Oxford
NOTECHS at least seems resistant to this contamination, even though the timing
of the Oxford NOTECHS ratings at the end of the operation, after the recording of
minor failures, means it would be the more likely measure to be contaminated.
In the UK, three groups working independently and more or less simultaneously
have derived observational techniques of team performance in the operating theatre
that have a great many resemblances. Indeed, we suspect that the differences largely
relate to decisions made about the appropriate trade-offs between conicting
demands. The ANTS (Fletcher et al. 2003) and NOTSS (Yule et al. 2006) systems,
for anaesthetists and surgeons respectively, have been developed at the University
of Aberdeen to assess individuals, and primarily as training aids, and they are
both extremely well designed for these purposes (see Chapters 2, 11 and 12 in this
volume). The OTAS system (Undre et al. 2007), developed by the Clinical Safety
Research Unit at Imperial College, London, has been developed to measure team
behaviour in the operating theatre. It is prescriptive – so relatively easily trained
– but requires the full attention of a single observer (see Chapter 6). Our Oxford
NOTECHS tool has been developed to allow assessment of team performance
simultaneously with other intra-operative parameters, and has evolved from
assessing whole team performance to allow the rating of sub-teams. We have
demonstrated here that, though it may require observational experience and some
calibration, it can indeed be used reliably even by non-specialists and may relate
to other aspects of intra-operative performance. We feel that establishing the
relationship between teamwork and quality of care or operative duration may be

one way to evaluate whether training for these skills in healthcare is valuable, and
to engage front line staff in thinking differently about how they practice.
The complexity and difculty of observing performance and behaviour in the
operating theatre should not be underestimated. Meaningful, useful and reliable
data are dependent upon the skills of the observer – who is the basic tool in the
observation – and the design of the observational method to be appropriate for the
type of operation, the parameters being studied, and the purpose of the observations.
We have described the evolution and substantive afrmation of the value of the
Oxford NOTECHS technique for evaluating non-technical skills in the operating
theatre, and have demonstrated an iterative cycle that swings between measurement
and theoretical development. Thus, by building on the excellent research work
reported in this volume, we hope to work with others to develop better measurement
methods, that require fewer trade-offs to be made, that examine these skills in more