A rationale to unify measurements of effectiveness for animal health surveillance

Preventive Veterinary Medicine 120 (2015) 70–85

Contents lists available at ScienceDirect

Preventive Veterinary Medicine
journal homepage: www.elsevier.com/locate/prevetmed

A rationale to unify measurements of effectiveness for animal
health surveillance
Vladimir Grosbois a,∗ , Barbara Häsler b , Marisa Peyre a , Dao Thi Hiep c ,
Timothée Vergne b
a
UPR AGIRs, Animal and Integrate Risk Management, International Research Center in Agriculture for Development (CIRAD), TA C 22/E
Campus International Baillarguet, 34398 Montpellier Cedex 5, France
b
Veterinary Epidemiology, Economics and Public Health, The Royal Veterinary College, Hawkshead Lane, North Mymms, Hatfield, Herts
AL9 7TA, United Kingdom
c
Center for Interdisciplinary Research on Rural Development, Vietnam National University of Agriculture, Trau Quy, Gia Lam, Hanoi,
Viet Nam

a r t i c l e

i n f o

Article history:
Received 3 July 2014
Received in revised form 5 December 2014
Accepted 15 December 2014
Keywords:
Intervention

Disease surveillance
Decision making
Type I error
Type II error

a b s t r a c t
Surveillance systems produce data which, once analysed and interpreted, support decisions
regarding disease management. While several performance measures for surveillance are
in use, no theoretical framework has been proposed yet with a rationale for defining and
estimating effectiveness measures of surveillance systems in a generic way. An effective
surveillance system is a system whose data collection, analysis and interpretation processes lead to decisions that are appropriate given the true disease status of the target
population. Accordingly, we developed a framework accounting for sampling, testing and
data interpretation processes, to depict in a probabilistic way the direction and magnitude
of the discrepancy between “decisions that would be made if the true state of a population
was known” and the “decisions that are actually made upon the analysis and interpretation
of surveillance data”. The proposed framework provides a theoretical basis for standardised quantitative evaluation of the effectiveness of surveillance systems. We illustrate such
approaches using hypothetical surveillance systems aimed at monitoring the prevalence of
an endemic disease and at detecting an emerging disease as early as possible and with an
empirical case study on a passive surveillance system aiming at detecting cases of Highly
Pathogenic Avian Influenza cases in Vietnamese poultry.
© 2015 Elsevier B.V. All rights reserved.

1. Introduction
The past 20 years have seen wide-reaching economic,
social and political impact of large-scale animal disease
outbreaks such as bovine spongiform encephalopathy, foot
and mouth disease or avian influenza (Caspari et al., 2007;
Knight-Jones and Rushton, 2013; Otte et al., 2004). These

∗ Corresponding author. Tel.: +33 467593833; fax: +33 467593799.

E-mail address: (V. Grosbois).
/>0167-5877/© 2015 Elsevier B.V. All rights reserved.

shockwaves emphasize the need for well-developed and
adequately resourced health systems, including animal
health surveillance (Rushton and Upton, 2006). Moreover,
there are various endemic diseases that do not get the
same attention as large, unexpected outbreaks, but that
cause continuous losses for society in terms of human
disease, decreased productivity in animals and negative
consequences for animal welfare (Otte et al., 2004; KnightJones and Rushton, 2013). Importantly, to combat animal
disease, resources must be allocated to surveillance, prevention and intervention efforts that could otherwise be


V. Grosbois et al. / Preventive Veterinary Medicine 120 (2015) 70–85

used for alternative purposes (Häsler et al., 2011; Howe
et al., 2013). While the need for effective animal health
surveillance is widely recognized for the management of
animal health threats, investment is being constrained due
to financial budget restrictions. Therefore, there is strong
demand for frameworks that allow assessing the economic
value of surveillance programmes that inform decision
about investments for surveillance.
Surveillance has been defined as the systematic measurement, collection, collation, analysis, interpretation,
and timely dissemination of animal-health and -welfare
data from defined populations essential for describing
health-hazard occurrence and to contribute to the planning, implementation, and evaluation of risk-mitigation
actions (Hoinville et al., 2013). In other words, surveillance
provides information for decisions regarding the implementation of interventions. Together surveillance and

intervention achieve loss avoidance through the process
of making the effects of disease less severe by avoiding, containing, reducing or removing it – the outcome
decision-makers are ultimately interested in (Häsler et al.,
2011). If surveillance information shows that the disease
situation is not of concern, then a decision may be taken
not to do anything. In reality, the decision to implement
an intervention does not only depend on the disease situation and the information provided by the surveillance
system, but also on multiple other factors such as social
expectations, political will, or practical considerations. This
article focuses on one decision-factor only, namely the
quality of information provided by the surveillance system,
while fully acknowledging the multi-factorial complexity
of decision-making. Keeping other factors constant, this
article aims to provide a rationale for measurement of the
effectiveness of information produced by animal health
surveillance that is used to make a decision on disease
management.
Surveillance data are generated through reporting, diagnosing, sampling and testing processes. Often reporting
and/or sampling are not exhaustive, and sometimes can
be non-representative. Moreover diagnostic and/or sample testing procedures usually misclassify a fraction of
the examined units and tested samples. The data generated by surveillance systems are thus most of the time
non-exhaustive, partially distorted and sometimes nonrepresentative. Decisions regarding the implementation of
intervention measures nonetheless rely on the assessment
through the analysis and interpretation of such imperfect
data of the epidemiological status of target populations or
of focal units in target populations (Häsler et al., 2011;
Howe et al., 2013). Consequently, even with perfectly
tailored response mechanisms, ineffective surveillance
can result in misjudging an epidemiological situation
and adopting inappropriate intervention measures. Thus,

although surveillance systems produce imperfect data,
they should provide information that is reliable enough for
suitable decisions on intervention measures to be made.
The challenge for surveillance systems is therefore to
maximise the reliability of the information it produces
through the optimization of the data generation and interpretation processes. For doing so, attributes reflecting
information reliability need to be assessed and variation

71

in such attributes with regard to the characteristics of data
generation and interpretation processes need to be investigated. So far, numerous attributes such as sensitivity,
specificity, negative predictive value, positive predictive
value, bias, precision and timeliness have been proposed
to quantify such reliability (German et al., 2001; Hendrikx
et al., 2011; Drewe et al., 2012, 2015; Hoinville et al.,
2013). Moreover, effectiveness evaluations often aim at
optimizing a specific aspect of the surveillance process
which differs according to the objectives of the surveillance system considered. Evaluations of the effectiveness
of surveillance systems aiming at demonstrating freedom
from disease most often focused on the sampling process (random vs. risk-based, sample size, e.g. Martin et al.,
2007a). Such evaluation for systems aiming at detecting
early the introduction of an emerging pathogen commonly
focused on the comparison of the timeliness or componentlevel sensitivity of distinct surveillance components (e.g.
Yamamoto et al., 2008; Knight-Jones et al., 2010). Finally,
evaluations of the effectiveness of syndromic surveillance
mainly focused on statistical algorithms for the detection of anomalies in time series (e.g. Dórea et al., 2013).
None of these studies explored the meaning of performance attributes in general and did not establish a generic
theoretical foundation for the measurement of effectiveness independent of the surveillance objective or approach
used. Consequently, there is little guidance available about

what the common denominator is of the performance
measures listed above and what differentiates them. This
can not only lead to confusion among users, but also limit
standardisation and comparison of studies aiming to assess
surveillance performance.
Here we present a rationale which can be used to assess
effectiveness whatever is the objective of the surveillance
system considered. It is assumed that the primary effectiveness criterion is the ability of a surveillance system to
provide information that is reliable enough for decision
makers to implement mitigation measures similar to those
they would implement given a perfect knowledge of an
epidemiological situation. This rationale allows developing
optimization studies for any aspect of the surveillance process and forms an important basis for economic evaluation
of surveillance.
2. General overview of the rationale
The rationale we propose to assess the effectiveness of
a surveillance system relies on the principle that the decisions that are made based on the information produced by
surveillance should not differ greatly from the decisions on
interventions that would be made given perfect knowledge
of the epidemiological situation (i.e. of the epidemiological
state of the target population and of its components). It
requires reviewing several aspects of mitigation strategies
and processes, as detailed below and highlighted in Fig. 1.
2.1. Defining relevant epidemiological scales and state
variables
The epidemiological scale and the state variable(s) that
are relevant with regard to the objectives of surveillance


72

V. Grosbois et al. / Preventive Veterinary Medicine 120 (2015) 70–85

SURVEILLANCE DATA
Non-exhausƟve, non-representaƟve,
parƟally distorted

Data generaƟon process
Sampling, reporƟng,
diagnosing, tesƟng

Data analysis and
interpretaƟon

TRUE EPIDEMIOLOGICAL SITUATION

ASSESSMENT EPIDEMIOLOGICAL
SITUATION

IntervenƟon strategy
Defined based on epidemiological
modelling and cost-effecƟveness
and/or cost-benefit analyses

PREVENTION/CONTROL MEASURES
That would be implemented given a
perfect knowledge of the
epidemiological situaƟon

Decision making process

Surveillance EffecƟveness

EFFECTIVENESS AND ECONOMIC
EFFICIENCY
of the prevenƟon/control measures
that would be implemented
given a perfect knowledge of the
epidemiological situaƟon

PREVENTION/CONTROL MEASURES
That are actually implemented
(modaliƟes/intensity)

EFFECTIVENESS AND ECONOMIC
EFFICIENCY of the risk
prevenƟon/control measures that are
actually implemented

Fig. 1. Proposed approach for the evaluation of the effectiveness of a surveillance system.
Table 1
Examples of simple intervention strategies for various surveillance objectives.
Surveillance
objective

Scale

State variable

S−

I−

S+

I+

Monitoring
prevalence

Country/region

Yearly
prevalence of a
disease (Prev)

Prev ≤ Threshold

Do nothing

Prev > Threshold

Disease case
detection

Herd

Disease status

Do nothing

Demonstrate
freedom
from disease
Early detection
of an
emerging
disease

Country/region

Yearly
prevalence of a
disease (Prev)
Instantaneous
incidence rate
(IIR)

No infected
animal in the
herd
Prev ≤ Threshold

≥1 infected
animal in the
herd
Prev > Threshold

Implement
systematic

testing in
slaughterhouses before
products are
put on the
market
Cull the herd

IIR = 0

Do nothing

Country/region

Allow
exportations

IIR > 0

Ban
exportations
Launch
intensive
surveillance
and in depth
case
investigation.
Limit
movements

S− , S+ : epidemiological states for which the “no intervention” and “intervention” options, respectively, are required; I− , I+ : description of actions associated

with to the “no intervention” and “intervention” options.

form the basis for an intervention decision. The relevant
epidemiological scale is the scale at which decisions are
being made about implementing an intervention. Such
decisions can be for example to start vaccinating animals

in the target population if the disease prevalence crosses
a defined threshold or not to do anything if surveillance
to document freedom from disease delivers the expected
result (i.e. freedom). The scale can be animal, herd, country,


V. Grosbois et al. / Preventive Veterinary Medicine 120 (2015) 70–85

regional or global level. The relevant state variable is a
variable, such as prevalence or incidence, that reflects
the current epidemiological situation, and which value
determines the intervention measures considered as
appropriate by stakeholders and decision makers. Table 1
provides examples of relevant epidemiological scales and
associated state variables for surveillance systems with
distinct objectives.
2.2. Describing the intervention strategy
The proposed rationale relies on the comparison of
the decisions likely to be made based on the information
produced by surveillance with the decisions that would
be made given a perfect knowledge of the epidemiological situation. Consequently, the decisions that would be
considered as appropriate by stakeholders and decision
makers for a set of possible epidemiological situations need

to be described.
Planning response mechanisms to potential epidemiological situations constitutes an important measure to
improve preparedness towards threats posed by animal diseases (Rushton and Upton, 2006). Epidemiological
modelling and analysis in combination with economic
evaluation produce the scientific evidence for planning
intervention strategies. Such approaches have been widely
used for the definition of national and international
preparedness plans. Predefined intervention strategies
should thus in most instances exist and can be described
(Tomassen et al., 2002).
Intervention strategies can be described through the
relationship between the value(s) of the epidemiological state variable(s)that characterize an epidemiological
situation and the intervention measures considered as
appropriate for that epidemiological situation. Usually, the
possible values of the epidemiological state variable(s) are
classified into ordered categories of increasingly harmful
sanitary and economic consequences. In Table 1, contrived
examples of simple intervention strategies are presented
for distinct surveillance objectives. In these strategies, the
possible values of the relevant state variable are classified
according to two subsets referred to as S+ and S− . Each of
these subsets is associated with a pre-defined intervention
option (I+ and I− , respectively) considered as appropriate
by stakeholders and decision makers. S+ is the subset of values of the state variable that requires the implementation
of intervention measures (i.e. intervention option I+ ) and
S− is the subset of values of the state variable that requires
no intervention (i.e. intervention option I− ).
2.3. Describing the data generation and interpretation
processes
Once the epidemiological scale, the state variable

and the intervention strategy are defined, it is necessary
to describe the surveillance data generation and interpretation processes that produce the information upon
which decision makers rely for the implementation of
intervention measures (Fig. 1). Surveillance data generation processes include reporting (e.g. underreporting
rate and the factors influencing it), diagnostic (e.g. case

73

definition), sampling (e.g. coverage, stratification, intensity, frequency), and sample testing (e.g. sensitivity and
specificity of the tests used).
Surveillance data interpretation involves in most
instances the computation of statistics that provide an
assessment of the current epidemiological situation and
inform decisions regarding intervention. Considering the
potential intervention strategies presented in Table 1
where two subsets of values of the focal state variable are
considered, two subsets of values for such a statistic can
be defined (Table 2). A+ is the subset for which the focal
epidemiological state variable is assessed as falling into
the category requiring the implementation of intervention
measures (I+ ). A− is the subset for which the focal epidemiological state variable is assessed as falling into the category
requiring no intervention measure (I− ).
2.4. Effectiveness criteria
With S+ , S− , A+ , A− determined, it is possible to define
two types of errors, namely Type I and Type II errors, analogously to the types of error used in statistical or diagnostic
tests (Table 3). Type I error occurs when a surveillance
system produces information which results in the implementation of intervention measures while the true state of
the population would not require it. Type I errors imply that
costly mitigation measures are unnecessarily activated.
Type II error occurs when a surveillance system produces

information which results in no implementation of mitigation measures while the true state of the population would
require it. Type II errors result in increased risks of failure
to control a genuine disease threat or may lead to a delayed
response.
The effectiveness of a surveillance system can be
assessed by estimating for that system the probabilities of
Type I errors P(A+ |S− ) and the probabilities of Type II errors
P(A− |S+ ). Using the information on intervention strategies
as well as on data generation, analysis and interpretation
processes leading to decisions, probabilities of Type I and
Type II errors can be assessed either analytically using
sampling and probability theories or through simulations.
This is illustrated in the following section with two hypothetical and a real examples.
3. Illustrations of effectiveness assessment three
contrived surveillance system examples and an
empirical case study
In this section, the proposed rationale is further developed for four types of surveillance systems with the
objectives of: (1) demonstrating freedom from a disease,
(2) monitoring the prevalence of an endemic disease, (3)
detecting as early as possible the presence of an emerging
disease, and (4) detecting cases of a disease.
3.1. Demonstrating freedom from a disease
The case of surveillance systems aiming at demonstrating a territory as free from a disease is interesting because
the rationale proposed here has already been applied to
assess effectiveness of such surveillance systems. The state


74

V. Grosbois et al. / Preventive Veterinary Medicine 120 (2015) 70–85

Table 2
Examples of decision making rules relying on the analysis and interpretation of surveillance data. The decision rules correspond to the mitigation strategies
presented in Table 1.
Surveillance
objective

Scale

Statistics used
to assess epidemiological
status

A−

Decision I−

A+

Decision I+

Monitoring
prevalence

Country/region

Proportion of
positive tests in
the samples
collected over a

year P(+)

P(+) ≤ Threshold

Do nothing

P(+) > Threshold

Case detection
of disease
Demonstrate
freedom
from disease

Herd

Result of a
pooled test
Proportion of
positive tests in
the samples
collected over a
year P(+)
Case reporting

Negative test
result
P(+) ≤ Threshold

Do nothing

Positive test
result
P(+) > Threshold

Implement
systematic
testing in
slaughterhouses before
products are
put on the
market
Cull the herd

No case
reported

Do nothing

Early detection
of an
emerging
disease

Country/region

Country/region

Allow
exportations

≥1 case
reported

Ban
exportations

Launch
intensive
surveillance
and in depth
case
investigation.
Limit
movements

A− , A+ : assessments of epidemiological state for which the “no intervention” and “intervention” options, respectively, are implemented; I− , I+ : description
of actions associated with the “no intervention” and “intervention” options.
Table 3
The two types of error used as effectiveness criteria.
True epidemiological status
S+ intervention required

S− intervention not required

Assessment of the epidemiological status resulting from the generation, analysis and interpretation of surveillance data
A+ intervention implemented
Type I error
A− intervention not implemented
Type II error

S− , S+ : epidemiological states for which the “no intervention” and “intervention” options, respectively, are required; A− , A+ : assessments of epidemiological
state for which the “no intervention” and “intervention” options, respectively, are implemented.

variable which conditions decisions in terms of prevention/intervention measures is usually the prevalence of
the disease in the focal population. The prevalence categories considered as requiring distinct intervention options
are determined according to the so called “design prevalence”. Whenever the prevalence in the population is below
the design prevalence, the territory is considered “as free
from the disease” (S− ) and no measure to limit its spread
is implemented (for instance no limitations to animal
trading: I− ) whereas whenever the prevalence in the population is above the design prevalence, measures to limit
its spread are implemented (for instance animal trading is
restricted: I+ ). The crucial aspect of the mitigation strategy is the determination of the design prevalence. It can
be chosen based on the relative likelihood of prevalence
levels given the presence of the disease on the territory
or by considering how the magnitude of sanitary and economic consequences of the presence of the disease vary
as a function of the prevalence of that disease. So the
design prevalence can be the minimum expected prevalence of the disease provided it is present on the territory

or the maximum prevalence at which the sanitary and economic consequences of the presence of the disease are
considered as negligible. The statistics used to assess the
epidemiological situation from surveillance data is usually the binary variable reflecting whether at least one case
has been detected (A+ ) or no case has been detected (A− ).
In the numerous papers in which this approach has been
used to assess the effectiveness of surveillance systems
aiming at demonstrating the freedom of a territory from
a disease (e.g. Martin et al., 2007a; Martin, 2008; Frössling
et al., 2009; Hood et al., 2009; Christensen et al., 2011), the
effectiveness criterion used is the probability of a Type II
error P(A− |S+ ) which is the probability that the territory is
qualified as free of a disease while the prevalence of the disease is above the design prevalence. The method applied to

compute this probability is usually scenario tree modelling
(Martin et al., 2007b) although other methods have been
proposed (Hood et al., 2009). These published effectiveness
assessments for systems aiming at demonstrating freedom
from disease already follow the rationale proposed here
and support the logic outlined.


V. Grosbois et al. / Preventive Veterinary Medicine 120 (2015) 70–85

75

Table 4
Information for assessing the effectiveness of a contrived surveillance system aiming at monitoring the prevalence of an endemic disease.
Surveillance
objective

Relevant scale
Relevant epidemiological
variable
Intervention strategy

Surveillance
data
generation
process

Statistics
computed
from

surveillance
data
Decision rule 1 (test performances
not accounted for)

Decision rule 2 (test
performances accounted for)

Knowing how prevalent is an
endemic disease to inform
decisions about vaccination
strategy
Country (population of 100,000
animals)
Individual level prevalence (p)

S−
p ≤ 0.1
I−
no vaccination

S+
0.1 < p ≤ 0.2
I+
vaccination is
implemented
only in high
risk areas

S++

p > 0.2
I++
vaccination is
implemented
in all areas

A+
0.1 * n < np ≤ 0.2 * n
I+
targeted
vaccination is
implemented
A+
(0.1 * Se + (1 − 0.1) * (1 − Sp)) * n
(1 − 0.2) * (1 − Sp)) * n
I+
targeted
vaccination is
implemented

A++
np > 0.2 * n
I++
vaccination is
implemented
in all areas
A++
np > (0.2*Se
+ (1 − 0.2)*(1 − Sp)) * n

n = 100 randomly chosen
individuals are sampled over a
1 month period
(coverage = 0.1%). Each sample
is tested using a test with
sensitivity Se = 0.90 and
specificity Sp = 0.95
Number of sampled units
testing positive (np )

A−
np ≤ 0.1 * n
I−
no vaccination

A−
np ≤ (0.1 * Se + (1 − 0.1) * (1 − Sp)) * n

I−
no vaccination

I++
vaccination is
implemented
in all areas

S− , S+ , S++ : epidemiological states for which the “no intervention”, “low intensity intervention” and “high intensity intervention” options, respectively, are
required; I− , I+ , I++ : description of actions associated with the “no intervention”, “low intensity intervention” and “high intensity intervention” options,
respectively; A− , A+ , A++ : assessments of epidemiological state for which the “no intervention”, “low intensity intervention” and “high intensity intervention”

options, respectively, are implemented.

3.2. Monitoring the prevalence of an endemic disease
This section presents a contrived example an active
surveillance system aiming at monitoring prevalence of a
cattle disease to inform decision-makers on which vaccination strategy to implement at the national level.
3.2.1. Information required for assessing effectiveness
Table 4 summarises the information required to assess
the effectiveness of such a surveillance system. Potential epidemiological situations are categorised according
to three prevalence levels: at low prevalence, it is considered that vaccination is not necessary; at intermediate
prevalence, it is considered that targeted vaccination
should be implemented around detected outbreaks; at
high prevalence, it is considered that vaccination should
be implemented in all areas of the country. Surveillance
data are assumed to be generated through random sampling of individuals (which ensures homogeneous coverage

of the population) and assessment of individual disease
status with a test of known sensitivity and specificity
(Table 4). The data interpretation process consists in comparing the number of samples testing positive with the
expected numbers of diseased individuals in the sample
for the prevalence thresholds defined in the intervention
strategy (Table 4).
3.2.2. Assessment of effectiveness
The effectiveness of this surveillance system is determined by estimating the probabilities that the information
produced by the surveillance system leads to the implementation of inappropriate intervention measures.
Using the notations of Table 4, Pr(A− |S+ ), Pr(A− | S++ )
and Pr(A+ |S++ ) represent probabilities of more or less
severely under-sizing the intervention measures given
the true epidemiological situation (i.e. more or less severe
Type II errors) while, Pr(A+ |S− ), Pr(A++ |S+ ) and Pr(A++ |S− )

represent probabilities of more or less severely over-sizing


76

V. Grosbois et al. / Preventive Veterinary Medicine 120 (2015) 70–85

Fig. 2. Probabilities that data generation, analysis and interpretation processes result in the implementation of different intervention options as a function
of the true epidemiological state. n: sample size; Se: sensitivity of the test; Sp: specificity of the test; S− : vaccination is not required; S+ : targeted vaccination
is required; S++ : mass vaccination is required.

the intervention measures given the true epidemiological
situation (i.e. more or less severe Type I errors).
The probability distribution of the statistics examined to
make decisions about intervention measures (i.e. the number of samples testing positive, np ) is known: it is a binomial
distribution where the number of trials parameter is the
sample size (n) and the probability parameter is a function
of the real prevalence of the disease in the population and
of the test performance parameters (pSe + (1 − p)(1 − Sp)).
So Pr(X < np < Y|p, n, Se, Sp) can be computed for any value
of X, Y, p, n, Se and Sp.
Fig. 2 illustrates the surveillance effectiveness of this
hypothetical example. This figure displays for any given
value of true disease prevalence (p), the probabilities
that decision makers implement the “no vaccination”
(I− ),“targeted vaccination” (I+ )or “mass vaccination” (I++ )
options according to decision rule 1 in Table 4. It also
shows the ranges for true prevalence (p) requiring distinct
intervention measures to be implemented as defined by
decision-makers (S− : “no vaccination”, S+ : “targeted vaccination”, S++ : “mass vaccination”). When true prevalence

is just above 0.2 (thus where mass vaccination would be
required), it is estimated that given the sample size, the
diagnostic test characteristics, and the decision rule used,
the probability of actually implementing mass vaccination is 0.65, the probability of implementing only targeted
vaccination is 0.35 (moderate Type II error) and the probability of not implementing vaccination is 0 (severe Type II
error). When true prevalence is just above 0.05 (thus no
vaccination would be required), it is estimated that the
probability of nonetheless implementing mass vaccination
(severe Type I error) is 0, the probability of implementing
targeted vaccination is 0.32 (moderate Type I error) and
the probability of not implementing vaccination is around
0.68.

3.2.3. Sensitivity of effectiveness to the characteristics of
data generation and interpretation processes
The proposed approach allows assessing how probabilities of Type I and Type II errors change when reporting,
diagnostic, sampling, sample testing or data interpretation procedures are modified. In Fig. 3, modifications of the
surveillance process in terms of sample size and performance of the diagnostic test used to detect the disease in
each sampled unit are illustrated. Increasing sample sizes
and improving test performances results in reducing the
probabilities of Type I and Type II errors.
In Fig. 4 the sample size is 100, Se is 0.6 and Sp is 0.8
in the two panels but the surveillance data interpretation process differs between the two panels. In Fig. 4a the
decision regarding intervention relies on an assessment of
the population epidemiological status (i.e. the prevalence
level) that does not account for the fact that the test used
to assess individual disease status is imperfect (decision
rule 1 in Table 4). In Fig. 4b test sensitivity and specificity
are accounted for in the assessment of the population epidemiological status (decision rule 2 in Table 4). This figure
illustrates that changes in the data interpretation process

can improve dramatically the performance of a surveillance
system.
3.3. Early detection
In the case of surveillance systems aiming at detecting
the introduction of a pathogen in a territory or a population as early as possible, the state variables which condition
decisions regarding intervention measures are the binary
variable that indicates whether or not the pathogen infects
at least one unit in the focal host population and the time
elapsed since the occurrence of the index infection(s) in the
focal population. The latter underlies a number of other


V. Grosbois et al. / Preventive Veterinary Medicine 120 (2015) 70–85

77

Fig. 3. Sensitivity of surveillance effectiveness to changes in sampling and sample testing procedures. n: sample size; Se: sensitivity of the test; Sp: specificity
of the test.

Fig. 4. Sensitivity of surveillance effectiveness to changes in data analysis and interpretation procedures. n: sample size; Se: sensitivity of the test; Sp:
specificity of the test.


78

V. Grosbois et al. / Preventive Veterinary Medicine 120 (2015) 70–85

Table 5
Information for assessing the effectiveness of a contrived surveillance system aiming at detecting an emerging or exotic disease early.
Surveillance objective

Relevant scale
Relevant epidemiological variables

Intervention strategy

Surveillance data generation process

Statistics computed from surveillance data
Decision rule

Detecting an emerging disease
following its introduction in a
territory as soon as possible
Country (population of 10,000
animals)
Cumulative incidence
(correlated with time elapsed
since introduction and spatial
spread)
S− The disease has not yet been
introduced
I− Keep low intensity
surveillance with 50
individuals sampled daily

S+ The disease has been
introduced but cumulative
incidence is <0.5%
I+ Cull detected infectious cases

and reinforce surveillance with
100 individuals sampled daily

Randomly chosen individuals
are sampled daily. Samples are
screened for antibody using a
test which sensitivity is Se = 0.8
and specificity is Sp = 1.
Seropositive samples are
tested for pathogen detection
using a test which sensitivity
and specificity are 1
Cumulative number of
detected cases (np )
A− No case detected so far
I− Keep low intensity
surveillance with 50
individuals sampled daily

A+ One case detected
I+ Cull the case if it is infectious
and reinforce surveillance with
100 individuals sampled daily

S++ Cumulative incidence is
≥0.5%
I++ Cull detected infectious
cases, reinforce surveillance
with 200 individuals sampled
daily, and limit animal

movements

A++ At least two cases detected
I++ Cull detected infectious
cases, reinforce surveillance
with 200 individuals sampled
daily, and limit animal
movements

S− , S+ , S++ : epidemiological states for which the “no intervention”, “low intensity intervention” and “high intensity intervention” options, respectively, are
required; I− , I+ , I++ : description of actions associated with the “no intervention”, “low intensity intervention” and “high intensity intervention” options,
respectively; A− , A+ , A++ : assessments of epidemiological state for which the “no intervention”, “low intensity intervention” and “high intensity intervention”
options, respectively, are implemented.

important epidemiological state variables such as cumulative incidence, spatial spread, and average number of
transmission events that link a case to the index case.
As time elapsed since the occurrence of the index case
increases so do cumulative incidence and spatial spread.
Consequently, the later the implementation of intervention
measures (relative to the time of occurrence of the index
case) the larger are the losses already generated by the disease. This also means that costs of intervention measures
required will increase with more animals and/or holdings
being affected.
The performance of a surveillance system aiming at
detecting an emerging pathogen as early as possible could
thus be evaluated according to two components: its ability to detect at any point in time the presence of the focal
pathogen in the focal host population and its ability to evaluate the spatial spread and prevalence of the focal pathogen
once its presence has been detected. The first component
is probably the most important because a surveillance
system which performs well in terms of instantaneous

detection probability will allow implementation of prevention/control measures soon after the introduction of the
pathogen, when the losses already generated by the disease
as well as the resources required for mitigation measures to
control its further spread are limited. The second criterion

reflects the ability of the surveillance system, once detection has been achieved, to provide information that allows
the implementation of mitigation measures which nature
and intensity would be considered as adapted by stakeholders and decision makers given perfect knowledge of
the real epidemiological situation in terms of prevalence
and spatial spread. This second component is relates to the
evaluation of surveillance systems aiming at monitoring
the prevalence of a disease. Such attributes are presented
in Section 3.2.
3.3.1. Information required to assess effectiveness
We consider a contrived example of an active surveillance system aiming at detecting the introduction and
spread of an emerging disease in a cattle population as early
as possible. Table 5 includes the information required to
assess the effectiveness of such a surveillance system. In
this example the relevant population state variable is the
cumulative incidence of the disease in the focal country.
3.3.2. Intervention strategy
It is assumed that the intervention strategy has been
planned based on the analysis of the of a simple SEIR
(Susceptible, Exposed, Infectious, Recovered) model with
a daily time step. In this model, the host population size


V. Grosbois et al. / Preventive Veterinary Medicine 120 (2015) 70–85

79

Fig. 5. Effectiveness of a contrived surveillance system aiming at detecting the introduction of an emerging disease as early as possible.

is assumed to be 10,000. Given available knowledge of the
epidemiological characteristics of the host population and
of the pathogen, it has further been assumed that the host
population is free mixing, that the daily probability of transition from the “Infectious” to the “Recovered” state is 0.83,
that the daily probability of transition from the “Exposed”
to the “Infectious” state is 0.1, that each infectious individual can transmit the pathogen to any susceptible individual
with a daily probability of 0.00015. At time step 1 of a
simulation run, all the hosts in the population are in the
“Susceptible” category except one individual which is in
the “Exposed” category. Using these parameters, 10,000
epidemiological dynamics have been simulated over 400
days under the assumption that no intervention measure is
applied (Fig. 5a). It turns out that in 19% of the simulations,
the pathogen goes extinct before a single transmission
occurs. Moreover, in 12% of the simulations, the final size
of the epidemics is smaller than 0.5% of the size of the
host population, which is considered by decision makers
and stakeholders as an acceptable impact. Finally in 69%

of the simulations the final size of the epidemics is larger
than 0.5% of the population, which is considered by decision makers and stakeholders as an unacceptable impact.
Considering these modelling results, decision makers and
stakeholders have decided that only moderate intervention (I+ ) including culling of detected infectious cases and
reinforced surveillance would be required whenever at
least one transmission of the pathogen has occurred in the
population but cumulative incidence is lower than 0.5%
(S+ ) while intensive intervention (I++ ) consisting in movement restrictions, that are likely to result in a reduction

in transmission probability, and in a further reinforcement
of surveillance in addition to culling of detected infectious
cases would be required as soon as cumulative incidence
reaches 0.5% (S++ ).
3.3.3. Surveillance data generation and interpretation
processes
The test used for the screening of sampled animals
is a serological test that detects antibodies against the


80

V. Grosbois et al. / Preventive Veterinary Medicine 120 (2015) 70–85

focal pathogen and thus identifies infectious and recovered
individuals. The sensitivity and specificity of this test are
assumed to be 0.8 and 1, respectively. Samples in which
antibodies have been detected are subjected to a complementary test that detects the pathogen with perfect
sensitivity and specificity and thus allows distinguishing
infectious from recovered individuals among seropositive
sampled animals. Before the first detection of a seropositive animal (A− ), it is considered that the pathogen has not
yet been introduced in the population and low intensity
surveillance is maintained with 50 randomly sampled cattle tested daily (I− ). Whenever only one seropositive animal
has been detected (A+ ), it is considered that the pathogen
has been introduced in the population but that the epidemiological situation does not yet require high intensity
intervention measures (i.e. the cumulative incidence might
still be below 0.5%). Moderate intensity intervention measures (I+ ) are then implemented: surveillance is reinforced
with 100 animals sampled daily and the detected case is
culled if it has been identified as infectious by the complementary test. As soon as a second seropositive individual
has been detected (A++ ) it is considered that cumulative

incidence is likely to have reached 0.5% of the population. High intensity intervention measures (I++ ) are thus
implemented: surveillance is further reinforced with 200
animals sampled daily, animal movements are restricted
leading to a tenfold reduction in transmission probability
and animals identified as infectious are culled. The objective of such a decision rule where intensive intervention
measures are not implemented upon the detection of the
first case is to lower the probability of implementing costly
intervention measures in situation where such measures
are not necessarily required for the pathogen to get extinct
before having produced any noticeable impact.
3.3.4. Effectiveness evaluation
Effectiveness in this example can be evaluated by
assessing how probabilities of Type I (i.e. Pr(A++ |S+ ))
and Type II (i.e. Pr(A− |S+ ), Pr(A− |S++ ) and Pr(A+ |S++ )) errors
change as time since introduction of the disease in the focal
host population increases. For doing so, one needs to assess
the expected trajectories of the population epidemiological state and of the implemented intervention measures
in a population where the surveillance and intervention
strategy described above is applied. This can be done by
modifying the SEIR simulation model presented above to
account for the surveillance and intervention processes.
In this modification of the SEIR model detection of the
disease in the infectious and recovered compartments was
simulated at each time step of each simulation run. This
was done using a binomial process which parameters were
the number of individuals in the focal epidemiological
compartment (infectious or recovered) and the product
of individual-scale sampling probability (i.e. the ratio
of sample size to population size) and sensitivity of the
serological screening test. Sample size was 50 animals

as long as no detection had occurred, 100 animals as
soon as a first case (infectious or recovered) was detected
and 200 animals as soon as a second case (infectious
or recovered) was detected. In order to simulate the
culling process, detected infectious animals were removed

from the population. Detected recovered animals were
removed from the detectable recovered compartment to
avoid multiple detections of the same recovered animal.
Animal movement restriction following the detection of
the second case was simulated by setting the transmission
probability to 1.5 × 10−5 (i.e. tenfold reduction as compared to transmission probability before the detection
of a second case). Fig. 5a and b present the dynamics of
the probability of the possible population epidemiological
states after introduction of the disease (S+ , S++ ) without and
with surveillance and intervention measures, respectively.
Without intervention, the probability that cumulative
incidence exceeds 0.5% (i.e. the probability of S++ ) is 0
until around 20 days after introduction. It then increases
to stabilize at 0.69 around 150 days after introduction.
When the intervention strategy is applied, the probability
that cumulative incidence exceeds 0.5% is 0 until around
20 days after introduction. It then increases to stabilize at
0.32 around 150 days after introduction. The application
of the surveillance and intervention strategy can thus be
considered as quite efficient since it produces a twofold
decrease in the probability of the introduced disease having a strong impact. However this probability might still
be considered as too high and one reason could be that the
surveillance data generation and interpretation processes
are not effective enough. Moreover, decision makers and

stakeholders might also be interested in evaluating the
likelihood that the surveillance data generation and interpretation processes result in the implementation of the
costly high intensity intervention measures in situation
where such measures are not necessary (i.e. probability of
a Type I error). These issues can be addressed through the
evaluation of surveillance effectiveness.
Fig. 5c–d illustrates effectiveness of surveillance for this
example. It shows the probabilities that the surveillance
process results in the implementation of distinct intervention measures (i.e. I− , I+ , I++ ) conditionally on the true
population state (S+ : cumulative incidence <0.5% on Fig. 5c
or S++ : cumulative incidence ≥0.5% on Fig. 5d).
Fig. 5c shows that when cumulative incidence is lower
than 0.5% (S+ ) and the time elapsed since introduction is
less than 60 days, the most likely outcome of the surveillance process is the failure to detect any case (A− ) which
results in maintaining low intensity surveillance (I− ). On
the other hand, when cumulative incidence is lower than
0.5% (S+ ) and the time elapsed since introduction is more
than 60 days, the most likely outcome of the surveillance
process is the detection of at least two cases (A++ ) leading to
the implementation of high intensity surveillance, culling
of detected infectious animals and movement restriction
(I++ ). Thus, when the true state of the population is S+ which
requires only moderate intervention (I+ ) the considered
surveillance strategy generates either high probability of
a moderate Type II error P(A− |S+ ) or high probability of
a moderate Type I error P(A++ |S+ ) depending on the time
elapsed since introduction. The effectiveness of surveillance could thus probably be improved by increasing sample size before the detection of the first case (which would
lower the probability of a Type II error P(A− |S+ ) shortly
after introduction) and by adding a process that would
allow switching back from high intensity to moderate

V. Grosbois et al. / Preventive Veterinary Medicine 120 (2015) 70–85

intensity intervention when no new cases are detected over
a long time period (which would lower the probability of a
Type I error P(A++ |S+ ) long after introduction).
Fig. 5d shows that when cumulative incidence is larger
than 0.5% (S++ ) the most likely outcome of the surveillance process is the detection of at least two cases (A++ )
leading to the implementation of high intensity surveillance, culling of detected infectious animals and movement
restriction (I++ ). The considered surveillance system thus
performs well in situations where high intensity intervention is required, even when this epidemiological situation
occurs shortly after introduction.
3.4. An empirical case study: passive surveillance for the
detection of villages and holdings infected by Highly
Pathogenic Avian Influenza (HPAI) in Vietnam
Although the poultry population in Vietnam is partially
vaccinated against HPAI, epidemics still sporadically occur,
often around the Tet festival, the Vietnamese New Year. It is
suspected that HPAI viruses are maintained in local domestic duck populations and regularly introduced into Vietnam
from neighbouring countries. Vietnamese veterinary services have implemented a passive surveillance system to
detect holdings or villages infected by HPAI viruses (MARD
and MOH, 2011). However they acknowledge the fact that
limited effectiveness of this surveillance system represents
a major issue for proper implementation of control programs (Minh et al., 2009). The approach proposed here
is applied to assess the effectiveness of this surveillance
system that aims at detecting cases of a disease.
3.4.1. Information required to assess effectiveness
Table 6 includes the information required to assess the
effectiveness of this surveillance system. The relevant scale

is the epidemiological unit (the holding for commercial
production or the village for backyard production) and the
relevant state variable is the infectious status of a unit: a
unit including at least one infectious animal is infectious,
while a unit including no infectious animal is not infectious.
3.4.2. Intervention strategy
The intervention strategy is to implement culling and
ring vaccination (I+ ) in any infectious unit (S+ ). No intervention (I− ) is required for non-infectious units (S− ).
3.4.3. Surveillance data generation and interpretation
processes
Surveillance data are generated through passive reporting of case suspicions. According to the recommendations
of Vietnamese state veterinary services any village or holding in which mortality exceeds 0.05 over 2 days should
be reported to veterinary services. Samples are collected
in each suspected unit and a real time RT-PCR test which
sensitivity and specificity are estimated at 0.93 (95% CI:
0.91–0.96) and 0.98 (95% CI: 0.97–1), respectively (Peyre
et al., 2009) is used to confirm the infectious status of
each suspected unit. Culling and ring vaccination (I+ ) is
implemented in each unit confirmed as infectious (A+ ).
No intervention (I− ) is implemented in units that are not

81

reported as suspect or in suspected units for which confirmatory tests are negative (A− ).

3.4.4. Effectiveness evaluation
In order to assess the effectiveness of this surveillance
system one needs to estimate the probability of a noninfectious unit being reported as suspect and confirmed
as infectious (P(A+ |S− ): probability of a Type I error) so
that culling and ring vaccination would be unnecessarily implemented in that unit. Since the specificity of the

confirmatory test is very close to 1, the probability of a noninfectious unit being categorised as infectious is very close
to 0 and the probability of a Type I error can be considered
as negligible.
More importantly one needs to estimate the probability of an infectious unit not being reported as suspect or
not being confirmed as infectious when reported as suspect
(P(A− |S+ ): probability of a Type II error) so that culling and
ring vaccination would not be implemented in that infectious unit. The probability of a Type II error depends on the
distribution of maximum mortality over 2 days in infectious units, on how the probability of reporting a suspicion
varies as a function of maximum mortality over 2 days, and
on the sensitivity of the confirmatory test. Let us denote by
P(M > Rt) the probability that mortality in the infected unit
is above the reporting threshold, P(R|M > Rt) the probability of case reporting when mortality is above the reporting
threshold and Se the sensitivity of the confirmatory test.
The probability of a Type II error can be computed as:
P A− |S + = 1 − P(M > Rt) × P(R|M > Rt) × Se

(1)

Although it is known that the transmission of HPAI viruses
in poultry flocks is very fast and that mortality rates from
0.5 to 1 can be reached within 3–4 days from the onset
of the first clinical signs (Swayne, 2009), it is difficult to
obtain precise information on mortality rates over 2 days
in villages or holding infected by HPAI. However mortality rates over 2 days of 0.33 in poultry holdings infected
by H5N1 viruses in Thailand have been reported in Tiensin
et al. (2007). Furthermore the analysis of unpublished data
on mortality monitoring in 6 poultry flocks infected by
H9N2 viruses in an Egyptian holding revealed that maximum mortality over 2 days in these flocks ranged between
0.06 and 0.24. This information was used to define a distribution for maximum mortality over 2 days in infected
units. This distribution was assumed to be a Beta distribution of parameters ˛ = 1.85 and ˇ = 4.41. The 5th percentile

of this distribution is 0.06 and its median is 0.20. The probability P(M > Rt) of maximum mortality over 2 days being
higher than the reporting threshold is easily computed
using the cumulative density function of this distribution.
For a reporting threshold of 0.05, this probability equals
0.96.
Assuming that the recommendations of Vietnamese
state veterinary services are strictly respected, the probability that a unit is reported as suspicion is 1 when
maximum mortality over 2 days in that unit exceeds 0.05
and 0 when maximum mortality over 2 days in that unit
does not exceed 0.05. Thus, P(R|M > Rt) = 1.


82

V. Grosbois et al. / Preventive Veterinary Medicine 120 (2015) 70–85

Table 6
Information for assessing the effectiveness of the passive surveillance system aiming at detecting cases of Highly Pathogenic Avian Influenza (HPAI) in the
Vietnam poultry population.
Surveillance objective
Relevant scale
Relevant epidemiological variable

Intervention strategy

Surveillance data generation process

Statistics computed from surveillance data

Decision rule

Find villages/holdings infected
by HPAI
Industrial holding/village
Epidemiological status
(infected/uninfected) of the
unit (holding or village)
S−
The unit is not infected (no infected animal in
the unit)
I−
Do nothing
A unit is reported as suspicious whenever
mortality rate over 2-day exceeds 5% in that
unit and specific symptoms are observed. A
confirmatory tests which sensitivity is Se = 0.93
and specificity is Sp = 0.98 is applied to all the
units reported as suspicious
Infectious status of the unit as
perceived through the
reporting and testing processes
A−
The unit is not reported as suspicious
The unit is reported as suspicious but the
confirmatory tests have not detected HPAI
I−
Do nothing

S+
The unit is infected (at least

one infected animal in the unit)
I+
Cull the unit and implement
ring vaccination

A+
The unit is reported as
suspicious and the
confirmatory tests have
detected HPAI
I+
Cull the unit and implement
ring vaccination

S− , S+ : epidemiological states for which the “no intervention” and “intervention” options are respectively required; I− , I+ : description of actions associated respectively to the “no intervention” and “intervention” options; A− , A+ : assessments of epidemiological state for which the “no intervention” and
“intervention” options are respectively implemented.

Finally, given that the sensitivity of the confirmatory
test is 0.93, the probability of a Type II error according to
Eq. (1) is P(A− |S+ ) = 1 − 0.93 × 1 × 0.96 = 0.11.
The above evaluation of effectiveness suggests that HPAI
passive surveillance in Vietnam is quite effective (the probabilities of Type I and Type II errors are low). However,
assumptions such as strict adherence to the recommendations of Vietnamese state veterinary services or maximum
mortality following a Beta (1.85, 4.41) distribution should
be questioned and could be easily relaxed in a sensitivity analysis such as the one presented in Section 3.2. For
instance, unpublished data obtained from participatory
investigations of the reporting behaviour of backyard poultry owners suggest that suspicions are actually not reported
when mortality over 2 days is lower than 0.3, so that the
reporting threshold would be 0.3 rather than 0.05. According to Eq. (1), the probability of a Type II error in that
situation would be equal to 0.58. Finally, a thorough evaluation of effectiveness of this surveillance system should

also consider that mortality and reporting patterns differ
between types of units (industrial holdings and villages).
4. Discussion
This paper shows how the effectiveness of a surveillance system can be evaluated in terms of discrepancy
between the modalities and intensity of prevention and/or
control measures that would be implemented given a perfect knowledge of the true epidemiological status of a

population and of its components and the modalities and
intensity of prevention and/or control measures that are
likely to be actually implemented based on the analysis
and interpretation of the data produced by a surveillance
system. We have also shown that this rationale can be
used to conduct sensitivity analyses to establish which
changes in the surveillance system allow improved effectiveness. Importantly, it appears that information on data
generation processes alone does not allow thorough evaluations of surveillance effectiveness. Indeed, information
on planned mitigation strategies, on the processes through
which surveillance data are analysed and interpreted and
on the decision-making process leading to the implementation of mitigation strategies are also crucial.
4.1. Links with previously proposed effectiveness criteria
It is important to note that probabilities of Type I and
Type II errors have already been used as effectiveness
attributes for surveillance systems or components. However they are most often referred to as component-level
or system-level sensitivity (which is the complement to 1
of the probability of a Type II error) and false alarm rate
(which is the probability of a Type I error). We argue that
system-level sensitivity and false alarm rate are relevant
effectiveness criteria for any surveillance system. All the
other previously proposed effectiveness attributes matter
in that they influence system-level sensitivity and false
alarm rate.

V. Grosbois et al. / Preventive Veterinary Medicine 120 (2015) 70–85

The rationale proposed here identifies the analysis and
interpretation of surveillance data as an important aspect
of surveillance. The analysis and interpretation of surveillance data often imply estimations of epidemiological
variables such as incidence or prevalence. The accuracy
(or bias) and precision of such estimations have been considered as important surveillance effectiveness attributes
(e.g. Drewe et al., 2015; Hoinville, 2013). However, their
importance relies in that they ultimately influence probabilities of making decisions that differ from those that
would be made given a perfect knowledge of the true
value of the focal epidemiological variables. We thus argue
that accuracy and precision are in some instances important quantities for computing probabilities of Type I and
Type II errors which, ultimately, are the two most relevant
attributes for assessing effectiveness.
The positive predictive value (PPV) and negative predictive value (NPV) computed at system level have also
been proposed as relevant statistics for the evaluation of
surveillance systems (Drewe et al., 2012). As pointed out by
Martin et al. (2007b) PPV and NPV are related to probabilities of a focal population being in a given epidemiological
state given an assessment of the epidemiological state of
that population through the analysis and interpretation of
surveillance data (i.e. P(S|A) using the notation introduced
above). Indeed, the PPV is P(S+ |A+ ) (or 1 − P(S− |A+ )) and
the NPV is P(S− |A− ) (or 1 − P(S+ |A− )). NPV and PPV are
important quantities that inform decision makers about the
risk taken when making a decision and therefore constitute critical information in decision making (Martin et al.,
2007b). P(S+ |A− ) (i.e. 1 − NPV) informs the decision maker
on the probability that the true epidemiological situation
would require the implementation of mitigation measures in situations where surveillance evidence suggests

that no mitigation measures should be implemented (for
example the probability to declare a territory free of a
disease although the disease is present with a prevalence
higher than the design prevalence (Martin, 2008; Frössling
et al., 2009)). P(S− |A+ ) (i.e. 1 − PPV) informs the decision
maker on the probability that the true epidemiological status would not require the implementation of mitigation
measures in situations where surveillance evidence suggests that mitigation measures should be implemented (for
example the probability not to declare a territory free of
a disease although the disease is absent or present with a
prevalence lower than the design prevalence). PPV and NPV
are thus useful quantities for interpreting data produced by
surveillance rather than for evaluating the effectiveness of
a surveillance system. P(S+ |A− ) (i.e. 1 − NPV) and P(S− |A+ )
(i.e. 1 − PPV) are related to the probability of Type II error
(P(A− |S+ )) and the probability of Type I error (P(A+ |S− ))
through the Bayes formula for conditional probabilities
(Martin, 2008).
Timeliness is considered as an important attribute for
the effectiveness of surveillance systems aiming at detecting the introduction or the emergence of a pathogen
because the later the implementation of intervention
measures (relative to the time of occurrence of the index
case) the larger are the losses already generated by the
disease and the costs of intervention measures required
to control it. Thus early detection is important because it

83

insures detection before the disease has spread widely in
the population. The hypothetical example of a surveillance
system aiming at detecting a disease as early as possible presented above illustrates this point by considering

that decision makers plan their intervention strategy in
relation with cumulative incidence rather than with time
since introduction. It also shows that when intervention
strategies are elaborated in this way timeliness can be
incorporated in the evaluation of Type I and Type II by looking at how probabilities of such errors change as time since
introduction increases.

4.2. Limitations
4.2.1. Collecting information on mitigation strategies
The proposed rationale requires the characterization
of potential epidemiological situations in terms of categories considered by stakeholders and decision makers
as requiring distinct responses. This is a pre-requisite for
the computation of the probabilities of Type I and Type
II errors. It is possible that in some instances, intervention strategies are not defined yet and that responses to
threats posed by animal diseases follow conventional outbreak investigation activities. Depending on the technical
possibilities, interventions may then be implemented to
contain the disease. However, in some cases there are
either no technical intervention measures available, or the
disease is too widespread or not considered important
enough to warrant a reaction. Planning an intervention
for any type of unknown hazards poses a considerable
challenge for animal health services, because no information is available about the nature of any such hazard,
the population it affects, or its transmission and physiological characteristics. The EU Animal Health Strategy for
2007–2013 (available at />diseases/strategy/index en.htm) advocated the precautionary principle “where proportionate provisional measures should be taken to ensure a high level of health
protection pending further scientific information clarifying
the extent of the risk”. But in the absence of information
about what type of hazard emergence is to be expected,
the formulation of appropriate strategies and therefore
the assessment of early warning surveillance for emerging
diseases are severely constrained. More epidemiological

research is needed to estimate the likelihood of different categories of hazards, which then allows gathering
information on likely consequences and the necessary
response. The availability of such structured approaches
to support decision-making are critical to direct resources
towards hazards identified based on latest scientific evidence, which will avoid ‘fishing in the dark’. Participatory
approaches involving stakeholders and decision makers (e.g. using companion modelling) could for instance
be used to determine which management measures are
considered as appropriate for different epidemiological
scenarios (regarding the status of the focal population
and/or its components). In conclusion, the description of
an intervention strategy might not always be available but
is an essential pre-requisite to assess surveillance effectiveness through the approach presented here.


84

V. Grosbois et al. / Preventive Veterinary Medicine 120 (2015) 70–85

4.3. Further developments
4.3.1. Considering more complex mitigation strategies
For the sake of clarity we have only considered mitigation strategies in which the possible values of the
epidemiological state variable were categorized into subsets each of which was associated with a specific mitigation
option. Mitigation strategies in which the modalities and
intensity of mitigation measures vary continuously as a
function of the value of the epidemiological state variable
will be considered in further developments.
Surveillance-control is most often an adaptive management process in which surveillance informs decision
makers not only on the current epidemiological state of a
focal population but also on the effectiveness of the intervention measures implemented. The implementation of
an intervention measure is thus likely to be motivated

by a change in the epidemiological state of the population resulting from previous interventions rather than by
its current epidemiological state. It is thus necessary to
extend the proposed rationale to situations where surveillance iteratively produce information on the evolution of
an epidemiological situation that determine sequences of
intervention measures
4.3.2. Integration of an epidemiological component
As exemplified in the hypothetical surveillance system
aiming at monitoring an endemic disease presented above,
the proposed approach allows evaluating the effectiveness
of surveillance for a given epidemiological situation. One
possible limitation is thus that estimations of the probabilities of Type I and Type II errors are relevant with regard
only to this unique situation. A question that will shortly
be addressed is that of the integration of probabilities of
Type I and Type II errors over sets of potential epidemiological situations. Epidemiological models could be integrated
in our framework to derive relative probabilities of occurrence the potential situations in such sets. Such relative
probabilities could be used to compute weighted average
for probabilities of Type I and Type II errors.
4.3.3. Economic analysis and decision-making
With the probabilities of Type I and Type II errors
established, the next step is to assess what the economic
consequences are of each type of error. This would allow
assessing discrepancy between the cost, the effectiveness
and the benefits of intervention measures that would be
implemented given a perfect knowledge of the epidemiological situation and the cost, the effectiveness and the
benefits of intervention measures that are likely to be actually implemented based on the analysis and interpretation
of the data produced by a surveillance system.
Evaluating economic consequences of Type I and Type
II errors implies estimating the economic consequences
of either implementing costly interventions unnecessarily
or not implementing interventions when they would be

required because of disease presence. This includes valuating (1) the production losses that occur due to morbidity
and mortality in animals by for example multiplying physical losses such as reduction in litres of milk produced in
dairy cows by price coefficients; (2) all the financial and

other resources used for intervention measures (e.g. vaccines, veterinary services, drugs); and (3) wider impacts
including human health effects, spill-over to other sectors
(e.g. disruption to tourism), and impacts on downstream
and upstream businesses (e.g. breeders, feed and drug producers, slaughterhouses), and multiplying them by the
probability of the error in question. Consequently, it is
indispensable to have an idea of the number of holdings
or animals affected (or in other words of the spread of the
disease) as well as the activities comprised in mitigation
activities. In the absence of empirical data, epidemiological
modelling techniques that capture the dynamics and complexity of disease in populations can be used to generate
these data. Such models are often used to deliver important input data for economic analyses (Perry and Randolph,
2004). Ideally, the estimations of costs are made based
on a continuous function of, for example, the cumulative
incidence and the probability of the errors as well as the
surveillance costs associated with this error. Like this, the
economic consequences of a Type I or II error can be compared directly to the investment in surveillance needed
to reduce the probability of this error and the target or
target range can be determined where overall costs are
minimised. Consequently, such a target would be an epidemiologically and economically efficient one. Hence, the
rationale proposed is suitable both to establish the value
of the surveillance in relation to an already defined target,
as well as to refine a decision rule taking into account the
socio-economic consequences of Type I or II errors and the
additional surveillance costs that would accrue to reduce
these errors.
Conflicts of interest

There are no conflicts of interest.
Acknowledgments
The research leading to these results has received
funding from the European Union Seventh Framework
Programme (FP7/2007–2013) under grant agreement no.
310806 (RISKSUR project). We warmly thank all the participants of RISKSUR project without whom the completion of
this work would not have been possible.
References
Caspari, C., Christodoulou, M., Monti, E., 2007. Prevention and control of
animal diseases worldwide. Economic Analysis – Prevention Versus
Outbreak costs. Final Report Part I. The World Organisation for Animal
Health, pp. 198, Available at: />Christensen, J., Stryhn, H., Vallières, A., Allaki, F.E., 2011. A scenario tree
model for the Canadian Notifiable Avian Influenza Surveillance System and its application to estimation of probability of freedom and
sample size determination. Prev. Vet. Med. 99, 161–175.
Dórea, F.C., McEwen, B.J., McNab, W.B., Revie, C.W., Sanchez, J., 2013.
Syndromic surveillance using veterinary laboratory data: data preprocessing and algorithm performance evaluation. J. R. Soc. Interface
10, 20130114.
Drewe, J., Hoinville, L., Cook, A., Floyd, T., Stärk, K., 2012. Evaluation of
animal and public health surveillance systems: a systematic review.
Epidemiol. Infect. 140, 575–590.
Drewe, J., Hoinville, L., Cook, A., Floyd, T., Gunn, G., Stärk, K., 2015. SERVAL:
a new framework for the evaluation of animal health surveillance.
Transbound. Emerg. Dis. 62, 33–45.


V. Grosbois et al. / Preventive Veterinary Medicine 120 (2015) 70–85
Frössling, J., Ågren, E.C.C., Eliasson-Selling, L., Lewerin, S.S., 2009. Probability of freedom from disease after the first detection and eradication of
PRRS in Sweden: scenario-tree modelling of the surveillance system.
Prev. Vet. Med. 91, 137–145.
German, R.R., Lee, L., Horan, J., Milstein, R., Pertowski, C., Waller, M., 2001.

Updated guidelines for evaluating public health surveillance systems.
MMWR Recomm. Rep. 50, 1–35.
Häsler, B., Howe, K.S., Stärk, K.D., 2011. Conceptualising the technical relationship of animal disease surveillance to intervention and mitigation
as a basis for economic analysis. BMC Health Serv. Res. 11, 225.
Hendrikx, P., Gay, E., Chazel, M., Moutou, F., Danan, C., Richomme, C., Boue,
F., Souillard, R., Gauchard, F., Dufour, B., 2011. OASIS: an assessment
tool of epidemiological surveillance systems in animal health and food
safety. Epidemiol. Infect. 139, 1486–1496.
Hoinville, L., 2013. Animal Health Surveillance Terminology Final Report
from Pre-ICAHS Workshop, Available at: />ahvla-en/files/icahs-workshop-report.pdf
Hoinville, L., Alban, L., Drewe, J., Gibbens, J., Gustafson, L., Häsler, B.,
Saegerman, C., Salman, M., Stärk, K., 2013. Proposed terms and
concepts for describing and evaluating animal-health surveillance
systems. Prev. Vet. Med. 112, 1–12.
Hood, G., Barry, S., Martin, P., 2009. Alternative methods for computing the
sensitivity of complex surveillance systems. Risk Anal. 29, 1686–1698.
Howe, K., Häsler, B., Stärk, K., 2013. Economic principles for resource allocation decisions at national level to mitigate the effects of disease in
farm animal populations. Epidemiol. Infect. 141, 91–101.
Knight-Jones, T., Rushton, J., 2013. The economic impacts of foot and
mouth disease – what are they, how big are they and where do they
occur? Prev. Vet. Med. 112, 161–173.
Knight-Jones, T.J., Hauser, R., Matthes, D., Stärk, K.D., 2010. Evaluation
of effectiveness and efficiency of wild bird surveillance for avian
influenza. Vet. Res. 41, 50.
MARD, MOH, 2011. Vietnam Integrated National Operational Program
on Avian Influenza, Pandemic Preparedness and Emerging Infectious
Diseases, 2011–2015. Ministry of Agriculture and Rural Development and Ministry of Health, Hanoi, Vietnam, Available at: http://
animaldiseaseintelligence.com/wordpress/wp-content/uploads/
2011/12/AIPED.pdf
Martin, P., 2008. Current value of historical and ongoing surveillance for

disease freedom: surveillance for bovine Johne’s disease in Western
Australia. Prev. Vet. Med. 84, 291–309.

85

Martin, P., Cameron, A., Barfod, K., Sergeant, E., Greiner, M., 2007a. Demonstrating freedom from disease using multiple complex data sources:
2: Case study – classical swine fever in Denmark. Prev. Vet. Med. 79,
98–115.
Martin, P., Cameron, A., Greiner, M., 2007b. Demonstrating freedom from
disease using multiple complex data sources: 1: A new methodology
based on scenario trees. Prev. Vet. Med. 79, 71–97.
Minh, P.Q., Morris, R.S., Schauer, B., Stevenson, M., Benschop, J., Nam, H.V.,
Jackson, R., 2009. Spatio-temporal epidemiology of highly pathogenic
avian influenza outbreaks in the two deltas of Vietnam during
2003–2007. Prev. Vet. Med. 89, 16–24.
Otte, M., Nugent, R., McLeod, A., 2004. Transboundary Animal Diseases:
Assessment of Socio-Economic Impacts and Institutional Responses,
at:
/>Available
publications/sector discuss/pp nr9 final.pdf
Perry, B.D., Randolph, T.F., 2004. Integrated epidemiology and economics
modelling for the management of animal health. Dev. Biol. (Basel) 119,
389–402.
Peyre, M., Fusheng, G., Desvaux, S., Roger, F., 2009. Avian influenza
vaccines: a practical review in relation to their application in the
field with a focus on the Asian experience. Epidemiol. Infect. 137,
1–21.
Rushton, J., Upton, M., 2006. Investment in preventing and preparing for
biological emergencies and disasters: social and economic costs of
disasters versus costs of surveillance and response preparedness. Rev.

Sci. Tech. 25, 375–388.
Swayne, D.E., 2009. Avian Influenza. John Wiley & Sons.
Tiensin, T., Nielen, M., Vernooij, H., Songserm, T., Kalpravidh, W.,
Chotiprasatintara, S., Chaisingh, A., Wongkasemjit, S., Chanachai, K.,
Thanapongtham, W., 2007. Transmission of the highly pathogenic
avian influenza virus H5N1 within flocks during the 2004 epidemic
in Thailand. J. Infect. Dis. 196, 1679–1684.
Tomassen, F., De Koeijer, A., Mourits, M., Dekker, A., Bouma, A., Huirne,
R., 2002. A decision-tree to optimise control measures during the
early stage of a foot-and-mouth disease epidemic. Prev. Vet. Med. 54,
301–324.
Yamamoto, T., Tsutsui, T., Nishiguchi, A., Kobayashi, S., 2008. Evaluation
of surveillance strategies for bovine brucellosis in Japan using a simulation model. Prev. Vet. Med. 86, 57–74.