Predictive nuclear power plant outage control through computer vision and data-driven simulation

Progress in Nuclear Energy 127 (2020) 103448

Contents lists available at ScienceDirect

Progress in Nuclear Energy
journal homepage: />
Predictive nuclear power plant outage control through computer vision and
data-driven simulation
Zhe Sun a, Cheng Zhang b, Jiawei Chen a, Pingbo Tang c, *, Alper Yilmaz d
a

School of Sustainable Engineering and the Built Environment, Arizona State University, 660 S College Avenue, Tempe, AZ, 85281, USA
The Zachry Department of Civil Engineering, Texas A&M University, 201 Dwight Look Engineering Building, College Station, TX, 77843, USA
c
Department of Civil and Environmental Engineering, Carnegie Mellon University, 5000 Forbes Avenue, Pittsburgh, PA, 15213, USA
d
Department of Civil, Environmental and Geodetic Engineering, The Ohio State University, 470 Hitchcock Hall, 2070 Neil Avenue, Columbus, OH, 43210, USA
b

A R T I C L E I N F O

A B S T R A C T

Keywords:
Nuclear power plant outage
Computer vision
Simulation

Field operation and preparation (FO & P) processes in the outages of nuclear power plants (NPPs) involve tedious
team coordination processes. This study proposed a predictive NPP outage control method through computer
vision and data-driven simulation. The proposed approach aims at automatically detecting abnormal human/

team behaviors and predicting delays during outages. Abnormal human/team behaviors, such as prolonged task
completion and long waiting time, could induce delays. Timely capturing these field anomalies and precisely
predicting delays is critical for guiding schedule updates during outages. Current outage control relies heavily on
manual observations and experience-based field adjustments, which require extensive management efforts. Realtime field videos that capture abnormal human/team behaviors could provide information for supporting the
prognosis of abnormal FO & P processes. However, manual video analysis could hardly provide timely infor­
mation for diagnosing delays. Previous studies show the potentials of using real-time videos for capturing field
anomalies. These studies fell short in examining automatic video analysis in compact work environments with
significant occlusions. Besides, limited studies revealed how the captured field anomalies trigger delays during
outages.
Computer vision techniques have the potential for automating field video analysis and detections of prolonged
task completions and long waiting times. This paper aims at automating the integrated use of 1) real-time
computer vision and spatial analysis algorithms, and 2) data-driven simulations of FO & P processes for sup­
porting predictive outage control. The authors first use the video-based human tracking algorithm to detect
human/team behaviors from field videos. Then, the authors formalized detailed human-task-workspace in­
teractions for establishing a simulation model of FO & P processes during outages. The simulation model takes
the field anomalies captured from videos as inputs to adjust model parameters for achieving reliable predictions
of workflow delays. Major observations show that 1) task delays often occur at the initial stage of the workflow,
and 2) waiting line accumulates due to excessive resource sharing during handoffs at the middle stage of the
workflow. The simulation results show that tasks on the critical-path are more sensitive to these anomalies and
cause up to 5.53% delays against the as-planned schedule.

1. Introduction
Aging nuclear power plants (NPPs) in the United States require
routine maintenance shutdowns (known as “outages”) to ensure
continuous power supplies (Lloyd, 2003). Outages are necessary to
ensure efficient NPP operations by refueling the reactor and executing
essential repairs. Abnormal human/team behaviors (e.g., prolonged task
completion, long waiting) during field operation and preparation (FO &

P) processes bring significant challenges in controlling delays during

outages. Real-time monitoring of FO & P processes for capturing
abnormal human/team behaviors and estimating potential delays is
vital to ensure resilient outage control. Such a monitoring system aims to
support the outage control center (OCC) for making appropriate
schedule updates and reducing delays (Fig. 1). A typical NPP outage
requires hundreds of contract workers to complete thousands of refu­
eling and maintenance activities within 30 days (B.N. Spring 2009).

* Corresponding author.
E-mail address: (P. Tang).
/>Received 1 November 2019; Received in revised form 25 May 2020; Accepted 10 July 2020
Available online 27 July 2020
0149-1970/© 2020 The Authors.
Published by Elsevier Ltd.
This is an
( />
open

access

article

under

the

CC

BY-NC-ND

license


Z. Sun et al.

Progress in Nuclear Energy 127 (2020) 103448

However, workers with diverse backgrounds and prior outage experi­
ences require extensive training to fulfill the productivity requirements
during NPP outages (Sun et al., 2018a; Zhang et al., 2018a). Poor
human/team performance that causes prolonged task completion and
long waiting time could jeopardize NPP outages. Timely capturing such
poor human/team behaviors that cause performance bottlenecks during
FO &P processes is thus crucial for predictive control of NPP outages.
Integrated uses of automatic video analysis algorithms and compu­
tational simulation models for automated monitoring, diagnosis, and
control of teamwork processes during outages is becoming possible with
the development of computer vision techniques and computational
simulation methods (Luo et al., 2014, 2019; Sun et al., 2020). Field
videos of outage workspaces are becoming widely available for FO &P
process monitoring with the increasing use of cameras in workspaces
(Bolton, 2015). Current outage control practices rely heavily on tedious
and error-prone manual inspections and experience-based judgments of
outage professionals (Germain et al., 2014). Video data collection has
the advantage of capturing rich spatiotemporal details of human mo­
tions without having to install any contact sensors on workers. A
video-based approach for human/team behavior analysis in outages is
thus of great potential for supporting human factors analysis and pre­
dictive NPP outage control (Fang et al., 2018). Such a video-based
approach can help to capture the prolonged task completion and long

waiting time during FO & P processes. The captured anomalies could
become rich data sources for OCC to decide schedule updates for miti­
gating delays.
Formal models and simulations have the potential to resolve the
difficulties of assessing the captured abnormal human/team behaviors
and predicting delays caused by those anomalies (Bolton, 2015; Pan and
Bolton, 2015). Specifically, agent-based simulation models could
represent 1) human agents – representing behaviors of workers and the
outage supervisors (Zhang et al., 2019; Bonabeau, 2002); and 2) work­
flows – representing the maintenance workflow based on given sched­
ules (Wang et al., 2014; Chen et al., 2012). Some simulation models
show the potential of being able to take the captured anomalies from the
real-time videos as inputs for predicting workflow delays (Chen et al.,
2012; Mohamed et al., 2017; Han et al., 2013; Alzraiee et al., 2015). For
example, Chen et al. established an intelligent scheduling system for
optimizing resource sharing during construction projects through sim­
ulations (Chen et al., 2012). This study established a simulation model
that includes representations of a construction schedule and relation­
ships between available resources (e.g., workforce, equipment) and
workspaces. However, these simulation models fell short in providing

detailed attributes when modeling human agents in the simulation. Such
attributes, such as traveling activities, communication behaviors, and
waiting at a station, are pivotal for capturing abnormal human/team
behaviors. A data-driven simulation model that represents detailed
human/team behaviors during outages is vital for predicting delays
using the captured field anomalies from videos.
Previous studies have examined the use of computer vision and
machine learning techniques to detect abnormal human/team behaviors
using real-time videos (Fang et al., 2018; Wang et al., 2018; Wang and

Liu, 2018). However, limited studies have examined the use of real-time
field videos in a compact workspace with significant occlusions for
capturing abnormal human/team behaviors. Challenges remain in
automatically capturing field teamwork anomalies and related produc­
tivity losses for predicting delays. Such challenges include 1) significant
occlusions in the field videos recorded in a compact indoor workspace
during outages, and 2) modeling of interwoven human-task-workspace
interactions during FO & P processes for predicting delays due to the
captured anomalies. This study proposed an integrated use of computer
vision and simulation for capturing abnormal human/team behaviors
and predicting delays caused by the captured anomalies (Fig. 2). The
proposed method includes using a single-camera based trajectory anal­
ysis approach for increasing the coverage of monitoring with the same
number of cameras during indoor monitoring. Specifically, this study 1)
examined real-time human tracking and spatiotemporal analysis
methods for automatically diagnosing abnormal human interactions and
unexpected trajectories of workers, and 2) developed a data-driven
agent-based simulation model for using the detected abnormal
human/team behaviors as input for predicting delays during NPP
outages.
The proposed framework integrates knowledge from human factors,
computer vision, and simulations that enable engineers to fully discover
the interactions between humans, resources, and workflow that influ­
ence outage productivities. The goal is to advancing disciplines of
human systems integration and computer vision in the construction
management domain. First, the authors conducted an extensive litera­
ture review about human factors in NPP outages. The review aims to
identify abnormal human/team behaviors in FO & P processes during
NPP outages. The authors then developed and tested state-of-the-art
computer vision algorithms that help to monitor and capture field

anomalies in compact workspaces during FO & P processes of NPP
outages. The developed computer vision algorithms enable timely and
detailed monitoring of the FO & P processes for supporting real-time
schedule adjustments and resource allocations during NPP outages.

Fig. 1. Real-time field information acquisitions for capturing waiting time and task duration variations.
2


Z. Sun et al.

Progress in Nuclear Energy 127 (2020) 103448

Fig. 2. The proposed predictive NPP outage control framework.

Last, the presented research study established a data-driven agent-based
simulation model based on typical FO & P processes during NPP outages.
The model integrates the synthesized findings of human errors in outage
control to model “difficult” FO & P processes during NPP outages that
challenge teamwork performance. Running simulations could help to
examine the occurrence and propagation of abnormal human/team
behaviors during NPP outages.
The organization of the remaining parts of the paper is: Section 2
provides a detailed literature review about three practical problems in
NPP outages. Section 3 illustrates the developed agent-based model of
human/team behaviors within outage workflows. Section 4 introduces
the developed computer vision algorithms for automatic human
behavior data acquisition and analysis. Section 5 describes the frame­
work of the data-driven simulation model for assessing the impact of
abnormal human/team behaviors on workflow delays. Section 6 sum­

marizes the major research findings and technical challenges. Section 7
concludes and synthesizes the future research directions.

window (Tang et al., 2016). Effective FO & P processes at both indi­
vidual and team levels are increasingly necessary for accomplishing
complex tasks and avoid delays (Sun et al., 2020). At the individual
level, schedule deviations due to workers’ operational errors could also
bring significant risks of delays. Such deviations often cause prolonged
task completion against the as-planned schedule (Jang et al., 2013).
Delays on these individual tasks could propagate to delays to larger
workflows or even the complete outage, especially when the scheduled
task has limited floats. At the team level, NPP outage control is one of
those tasks that need multiple professionals to work together for col­
lective decision-making. Workers need to go through Radiation Pro­
tection Island (RPI), where the space connecting the containment and
outside environment for getting prepared for the scheduled task. Coor­
dinating workers from different teams during handoffs require precise
estimations of delays inside the RPI. Poor coordination can cause un­
necessary waiting and cause significant delays in RPI. Previous studies
and practices tried to examine the impact of waiting in such task prep­
aration processes on workflow delays (Zhang et al., 2018a; Sun et al.,
2020). However, most of these studies are Monte Carlo simulations with
limited use of real-time data (waiting time) collected in field operations
as inputs for reliable delay diagnosis based on real data.

2. Literature review
This section provides a systematic literature review about 1) prac­
tical problems during NPP outages and 2) challenges in capturing
abnormal human/team behaviors and predicting outage delays.
Abnormal human/team behaviors during NPP outages could induce

severe delays and cause significant financial losses. How to capture these
anomalies and assess the impacts on outage delays is thus crucial for
supporting the outage management efforts. Three significant aspects
need to be solved to achieve such goal: 1) a better understanding of the
interactions between human/team behaviors and workflows of NPP
outages; 2) a technology that can effectively capture abnormal human/
team behaviors, and 3) a method that can use the captured human/team
anomalies for predicting delays during outages. The following sections
thus provide reviews on 1) human and team behaviors in NPP outages;
2) computer vision for real-time human/team behavior monitoring; and
3) effective use of field information for resilient outage control.

2.2. Practical problem two: computer vision for real-time human/team
behavior monitoring
FO & P surveillance important for determining whether a project can
complete on time and avoiding budget overruns (Ghanem and Abdel­
Razig, 2006; Cheng et al., 2013; Girardeau-Montaut et al., 2005).
Existing sensing techniques show the potential for tracking real-time
locations of construction entities during FO & P processes. Examples
of such technologies include Radio Frequency Identification (RFID),
Global Positioning Systems (GPS), and Ultra-Wideband (UWB) (Ghanem
and AbdelRazig, 2006). However, all these sensors are usually not
applicable for NPP outages due to confidentiality issues (Zhang et al.,
2017). Some studies proposed the use of closed-circuit television (CCTV)
installed at job sites to monitor construction workers’ behaviors and
locations of equipment to ensure safe operation (Shrestha et al., 2015;
Hinze and Teizer, 2011). However, such monitoring methods risk
exposing workers’ faces and cause confidentiality issues. A
confidential-protective progress monitoring method is thus necessary to

2.1. Practical problem one: human and team behaviors in NPP outages
NPP outages require coordinating hundreds of contract workers with
diverse backgrounds to complete thousands of tasks within a tight
3


Z. Sun et al.

Progress in Nuclear Energy 127 (2020) 103448

capture field workers’ behaviors and avoid exposing sensitive produc­
tivity information. Such a monitoring method could target monitoring
areas that are mostly preparation activities where workers are mostly
waiting and preparing to help the manager to understand the overall
productivity and bottlenecks without directly measure the workers’ task
performance. The confidential-protective monitoring method could
target at detecting and tracking workers’ body joints without exposing
their faces and other identity information. Besides, such a technique
could also capture anomalies during the task preparation processes (e.g.,
extended duration, waiting time) without leaking extensive human
privacy. At the same time, such preparation and waiting time informa­
tion could still be useful for inferring possible delays, identifying poor
process arrangements, and suggesting schedule updates.
Monitoring FO & P processes are critical for ensuring a better team
situation awareness during outages. However, human gestures during
refueling and maintenance activities vary significantly. On the other
hand, human/team behaviors during handoff processes have relatively
fewer uncertainties and variations from the predefined procedures in the
controlled indoor environments. For example, the Radiation Protection
Island (RPI), which is the space connecting the containment and outside

environment, contains a lot of handoff activities. Such handoffs usually
involve dosimetry checking, technical debriefing, and tool pick-up/
drop-off. Workers have limited options while deciding how to go
through their handoffs. These limited options can be as waiting
(standing still or sitting), walking between stations in the RPI, or talking
to each other. Monitoring such simple behaviors in an RPI could still be
useful for inferring workflow delays during outages. An effective and
efficient method for preparation and waiting behavior monitoring in
indoor environments could thus bring benefits to NPP outage control.

frequently cause delays during outages. The critical path method (CPM)
has widely been adopted by the construction industry to control the
schedule and estimate delays by identifying the longest path of depen­
dent activities and measuring the time.
3.1. Modeling of detailed spatiotemporal human-task-workspace
interactions in a valve maintenance workflow during NPP outage
Modeling detailed human-task-workspace interactions in a valve
maintenance workflow during NPP outages requires setting up
numerous constraints to specify the relationships between humans, task,
and workspace. The developed model contains a workflow model and a
human activity model. The workflow model captures the spatiotemporal
relationship between tasks. The human activity model specifies the re­
sponsibilities of workers on the scheduled tasks. Fig. 3 visualizes the
developed human-task-workspace model.
Valves are the critical mechanical component for nuclear reactors.
The authors model the valve maintenance activities at two job sites (Site
A, and B, respectively) during a typical NPP outage (Fig. 3). Three
workers (insulator, electrician, and mechanic) need to complete five
tasks on each site with the information from the supervisor. All workers
need to go through RPI for 1) checking available work packages, 2)

technical briefing, and 3) picking up tools (e.g., earplugs) before the
workers start their work at Site A. Besides, once the workers complete
their tasks, they need to 1) get back to RPI for dosimetry checking; 2)
dropping off tools, and 3) check other available work packages. Work­
spaces and workers are all shared resources, which means workers
cannot work at two sites at the same time. If a worker occupies a
workstation for a prolonged duration, other workers have to wait in line
for using the workstation. Extended task durations and long waiting due
to resource sharing could be indicators of delays in the valve mainte­
nance workflow.

2.3. Practical problem three: effective use of field information for resilient
outage control
Continuously monitoring of FO & P processes and use the captured
field anomalies for predicting workflow delays is critical to ensure a
resilient outage control (Sun et al., 2020; Yoo et al., 2016). A detailed
human-task-workspace model that captures interwoven relationships
between human/team behaviors and workspaces can support
decision-makers for making prompt field adjustments (e.g., schedule
updates) (Sun et al., 2018b). However, the lack of representations of
detailed human/team behaviors in current outage scheduling processes
causes challenges in risk assessments with full considerations of human
factors. Such a situation impedes engineers and researchers from using
computer algorithms for assessing problematic outage scenarios and
mitigation strategies. Besides, tedious communications and traveling
activities during NPP outages raised additional challenges for precise
risk assessments. For example, the talking behaviors and the perception
processes of the information received are challenging to model in any
mathematical model. Therefore, project managers chose not to consider
complex human/team behaviors as a factor for analytical modeling in

the current practice of project management. Instead, the use of buffering
approaches is favored in the current project management processes to
mitigate delays due to abnormal human/team behaviors. Besides, the
uncertainty of the task durations greatly influences the performance of
scheduling techniques. In brief, most of the current scheduling tech­
niques fell short in reducing delays and achieving NPP outage resilience.

3.2. Modeling of human/team behaviors
In a valve maintenance workflow, the supervisor needs to coordinate
several workers to work on several tasks at the same time through
tedious communications. In this study, the authors have created two
types of agents (the worker agent and the supervisor agent) for modeling
the human behaviors during NPP outages. The supervisor agent collects
field information from the worker agent through communications. Field
information is critical for the supervisor to make appropriate decisions,

3. Agent-based modeling of human/team behaviors within
outage workflows
This section presents lab experiments for modeling and capturing
human/team behaviors during a FO & P process in a typical valve
maintenance workflow during NPP outage. The developed agent-based
simulation model helps to formulate a basis for understanding the im­
pacts of abnormal human/team behaviors on workflow delays. The
valve maintenance workflow contains critical-path activities that

Fig. 3. The spatial and temporal relationship between tasks in the
outage workflow.
4

Z. Sun et al.

Progress in Nuclear Energy 127 (2020) 103448

such as add additional tasks if discoveries found in the field. Field
conditions change frequently during outages due to various un­
certainties (e.g., variations of the task duration, field discoveries).
Effective coordination between the supervisor and workers is thus
necessary to keep each other informed. The authors model such infor­
mation exchanging processes as the communication behaviors between
workers and the supervisor (Fig. 4). The communication behaviors in
the model considered two communication aspects, 1) communication
network patterns (e.g., network structure); and 2) characterizations of
communication links (e.g., channel and timing). Specifically, the au­
thors use a centralized communication network to model the commu­
nication behaviors that allows 1) the supervisor to assigned work
packages to workers when there are available tasks, and 2) the workers
to report the task completion status to the supervisor when the current
task has completed.
The authors define four behaviors for the “worker” agents, 1)
working on scheduled tasks according to the as-planned schedule, 2)
reporting to the “supervisor” agent when the current task has completed,
3) traveling to the next job site for the successor task, and 4) waiting for
task availability information from the supervisor. The “worker” agents
will only enter into the “working” status to execute the scheduled task
only when the “supervisor” agent has assigned a task to them. When in
working status, the “worker” agents will work on the tasks planned by
following the as-planned task duration specified in the schedule. How­
ever, uncertainties (e.g., task duration variation) in the “working” status
may cause prolonged task completion due to poor human behaviors.

After the “worker” agents finish the task, they will enter the communi­
cation status. The communication status requires the “worker” agents to
report the task completeness to the “supervisor” agent. In the meantime,
the “supervisor” agent will be in the communication status as well.
Based on the field information reported by the “worker” agents, the
“supervisor” agent can mark the task as complete and identify new
available tasks. Once the “supervisor” agent identified the available
tasks, he/she can call the “worker” agent about available tasks. The
“worker” agents can travel to a different job site to work on the available
task informed by the “supervisor” agent during communications.

human/team behaviors in an indoor workspace and capturing abnormal
poses during NPP outages. The authors then introduced the basic
graphical user interface (GUI) design to highlight the developed system.
4.1. Human joint detection
The human joint detection algorithm uses the layout of an indoor
workspace (e.g., RPI) for mapping the locations on video frames to the
layout (Fig. 5). The algorithm first maps the image spaces of raw images
to the 2D trajectory on the RPI room layout. The authors then use a topdown method for tracking the workers’ poses (Cao et al., 2017) and
apply a two-branch Convolutional Neural Network (CNN) network for
detecting body joints of multiple workers in the indoor workspace (Cao
et al., 2017). The algorithm then outputs the detected body joints of
multiple workers in the scene through a refining process (Cao et al.,
2017). As shown in Figs. 5 and 6, the detected skeletons consist of body
joints of workers in the indoor workspace.
In the human joint detection process, the authors use a graphmatching algorithm for matching the body joints of a worker in the
scene. The graph-matching algorithm recognizes all body joints of a
worker by using the orientation of the worker’s body and the workers’
limbs as the edge weights of the k-partite graph (Cao et al., 2017). The
detection randomly chooses an ID for a worker in the video per frame

when the worker first appears in the scene. However, keeping the same
IDs on workers during the tracking process is still a challenge due to
significant occlusions that might cause ID switches. The outputs (Fig. 6)
of the human joint detection process grouped the labeled body joints of
multiple workers into skeletons. These skeletons then serve as inputs to
the virtual planes generated based on the anthropometric measures of a
typical worker (Zhang et al., 2018b).
4.2. Video projection to layout map
Using a single-camera-based approach for detecting and tracking
body joints of multiple workers in a compact indoor workspace is
challenging due to significant occlusions (Zhang et al., 2018a). An
effective tracking algorithm using human body joints is thus necessary
to track workers’ movements precisely in a compact workspace. How­
ever, the displacements in the image space become larger when workers
getting closer to a camera and result in a higher velocity in the image
space. Besides, multiple workers might walk across the room, running,
and standstill. Such multi-worker scenarios bring challenges for effec­
tive human tracking. Using a single-camera-based approach causes dif­
ficulties in reliably tracking multiple workers that are moving inside a

4. Computer vision algorithms for automatic human behavior
data acquisition and analysis
In this section, the authors introduce the developed automatic video
surveillance system. The developed system uses state-of-the-art com­
puter vision algorithms to achieve effective situation awareness of the
outage progress. Besides, the surveillance system aims at monitoring

Fig. 4. Agent-based communication behavior modeling (worker agent; supervisor agent).
5

Z. Sun et al.

Progress in Nuclear Energy 127 (2020) 103448

Fig. 5. Human joint detection.

Fig. 6. Body joint detection of workers.

compact workspace. The algorithms transform the detected joints from
the image space to the joint space on virtual planes to mitigate the risks
of losing depth information by using one single camera. Luo et al. pro­
posed a method for generating virtual “Anthropometric Planes” through
homograph transformation (Luo et al., 2019). All these virtual planes are
in parallel with the horizontal plane of the ground of the indoor work­
space. Then the algorithms generate multiple virtual planes at levels of
workers’ body joints. The authors then applied Kalman Filter to track
those detected joints of multiple workers in the indoor workspace on
these “Anthropometric Planes”.

preparation processes in the indoor workspace (e.g., technical debrief­
ing, tool pick-up/drop-off). The GUI allows the outage manager to use
layout maps of any indoor workspace and select the area of interest for
monitoring bases on the layout. In this study, the authors use the layout
map of an RPI for testing the developed human-tracking algorithms and
the developed GUI. The GUI shows that the average waiting time will
start counting when a worker enters a station and starts the handoff
process (e.g., technical briefing). The counting will stop until the worker
finishes the handoff and moves on to the next station.
In this GUI, all stations have different thresholds (alarming and alert

times) with the time unit due to the nature of the different tasks during
NPP outages. The management team can modify the alarming and alert
thresholds based on the urgent levels of specific tasks when coordinating
activities during NPP outages. Besides, the GUI displays the overall
waiting time inside the RPI as well. With such information, the man­
agement team will be able to monitor the waiting time inside the RPI
and make proper decisions on when to send workers to the RPI for
preparing the successor tasks. This visualization of the computer vision
system enables outage manager to quickly identify the status of multiple
stations and spot the field anomalies.

4.3. Design of graphical user interface (GUI)
In this section, the authors develop a graphical user interface GUI
(Fig. 7) that displays multiple simultaneously tracked workers in the RPI
and identifies field anomalies (e.g., extended task duration, long waiting
time) in the workflow. The GUI visualizes the human/team behaviors in
an indoor environment by showing 1) the moving patterns between
stations, 2) the duration each worker spends at each workstation, and 3)
wait time for each station. Such visualization enables engineers to use
the developed tracking algorithm for real-time visualizing of the
tracking results. The developed GUI allows outage managers to visualize
the FO & P processes in the indoor workspace and make proper field
adjustments for mitigating delays caused by field anomalies.
Fig. 7 shows the detailed GUI design for visualizing the task

5. Data-driven simulation framework for assessing the impact of
human/team behaviors on workflow delays
The proposed data-driven agent-based simulation framework
6

Z. Sun et al.

Progress in Nuclear Energy 127 (2020) 103448

Fig. 7. Real-time monitoring and statistics output (Red cell indicates the time the worker spent in the station exceeded the alert limits).

contains 1) an agent-based simulation platform consists of a process
model of an NPP outage workflow, and 2) a human activity model. The
data-driven simulation platform uses the captured anomalies from the
computer vision algorithm (e.g., prolonged task completion, long
waiting time) as input to simulate the impact of these detected anom­
alies on workflow delays (Fig. 8). The authors conducted a series of lab
experiments by using the valve maintenance workflow to validate the
proposed framework. During the experiments, the authors implemented
the developed computer simulations to examine 1) the developed
computer vision algorithm in capturing task duration variations and
waiting for lines in RPI; and 2) the proposed simulation framework in
predicting potential workflow delays.

5.1. Simulation for assessing the impact of task duration variance on
workflow delays
The authors carried out a series of lab experiments with participants
recruited from the construction engineering program at Arizona State
University. All participants have profound knowledge about the con­
struction schedule and were provided with extensive training sessions to
get familiar with the experiments. The experiment was set up based on
the established spatial and temporal relationship between tasks in the
outage workflow (Fig. 3) and the human behavior models (Fig. 4). The
supervisor will coordinate with three workers (insulator, electrician,

and mechanic) to complete five tasks at two job sites (Table 1 listed all
the task information). Besides, all workers have to go through RPI for
completing the handoff activities (Fig. 3). Each task will have an asplanned task duration that requires the participant to follow (partici­
pants will use the timer provided to count the time for their tasks).
However, participants might have different behaviors (e.g., prolonged
task completion, waiting) during handoffs and cause delays to the asplanned task duration (Fig. 9). Besides, such delays could be critical if
the successor tasks are on the critical path of the schedule. According to
the as-planned schedule, the authors derived the critical path of this
workflow to better interpret the delays captured during the experiments.
According to the observations during experiments, the authors found
that some participants could not strictly follow the as-planned schedule
and complete the task in time. Besides, delays on one task can easily
propagate to other tasks and jeopardize the workflow. The authors have
recorded all such delays during the experiments and incorporated the

Fig. 8. The data-driven agent-based simulation framework.
7


Z. Sun et al.

Progress in Nuclear Energy 127 (2020) 103448

5.2. Simulation for assessing the impact of waiting line during handoff on
workflow delays

Table 1
Delays captured during lab experiments.
Site

A

Task

Task 1
(A)
Task 2
(A)
Task 3
(A)
Task 4
(A)
Task 5
(A)
B
Task 1
(B)
Task 2
(B)
Task 3
(B)
Task 4
(B)
Task 5
(B)
Total Duration
Delay

Worker
Team

As-planed
Duration
(minutes)

Avg. Delay
(minutes)

Delays in
Simulation
(minutes)

Insulator

3

0:25

4:10

Electrician

4.5

0:20

3:20

Mechanic

6

0

0

Electrician

4.5

0

0

Insulator

6

0

0

Insulator

3

0:37

6:10

Electrician

4.5

0:21

3:30

Mechanic

6

0

0

Electrician

4.5

0

0

Insulator

6

0:20

3:20

The waiting time during RPI is essential for estimating the delays to
the valve maintenance workflow. During the lab experiments, the au­
thors observed that the waiting line in the RPI could be up to 30 min,
which causes a late-start of the successor task. Such delays could prop­
agate to all following tasks. As shown in Table 2, the last column in­
dicates the percentage of delays to the overall workflow due to waiting
time during handoffs. For example, the authors added a 30-min delay
after the insulator finished Task 1 (A) due to the late-start of Task 1 (A).
Such added delay causes a 4.32% delay (30 min) against the as-planned
schedule since Task 1 (A) is on the critical path.
The simulation results suggest that tasks on the critical-path are more
vulnerable to delays during handoffs. For example, Table 2 shows that a
30-min waiting during the handoff of Task 3 (A) contribute to 5.53%
delays against the as-planned schedule. Delays on Task 5 (A) had the
least impact on the as-planned workflow duration. Additionally, such
added delays not only affect individual tasks but also affect the prepa­
ration processes in the RPI. If specific tasks had delays, the probability of
having conflicts between different workers while in the briefing process
would increase. Moreover, the waiting time in the RPI would increase
due to the resource sharing between multiple workers during handoffs in
the RPI. Such resource sharing issues could occur when multiple workers
traveled to the same station in the briefing process. However, one station
can only be occupied by one worker at a time. Additional delays to the
workflow could arise due to such resource sharing during handoffs.

11.86 (hour)
0.29 (hour) 2.5%

recorded delays into the simulation model. By running the simulation,
the authors tried to understand the impact of individual task delays on
workflow delays. The last columns of Table 1 indicate the average delays
captured during the lab experiments and the delays during the simula­
tion (duration in the lab experiments are scaled). The average total
duration of is 11.57 h after 1000 runs of the simulation model. Compare
to the as-planed workflow duration; the delay is 0.29 h (2.5%).

6. Discussion
NPP outages are accelerated construction projects that pose signifi­
cant challenges to the limits and requirements of both human and
physical environments. A more resilient outage control should imple­
ment robust control methods to fully assess the reliability of humanphysical interactions and propose mitigating strategies accordingly.

Fig. 9. Images captured during lab experiment (a: experiment scene; b: the insulator is traveling between stations; c: the insulator is working at Station #1; d: a
waiting line occurs at Station #3).
8


Z. Sun et al.

Progress in Nuclear Energy 127 (2020) 103448

preparation processes are critical to ensure the smooth transitions be­
tween tasks and poor handoffs could result in substantial delays to NPP
outages. Non-value added activates during handoffs demand a more
accurate and flexible schedule updating methods for sending the worker
team to the RPI when workstations are available. Such demand thus
requires effective real-time monitoring of the handoff processes in RPI
and accurate estimation of the waiting time.

The proposed single-camera based trajectory analysis approach can
significantly increase the coverage of monitoring with the same number
of cameras during indoor monitoring. In the past, researchers can only
analyze 3D trajectories in regions covered by both two cameras; such
areas are much smaller than areas that only need to be visible to one
camera. Several technical challenges need further investigations to
quantify the costs and benefits of using a single-camera-based approach
for locating and tracking multiple workers, and how such techniques
could enhance the performance of a network of cameras in monitoring
numerous workers. First, implementing the computer vision algorithms
with one camera for detecting and tracking human/team behaviors in
compact indoor space with severe occlusion issues is still challenging.
Besides, using multiple cameras could solve the occlusion problems if
such resource is available in a packed indoor workspace like RPI, but
would raise additional challenges.
Specifically, challenges of using one single camera mainly lie in 1)
the use of one single camera for 3D localization of human individuals
moving in the indoor workspace; 2) potential information losses (e.g.,
depth information) due to the use of one single camera for tracking
moving patterns of individuals; 3) the real-time monitoring of moving
patterns of multiple individuals in the compact indoor workspace with
significant occlusions; and 4) frequent ID switch of multiple individuals
for accurate estimating the task duration and waiting time in the indoor
workspace. Besides, using a dual-camera system to get the 3D locations
still has some challenges. In certain areas, which are out of the over­
lapped field of view of multiple cameras, getting 3D locations becomes
difficult. Another reason for using a layout map to assist the human
tracking is that a layout map of RPI is quite accessible and reliable. Using
multiple cameras for human tracking could raise additional problems,
such as 1) calibration of multiple cameras, 2) coordinating the field-ofviews and locations of multiple cameras for ensuring the coverage of the

team processes with enough spatiotemporal details.

Table 2
Delays while considering 30-min waiting-line in RPI for each task.
Site

Task

Worker

As-planed Duration
(minutes)

Percentage of
Delays

A

Task 1
(A)
Task 2
(A)
Task 3
(A)
Task 4
(A)
Task 5
(A)
Task 1
(B)

Task 2
(B)
Task 3
(B)
Task 4
(B)
Task 5
(B)

Insulator

30

4.32%

Electrician

45

4.49%

Mechanic

60

5.53%

Electrician

45

4.32%

Insulator

30

1.38%

Insulator

30

4.32%

Electrician

45

3.46%

Mechanic

60

4.06%

Electrician

45

4.41%

Insulator

30

4.32%

B

This study developed a predictive outage control method, which in­
tegrates knowledge and strength from three domains, 1) human factors,
2) computer vision, and 3) computational simulation. The proposed
system aims at capturing abnormal human/team behaviors and predict
delays during outages. However, the authors have discovered challenges
while integrating human factors, computer vision, and simulation for
effective control of NPP outages. This section illustrates specific tech­
nical challenges associate with every element in the proposed predictive
control method.
6.1. Challenges of human factors analysis for predictive NPP outage
control
Human factors in NPP outages are critical for ensuring the safety and
productivity of the operation and maintenance activities. Human/team
behaviors, such as cognitive behaviors, communications, and travel
activities, could significantly affect NPP outages. Previous human fac­
tors studies focused more on conducting tedious lab experiments for
discovering human/team behavior deviations with limited consider­
ation of the impact of such deviations to the physical environment and
cause severe safety and productivity concerns to the NPP industries.

Besides, such experiments always tailored to fit into a certain scenario,
and the data collected cannot be used for other cases. Also, quantita­
tively defining “normal” interactions among individuals is challenging.
The OCC has established a detailed procedure for individuals to follow
during NPP outages with limited details of the expected behaviors at the
individual and team level. Specifically, challenges mainly fall into four
categories, 1) lack of formalized representations for modeling human/
team behaviors during NPP operation and maintenance activities; 2)
lack of comprehensive categorization of normal/abnormal human/team
behaviors in the NPP industry; 3) lack of a detailed reasoning method to
fully understand the arising and propagation processes of human/team
errors in NPP operation and maintenance activities; 4) lack of specific
human-task-workspace models for an accurate estimate the impact of
abnormal human/team behaviors on the productivity of NPP operation
and maintenance activities.

6.3. Challenges of data-driven simulation for predictive NPP outage
control
Data-driven simulation is a powerful tool for assessing the impacts of
abnormal human/team behaviors (e.g., prolonged task completion, long
waiting time) on NPP outage productivity issues. Such simulation plat­
forms can also examine possible mitigation strategies proposed by NPP
professionals in mitigating the risks of delays caused by abnormal
human/team behaviors. However, frequent schedule updates and
changes in work packages require effective outage team coordination for
ensuring proper executions of new work packages without elevating the
risks. For example, discoveries of new tasks due to maintenance failures,
defects on mechanical parts, or delays that occur while ordering new
parts for maintenance can cause severe delays. Unfortunately, current
outage control heavily rely on tedious manual inspection and adjust­

ment based on the control manager’s experience and knowledge. A more
resilient outage control system through data-driven simulation is thus
necessary to automatically 1) propose contingency plans to reduce risks
of delays in real-time; 2) and evaluate the performance of the proposed
contingency plan in terms of resource allocation, schedule delays and
cost overrun. Challenges still exist and lie in 1) lack of capabilities for
achieving real-time data-driven simulation that can use the captured
abnormal human/team behaviors as input to update the simulation
model rapidly; 2) lack of formalized methods to consider all extreme
events and assess the impacts on outage safety and productivity; 3) lack
of systematic approaches for examining mitigation strategies and

6.2. Challenges of computer vision techniques for predictive NPP outage
control
The developed computer vision algorithms in this study enabled the
real-time monitoring of human/team behaviors during FO & P processes
in a compact indoor workspace with significant occlusions. The
9


Z. Sun et al.

Progress in Nuclear Energy 127 (2020) 103448

optimizing parameters in the simulation models for reducing the safety
and productivity risks during outages.

Declaration of competing interest
The authors declare that they have no known competing financial
interests or personal relationships that could have appeared to influence

the work reported in this paper.

7. Conclusion and future research
Abnormal human/team behaviors that lead to frequent schedule
updates bring significant difficulties to achieve resilient outage control.
Even NPP outage professionals with extensive field experience could
hardly discover such abnormal human/team behaviors and assess the
impacts on outage delays. The developed computer vision algorithms
proved to be able to detect and track multiple individuals with a single
camera in a compact workspace with severe occlusions. The algorithm
then achieves a precision of 70% and a recall of 38%. The developed
data-driven simulation platform proved to be able to predict delays
using the anomalies captured by the computer vision algorithm during
handoffs. Major observations show that 1) task delays often occur at the
initial stage of the workflow, and 2) waiting line accumulates due to
excessive resource sharing during handoffs at the middle stage of the
workflow. The simulation results show that tasks on the critical-path are
more sensitive to these anomalies and cause up to 5.53% delays against
the as-planned schedule.
The authors envision the extension of the proposed computer vision
algorithms that could also be useful for 1) psychological assessment (e.
g., situation awareness, workload) of workers; 2) physical capability (e.
g., fatigue) evaluation of workers. Overall, the contributions of the
proposed method could be at two levels. First, the proposed method
could detect abnormal human/team behaviors by using body joints for
inferring delays without exposing the identity information. Second, the
proposed method could capture detailed human/team behaviors for
evaluating human/team performances on safety-critical tasks if no
confidentiality concerns. For example, heavy crane-lifting activities
during NPP outages usually require a lot of moving spaces in a compact

job site. Effective coordination to separate workers and cranes in a safe
spatial distance is critical to avoid crane-related accidents. Besides, vi­
sual verification and maintenance on pumps and valves are crucial for
ensuring enough coolant and adequate pressure to prevent a core
meltdown during NPP outages. However, a great number of valves and
pumps are in high proximity at certain locations become obstacles for
workers to recognize the correct valves. Performing scheduled tasks on
the wrong valves could be fatal for NPP outages.
Enhancing the proposed computer vision algorithm is thus necessary
to be able to 1) locate workers’ locations in real-time; 2) provide safe
navigation routes for workers to stay away from the crane; 3) auto­
matically match the worker’s location and the physical location of the
scheduled task based on the as-planned schedule and the site layout, and
4) generate alarms if a worker presented at the wrong location and
performed the task on the wrong object. Moreover, the developed
computer vision algorithms could also serve as powerful tools for the
post-hoc assessments of NPP outage efficiency. The captured human and
task-related anomalies are thus necessary for determining sections of the
outage that have low efficiency when multiple sections are included at
one job site. Overall, this paper summarizes the challenges and future
research directions that could benefit more broad research communities
composed of construction engineering and management and computer
science researchers.

Acknowledgment
This material is based upon work supported by the U.S. Department
of Energy (DOE), Nuclear Energy University Program (NEUP) under
Award No. DE-NE0008403. DOE’s support is acknowledged. Any opin­
ions and findings presented are those of authors and do not necessarily
reflect the views of DOE.

References
Alzraiee, H., Zayed, T., Moselhi, O., 2015. Dynamic planning of construction activities
using hybrid simulation. Autom. ConStruct. 49, 176–192. />autcon.2014.08.011.
B N Spring, S. (Ed.), 2009. Nuclear Outage Operational Excellence 08/01/2009.
Bolton, M.L., 2015. Model checking human-human communication protocols using task
models and miscommunication generation. J. Aero. Inf. Syst. 12, 476–489. https://
doi.org/10.2514/1.I010276.
Bonabeau, E., 2002. Agent-based modeling: methods and techniques for simulating
human systems. Proc. Natl. Acad. Sci. U.S.A. />pnas.082080899.
Cao, Z., Simon, T., Wei, S.-E., Sheikh, Y., 2017. Realtime multi-person 2D pose estimation
using Part Affinity fields. IEEE Conf. Comput. Vis. Pattern Recognit. />10.1109/CVPR.2017.143.
Chen, S.M., Griffis, F.H., Chen, P.H., Chang, L.M., 2012. Simulation, and analytical
techniques for construction resource planning and scheduling. Autom. ConStruct.
/>Cheng, T., Teizer, J., Migliaccio, G.C., Gatti, U.C., 2013. Automated task-level activity
analysis through fusion of real-time location sensors and worker’s thoracic posture
data. Autom. ConStruct. 29, 24–39. />Fang, W., Ding, L., Luo, H., Love, P.E.D., 2018. Falls from heights: a computer visionbased approach for safety harness detection. Autom. ConStruct. 91, 53–61. https://
doi.org/10.1016/j.autcon.2018.02.018.
St Germain, S.W., Farris, R.K., Whaley, A.M., Medema, H.D., Gertman, D.I., 2014.
Guidelines for implementation of an advanced outage control center to improve
outage coordination, problem resolution, and outage risk management (No. INL/
EXT-14-33182). Idaho National Lab.(INL), Idaho Falls, ID (United States).
Ghanem, A.G., AbdelRazig, Y.A., 2006. A framework for real-time construction project
progress tracking. Earth Space 1–8.
Girardeau-Montaut, D., Roux, M., Marc, R., Thibault, G., 2005. Change detection on
points cloud data acquired with a ground laser scanner. Int. Arch. Photogram. Rem.
Sens. Spatial Inf. Sci. 36, W19, 10.1.1.221.8313.
Han, S., Love, P., Pe~
na-Mora, F., 2013. A system dynamics model for assessing the
impacts of design errors in construction projects. Math. Comput. Model. 57,
2044–2053. />Hinze, J.W., Teizer, J., 2011. Visibility-related fatalities related to construction

equipment. Saf. Sci. 49, 709–718. />Jang, I., Ryum, A., Ali, M., Al, S., Jun, S., Gook, H., Hyun, P., 2013. An empirical study on
the basic human error probabilities for NPP advanced main control room operation
using soft control. Nucl. Eng. Des. 257, 79–87. />nucengdes.2013.01.003.
Lloyd, R.L., 2003. A survey of crane operating experience at US nuclear power plants
from 1968 through 2002. Division of Systems Analysis and Regulatory Effectiveness,
Office of Nuclear Regulatory Research, US Nuclear Regulatory Commission.
Luo, W., Xing, J., Milan, A., Zhang, X., Liu, W., Zhao, X., Kim, T.-K., 2014. Multiple
Object Tracking: A Literature Review.
Luo, X., Li, H., Wang, H., Wu, Z., Dai, F., Cao, D., 2019. Vision-based detection, and
visualization of dynamic workspaces. Autom. ConStruct. 104, 1–13. />10.1016/j.autcon.2019.04.001.
Mohamed, E., Jafari, P., Siu, M.-F., Francis, AbouRizk, S., 2017. Data-driven simulation
model for planning roadway operation and maintenance projects. In: Proc. 2017
Winter Simul. Conf., pp. 3323–3334.
Pan, D., Bolton, M.L., 2015. Properties for formally assessing the performance level of
human-human collaborative procedures with miscommunications and erroneous
human behavior. Int. J. Ind. Ergon. 1–14. />ergon.2016.04.001.
Shrestha, K., Shrestha, P.P., Bajracharya, D., Yfantis, E.A., 2015. Hard-Hat Detection for
Construction Safety Visualization, 2015. />Sun, Z., Zhang, C., Tang, P., 2018a. Simulation-Based Optimization of Communication
Protocols for Reducing Delays during Nuclear Power Plant Outages, pp. 455–464.
/>Sun, Z., Zhang, C., Tang, P., 2018b. Simulation-based optimization of communication
protocols for reducing delays during nuclear power plant outages. In: Constr. Res.
Congr. 2018 Infrastruct. Facil. Manag. - Sel. Pap. From Constr. Res. Congr. 2018.
/>
Credit author statement
Zhe Sun: Conceptualization, Methodology, Writing - original draft.
Cheng Zhang: Writing - review & editing. Jiawei Chen: Writing - review
& editing, Computer Vision algorithm development and testing. Pingbo
Tang: Conceptualization, Methodology, Writing - original draft, Writing
- review & editing. Alper Yilmaz: Computer Vision algorithm develop­
ment and testing, Writing - review & editing.

10


Z. Sun et al.

Progress in Nuclear Energy 127 (2020) 103448

Sun, Z., Zhang, C., Tang, P., 2020. Modeling and simulating the impact of forgetting and
communication errors on delays in civil infrastructure shutdowns. Front. Eng.
Manag. 1–13.
Tang, P., Zhang, C., Yilmaz, A., Cooke, N., 2016. Automatic imagery data analysis for
diagnosing human factors in the outage of a nuclear plant. Lect. Notes Comput. Sci. Digit. Hum. Model. Appl. Heal. Safety, Ergon. Risk Manag. 9745.
Wang, Y., Liu, Y., 2018. A novel Bayesian Entropy Network for probabilistic damage
detection and classification. In: In 2018 AIAA Non-Deterministic Approaches
Conference (p. 1407).
Wang, W.C., Weng, S.W., Wang, S.H., Chen, C.Y., 2014. Integrating building information
models with construction process simulations for project scheduling support. Autom.
ConStruct. />Wang, Y., Liu, Y., Sun, Z., Tang, P., 2018. A Bayesian-entropy network for information
fusion and reliability assessment of national airspace systems. Proc. Annu. Conf.
Progn. Heal. Manag. Soc. PHM.

Yoo, W.S., Yang, J., Kang, S., Lee, S., 2016. Development of a computerized risk
management system for international NPP EPC projects. KSCE J. Civ. Eng. 21, 1–16.
/>Zhang, C., Tang, P., Cooke, N., Buchanan, V., Yilmaz, A., Germain, S., Boring, R., AkcaHobbins, S., Gupta, A., 2017. Human-centered automation for resilient nuclear
power plant outage control. Autom. ConStruct. 82, 179–192. />10.1016/j.autcon.2017.05.001.
Zhang, C., Sun, Z., Tang, P., Germain, S.W. St, Boring, R., 2018a. Simulation-based
optimization of resilient communication protocol for nuclear power plant outages.
Adv. Intell. Syst. Comput. 20–29. />Zhang, B., Zhu, Z., Hammad, A., Aly, W., 2018b. Automatic matching of construction
onsite resources under camera views. Autom. ConStruct. 91, 206–215. https://doi.

org/10.1016/j.autcon.2018.03.011.
Zhang, P., Li, N., Jiang, Z., Fang, D., Anumba, C.J., 2019. An agent-based modeling
approach for understanding the effect of worker-management interactions on
construction workers’ safety-related behaviors. Autom. ConStruct. />10.1016/j.autcon.2018.10.015.

11