VIETNAM NATIONAL UNIVERSITY, HANOI
-----------------------------------------VNU UNIVERSITY OF ENGINEERING AND TECHNOLOGY
Phung Manh Duong
STUDY ON SUPERVISION AND CONTROL OF ROBOT OVER
COMPUTER NETWORK
DOCTORAL THESIS IN ELECTRONICS AND
TELECOMMUNICATIONS TECHNOLOGY
Hanoi - 2015
VIETNAM NATIONAL UNIVERSITY, HANOI
-----------------------------------------VNU UNIVERSITY OF ENGINEERING AND TECHNOLOGY
Phung Manh Duong
STUDY ON SUPERVISION AND CONTROL OF ROBOT OVER
COMPUTER NETWORK
Major: Electronic Engineering
Code: 62 52 70 01
DOCTORAL THESIS IN ELECTRONICS AND
TELECOMMUNICATIONS TECHNOLOGY
SUPERVISOR:
Assoc. Prof. Dr Tran Quang Vinh
Hanoi - 2015
ĐẠI HỌC QUỐC GIA HÀ NỘI
-----------------------------------------TRƯỜNG ĐẠI HỌC CÔNG NGHỆ
Phùng Mạnh Dương
NGHIÊN CỨU VẤN ĐỀ GIÁM SÁT VÀ ĐIỀU KHIỂN ROBOT QUA MẠNG
MÁY TÍNH
Chuyên ngành: Kỹ thuật Điện tử
Mã số: 62 52 70 01
LUẬN ÁN TIẾN SĨ NGÀNH CÔNG NGHỆ ĐIỆN TỬ - VIỄN THÔNG
NGƯỜI HƯỚNG DẪN KHOA HỌC:
PGS.TS Trần Quang Vinh
Hà Nội - 2015
DECLARATION BY CANDIDATE
I hereby declare that this thesis is my own work and effort and that it has not been
submitted anywhere for any award. Where other sources of information have been
used, they have been acknowledged.
Author
Phung Manh Duong
i
Acknowledgements
This thesis has been completed with the contributions of many people.
First of all, I would like to express my special thanks Prof. Tran Quang Vinh, my
supervisor, for his guidance, encouragement, and support. I was truly fortunate to
have the opportunity to work with him as a PhD student.
I am greatly indebted to Prof. Kok Kiong Tan and the members of Mechatronics and
Automation Laboratory for their support and advices during my research in National University of Singapore.
I am grateful to Ms. Nguyen Thi Thanh Van and Mr. Tran Thuan Hoang for their
supports on simulation and experiment in my research.
I would like to express my appreciation to the teachers and students in VNU-UET.
When I started working in this college, I immediately noticed and valued the high
level of communication and the free flow of ideas here.
I do appreciate the National Foundation for Science and Technology Development
(NAFOSTED) for their financial support for my attendance of international conferences.
Finally, I owe special thanks to my family, who always accept and encourage my
decisions in the professional and the personal fields over the past years.
ii
Contents
List of Abbreviations ...............................................................................................
List of Tables ..........................................................................................................
List of Figures ..........................................................................................................
Chapter 1: Introduction ............................................................................................
1.1Introduction to networked robot systems ....
1.2Applications of networked robot systems ...
1.2.1Industrial
1.2.2Education
1.2.3Medical n
1.2.4Service ne
1.2.5Other netw
1.2.6Networke
1.3Related works ..............................................
1.3.1Study of N
1.3.2Study of N
1.3.3Study of N
1.4The goal of the research .............................
1.5The organization of this thesis ....................
Chapter 2: System Model .......................................................................................
2.1State-space representation of the NRS ......
2.2The communications network ...................
2.2.1Network t
2.2.2Network c
2.3The Robot ..................................................
2.3.1Hardware
2.3.2Data com
2.4Conclusion ..................................................
Chapter 3: Localization Using Optimal Filter .....................................................
iii
3.1Robot localization ...............................................
3.2Localization of NRSs ..........................................
3.3 Localization of NRSs using past-observation b
45
3.3.1 T
3.3.2 O
3.3.3 O
3.4Implementation of the PO-EKF for the differenti
3.4.1 M
3.4.2 I
3.4.3 I
3.5Simulations .........................................................
3.5.1 S
3.5.2 S
3.6Experiments ........................................................
3.6.1 E
3.6.2 E
3.7Discussion ...........................................................
3.8Conclusion ..........................................................
Chapter 4: Motion Control Using Lyapunov Stability Theory and Predictive
Filter
..............................................................................
4.1Introduction .........................................................
4.2Problem formulation ...........................................
4.3Stabilization of non-networked robot system .....
4.4Stabilization of NRS ...........................................
4.5Simulations .........................................................
4.6Experiments ........................................................
4.7Discussion ...........................................................
4.8Conclusion ..........................................................
Chapter 5: Navigation Using Behavior-based Model ..........................................
iv
5.1Introduction ................................................
5.2Behavior-based navigation for NRSs .........
5.2.1User fo
5.2.2Obstac
5.2.3Goal re
5.2.4Superv
5.3Simulations ................................................
5.4Experiments ...............................................
5.5Conclusion .................................................
Chapter 6: Conclusion ..........................................................................................
List of Publications Related to This Thesis .........................................................
References...........
v
List of Abbreviations
2D
Two-dimensional
3D
Three-dimensional
ADSL
Asymmetric digital subscriber line
ALCAP
Access link control application part
ARP
Address resolution protocol
CAN
Controller area network
CCD
Charge-coupled device
CSMA/CD
Carrier sense multiple access with collision detection
DARPA
Defense advanced research projects agency
EKF
Extended Kalman filter
FIP
Factory instrumentation protocol
GPS
Global positioning system
GUI
Graphic user interface
HTTP
Hypertext transfer protocol
ICMP
Internet control message protocol
ID
Identifier
IETF
Internet engineering task force
IGMP
Internet group management protocol
IP
Internet protocol
IRTP
Interactive real-time protocol
ISP
Internet service providers
LAN
Local area network
LAS
Link active scheduler
LEKF
Lucas-extended Kalman filter
LRF
Laser range finder
MAC
Media access control
vi
MSSR
Multi-sensor smart robot
NRS
Networked robot system
NRSs
Networked robot systems
NS2
Network simulator ns-2
PC
Personal computer
PID
Proportional-integral-derivative
PO-EKF
Past observation-based extended Kalman filter
PO-KF
Past observation-based Kalman filter
PROFIBUS
Process field bus
QoS
Quality of service
RARP
Reverse address resolution protocol
RMSEs
Root mean square errors
RTCP
RTP control protocol
RTP
Real-time transport protocol
RTT
Round-trip time
STCP
Stream control transmission protocol
SNRP
Simple network robot protocol
TCP
Transmission control protocol
TEAR
TCP emulation at receivers
TICP
Transparent inter-process communication
UDP
User datagram protocol
VINT
Virtual internetwork test
VPN
Virtual private network
vii
List of Tables
Table 2.1: Summary of data classification and transport protocols........................40
Table 3.1: Normalized computational burden of filters........................................... 64
Table 3.2: Normalized computational burden of filters........................................... 66
Table 3.3: Network parameters during experiments................................................ 71
Table 5.1: Fuzzy rules of the user following behaviors.......................................... 102
Table 5.2: Fuzzy rules of the obstacle avoidance behavior.................................... 105
Table 5.3: Rules for determination of behaviors.................................................... 106
viii
List of Figures
Figure 1.1: The coal mining networked robot system[36].........................................3
Figure 1.2: A telerobotic laboratory platform[82]..................................................... 4
Figure 1.3: The telesurgical system [14]................................................................... 5
Figure 1.4: Service networked robots in urban areas [62]........................................ 6
Figure 1.5: Schematic structure of the thesis........................................................... 15
Figure 2.1: The robot’s pose and parameters.......................................................... 17
Figure 2.2: Model of the NRS.................................................................................. 20
Figure 2.3: Timing of signals in the NRS................................................................. 21
Figure 2.4: Clock offset estimation.......................................................................... 27
Figure 2.5: Overview of the developed NRS............................................................ 29
Figure 2.6: Hardware configuration of the robot.................................................... 30
Figure 2.7: Operation of the compass sensor........................................................... 31
Figure 2.8: Operation of the omni-directional camera............................................ 32
Figure 2.9: Operation of the laser range finder [68]............................................... 32
Figure 2.10: Communications scheme between sensors and actuators...................33
Figure 2.11: Graphic user interface of the control software written in Microsoft
Visual C++.............................................................................................................. 34
Figure 2.12: Network topology in simulations......................................................... 37
Figure 2.13: Characteristics of RTP, TCP and UDP in case of no network
congestion............................................................................................................... 37
Figure 2.14: Characteristics of RTP, TCP and UDP in case of network congestion
38
Figure 2.15: Data communications in the NRS using multi-protocol model............40
Figure 2.16: Network state during the experiment................................................... 41
Figure 2.17: Images transmitted by two different transport protocols.....................41
Figure 3.1: An ordinary theory trajectory of the robot in the motion plane.............62
ix
Figure 3.2: An extreme theory trajectory of the robot in the motion plane..............62
Figure 3.3: The theory, true and observation trajectories in X direction.................62
Figure 3.4: The EKF, LEKF, PO-EKF, and true trajectories in the motion plane....63
Figure 3.5: RMSE of the EKF, LEKF, PO-EKF estimation and the true trajectories
in X direction........................................................................................................... 63
Figure 3.6: RMSE between the EKF, LEKF, PO-EKF and the true trajectories in Y
direction.................................................................................................................. 64
Figure 3.7: RMSE between the EKF, LEKF, PO-EKF and the true trajectories in
orientation............................................................................................................... 64
Figure 3.8: The EKF, LEKF, PO-EKF, and true trajectories of the robot in the
motion plane............................................................................................................ 65
Figure 3.9: RMSE between the EKF, LEKF, PO-EKF and the true trajectories in X
direction.................................................................................................................. 65
Figure 3.10: RMSE between the EKF, LEKF, PO-EKF and the true trajectories in Y
orientation............................................................................................................... 65
Figure 3.11: RMSE between the EKF, LEKF, PO-EKF and the true trajectories in
orientation............................................................................................................... 65
Figure 3.12: Experimental configuration with local Internet service providers......68
Figure 3.13: Experimental configuration with VPN connections to the servers
located at the United State...................................................................................... 68
Figure 3.14: Comparison between the EKF, LEKF and PO-EKF in local
configuration........................................................................................................... 70
Figure 3.15: Comparison between the EKF, LEKF and PO-EKF in VPN
configuration........................................................................................................... 72
Figure 3.16: Estimate by the PO-EKF with uniform distribution noise...................73
Figure 3.17: Estimate by the PO-EKF with non-zero mean noise...........................74
Figure 4.1: The goal of the controller...................................................................... 77
Figure 4.2: The robot poses and navigation variables............................................. 80
Figure 4.3: NRS with the presence of a state estimator............................................ 84
x
Figure 4.4: Stable control of the non-networked robot system.................................87
Figure 4.5: Velocities during the stable control of the non-networked robot system
88
Figure 4.6: Stable control of the NRS without using the predictive filter.................88
Figure 4.7: Velocities of the robot during the stable control without using the
predictive filter........................................................................................................ 89
Figure 4.8: Stable control of the NRS with the use of the predictive filter...............89
Figure 4.9: Velocities of the robot during the stable control with the use of the
predictive filter........................................................................................................ 90
Figure 4.10: Setup of the experimental environment............................................... 91
Figure 4.11: Stable control of NRS with the use of the predictive filter...................91
Figure 4.12: Velocities of the robot during the stable control with the use of the
predictive filter........................................................................................................ 92
Figure 4.13: Trajectories of the robot during the stable control with different control
parameters.............................................................................................................. 93
Figure 4.14: RMSE of the predictive filter with the time-out value set to 5 seconds
94
Figure 5.1: General navigation scheme for mobile robot systems...........................97
Figure 5.2: An architecture of the behavior-based navigation.................................98
Figure 5.3: The architecture of our behavior-based navigation system...................99
Figure 5.4: Membership function of the network delay input.................................101
Figure 5.5: Membership function of the command state input............................... 101
Figure 5.6: Membership function of the adjusting ratio output.............................101
Figure 5.7: The membership functions of input variables Dright, Dforward, and Dleft. 104
Figure 5.8: The membership functions of output variables ωL and ωR..................104
Figure 5.9: Configuration of sonar sensors in simulations....................................108
Figure 5.10: Result of the behavior-based navigation........................................... 109
Figure 5.11: Network state during navigation....................................................... 110
Figure 5.12: Angular velocities of the left and right wheels during navigation.....110
xi
Figure 5.13: Experimental setup with the start position, target position,...............111
Figure 5.14: Result of the behavior-based navigation........................................... 112
Figure 5.15: Network state during the user following process............................... 113
Figure 5.16: Angular velocities of the left and right wheels during the userfollowing process................................................................................................... 113
Figure 5.17: Distance to objects measured by the front sonar during navigation. . 114
Figure 5.18: Angular velocities of the left and right wheels during navigation.....115
Figure 5.19: A sequence of images showing the motion of robot during the
navigation.............................................................................................................. 115
xii
Chapter 1
Introduction
1.1 Introduction to networked robot systems
Appeared in 1994, the first networked robot system (NRS) had received over 2.5
million accesses for seven months. The system namely Mercury project permitted
Internet users to operate a manipulator to excavate artifacts buried in a nuclear contaminated region to look for evidence of ancient water flows [27]. In the next seven
years, over forty such systems had been developed allowing users to remotely visit
museums, tend gardens, navigate undersea, float in blimps, and handle protein crystals [25, 26, 48, 63, 76]. This new trend attracted the IEEE Society of Robotics and
Automation and the name Networked Robots was created in 2004 with the following definition [62]:
“A networked robot is a robotic device connected to a communications network such as the Internet or LAN. The network could be wired or wireless,
and based on any of a variety of protocols such as TCP, UDP, or 802.11.”
To reflect a broader set of problems and applications, the definition was extended to include two subclasses of networked robot systems (NRSs): the autonomous
systems and the teleoperated (manual) systems.
•
Autonomous NRSs: In autonomous systems, the controller generates control
signals to operate the robot. The robot (including actuators and sensors) communicates with the controller via the network. The network extends the effective
operating range of the robot, allowing multiple robots to communicate with each
other over long distances to coordinate their activity.
1
•
Teleoperated NRSs: In teleoperated systems, human operators send commands
to the robot. The robot executes the commands and returns feedback to the operators via the network. Input devices and user graphic interfaces are used to support the operator. They act as a passive controller to translate operator’s commands to control signals.
NRSs call for the integration of several fields: robotics, perception (sensor sys-
tems), ubiquitous computing, artificial intelligence, and network communications.
The topics of NRSs transcend ‘‘conventional’’ robotic problems such as localization
and motion control to the type of distributed systems over heterogeneous communications networks. The challenging issues include the guarantee of system reliability
and performance under the influence of time-varying transmission delay, message
loss, out-of-order delivery, and non-guaranteed transmission bandwidth. Many new
applications are now being developed ranging from automation to exploration.
1.2 Applications of networked robot systems
Applications of NRSs can be classified into five groups including industrial robots,
educational robots, medical robots, service robots, and other various robots.
1.2.1 Industrial networked robots
Networked robots are applicable in industry. In [36], a teleoperated robot system for
coal mining was developed. The robot is equipped with two cameras, two laser
scanners, and some other sensors to extract the environmental parameters [figure
1.1(a)]. These parameters are then transmitted via the network to the human opera-tor.
The human operator, in a safe location, controls the robot to complete tasks such as
shoveling and breaking [figure 1.1(b)]. Advantages of the system include the iso-lation
of human labor from the harsh environments and the endurance of the robot in hard
work. Difficulties relate to the problem of sensor fusion and localization.
2
(a)
(b)
Figure 1.1: The coal mining networked robot system[36]: (a) The robot with actuators and sensors; (b) The operator with input devices and user graphic interfaces.
1.2.2 Educational networked robots
Networked robots enable the development of telelaboratory. In [82], a telelaboratory
was developed for the training of robotics. Via the Internet and world wide web,
students are able to connect to the laboratory to operate a real manipulator (figure
1.2). The students also have access to other facilities such as the teach pendant and
control computer to fulfill the practical lesson. Advantages of telelaboratory include
the sharing of expensive systems among educational units, the ease in evaluation
process, and the flexibility in practice.
3
(a)
(b)
Figure 1.2: A telerobotic laboratory platform[82]: (a) The telelaboratory with robotic devices; (b) The web-based interface to interact with the telelaboratory.
1.2.3 Medical networked robots
Medical networked robots bring users chance to be diagnosed and treated remotely
by experienced doctors. In [53], Masuda developed a telerobot for echography. Remote operators use two joysticks to control: one moves the robot; the other adjusts
the ultrasonic probe’s angle. At each stop position, the robot captures an ultrasonic
image, compresses it, and sends it over the Internet to the examiner. In [14], a telesurgical system was developed. Two slave manipulators are placed at patient’s site.
They are controlled by two joysticks located at operator’s site. The interaction between the operator and patient is conducted with the support of tactile and force
feedback sensors (figure 1.3).
4
(a)
(b)
Figure 1.3: The telesurgical system [14]: (a) Setup of slave manipulators around
the operating table; (b) An experiment with the surgeon and patient.
1.2.4 Service networked robots
Networked robots can supply services at home, office, and public areas. In [62],
Sanfeliu et al developed new ways of cooperation between networked robots and
human beings. Robots have tasks of guidance, assistance, transportation, and surveillance in urban areas. Their architecture integrates cooperating urban robots, intelligent sensors (video cameras, acoustic sensors, etc.), intelligent devices (PDA,
mobile telephones, etc.) and communications (figure 1.4).
5
(a)
(b)
(c)
Figure 1.4: Service networked robots in urban areas [62]: (a) A service robot; (b)
The robot interacts with human to supply information; (c) Robots cooperate in a
guidance task.
1.2.5 Other networked robots
Networked robots can be applied in other various applications. In [50], Luo introduced a robot for remote surveillance. The robot has five sensory subsystems for the
fire detection, intruder detection, motor diagnosis, obstacle detection, and environment detection. In [4], the robot named GARBI was developed for the tasks of observation and manipulation in shallow water environments. It has a 3D vision system to support the remote control. In [65], a prototype rover was designed as a testand-validation vehicle for a Mars technology program. Other applications of networked robots include pet robot, tour-guide robot, intelligent wheelchair, etc. [49,
51, 81].
1.2.6 Networked robots in Vietnam
In Vietnam, several works on NRSs have been proposed recently. In [1], Cong
Thanh developed a telemanipulator system. The system consists of two manipulators (master and slave) communicated through a local area network. During the operation, human operators act on the master manipulator. At remote site, the slave
6
manipulator mimics the master based on signals received via the network. In [78], a
telehealthcare system was developed. In the system, cameras are arranged at patient’s house to record and transmit the vision data to a remote processing central. If
abnormal behaviors are detected, the system will send messages to doctors. In [37],
a NRS for searching high radioactivity regions in a disaster area was introduced.
The system uses a partial swarm optimization algorithm to enable the cooperation
between the robots to fulfill the task.
Though the proposed works are mainly preliminary results, they prove the applicability of NRSs in Vietnam. In fact, NRSs would be also applicable to emerging
fields in Vietnam such as traffic control, ubiquitous healthcare, and underwater exploration.
1.3 Related works
Adapting to this emerge field of robotics, there have been a number of projects dealing with problems involved in networked robots. This section reviews studies on the
localization, stabilization control, and navigation which are topics covered in this
thesis.
1.3.1 Study of NRSs on localization
Localization is fundamental to the functioning operation of robotic systems. In order to complete a given task, the robot needs to know its own state in the environment. In NRSs, there are two approaches in localization including advance interface
techniques and optimal filters.
A. Localization using advance interface techniques
Advance interface techniques locate the robot by reconstructing the operating environment. In [86], a virtual environment which is proportional to the real dimension
7
of the working environment is constructed at client site. Before commands are sent
to the server program, they are processed in the virtual environment to predict the
upcoming position of the robot. Based on it, the system can tolerate a certain
amount of time delays as well as allow users to experience the interaction with the
robot as in the real environment. In [74], the model-based virtual reality technique
was used to construct the polygonal model of the complete 3D world. The operator
controls the robot by issuing commands to the model. An extension of this technique called ecological interface paradigm was introduced in [55]. It uses multiple
sets of information to create a 3-D virtual environment that is augmented with the
real video information.
On another note, Hou and Su put four cameras in the robot field as external visual sensors to formulate four adjacent grids of vision without the dead zone [34]. A
recognition algorithm is implemented to recognize and give out the relative location
of the robot, target, landmark and obstacle symbols. In [31], the robot pose is estimated at the local site using odometry, sonar, and compass sensors. This information is then transmitted to the remote site as the pose of the robot without considering the its change during the communications. In [41], a map-based localization
method was proposed. The absolute position of the robot is determined by comparing a reference map of the local site with the one built by the GPS map building
technique at the remote site.
The limitation of advance interface techniques is that they avoid dealing with
the key issues of communications networks, the network delay, message loss, and
out-of-order delivery. The data transmission between the remote controller and the
actuator is often treated as a given condition and rarely touched. From the viewpoint
of estimation theory, significant delay and loss are equivalent to the inaccuracy in
state estimation and control that can easily downgrade the system performance.
8
B. Localization using optimal filters
The optimal filtering method incorporates parameters of the network into the statespace representation of the NRS and then uses the estimation theory to locate the
robot. In [71], Sinopoli et al. introduced an augmented Kalman filter for state estimation with intermittent observations. The arrival of observations is modeled as a
binary random variable λt. Its variance is defined as the variance of output noise if
2
λt is 1, and as σ I otherwise, where σ is a real constant and I is the identity matrix.
The Kalman filter is then reformulated using a “dummy” observation (if a real
observa-tion does not arrive) and takes the limit as → ∞. This approach was
expanded in
[64] to simultaneously deal with random delay and packet drop. In this work, the
arrival of observations is also defined as a binary random variable, but an infinite
buffer is introduced to store and rearrange delayed measurements. The estimation is
then computed from the initial to the latest available measurements at each step.
Due to the iteration, this filter is very computationally demanding. A finite buffer
can be used to lighten the problem, but it requires the time delay to be bounded.
Recently, Moayedi et al. [54] have introduced an adaptive Kalman filter for
networked control systems with mixed uncertainties of random delays, packet dropouts, and missing measurements. This work uses a set of matrices to represent the
mixed uncertainties and add them into the system model. The filter gain is then derived by solving a set of recursive discrete-time Riccati equations. On a similar
note, Ma and Sun [52] proposed an optimal filter to deal with random sensor delays,
packet dropouts, and uncertain observations. Three Bernoulli distributed random
variables with known distributions are introduced to generate a unified model of the
mixed uncertainties. Based on it, a Riccati equation and a Lyapunov equation are
solved to retrieve the estimator. Both filters have advantages in dealing with network induced problems. They are however not optimal for the out-of-order delivery.
In addition, those filters are designed for linear systems. Further modifications are
required to extend them to nonlinear systems such as NRSs.
9
1.3.2 Study of NRSs on stabilization control
Similar to classic robot systems, stability is an important criteria of NRSs. In the
literature, several works have been proposed for this problem. Wargui et al. developed a stable controller for NRSs with nonholonomic constraints [88]. In this work,
control laws that stabilize the system in delay-free scenario are first derived. They
are then extended to work with the time delay by adding a multistep predictive estimator between the system output and the controller. In [15], the maximum time
delay allowed at the control input without losing the system stability is estimated.
Based on it, a single layered neural network is designed to stably control the networked robot. In [47], Luck used a time buffer which was longer than the worst delay to make the system to be time-invariant and then applied the classic control theory. In [89], Xi and Tarn proposed an event-based (non-time) control scheme to reduce the impact of time delay on the system operation.
In addition to the above main approaches, some other interesting works were also introduced. In [72], the minimum jerk motion model is employed to predict the
input from the user a time into the future. This predicted input is then used to stabilize the remote robot. In [35], multiple commands with all possible happened delay
are sent in one packet to ensure the relevant of control signal.
It is recognizable from the above approaches that they mainly focus on dealing
with the time delay. However, communications over the network is influenced by
not only the time delay but also other various problems such as the message loss,
out-of-order delivery, and limited transmission bandwidth. These problems affect
the control in such a complicated way that may cause the overall system to be unstable.
10