SENSING, INTELLIGENCE, MOTION pot

TLFeBOOK
TLFeBOOK
SENSING,
INTELLIGENCE,
MOTION

SENSING,
INTELLIGENCE,
MOTION
HOW ROBOTS AND HUMANS MOVE
IN AN UNSTRUCTURED WORLD
Vladimir J. Lumelsky
A JOHN WILEY & SONS, INC., PUBLICATION
Copyright 2006 by John Wiley & Sons, Inc. All rights reserved.
Published by John Wiley & Sons, Inc., Hoboken, New Jersey.
Published simultaneously in Canada.
No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any
form or by any means, electronic, mechanical, photocopying, recording, scanning, or otherwise,
except as permitted under Section 107 or 108 of the 1976 United States Copyright Act, without
either the prior written permission of the Publisher, or authorization through payment of the
appropriate per-copy fee to the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers,
to the Publisher for permission should be addressed to the Permissions Department, John Wiley &
Sons, Inc., 111 River Street, Hoboken, NJ 07030, (201) 748-6011, fax (201) 748-6008, or online at
/>Limit of Liability/Disclaimer of Warranty: While the publisher and author have used their best
efforts in preparing this book, they make no representations or warranties with respect to the
accuracy or completeness of the contents of this book and specifically disclaim any implied
warranties of merchantability or fitness for a particular purpose. No warranty may be created or
extended by sales representatives or written sales materials. The advice and strategies contained
herein may not be suitable for your situation. You should consult with a professional where
appropriate. Neither the publisher nor author shall be liable for any loss of profit or any other
commercial damages, including but not limited to special, incidental, consequential, or other

damages.
For general information on our other products and services or for technical support, please contact
our Customer Care Department within the United States at (800) 762-2974, outside the United
States at (317) 572-3993 or fax (317) 572-4002.
Wiley also publishes its books in a variety of electronic formats. Some content that appears in print
may not be available in electronic formats. For more information about Wiley products, visit our
web site at www.wiley.com.
Library of Congress Cataloging-in-Publication Data:
Lumelsky, Vladimir.
Sensing, intelligence motion : how robots and humans move in an unstructured world /
Vladimir L. Lumelsky.
p. cm.
“A Wiley-Interscience publication.”
Includes bibliographical references and index.
ISBN-13 978-0-471-70740-0
ISBN-10 0-471-70740-6
1. Robots—Motion. 2. Manipulators (Mechanism) I. Title.
TJ211.L85 2005
629.8

92—dc22
2005041748
Printed in the United States of America.
10987654321
MA 01923, (978) 750-8400, fax (978) 750-4470, or on the web at www.copyright.com. Requests
To Rakhil, Nadya, Michael, and Anna

CONTENTS
Preface xiii
Acknowledgments xxiii

1 Motion Planning—Introduction 1
1.1 Introduction 1
1.2 Basic Concepts 13
1.2.1 Robot? What Robot? 13
1.2.2 Space. Objects 15
1.2.3 Input Information. Sensing 15
1.2.4 Degrees of Freedom. Coordinate Systems 18
1.2.5 Motion Control 20
1.2.6 Robot Programming 21
1.2.7 Motion Planning 24
2 A Quick Sketch of Major Issues in Robotics 27
2.1 Kinematics 29
2.2 Statics 33
2.3 Dynamics 33
2.4 Feedback Control 37
2.5 Compliant Motion 40
2.6 Trajectory Modification 44
2.7 Collision Avoidance 48
2.8 Motion Planning with Complete Information 51
2.9 Motion Planning with Incomplete Information 55
2.9.1 The Beginnings 59
2.9.2 Maze-to-Graph Transition 66
vii
viii CONTENTS
2.9.3 Sensor-Based Motion Planning 66
2.10 Exercises 71
3 Motion Planning for a Mobile Robot 73
3.1 The Model 78
3.2 Universal Lower Bound for the Path Planning Problem 80
3.3 Basic Algorithms 84

3.3.1 First Basic Algorithm: Bug1 84
3.3.2 Second Basic Algorithm: Bug2 90
3.4 Combining Good Features of Basic Algorithms 100
3.5 Going After Tighter Bounds 103
3.6 Vision and Motion Planning 104
3.6.1 The Model 106
3.6.2 Algorithm VisBug-21 110
3.6.3 Algorithm VisBug-22 120
3.7 From a Point Robot to a Physical Robot 123
3.8 Other Approaches 124
3.9 Which Algorithm to Choose? 127
3.10 Discussion 130
3.11 Exercises 135
4 Accounting for Body Dynamics: The Jogger’s Problem 139
4.1 Problem Statement 139
4.2 Maximum Turn Strategy 144
4.2.1 The Model 144
4.2.2 Sketching the Approach 146
4.2.3 Velocity Constraints. Minimum Time Braking 148
4.2.4 Optimal Straight-Line Motion 149
4.2.5 Dynamics and Collision Avoidance 152
4.2.6 The Algorithm 154
4.2.7 Examples 157
4.3 Minimum Time Strategy 159
4.3.1 The Model 160
4.3.2 Sketching the Approach 161
4.3.3 Dynamics and Collision Avoidance 164
CONTENTS ix
4.3.4 Canonical Solution 166
4.3.5 Near-Canonical Solution 169

4.3.6 The Algorithm 170
4.3.7 Convergence. Computational Complexity 172
4.3.8 Examples 175
5 Motion Planning for Two-Dimensional Arm Manipulators 177
5.1 Introduction 177
5.1.1 Model and Definitions 183
5.2 Planar Revolute–Revolute (RR) Arm 187
5.2.1 Analysis 189
5.2.2 Algorithm 210
5.2.3 Step Planning 211
5.2.4 Example 212
5.2.5 Motion Planning with Vision and Proximity Sensing 213
5.2.6 Concluding Remarks 218
5.3 Distinct Kinematic Configurations of RR Arm 220
5.4 Prismatic–Prismatic (PP, or Cartesian) Arm 226
5.5 Revolute–Prismatic (RP) Arm with Parallel Links 229
5.6 Revolute–Prismatic (RP) Arm with Perpendicular Links 234
5.7 Prismatic–Revolute (PR) Arm 234
5.8 Topology of Arm’s Free Configuration Space 245
5.8.1 Workspace; Configuration Space 249
5.8.2 Interaction Between the Robot and Obstacles 252
5.8.3 Uniform Local Connectedness 255
5.8.4 The General Case of 2-DOF Arm Manipulators 256
5.9 Appendix 258
5.10 Exercises 267
6 Motion Planning for Three-Dimensional Arm Manipulators 271
6.1 Introduction 271
6.2 The Case of the PPP (Cartesian) Arm 276
6.2.1 Model, Definitions, and Terminology 276
6.2.2 The Approach 283

6.2.3 Topology of W -Obstacles and C-Obstacles 285
6.2.4 Connectivity of C 295
x CONTENTS
6.2.5 Algorithm 301
6.2.6 Examples 304
6.3 Three-Link XXP Arm Manipulators 305
6.3.1 Robot Arm Representation Spaces 307
6.3.2 Monotonicity of Joint Space 312
6.3.3 Connectivity of J
f
313
6.3.4 Retraction of J
f
317
6.3.5 Configuration Space and Its Retract 318
6.3.6 Connectivity Graph 321
6.3.7 Lifting 2D Algorithms into 3D 325
6.3.8 Step Planning 326
6.3.9 Discussion 327
6.4 Other XXX Arms 327
7 Human Performance in Motion Planning 329
7.1 Introduction 329
7.2 Preliminary Observations 332
7.2.1 Moving in a Maze 332
7.2.2 Moving an Arm Manipulator 339
7.2.3 Conclusions and Plan for Experiment Design 345
7.3 Experiment Design 349
7.3.1 The Setup 349
7.3.2 Test Protocol 352
7.4 Results—Experiment One 353

7.4.1 Principal Components Analysis 355
7.4.2 Nonparametric Statistics 358
7.4.3 Univariate Analysis of Variance 362
7.4.4 Two-Way Analysis of Variance 365
7.4.5 Implementation: Two-Way Analysis for Path Length 367
7.4.6 Implementation: Two-Way Analysis for Completion
Time 369
7.5 Results—Experiment Two 371
7.5.1 The Technique 372
7.5.2 Implementation Scheme 375
7.5.3 Results and Interpretation 377
7.6 Discussion 381
CONTENTS xi
8 Sensitive Skin—Designing an All-Sensitive Robot
Arm Manipulator 389
8.1 Introduction 389
8.2 Salient Characteristics of a Sensitive Skin 392
8.3 Skin Design 403
8.4 Examples 407
9 Suggested Course Projects 417
References 421
Index 429

PREFACE
We humans are good at moving around in this world of ours. If we are serious
about the ubiquity of robots’ help to humankind, we must pass this skill to our
robots. It also turns out that in some tasks, robots can find their way better than
humans. This suggests that it is time for humans and robots to join forces.
Imagine you arrive at a party. You are a bit late. The big room is teeming with
voices and movement. People talk, drink, dance, walk. As you look around, you

notice a friend waving to you from the opposite side of the room. You fill two
glasses with wine, glance quickly across the room, and start on your journey. You
maneuver between people, bend your body this way and that way to avoid colli-
sion or when shoved from the side, you raise your hands and squeeze your shoul-
ders, you step over objects on the floor. A scientifically minded observer would
say that you react to minute disruptions on your path while also keeping in mind
your global goal; that you probably make dozens of decisions per second, and a
great many sensors are likely involved in this process; that you react not only to
what you see, but also to what you sense at your sides, your back, your feet. In
a minute’s time you happily greet your friend and hand him a glass of wine.
You may be surprised to hear that in your trip across the room you planned
and executed a complex motion planning strategy whose emulation in technology
is a yet unachieved dream of scientists and engineers. Providing a robot with a
seemingly modest skill that you just demonstrated, an ability to move safely
among surrounding objects using incomplete sensing information about them
would be a breakthrough in science and technology whose consequences for
society is hard to overestimate. This would be the beginning of a new era,
with a great number of machines of unimaginable variety moving quietly and
productively in the world around us.
The main reason that we desire such technology is not, of course, the conve-
nience of a wine-serving automatic maid. A machine’s ability to safely operate
in a reasonably arbitrary environment will lead to our automating a wide span of
tasks that have eluded automation so far—from the delivery of drugs and food
to patients in hospitals and nursing homes, to a robot “nurse” in the homes of
elderly people, and to such indispensable tasks as cleaning chemical and nuclear
waste sites, demining of old and new mine fields, planetary exploration, repair of
faraway space satellites, and a great number of other tasks in agriculture, under-
sea, deep space, and so on. Equipped with this skill, the recent Mars rovers Spirit
and Opportunity would have accomplished in hours what took them weeks.
xiii

xiv PREFACE
We do not have such automation today. Today, humans are not even allowed to
share space with serious robots, though a good number of the tasks above would
require this. The only reason for this constraint is that today’s robot bodies are
too insensitive, too oblivious to their surroundings, and hence too dangerous to
themselves and to objects and people around them.
Looking ahead to the near future, however, there are at least three good reasons
for optimism. One is social: The problem will not go away and so the pressure
on scientists and engineers will stay strong. The need for machines capable of
working in our midst or far away with little or no supervision will only grow with
time. The value of human life and the increasing costs of human labor combined
with ever riskier undertakings in space, undersea, and in rough places on Earth
will continue the push for more automation. A very good example of this trend is
the recent unique “attempt for on attempt” for a robot mission to save the ailing
Hubble Telescope.
One may say that having a painful problem is not enough to find a solution.
True, but then there are the other two reasons. The second reason for optimism is
the successes of robot systems in recent years. Almost 1,000,000 highly reliable
industrial robots are doing useful, sometimes quite complex, work worldwide.
True, almost none of these robots can operate outside of their highly specialized
man-made environment, and those few that do are too simplistic to be taken
seriously. Hence the third reason for optimism: Research laboratories around the
world report more and more sophistication in robot systems operating outside the
“sanitized” factory environment. Robots have been shown to be as good as or bet-
ter than humans in some tasks that require spatial reasoning and motion planning.
Systems have been demonstrated where synergistic human–robot teams operate
better, even smarter, than each of them separately. This trend is bound to continue.
It is the ability to plan its own motion that makes a robot qualitatively different
from other machines. After all, the mechanical parts, electronics, computers,
some functional abilities, and sophistication that robots possess are present in

many other digitally controlled machines. Thus the half-humorous debates of the
1960s and 1970s when designers of digitally controlled factory machinery were
accusing specialists in robotics of inflating the prestige of their field by calling
their machines robots—aren’t these just slightly modified digitally controlled
machines? There is truth to it. Now we are approaching a time when the field
of robotics will be able to say that it is the ability to plan its own motion that
makes a robot a robot.
Doesn’t such technology already exist? Haven’t we read about robots that paint
and weld and do assembly in automotive and computer manufacturing factories?
For factories, yes, but for tasks outside the factory floor—hospitals and outer
space and mine fields—no, not really, except perhaps in a few simplistic cases.
What is the difference?
For you and me, the success of, say, returning a bottle to the refrigerator
depends little on whether at this very instant the arrangement of objects in the
refrigerator differs from what it was half hour ago when the bottle was taken out.
This is not so for today’s robots.
PREFACE xv
If the required motion is to be repeated over and over again and if all the
objects in the robot workspace can be described precisely—as they are, for
example, on the car assembly line or in an automatic painting booth—using
robots to automate the task presents no principal difficulties today. Designing
the required trajectories for the tool in the robot hand is a purely geometric
problem, fully solvable by computer. (Depending on the task specifics, it may
of course require an unrealistically large amount of computation time, but this
is another matter.) Once the car model changes next year, the new data are fed
into the computer, and the required motion is recalculated. This is an example
of a structured task, and it takes place in a structured environment.Theword
“structured” is roughly equivalent to “well-organized,” “known precisely,” “man-
made.” Objects in a structured environment can be safely assumed fully known
in space and time.

As a rule, a structured environment is designed, carefully and often at great
cost, by highly qualified professionals. From the standpoint of motion planning,
the input information that the robot needs in order to generate the desired motion
is available before the motion starts. What is needed is appropriate algorithms
for transforming this information into proper motion trajectories. Today there are
plenty of such algorithms. This setup represents the Intelligence–Motion planning
paradigm.
This algorithmic paradigm was formulated right at the beginning of robotics
as a field of science and technology, around the mid-1960s. Today the Intelli-
gence–Motion paradigm boasts a large literature, appearing under such names
as motion planning with complete information,ormodel-based motion planning,
or the Piano Mover’s model. The symbolism behind the latter term is that when
movers set out to move a piano, they can first sit down and figure out the whole
sequence of moves and turns and raisings and lowerings, before they start the
actual motion. After all, the physical setting that encompasses this information is
right there before them. (Except, one might comment, “Who in this world would
ever do it this way?” More likely the movers just say, “Let’s do it!”, and they
discuss every move as they get to it—thereby losing an opportunity to contribute
to a great theory.)
On the theoretical level, the problem of motion planning with complete infor-
mation is more or less closed: remarkably complete and enlightening studies of
the problem have provided computational complexity bounds, motion planning
algorithms, and deep insights into the problem. Which is not to say that all prob-
lems in this area are solved. Most of today’s work in this area is devoted to
special cases and to struggling with computational issues in realistic settings.
Somewhat ironically, applications where such techniques are used today relate
not so much to robotics as to other areas: computer-aided design (CAD, e.g., to
design an aircraft engine such as to allow quick removal or replacement of a
given unit), models of protein folding in biology, and a few others. The major
property of such tasks is that the required motion is designed in a database rather

than in a physical setting. Given the wealth of published work in this area, this
book reviews the Piano Mover’s paradigm only cursorily.
xvi PREFACE
The focus of this book is on unstructured tasks—tasks that unfold in an
unstructured environment, an environment that is not predesigned and has to be
taken as is. Most of the motion planning examples above (homes, outdoors, deep
space, etc.) refer to unstructured tasks. Until recently, robotics practitioners have
either ignored this area or have limited their efforts to grossly simplified tasks with
robot hands or with mobile robots. Even in the latter cases the operation is mostly
limited to a tight human teleoperation, with a minimum of robot autonomy (as in
the case of recent Mars rovers). All kinds of helpful “artificial” measures—for
example, an extremely slow operation—are taken to allow the operator to precede
commands with a careful analysis.
Automating motion planning for mobile robots will be considered in the first
sections of this text. We will also see later that teaching a robot arm manipulator
to safely move in an unstructured environment is a much taller order than the
same request for a mobile robot. This is unfortunate because a large number of
pressing applications require manipulators. Today people use a great deal more
arm manipulators than mobile robot vehicles. An arm manipulator is a device
similar to a human arm. If the task is to just move around and sense data or
take pictures, that is a job for a mobile robot. But if the task requires “doing
things”—welding, painting, putting things together or taking them apart—one
needs an arm manipulator. Interestingly, while collision avoidance is a major
bottleneck in the use of robot manipulators, there is minuscule literature on the
subject. This book attempts to fill the gap.
Objects in an unstructured robot workspace cannot be described fully—either
because of their unyielding shape, or because of lack of knowledge about them,
or because one doesn’t know which object is going to be where and when, or
because of all three. In dealing with an environment that has to be taken as is,
our robots have a good example to follow: The evolution has taught us humans

how to move around in our messy unstructured world. We want our robots to
leap-frog this process.
And then there are tasks—especially, as we will see, with motion planning
for arm manipulators—where human skills and intuition are not as enviable. In
fact, not enviable at all. Then not only do we need to enter unchartered terri-
tories and synthesize new robot motion planning strategies that are way beyond
human spatial reasoning skills, but also we must built a solid theoretical founda-
tion behind them, because human experience and heuristics cannot help ascertain
their validity.
If the input information about one’s surroundings is not available beforehand,
one cannot of course calculate the whole motion at once, or even in large pieces.
What do we humans and animals do in such cases? We compensate by real-time
sensing and sensor data processing: We look, touch, listen, smell, and continu-
ously use the sensing information to plan, execute, and replan our motion. Even
when one thinks one knows by heart how to move from point A to point B—say,
to drive from home to one’s office—the actual execution still involves a large
amount of continuous sensor-based motion planning.
PREFACE xvii
Hence the names of approaches to motion planning in an unstructured envi-
ronment that one finds in the literature are: motion planning with incomplete
information,orsensor-based motion planning. Another good name comes from
the crucial role that this paradigm assigns to sensing: Similar to the phrase Intel-
ligence–Motion for motion planning with complete information, we will use the
name Sensing–Intelligence–Motion (SIM) for motion planning with incomplete
information. The SIM approach will help open the door for robotics into automa-
tion of unstructured tasks. (Recall “Open door, Simsim!” in the Arabian tale “Ali
Baba and the Forty Thieves.”)
The described differences in how input information appears in the Piano
Mover’s and SIM paradigms affect their approach to motion planning in cru-
cial ways—so much so that attempted symbiosis of some useful features of

“structured” and “unstructured” approaches have been so far of little theoretical
interest and little practical use.
While techniques for motion planning with complete information started in
earnest in the first years of robotics, sometime in early 1960s, the work on
SIM approaches started later, in the late 1980s, and has proceeded more slowly.
The slow pace is partly due to the fact that the field of robotics in general
and the area of motion planning in particular have been initiated primarily by
computer scientists. The combinatoric–computational professional inclinations
of these visionaries made them more enthusiastic about geometric and compu-
tational issues in robotics than about real-time control and the algorithmic role
of sensing. Another important reason is the tight connection between algorithms
and hardware that the SIM approach espouses. As we will see later, some of this
(sensing) hardware has only started appearing recently. Finally, a quick look at
this book’s table of contents will show that the work on SIM approaches requires
from its practitioners a somewhat unusual combination of background: topology,
computational complexity, control theory, and a rather strange sensing hardware.
Whatever the reasons, in spite of its great theoretical interest and an immense
practical potential, the literature on the sensor-based motion planning paradigm
is small, especially for arm manipulators. In fact, today there are no textbooks
devoted to it.
Our goals in this book are as follows:
(a) Formulate the problem of sensor-based motion planning. We want to
explore why the relevant issues are so hard—so much so that in spite
of hard work and some glorious successes of robotics, there is no robot
today that can be left to its own devices, without supervision, outdoors
or in one’s home. Build a theoretical foundation for sensor-based motion
planning strategies.
(b) Study in depth a variety of particular algorithmic strategies for mobile
robots and robot arm manipulators, and try to identify promising directions
for conquering the general problem.

(c) Given the similarity of underlying tasks and requirements, compare robot
performance and human performance in sensor-based motion planning.
xviii PREFACE
The hope is that by doing so we can get a better insight into the nature
of the problem, and can help build synergistic human–robot teams for
tele-operation tasks.
(d) Review sensing hardware that is necessary to realize the SIM paradigm.
The book is intended to serve three purposes: (1) as a course textbook; (2) as
a research text covering in depth one particular area of robotics; (3) as a program
of research and development in robotic automation of unstructured tasks.
As a Textbook. A good portion of this book grew out of graduate and senior
undergraduate courses on robot motion planning taught by the author at Yale
University and the University of Wisconsin—Madison. As often happens with
research-oriented courses, the course kept changing as more research material
appeared and our knowledge of the subject expanded.
The text assumes a basic college background in mathematics and computer
science. A prior introductory course in robotics and some knowledge in topology
will be helpful but are not required. Some more exposure to topology is advised
for mastering the analysis that appears in Section 5.8 (Chapter 5) and the first two
pages of Section 6.2.4 (Chapter 6). Conclusions from this analysis, in particular
the formulation of algorithms, are written at the level compatible with the rest
of the book, though. The instructor is advised to glance through the chapters
beforehand to decide which level of what background a given chapter or section
requires.
Homework examples are provided as needed. In my view, a good home-
work structure for an advanced course like this one includes two components:
(a) ordinary homework assignments that dig deeper in the student’s knowledge,
are modest in number, and require a week or two to complete each assignment;
and (b) a course project that is initiated in the course’s first few weeks, goes in
parallel with it, and is defended at the end of the course, with the defense treated

as the final exam. The weights of those components in the student final grade can
be, say, 50% for the homework, 20% for the midterm assessment of the project,
and 30% for the final text-plus-presentation-before-class of the project. A list of
ideas for course projects is provided in Chapter 9.
Assuming a conventional two-semester school year, this book has about two
semesters worth of material. A one-semester course hence calls for choices. A
typical structure that covers ideas and computational schemes of the sensor-
based motion planning paradigm will include Chapters 1, 2, 3, 5, and 6 (Motion
Planning—Introduction, A Quick Sketch of Major Issues in Robotics, Motion
Planning for a Mobile Robot, Motion Planning for Two-Dimensional Arm Manip-
ulators, Motion Planning for Three-Dimensional Arm Manipulators). Let us call
this sequence the core course. The sequence contains no control theory or elec-
tronics, and it allows for the widest audience in terms of students’ majors.
For a strictly engineering class where students have already had courses in
controls and electronics, the instructor may want to sharply contract the time
for Chapter 2 and provide instead a deeper understanding of the effects of robot
PREFACE xix
dynamics on motion planning, covered in Chapter 4, plus a cursorial review
of principles of design of sensing devices necessary for realizing sensor-based
motion planning strategies, Chapter 8. Any group can benefit from Chapter 7,
which is devoted to human performance in motion planning and spatial reason-
ing tasks. A two-semester sequence will comfortably cover all those chapters
(with the danger of one’s noticing some repetitions necessitated by the foreseen
different uses of the book).
The decision to include in the course the topics covered in Chapters 4, 7, and
8, as well as the time devoted to the introductory Chapters 1 and 2 will depend
much on the mixture of students in class, in particular their prior exposure to
robotics, control theory, and electronics. Mandating prior courses on these topics
may introduce interesting difficulties. In my experience, a significant percent-
age of graduate students attracted to this course come from disciplines outside

of engineering, computer science, physics, and mathematics—such as business
administration, psychology, and even medicine. This is not surprising since the
course material touches upon the future of their disciplines rather deeply. Stu-
dents from some areas, especially the latter three above, are usually interested
in ideas and cognitive underpinnings of the subject. These students are often
extremely good, quick, and knowledgeable and have a reasonably good back-
ground in mathematics. Often such students do well in homework assignments,
bring in new ideas, and come up with wonderful course projects in their appro-
priate areas. Denying their participation would be a pity, in my view—after all,
robotics is a wide and widely connected field.
With such students in class, the instructor may choose to spend a bit more
time on the introductory sections, in order to bring up to speed students who
have had no past exposure to the robotics field. The instructor may also want to
complement introductory material with a relevant textbook (some such textbooks
are mentioned in Chapters 1 and 2). Students’ grades in the homework at the end
of Chapter 2 will give the instructor a good indication of how prepared they are
for the core course.
As a Research Text. This book is targeted to people who are interested in
or are directly involved in research and development of robot and human–robot
interaction systems. If one’s goal is to understand the underlying issues or design
a system capable of purposeful motion in an unstructured environment while
protecting the robot’s whole body—in streets, homes, undersea, deep space,
agriculture, and so on—today SIM is the only consistent approach one can count
on. This is not to say that the book contains answers to all questions. It provides
some constructive answers, and it calls for continuation.
The book should also be of interest to people working in areas that are
tangentially connected to robotics, such as sensor development and design of
tele-operated systems. And finally, the book will hopefully appeal to people
interested in the wide complex of underlying issues in robotics and human–robot
interaction, from mathematical and algorithmic questions to cognitive science to

advanced robot applications.
xx PREFACE
As a Program for Continued Research and Development. To repeat the
statement above, today the Sensing–Intelligence–Motion (SIM) approach seems
to be the only paradigm that holds promise to bring about robot automation of
unstructured tasks. This is not because of some special sophistication of SIM
techniques, but simply because only SIM techniques take care of the necessary
whole body awareness of the robot and do it “on the fly,” in real time, making it
possible to handle a high level of uncertainty. And only this approach guarantees
results in this area when human intuition breaks down.
And yet, as one will see later, only a limited number of SIM algorithms and
sensing schemes for real-world robot systems have been explored so far. Much
of the theory and of algorithmic and hardware machinery that is necessary to
bring the SIM approach to full fruition lies ahead of us. The book starts on the
misty route that lies ahead and that has to be traversed if we are serious about
bringing automation into unstructured tasks. With the risk of being seen less than
balanced, I suggest that not many areas of computer science and engineering
can compete with the excitement, the required breadth of knowledge, and the
potential impact on society of the topics covered in this book.
Professional and commercial importance of robotics aside, robots have been
always of immense interest to the general public. Isaac Asimov’s robot heros are
household names. Crowds invariably surround fake robots (controlled by humans
from nearby buildings) on the Disneyland streets. Robot exploits on Mars or on
the Space Shuttle or in a minefield disarming operation make front pages of
newspapers. What excites laymen is a human-like behavior potential of a robot.
This book takes the reader further in this same direction by providing a solid
foundation behind one human-like ability of robots that was so far assumed to
be an inherent monopoly of humans—namely, the ability to think of and plan
one’s motion in an unstructured world.
Robots are often referred to derisively: “He moves like a robot,” “Yours is

a robot reaction,” “Hey, don’t behave like a robot.” What is meant is crude,
unintelligent, and mechanical; even the word “mechanical” signifies here crude
and unintelligent. Many mimes entertain the crowd on the street corners by
moving “like a robot”—that is, switching sharply from one movement to the
other and being oblivious to the surroundings.
That is not what robots should be and even are today. Examples in Chapter 8
will show that when equipped with means for self-awareness and with strategies
to use it, robots become sensitive to their surroundings, “pensive,” and even
gentle in how they “mind” their movement.
1
A nonprofessional reader curious
about the possibilities of intelligent robots will find long layman-level passages in
1
Sharp “robot-like” movements have been a persistent science fiction-maintained myth. Many robot
applications—car painting is a good example—require smooth motion and simply cannot tolerate
sharp turns. Today’s industrial robots can generate a motion that is so smooth and delicate that it
may be the envy of “Swan Lake” ballerinas. For those who know calculus, what dancer can promise,
for example, a motion so smooth that both its derivatives have guaranteed continuity!
PREFACE xxi
the Introduction, introductory sections to other chapters, discussions, examples,
and simplified explanations of the underlying ideas throughout the text.
Designing a whole-sensitive robot is almost like designing a friend. One day
you move your hand in a stroking movement along the robot’s skin, and it
responds with a gentle appreciative movement. This gives you a strange feeling:
We humans are totally unprepared to see a machine exhibit a behavior that we
fully expect from a cat or a dog. I hope that both professional and layman readers
will share this gratifying feeling. And, of course, I hope the book will further our
attempts toward populating our environment with helpful and loyal robot friends.
Vladimir J. Lumelsky
Madison, Wisconsin

Washington D.C.
April 2005

ACKNOWLEDGMENTS
When pieces of a large multiyear project start falling into place, a sign that it
functions right is that the pieces “know by themselves” what to do and when to
do it. A product of one section logically invites and defines the other; theory calls
for the experiment to confirm its correctness; experiments beg for turning theory
into useful products. The project then operates as a leisurely human walk: As
the right foot is thrown forward, the left foot knows it should stay behind on the
ground, the body bends slightly forward as if ready to fall, the left arm moves
forward, and the right arm heads back—all at once, seemingly effortlessly, and
then they switch, one-two, one-two, a pleasure to watch, so hard to emulate,
one-two, one-two.
A piece of science or new technology cannot be like this, not that perfect,
simply because there is always more unknown and yet undiscovered than known
and understood. But the feeling is similar: All of a sudden, things fall into place.
This picture fully applies to this book. While the knowledge that it treats will
be always incomplete, a moment came when individual smaller projects started
looking as parts of a tightly coordinated organism.
This would not be possible without my graduate students. Much of today’s
science is produced this way. It is the graduate students’ sleepless nights, enthu-
siasm, and unwavering commitment to science that help cover the skeleton of
ideas with flesh and blood of details of design and proofs and tests and computer
simulations. They help turn the skeleton’s jerky squeakiness into smooth and
coordinated and pleasing to the professional eye elegant whole.
“What if” is rarely a reliable game. There is no way of knowing what this book
would look like if I had different students, not those I was privileged to have. I do
think that some pieces would have been quite different, because the personalities
and prior background of my students invariably left a strong trace on my choice

of projects for them and hence the joint papers that became the foundation of
this book. I am grateful to them for sharing with me the joy of doing science.
With all those different personalities, there was also something in common that
emerged in them as the work progressed–perhaps the desire for dry precision,
for doing things right. In thanking them for sharing with me our life in the lab
and discussions in seminars and at the blackboard, I am mentioning here only
those whose work was pivotal for this book: Kang Sun, Timothy Skewis, Edward
Cheung, Susan Hert, Andrei Shkel, Fei Liu, Dugan Um. Other students helped
as well, but their main work centered on topics that are beyond our subject here.
xxiii
xxiv ACKNOWLEDGMENTS
From the beginning of this research in the late 1980s, the National Science
Foundation was incredibly generous to me, funding in parallel the theory/software
and the hardware/sensing lines of this work. I am also indebted to the Sandia
Laboratories and Hitachi Corporation for providing necessary resources.
Every book has to be started, and that moment calls for an appropriate setting.
My thanks go to the Rockefeller Foundation, whose invitation to spend a month
at the incomparable Villa Serbelloni in the village of Bellagio, Lake Como, Italy,
made the start of this book quick and easy. Putting in a day of work, along with a
couple more hours in the evening, was tiring but easy, in anticipation of the game
of bocce on the lake by 5 o’clock and then dressing up for drinks and dinner
with the Villa’s guest artists and writers and scientists, among the seventeenth-
century rugs and furniture. It is not for nothing that the Villa Serbelloni’s library
is crammed with books authored by many of its visitors from all over the world.
V. J. L.