Handbook of Multimedia for Digital Entertainment and Arts- P17 pptx

21 Projector-Camera Systems in Entertainment and Art 479
Physically Viewing Interaction
By projecting images directly onto everyday surfaces, a projector-camera system
may be used for creating augmentation effects, such as virtually painting the ob-
ject surface with a new color, new texture, or even an animation. Users can interact
directly with such projector-based augmentations. For example, they may observe
the object from different sides, while simultaneously experiencing consistent occlu-
sion effects and depth, or they can move nearer or further from the object, to see
local details and global views. Thus, the intuitiveness of physical interaction and
advantages of digital presentation are combined.
This kind of physically interactive visualization ability is suitable for use in
situations when virtual content is mapped as a texture on real object surfaces.
View-dependent visual effects such as highlighting to simulate virtually shiny sur-
faces require tracking of the users’ view. Multi-user views can also be supported
by time-multiplexing the projection for multiple users, with each user wearing a
synchronized shutter glass allowing the selection of individual views. But this is
only necessary for view-dependent augmentations. Furthermore, view tracking and
stereoscopic presentation ability enables virtual objects to be displayed not only
on the real surface, but also in front of or behind the surface. A general geometric
framework to handle all these variants is described in [26].
The techniques described above, only simulate the desired appearance of an aug-
mented object which is supposed to remain fixed in space. To make the projected
content truly user-interactive, more information apart from viewpoint changes is
required. After turning an ordinary surface into a display, it is further desirable to ex-
tend it to become a user interface with an additional input channel. Thereby, cameras
can be used for sensing. In contrast to other input technologies, such as embedded
electronics for touch screens, tracked wand, or stylus and data gloves often used
in virtual environments; vision-based sensing technology has the flexibility to sup-
port different types of inputting techniques without modifying the display surface
or equipping the users with different devices for different tasks. Differing from in-
teraction with special projection screens such as electronically enabled multi-touch

or rear-projected screens, some of the primary issues associated with vision-based
interaction with front-projected interfaces are the illuminations on the detected hand
and object, as well as cast of shadows.
In following subsections, two types of typical interaction approaches with spatial
projector-camera systems will be introduced, namely near distance interaction and
far distance interaction. Vision based interaction techniques will be the main focus
and basic interaction operations such as pointing, selecting and manipulation will
be considered.
Near Distance Interaction
In near-distance situations where the projection surface is within arm’s length of the
user, finger touching or hand gestures are intuitive ways to select and manipulate the
480 O. Bimber and X. Yang
interface. Apart from this, the manipulation of physical objects can also be detected
and used for triggering interaction events.
Vision-based techniques may apply a visible light or infrared light camera to
capture the projected surface area. To detect finger touching on a projected surface a
calibration process, similar to the geometric techniques presented in section “Geo-
metric Image Correction”, is needed to map corresponding pixels between projector
and camera.
Next, fingers, hands and objects need to be categorized as part of the foreground
in order to separate them from the projected surface background. When interactions
take place on a front-projected surface, the hand is illuminated by the displayed
images and thus the appearance of a moving hand changes quickly. This renders
segmentation methods, based on skin color or region-growing methods as useless.
Frequently, conventional background subtraction methods are also unreliable, since
the skin color of a hand may become buried in the projected light.
One possible solution to this problem is to expand the capacity of the background
subtraction. Despite, its application to an ideal projection screen which assumes
enough color differences from skin color as in [27], the background subtraction
can also be used to take into account different background and foreground re-

flectance factors. When the background changes significantly, a segmentation may
fail. An image update can be applied to keep the segmentation robust, where an
artificial background may be generated from the known input image for a pro-
jector with geometric and color distortions corrected between the projector and
camera.
Another feasible solution is to detect the changing pixel area between the frames
of the captured video to obtain a basic shape of the moving hand or object. Noise
can then be removed using image morphology. Following this, a fingertip can be
detected by convolution with a fingertip-shaped template over the extracted image,
as in [28].
To avoid the complex varying illumination problem for visible light, an infrared
camera can be used instead, together with an infrared light source to produce in-
visible shadow of a finger on the projected flat surface, as shown in [29]. The
shadow of the finger can then be detected by the infrared camera and can thus be
singularly used to detect the finger region and fingertip. To enable screen intera-
tion by finger touching, the positioning of the finger, either touching the surface
or hovering above it, can be further determined by detecting the occlusion ratio of
the finger shadow. When the finger is touching the surface, its shadow is fully oc-
cluded by the finger itself; while the finger is hovering over the surface, its shadow
is larger.
It is also possible to exclude the projected content from the captured video by
interlacing the projecting images and captured camera frames using synchronized
high-speed projectors and cameras, so that more general gesture recognition algo-
rithms can be adopted as those reviewed in [30]. To obtain more robust detection
results, specific vision hardware can also be utilized, such as real-time depth cam-
eras that are based on the time-of-flight principle [31].
21 Projector-Camera Systems in Entertainment and Art 481
Far Distance Interaction
In a situation where the projection surface is beyond the user’s arm length, laser
pointer interaction is an intuitive way to select and manipulate projected interface

components. Recently, laser pointer interaction has used for interacting with large
scale projection display or tiled display at a far distance [32].
To detect and track a laser dot on a projection surface in projector-camera sys-
tems, a calibrated camera covering the projecting area is often used. The location
and movement of a laser dot can be detected simply by applying an intensity thresh-
old to the captured image – assuming that the laser dot is much brighter than the
projection. Since the camera and the projector are both geometrically calibrated, the
location of the laser dot on the camera image can be mapped to corresponding pixels
on projection image. The “on” and “off” status of the laser pointer can be mapped to
mouse click events for selecting particular operations. One or more virtual objects
that are supposed to be intersected with the laser dot or a corresponding laser ray
can be further calculated from the virtual scene geometry.
More events for laser pointer interaction can be triggered by temporal or spa-
tial gestures, such as encircling, or simply by adding some hardware on laser
pointers, such as buttons and embedded electronics for wireless communication.
Multiple user laser pointer interaction can also be supported for large projection
areas where each user’s laser pointer is distinguishable. This can be supported by
time-multiplexing the laser or by using different laser colors or patterns. User stud-
ies have been carried out to provide optimized design parameters for laser pointer
interaction [33].
Although laser pointing is an intuitive technique, it also suffers from issues
such as hand-jittering, inaccuracy and slow interaction speeds. To overcome the
hand-jittering problem, which is compounded at greater distances, filtering-based
smoothing techniques can be applied, though may lead to discrepancy between the
pointing laser dot and the estimated location. Infrared laser pointers may solve this
problem, but according to user study results, visible laser lights are still found to be
better for interaction.
Apart from laser pointing, other tools such as a tracked stylus or specially de-
signed passive vision wands [34] tracked by a camera have proven to be flexible and
efficient when interacting with large scale projection displays over distances.

Gesture recognition provides a natural way for interaction in greater distances
without using specific tools. It is mainly based on gesture pattern recognition with
or without hand model reconstruction. Evaluating body motions is also an intuitive
way for large scale interaction, where the body pose and motion are estimated and
behavior patterns may be further detected. When gesture and body motion are the
dominant modes of interaction with projector-camera systems, shadows and varying
illumination conditions are the main challenges, though shadows can also be utilized
for detecting gesture or body motion.
In gesture or body interaction, background subtraction is often used for detect-
ing the moving body from the difference between the current frame and a reference
background image. The background reference image must be regularly updated so
482 O. Bimber and X. Yang
as to adapt to the varying luminance conditions and geometry settings. More com-
plex models have extended the concept of background subtraction beyond its literal
meaning. A thorough review of the background extraction methods is presented
in [35].
Vision-based human action recognition approaches can be generally divided into
four phases. The model initialization phase ensures that a system commences its
operation with a correct interpretation of the current scene. The tracking phase seg-
ments and tracks the human bodies in each camera frame. The pose estimation phase
estimates the pose of the users in one or more frames. The recognition phase can
recognize the identity of individuals as well as the actions, activities and behaviors
performed by one or more user. Details about video based human action detection
techniques are reviewed in [36].
Interaction with Handheld Projectors
Hand-held projectors may display images on surfaces anywhere at anytime while
they are being moved by the user. This is especially useful for mobile projector-
based augmentation, which superimposes digital information in physical environ-
ments. Unlike other mobile displays such as provided by PDAs or mobile phones,
hand-held projectors offer a consistent visual combination of real information gather

from physical surfaces with virtual information. This is possible without context
switching between information space and real space, thus seamlessly blurring the
virtual and real world. They can be used, for instance, as interactive information
flashlights [37] – displaying registered image content on surface portions that are
illuminated by the projector.
Although hand-held projectors provide great flexibility for ubiquitous computing
and spontaneous interaction, there are fundamental issues to be addressed before a
fluid interaction between the user and the projector is possible. When using a hand-
held projector to display on various surfaces in a real environment, the projected
image will be dynamically modulated and distorted by the surfaces as the user
moves. When the user stops moving the projector, the presented image still suf-
fers from shaking by the user’s unavoidable hand-jitter. Thus, a basic requirement
for hand-held projector interaction is to produce stable projection.
Image Stabilizing
One often desired form of image stabilization is to produce a rectangular 2D image
on a planar surface – independently of the projector’s actual pose and movement.
In this case, the projected image must be continuously warped to keep the correct
aspect ratio and to remain undistorted. The warping process is similar to the geo-
metric correction techniques described earlier. The difference, however, is that the
21 Projector-Camera Systems in Entertainment and Art 483
target viewing perspective is usually pointing towards the projection surface along
its normal direction, while the position of the hand-held projector may keep on
changing.
To find the geometric mapping between the projector and the target perspective,
the projector’s six degrees of freedom may be obtained from an attached tracking
device. The homography is an adequate method to represent this geometric mapping
when the projection surface is planar. Instead of using the detected four vertices
of the visible projection area to calculate the homography matrix, another practical
technique is to identify laser spots displayed from laser-pointers that are attached
to the projector-camera system. The laser spots are brighter and therefore easier to

detect.
In [38], hand-jittering was compensated together with the geometry correc-
tion, by continuously tracking the projector’s pose and warping the image at each
time-step. A camera attached to the projector detects visual markers on the projec-
tion surface, that are used for warping the projected image accordingly. In [42]a
similar stabilization approach is described. Here, the projector pose relative to the
display surface is recovered up to an unknown translation in the display plane.
Pointing Techniques
After the stabilization of the projector images, several techniques can be adopted to
interact with the displayed content. Controlling a cursor by laser pointing (e.g., with
a projector-attached laser pointer) represents one possibility. In this case, common
desktop mouse interaction techniques can be mapped directly to hand-held projec-
tors. The projector’s center pixel ray can also be used instead of a laser pointer to
control the mouse cursor. One of the biggest problems associated with these meth-
ods are size reductions and cropping of the display area, caused by the movement of
the projector when controlling the cursor. Using a secondary device such as a tracked
stylus or a separate laser pointer can overcome these limitations, however the user
needs both hands for interaction. Mounting a touch pad or other input devices on
the projector is also possible, but might not be as intuitive as a direct pointing with
the projector itself.
Selection and Manipulation
Based on the display and direct pointing ability described above, mouse like interac-
tion can be emulated such as selecting a menu or performing a cut-and-paste oper-
ation by pointing the cursors on the projected area and pressing buttons mounted
on the projector. However, in this scenario, the hand jitter problem, similar to
laser pointer interaction, also exists – making it difficult to locate the cursor in
specific and small areas. The jitter problem is intensified when cursor pointing
is combined with mouse button-pressing operations. Adopting specially designed
interaction techniques rather than emulating common desktop GUI methods, can
alleviate this problem.

484 O. Bimber and X. Yang
One proven and efficient interaction technique for hand-held projectors is the
crossing based widget technique [37]. Crossing based widget is operated by moving
the cursor to cross the widget in a specific direction (e.g. from outside to inside, or
from top to bottom), while holding the mouse button. This technique avoids point-
ing the cursor and pressing a button at the same time. Crossing widget can be used
for hand-held projectors to support commonly used desktop GUI elements, such as
menus and sliders. Crossing based menu items can be activated by crossing from
one direction; and deactivated by crossing from the opposite direction. All actions
are executed by releasing the mouse button. Different colors can be used to indicate
the crossing directions. Hierarchical menus can also be supported. Similarly, the
crossing based slider is activated by crossing the interface in one direction, deacti-
vated by crossing it in the opposite direction, and adjusted according to the cursor
movement parallel to the slider.
Another specially designed interaction technique is called zoom-and-pick wid-
get, proposed by [39]. It was designed to implement the simultaneous use of stable
high-resolution visualization and pixel-accurate pointing for hand-held projectors.
The widget is basically a square magnification area, located around the current
pointing cursor position. A circular dead zone is defined within this area. The center
of the dead zone is treated as an interaction hot-spot. The widget remains static when
the pointing cursor is moving within the dead zone. To gain pixel-accurate pointing
ability, a rim is defined around the dead zone. Each crossing of the cursor from the
dead zone into the rim triggers a single pixel movement of the widget in the direc-
tion of the pointer movement. If the pointer is moving beyond the dead zone and the
rim, the widget will be relocated to include the pointer in its dead zone again.
Multi-user Interaction
Hand-held projectors also pose new chances and challenges for multi-user interac-
tion. In contrast to other multi-user devices such as tabletop displays, primarily used
for sharing information with others, or other mobile devices such as personal mo-
bile phones; hand-held projectors, due to their portability, and personal usage, are

suitable both for shared and individual use. Multiple hand-held projectors combine
the advantages of public and personal display systems.
The main issues associated with multi-user interaction and hand-held projec-
tors are primarily concerned with design for ownership, privacy control, sharing,
and so on. The name of the owner of a displayed object can be represented by spe-
cially designed label widgets placed on the object and operated using crossing based
operations. The overlap of two or more cursors can signify consent from multiple
users to accomplish collaborative interactive task, such as coping a file or blending
two images between the users. Snapping and docking actions can be performed by
multiple users in order to quickly view or modify connected information between
multiple objects. Multiple displayed images from more than one user can be blended
directly or semantically. By displaying high resolution images when the user moves
21 Projector-Camera Systems in Entertainment and Art 485
closer to the display surface, a focus-and-context experience can be achieved by
providing refined local details. More details can be found in [40].
Environment Awareness
Due to their portability, hand-held projectors are mainly used spontaneously. There-
fore, it is desirable to enhance the hand-held projectors with environment awareness
abilities. Geometric and photometric measurement and object recognition and track-
ing capacities, would enable the projector to sense and respond to the environment
accordingly.
Geometric and photometric awareness can be implemented using, for example,
structured light techniques, as described in section “Structured Light Scanning”. For
object recognition and tracking, the use of a passive fiducial marker (e.g., supported
with open source computer vision toolkits such as ARToolkit[41]) is a cheap solu-
tion. However, it is not visually attractive which may disturb the appearance of the
object and may fail as a result of occlusion or low illumination. Unpowered pas-
sive RFID tags can be detected via a radio frequency reader without being visible.
They represent another inexpensive solution for object identification. However, they
do not support pose tracking. The combination of RFID tags with photo-sensors,

called RFIG, has been developed in order to obtain both – object identification and
object position. The detection of the object position is implemented by projecting
Gray codes onto the photo-sensors. In this way the Gray code is sensed by each
photo-sensor and allows computing the projection of the sensors to the projector
image plane, and consequently enables projector registration. More details about
RFIG are referred to [42].
Interaction Design and Paradigm
In the sections above, techniques for human interaction with different configura-
tions of projector-camera systems were presented. This subsection, however, will
introduce higher level concepts and methods for interaction design and interaction
paradigms for such devices. Alternative configurations such as steerable projector
and moveable surfaces will also be discussed briefly.
Projector-based systems for displaying virtual environments assume high qual-
ity, large field of view, and continuous display areas which often evoke feelings
of immersion and presence, and provide continuous interaction spaces. In contrast,
spatial projector-camera systems that display on everyday surfaces may produce
blended and warped images with average quality and a cropped field of view. The
cropped view occurs as a result of the constricted display area, discontinuous im-
ages on different depth levels, and surfaces with different modulation properties.
Due to these discrepancies, it is not always possible to directly adopt interaction
techniques from immersive virtual environments or from conventional augmented
reality applications.
486 O. Bimber and X. Yang
For example, moving a virtual object using the pointing-and-drag technique,
which is often adopted in virtual environments, may not be the preferred method
in a projector-based augmented environment, since the appearance of the virtual ob-
ject may vary drastically as it is moved and displayed on discontinuous surfaces with
different depths and material properties. Instead, grasp-and-drop techniques may be
better suited to this situation, as discussed in [43].
Furthermore, the distance between the user and display surface is important for

designing and selecting interaction techniques. It was expected that pointing interac-
tion is more suitable for manipulating far distance objects, while touching is suitable
for near distance objects. However, contradictory findings, derived from user studies
for interaction with projector-camera systems aimed for implementing augmented
workspace [43], have proven otherwise. Users were found unwilling to touch the
physical surfaces even at close range distances after they learned distance gestures
such as pointing. Instead, users frequently continued using the pointing method,
even for surfaces located in close proximity to them. The reason for this behavior
may be two-fold. Firstly, users may prefer to use a consistent technique for manipu-
lation such as pointing. Secondly, it seems that the appearance and materials of the
surfaces affect the user’s willingness to interact with them [44].
Several interaction paradigms have been introduced with or for projector-camera
systems. Tangible user interfaces were developed to manipulate projected content
using physical tangible objects [45]. Vision based implicit interaction techniques
have also been applied to support subtle and persuasive display concepts derived
from ubiquitous computing [46]. The peephole paradigm is discussed as a concept
to describe the projected display as a peephole for the physical environment [47].
Varying bubble-like free-form shapes of the projected area based on the environment
enables a new interface that moves beyond regular fixed display boundaries [48].
Besides hand-held projectors which enable ubiquitous display, steerable projec-
tors also bring new interaction concepts, such as everywhere displays. Such systems
enable projections on different surfaces in a room, and to turn them into an interac-
tion interfaces. The best way to control a steerable projector during the interaction,
however still needs to be determined. Body tracking can be combined with steer-
able projections to produce a paradigm called user-following display [49], where
the user’s position and pose are tracked. Projection surfaces are then dynamically
selected and modulated accordingly, based on a measured and maintained three-
dimensional model of the surfaces in the room. Alternatively, laser pointers can be
used and tracked by a pan/tilt/zoom camera to control and interact with a steer-
able projector unit [50]. Another issue for interaction with steerable projectors is

the question of how to support a dynamic interfaces which can change form and
location on the fly. A vision-based approach can solve this problem by decoupling
interface specifications from its location in space and in the camera image [51].
Besides the projectors themselves, projection surfaces might also be moveable
rather than remain static in the environment. They may be rigidly moveable flat
screens, semi-rigidly foldable objects such as a fan or an umbrella, or deformable
objects such as paper and cloth. Moveable projection surfaces can provide novel
interfaces and enable unique interaction paradigms such as foldable displays or
21 Projector-Camera Systems in Entertainment and Art 487
organic user interfaces [52]. Tracking the pose or deformation of such surfaces, how-
ever, is an issue that still needs to be addressed. Cheap hardware trackers have been
used recently to support semi-rigid surfaces [53]. Vision-based deformation detec-
tion algorithms may be useful in future for supporting deformable display surfaces.
Application Examples
The basic visualization and interaction techniques that have been presented in
the sections above enable a variety of new applications in different domains. In
general, projector-camera systems can be applied to interactive or non-interactive
visual presentations in situations where the application of projection screens is not
possible, or not desired. Several examples are outlined below.
Embedded Multimedia Presentations
Many historic sites, such as castles, caves, or churches, are open to public. Flat
panel displays or projection screens are frequently being used for presenting vi-
sual information. These screens, however, are permanently installed features and
unnecessarily cover a certain amount of space. They cannot be temporally disas-
sembled to give the visitors an authentic impression of the environment’s ambience
when required.
Being able to project undistorted images onto arbitrary existing surfaces offers
a potential solution to this problem. Projectors can display images that are much
larger than the device itself. The images can be seamlessly embedded, and turned
off any time to provide an unconstrained experience. For these reasons, projector-

camera systems and image correction techniques are applied in several professional
domains, such as historic sites, theater, festivals, museums, public screen presenta-
tions, advertisement displays, theme parks, and many others. Figure 2 illustrates two
examples for a theater stage projection at the Karl-May Festival in Elspe (Germany),
and an immersive panoramic projection onto the walls of the main tower of castle
Osterburg in Weida (Germany). Both are used for displaying multimedia content
which is alternately turned on and off during the main stage performance and the
museum presentation respectively. Other examples of professional applications can
be found at www.vioso.com.
Superimposing Museum Artifacts
Projector-camera systems can also be used for superimposing museum artifacts with
pictorial content. This helps to communicate information about the displayed ob-
jects more efficiently than secondary screens.
488 O. Bimber and X. Yang
Fig. 2 Projection onto physical stage setting (top), and 360 degree surround projection onto natu-
ral stone walls in castle tower (bottom). Image courtesy: VIOSO GmbH, www.vioso.com
In this case, a precise registration of the projector-camera system is not only nec-
essary to ensure an adequate image correction (e.g., geometrically, photometrically,
and focus), but also for displaying visual content that is geometrically registered to
the corresponding parts of the object.
Figure 3 illustrates two examples for superimposing visual content, such as
color, text and image labels, interactive visualizations of magnifications and un-
derdrawings, and visual highlights on replicas of a fossil (primal horse displayed
by Senckenberg Museum Frankfurt, Germany) and paintings (Michaelangelo’s
Creation of Adam, sanguine and Pontormo’s Joseph and Jacob in Egypt, oil on
wood) [22].
In addition to augmenting an arbitrary image content, it is also possible to boost
the contrast of low contrast objects, such as paintings whose colors have faded after
a long exposure to sun light. The principle techniques describing how this can be
achieved are explained in [19].

Spatial Augmented Reality
Projector-camera systems cannot only acquire parameters that are necessary for im-
age correction, but also higher level information, such as the surrounding scene
geometry. This, for instance, enables corrected projections of stereoscopic images
21 Projector-Camera Systems in Entertainment and Art 489
Fig. 3 Fossil replica superimposed with projected color (top), and painting replicas augmented
with interactive pictorial content (bottom) [22]
onto real-world surfaces which allows the augmentation of three-dimensional in-
teractive content. Active stereoscopic shutter glasses and head-tracking technology
supports correct depth viewing of virtual content in precise alignment with the phys-
ical environment. This is a projector-based variation of what is referred to as spatial
augmented reality [23]. In contrast to mobile augmented realities, the display tech-
nology for spatial augmented reality applications is not hand-held or head-worn, but
fixed in the environment. This has several technological advantages, but also limits
the applications to non-mobile ones.
Figure 4 illustrates two projector-based spatial augmented reality examples: An
architectural global lighting simulation is projected directly within the real environ-
ment enabling a more realistic and immersive visualization than possible with only
a monitor. Stereoscopically projected game content can interact with real objects. A
physical simulation of the virtual car allows realistic collisions with real items. This
is possible through the scanned scene geometry, which also enables correct occlu-
sion effects. Object recognition techniques that are applied to the acquired scene
geometry and to the captured camera image enable the derivation of contextual
information that is used in the game logic. Motorized pan-tilt projector-camera units
allow using large parts of an entire room as playground for such spatial augmented
reality games. More information on spatial augmented reality can be found in [23].
A free e-book is available at www.SpatialAR.com.
490 O. Bimber and X. Yang
Fig. 4 Examples for spatial augmented reality applications with projector camera systems. An
immersive in-place visualization of an architectural lighting simulation (left), and a stereoscopi-

cally projected spatial augmented reality game (right). Door, window, illumination and the car are
projected
Flexible Digital Video Composition
Blue screens and chroma keying technology are essential for digital video compo-
sition. Professional studios apply tracking technology to record the camera path for
perspective augmentations of original video footage. Although this technology is
well established, it does not offer a great deal of flexibility.
For shootings at non-studio sets, physical blue screens can be installed and takes
might have to be recorded twice (with and without blue screens), or parts have to be
re-recorded in a studio.
In addition, virtual studio technology itself still faces limitations. Chroma-keying
and studio illumination, for instance, are difficult to harmonize. Moderators or actors
have to spend a fair amount of practice time before interacting with invisible virtual
components naturally. Spill on the foreground and disadvantageous foreground col-
ors lead to low-quality or even unusable keying results.
Temporally synchronized projector-camera systems can be used to project cor-
rected keying patterns and other spatial codes onto arbitrary diffuse (real-world)
surfaces. Therefore the reflectance of the underlying real surface is widely neutral-
ized by applying the image correction techniques that have been explained above.
The main difference to the application examples that have been described so far,
is that projector-camera systems are used for recording visual effects, and not for
presenting corrected visual content directly to human observers.
A temporal multiplexing between projection (p-frames) and flash illumination
(i-frames) allows capturing the fully lit scene, while still being able to key the fore-
ground objects. This is illustrated in figure 5.
Since the entire scene is recorded when physical blue screens do not block the
view, the footage of the full background scene can be used for video composition.
Thus, recordings need not be taken twice, and keying is invariant to foreground
colors. In addition, other spatial codes can be embedded into the projected im-
ages to enable tracking of the camera, environment matting, and displaying in-place

21 Projector-Camera Systems in Entertainment and Art 491
Fig. 5 VirtualStudio2Go: Odd (i-) frames record the fully illuminated scene. Even (p-) frames
record the non-illuminated scene with projected images that neutralize the appearance of a real
background surface and display code patterns. Repeating this at HD scanning speed (59.94Hz) and
registering both sub-frames during post-processing supports high quality digital video composition
effects for real (non-studio) environments
moderator information. Furthermore, the reconstruction of the scene geometry is
implicitly supported, and allows special composition effects, such as shadow casts,
occlusions and reflections.
A concept that combines all of these techniques into one single compact and
portable system that is fully compatible with common digital video composition
pipelines, and offers an immediate plug-and-play applicability is presented in [24]. It
enables professional digital video composition effects in real indoor environments.
Interactive Attraction Installations
Today, the most popular applications of projector-camera systems are perhaps in-
teractive attractions as public installations. By projecting interactive graphics onto
everyday surfaces in public places, such as walls in museums, floors in shop-
ping mall, subway tunnels, and even dining tables in restaurants, projector-camera
systems emerge as an effective attraction tool by creating vivid interactive art, enter-
tainment, and advertisement experience for people. Vision based sensing technology
can be mainly adopted in such interactive art systems to detect people’s presence and
activity in an unobtrusive way, and implicitly engaging people with the artificially
augmented environment through large scale human body motions, hand gesture, or
finger touching interactions with such installations.
Figure 6 illustrates two projector-based interactive attraction installations. In the
left example, a realistically rendered water pool is projected directly onto the ground
of the Lou Dong Chinese Painting Museum at Tai Cang (China) with a physically
built pool boundary, where the rendered water, lotus, and fishes in the pool are all
responsive to the visitors who step into it. Ripple effects in the water, blooming
lutoses, and escaping fishes, have been rendered in this way [54]. A Chinese painting

is mapped as texture on the ground of the pool to compliment the artwork. Since
this system has been installed in the museum, its realistic appearance and vivid
492 O. Bimber and X. Yang
Fig. 6 Examples for interactive attraction installations. An interactive water pool installed in a
traditional art museum (left) and an interactive augmented physical map installation for tourist
attraction (right)
interactivity have attracted many visitors, especially young children who have little
contact with traditional culture.
In another installation which was exhibited for an art-science festival in
Shanghai’s Oriental Pearl Tower (China), a shining icon is projected onto a tra-
ditional physical tourist map. The tourists can select different sites by hands or
props on the map to see related video information. A tour guide can then create
a touring path on the map with a laser pointer, or a visitor could produce a path
by walking on a projected map on the ground. A three dimensional walk-through
of the tour scene can then be triggered along the created path [55]. By integrating
the traditional tangible map with the augmented digital information, and enabling
vision-based and tangible interaction techniques, the projector-camera system can
provide tourists and tour guides with a fresh sightseeing experience.
The Future of Projector-Camera Systems
Projector-camera systems have already found practical applications in theater, mu-
seums, historic sites, open-air festivals, trade shows, advertisement, visual effects,
theme parks, and art installations. With advances in technology and techniques, they
will be applied in many more areas in future.
Future projectors will become more compact in size and will require little power
and cooling. Reflective technology (such as DLP or LCOS) will increasingly replace
transmissive technology (e.g., LCD). This will leads to an increased brightness and
extremely high update rates. GPUs for real-time graphics and vision processing
will also be integrated. While resolution, contrast and speed will keep increas-
ing, production costs and market prizes will continue to fall. Conventional UHP
lamps will be replaced by powerful LEDs or multi-channel lasers. This will make

them suitable for mobile applications. Projector-camera technology is currently be-
ing integrated into mobile devices, such as cellphones, and supports truly flexible
21 Projector-Camera Systems in Entertainment and Art 493
presentations methods. Image correction techniques, such as the ones explained
above are essential for these devices, since projection screens will most likely not
become mobile.
But projector-camera systems will not only be used as display devices. In future,
they will also enable intelligent, spatially and temporally controllable light sources.
Projector-based illumination techniques will not only solve problems in professional
domains, such as microscopy or endoscopy, but -one day- might also be applied in
more general contexts.
Imagine that networked projector-camera systems become as cheap and as com-
pact as light bulbs. They could not only be turned on and off, but would allow to
offer synthetic room illumination and interactive display capabilities everywhere.
For instance, they could produce individual mood profile and ambient light situa-
tions, as well as to enable internet access wherever you stand.
References
1. O. Bimber, D. Iwai, G. Wetzstein, and A. Grundhoefer, “The Visual Computing of Projector-
Camera Systems,” Computer Graphics Forum, Vol. 27, No. 8, pp. 2219–2245, 2008.
2. G. Wetzstein and O. Bimber, “Radiometric Compensation through Inverse Light Transport,”
Proceedings of Pacific Graphics, pp. 391–399, 2007.
3. J. Salvi, J. Pag
´
es, and J. Batlle, “Pattern Codification Strategies in Structured Light Systems,”
Pattern Recognition, Vol. 37, No.4, pp.827–849, 2004.
4. R. Raskar, G. Welch, M. Cutts, A. Lake, L. Stesin, and H. Fuchs, “The Office of the Future: A
Unified Approach to Image-Based Modeling and Spatially Immersive Displays,” Proceedings
of ACM SIGGRAPH, pp. 179–188, 1998.
5. O. Bimber, A. Emmering, and T. Klemmer, “Embedded Entertainment with Smart Projectors,”
IEEE Computer, Vol.38, No.1, pp.56–63, 2005.

6. S. K. Nayar, H. Peri, M. D. Grossberg, and P. N. Belhumeur, “A Projection System with Radio-
metric Compensation for Screen Imperfections,” Proceedings of IEEE International Workshop
on Projector-Camera Systems (ProCams), 2003.
7. T. Yoshida, C. Horii, and K. Sato, “A Virtual Color Reconstruction System for Real Heritage
with Light Projection,” Proceedings of International Conference on Virtual Systems and Mul-
timedia (VSMM), pp. 161–168, 2003.
8. M. Ashdown, T. Okabet, I. Sato, and Y. Sato, “Robust Content-Dependent Photometric Pro-
jector Compensation,” Proceedings of IEEE International Workshop on Projector-Camera
Systems (ProCams), 2006.
9. A. Grundhoefer and O. Bimber, “Real-Time Adaptive Radiometric Compensation,” IEEE
Transactions on Visualization and Computer Graphics (TVCG), Vol. 14, No. 1, pp.
97–108, 2008.
10. H. Park, M H. Lee, S J. Kim, and J I. Park, “Specularity-Free Projection on Nonplanar
Surface,” Proceedings of Pacific-Rim Conference on Multimedia (PCM) (2005), pp. 606–616.
11. R. Sukthankar, C. Tat-Jen, and G. Sukthankar, “Dynamic Shadow Elimination for Multi-
Projector Displays,” Proceedings of IEEE Conference on Computer Vision and Pattern
Recognition (CVPR), Vol. II, pp. 151–157, 2001.
12. C. Jaynes, S. Webb, and R. M. Steele, “Camera-Based Detection and Removal of Shadows
from Interactive Multiprojector Displays,” IEEE Transactions on Visualization and Computer
Graphics (TVCG), Vol. 10, No. 3, pp. 290–301, 2004.
494 O. Bimber and X. Yang
13. M. S. Brown, P. Song, and T. - J. Cham, “Image Pre-Conditioning for Out-of-Focus Projector
Blur,” Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR),
Vol. II, pp. 1956–1963, 2006.
14. Y. Oyamada and H. Saito, “Focal Pre-Correction of Projected Image for Deblurring Screen Im-
age,” Proceedings of IEEE International Workshop on Projector-Camera Systems (ProCams),
2007.
15. L. Zhang and S. K. Nayar, “Projection Defocus Analysis for Scene Capture and Image Dis-
play,” ACM Trans. Graph. (Siggraph), Vol. 25, No. 3, pp.907–915, 2006.
16. M. Grosse and O. Bimber, “Coded Aperture Projection,” Proceedings of Emerging Display

Technologies and Immersive Projection Technologies 2008 (EDT IPT08), 2008.
17. O. Bimber and A. Emmerling, “Multifocal Projection: A Multiprojector Technique for In-
creasing Focal Depth,” IEEE Transactions on Visualization and Computer Graphics (TVCG),
Vol. 12, No. 4, pp. 658–667, 2006.
18. K. Fuji, M. Grossberg, and S. Nayar, “A Projector-Camera System with Real-Time Photomet-
ric Adaptation for Dynamic Environments,” Proceedings of IEEE Conference on Computer
Vision and Pattern Recognition (CVPR), Vol. I, pp. 814–821, 2005.
19. O. Bimber and D. Iwai, “Superimposing Dynamic Range,” ACM Transactions on Graphics
(ACM Siggraph Asia), Vol. 27, No. 5, article 150, 2008.
20. D. Cotting, M. Naef, M. H. Gross, and H. Fuchs, “Embedding Imperceptible Patterns into
Projected Images for Simultaneous Acquisition and Display,” Proceedings of IEEE/ACM In-
ternational Symposium on Mixed and Augmented Reality (ISMAR), pp. 100–109, 2004.
21. S. Zollmann and O. Bimber, “Imperceptible Calibration for Radiometric Compensation,” Pro-
ceedings of Eurographics, pp. 61–64, 2007.
22. O. Bimber, F. Coriand, A. Kleppe, E. Bruns, S. Zollmann, and T. Langlotz, “Superimposing
Pictorial Artwork with Projected Imagery,” IEEE MultiMedia, Vol. 12, No.1, pp. 16–26, 2005.
23. O. Bimber and R. Raskar, “Spatial Augmented Reality: Merging Real and Virtual Worlds,”
A K Peters LTD (publisher), ISBN: 1-56881-230-2, free e-book: www.SpatialAR.com,
July 2005.
24. A. Grundhoefer and O. Bimber, “VirtualStudio2Go: Digital Videocomposition for Real En-
vironments,” ACM Transactions on Graphics (ACM Siggraph Asia), Vol. 27, No. 5, article
151, 2008.
25. D. A. Bowman, E. Kruijff, J. J. Laviola, and I. Poupyrev, “3D User Interfaces: Theory and
Practice,” Addison-Wesley, 2005.
26. R. Raskar and K. L. Low, “Interacting with spatially augmented reality,” Proceedings of the 1st
international conference on Computer graphics, virtual reality and visualisation(AFRIGRAPH
’01), pp. 101–108, 2001.
27. A. Licsar and T. Sziranyi, “Hand gesture recognition in camera-projector system,” Proceedings
of the ECCV 2004 Workshop on HCI - Computer Vision in Human-Computer Interaction,
pp. 83–93, 2004.

28. R. Kjeldsen, C. Pinhanez, G. Pingali, J. Hartman, T. Levas, and M. Podlaseck, “Interacting
with Steerable Projected Displays,” Proceedings of the 5th International Conference on Auto-
matic Face and Gesture Recognition, pp. 12–17, 2002.
29. A. D. Wilson, “Playanywhere: a compact interactive tabletop projection-vision system,” Pro-
ceedings of the 18th annual ACM symposium on User interface software and technology
(UIST ’05), pp. 83–92, 2005.
30. Y. Wu, T. S. Huang, and N. Mathews, “Vision-based gesture recognition: A review,” Proceed-
ings of the International Workshop on Gesture-Based Communication in Human-Computer
Interaction, pp. 103–115, 1999.
31. G. Yahav, G. Iddan, and D. Mandelboum, “3d imaging camera for gaming application,” In-
ternational Conference on Consumer Electronics (ICCE 2007) - Digest of Technical Papers,
pp. 1–2, 2007.
32. D. R. Olsen and T. Nielsen, “Laser pointer interaction,” Proceedings of the SIGCHI conference
on Human factors in computing systems (CHI ’01), pp. 17–22, 2001.
21 Projector-Camera Systems in Entertainment and Art 495
33. B. Myers, R. Bhatnagar, J. Nichols, C. Peck, D. Kong, R. Miller, and A. Long, “Interacting at
a distance: measuring the performance of laser pointers and other devices,” Proceedings of the
SIGCHI conference on Human factors in computing systems (CHI ’02), pp. 33–40, 2002.
34. X. Cao and R. Balakrishnan, “Visionwand: interaction techniques for large displays using
a passive wand tracked in 3d,” Proceedings of the 16th annual ACM symposium on User
interface software and technology (UIST ’03), pp. 173–182, 2003.
35. M. Piccardi, “Background subtraction techniques: a review,” Proceedings of the IEEE Inter-
national Conference on Systems, Man and Cybernetics, Vol. 4, pp. 3099–3104, 2004.
36. T. Moeslund, A. Hilton, and V. Kr¨uger, “A survey of advances in vision-based human mo-
tion capture and analysis,” Computer Vision and Image Understanding, Vol. 104, No. 2–3,
pp. 90–126, 2006.
37. X. Cao and R. Balakrishnan, “Interacting with dynamically defined information spaces using
a handheld projector and a pen,” Proceedings of the 19th annual ACM symposium on User
interface software and technology (UIST ’06), pp. 225–234, 2006.
38. P. Beardsley, J. Van Baar, R. Raskar, and C. Forlines, “Interaction using a handheld projector,”

IEEE Computer Graphics and Applications, Vol. 25, No. 1, pp. 39–43, 2005.
39. C. Forlines, R. Balakrishnan, P. Beardsley, J. van Baar, and R. Raskar, “Zoom-and-pick:
facilitating visual zooming and precision pointing with interactive handheld projectors,” Pro-
ceedings of the 18th annual ACM symposium on User interface software and technology
(UIST ’05), pp. 73–82, 2005.
40. X. Cao, C. Forlines, and R. Balakrishnan, “Multi-user interaction using handheld projectors,”
Proceedings of the 20th annual ACM symposium on User interface software and technology
(UIST ’07), pp. 43–52, 2007.
41. H. Kato and M. Billinghurst, “Marker Tracking and HMD Calibration for a Video-based Aug-
mented Reality Conferencing System,” Proceedings of the 2nd IEEE and ACM International
Workshop on Augmented Reality, Vol. 99, pp. 85–94, 1999.
42. R. Raskar, P. Beardsley, J. van Baar, Y. Wang, P. Dietz, J. Lee, D. Leigh, and T. Willwacher,
“RFIG Lamps: interacting with a self-describing world via photosensing wireless tags and
projectors,” Proceedings of SIGGRAPH 2004, pp. 406–415, 2004.
43. S. Voida, M. Podlaseck, R. Kjeldsen, and C. Pinhanez, “A study on the manipulation of 2D ob-
jects in a projector/camera-based augmented reality environment,” Proceedings of the SIGCHI
conference on Human factors in computing systems, pp. 611–620, 2005.
44. M. Podlaseck, C. Pinhanez, N. Alvarado, M. Chan, and E. Dejesus, “On interfaces projected
onto real-world objects,” Proceedings of the SIGCHI Conference on Human Factors in Com-
puting Systems, pp. 802–803, 2003.
45. J. Underkoffler, B. Ullmer, and H. Ishii, “Emancipated pixels: real-world graphics in the lumi-
nous room,” Proceedings of the 26th annual conference on Computer graphics and interactive
techniques (SIGGRAPH ’99), pp. 385–392, 1999.
46. P. Dietz, R. Raskar, S. Booth, J. van Baar, K. Wittenburg, and B. Knep, “Multi-projectors and
implicit interaction in persuasive public displays,” Proceedings of the working conference on
Advanced visual interfaces, pp. 209–217, 2004.
47. A. Butz and A. Kr
¨
uger, “Applying the Peephole Metaphor in a Mixed-Reality Room,” IEEE
Computer Graphics and Applications, Vol. 26, No.1, pp. 56–63, 2006.

48. D. Cotting and M. Gross, “Interactive environment-aware display bubbles,” Proceedings of
the 19th annual ACM symposium on User interface software and technology (UIST ’06),
pp. 245–254, 2006.
49. G. Pingali, C. Pinhanez, T. Levas, R. Kjeldsen, and M. Podlaseck, “User-Following Displays,”
Proceedings of IEEE International Conference on Multimedia and Expo, 2002.
50. D. Kurz, F. Hantsch, M. Grobe, A. Schiewe, and O. Bimber, “Laser pointer tracking in
projector-augmented architectural environments,” Proceeding of the 6th IEEE and ACM In-
ternational Symposium on Mixed and Augmented Reality (ISMAR), pp.1–8, 2007.
51. R. Kjeldsen, A. Levas, and C. Pinhanez, “Dynamically reconfigurable vision-based user inter-
faces,” Machine Vision and Applications, Vol. 16, No. 1, pp. 6–12, 2004.
496 O. Bimber and X. Yang
52. R. Vertegaal and I. P. Poupyrev, “Introduction to Special Issue of Organic user interfaces,”
Communications of ACM, Vol. 51, No. 6, 2008.
53. J. C. Lee, S. E. Hudson, and E. Tse, “Foldable interactive displays,” Proceedings of the
21st annual ACM symposium on User interface software and technology (UIST ’08),
pp. 287–290, 2008.
54. M. Jin, H. Zhang, X. Yang, and S. Xiao, “A Real-Time ProCam System for Interaction with
Chinese Ink-and-Wash Cartoons,” Proceedings of IEEE International Workshop on Projector-
Camera Systems (ProCams), 2007.
55. Y. Yuan, X. Yang, and S. Xiao, “A Framework for Tangible User Interfaces within Projector-
based Mixed Reality,” Proceedings of the 6th IEEE and ACM International Symposium on
Mixed and Augmented Reality (ISMAR), 2007.
Chapter 22
Believable Characters
Magy Seif El-Nasr, Leslie Bishko, Veronica Zammitto, Michael Nixon,
Athanasios V. Vasiliakos, and Huaxin Wei
Introduction
The interactive entertainment industry is one of the fastest growing industries in
the world. In 1996, the U.S. entertainment software industry reported $2.6 billion
in sales revenue, this figure has more than tripled in 2007 yielding $9.5 billion in

revenues [1]. In addition, gamers, the target market for interactive entertainment
products, are now reaching beyond the traditional 8–34 year old male to include
women, Hispanics, and African Americans [2]. This trend has been observed in
several markets, including Japan, China, Korea, and India, who has just published
their first international AAA title (defined as high quality games with high budget),
a 3D third person action game: Ghajini – The Game [3].
The topic of believable characters is becoming a central issue when designing
and developing games for today’s game industry. While narrative and character were
considered secondary to game mechanics, games are currently evolving to integrate
characters, narrative, and drama as part of their design. One can see this pattern
through the emergence of games like Assassin’s Creed (published by Ubisoft 2008),
Hotel Dusk (published by Nintendo 2007), and Prince of Persia series (published
by Ubisoft), which emphasized character and narrative as part of their design.
M.S. El-Nasr (

)
School of Interactive Arts and Technology, Simon Fraser University, Vancouver, BC, Canada
e-mail:
L. Bishko
Department of Animation, Emily Carr University of Art and Design, Vancouver, BC, Canada
e-mail:
V. Zammitto, M. Nixon, and H. Wei
School of Interactive Arts and Technology, Simon Fraser University, Vancouver, BC, Canada
e-mail: ; ;
A.V. Vasiliakos
University of Peloponnese, Nauplion, Greece
e-mail:
B. Furht (ed.), Handbook of Multimedia for Digital Entertainment and Arts,
DOI 10.1007/978-0-387-89024-1 22,
c

Springer Science+Business Media, LLC 2009
497
498 M.S. El-Nasr et al.
Beyond the entertainment industry, the use of virtual environments for learning,
health therapy, cultural awareness, and training is increasingly becoming a reality.
In the recent years, there has been an increase in the number of research initiatives
that use simulations and interactive 3D environments for a wide variety of appli-
cations [4–11]. Several great examples are displayed in the projects developed by
Institute of Creative Technologies at University of Southern California, where they
utilize 3D environments with rich characters to teach cultural norms and foreign
language, among other subjects. These applications provide a safe and comfortable
environment for participants to interact within and learn at their own pace. In or-
der to achieve their goals, however, such applications require realistic simulation of
culture, people, and space. Thus, again the topic of believable characters is gaining
more attention as a central topic that deserves further attention.
Since the above mentioned applications are typically interactive, animated be-
lievable characters are often required to adapt based on the interaction. Current
industry methods, however, rely on heavy scripting, where voice acting, dialogue
scripts, hand-coded animation routines, and hard-coded behaviors are used to por-
tray the desired character; To mention a few examples of games that employ very
detailed motion-captured characters, readers are referred to Assassins’ Creed and
Prince of Persia (developed by Ubisoft) and Fac¸ade (developed by Mateas and
Stern [12]), see Figure 1. In these games, artists work very diligently to detail
characters’ mannerisms and body motion to exhibit the right character character-
istics [3]. Such attention to detail of the non-verbal behaviors is a crucial element
for character believability [4].
As one can guess, this kind of scripting is labor intensive and rigid, as it does not
adapt to all variations induced by interaction. An alternative is to use artificial intel-
ligent algorithms and graphics techniques to adapt character behaviors to variations
in context induced by interaction. This alternative, however, is not as simple as it

sounds, as it has been under research for many years and is still an open problem.
Researchers have been working on several fronts to create believable expressive
characters that can dynamically adapt within interactive narratives. Graphics re-
searchers, for example, explored the integration of emotions and personality as
parameters to modify virtual character animations [5–7]. Researchers working on
developing conversational agents focus on building articulate virtual characters
that can automatically synchronize gesture and speech [8]. Artificial intelligence
Fig. 1 Screenshots from games and interactive media featuring characters
22 Believable Characters 499
researchers focus on integrating models of emotion and personality to build
characters that have the ability to improvise [9–11].
As researchers tackle different aspects of this open problem, gaps between these
different directions start to appear. One important gap is the gap between character
models or attributes, such as personality, physical appearance, and emotions, and
how characters use nonverbal behaviour to portray these attributes. In order for a
character to adapt, it needs to not only be able to automatically select its motions
and execute its actions, but to also select nonverbal behaviours that convey and
maintain its attributes. Let’s take a character within a soccer game, for example.
Such a character is required to adapt and blend between different actions, such as
dribbling, diving, running, walking, scoring, yelling, and arguing with the referee,
to mention a few actions. This character should be able to select its actions based on
the current context and its own goals. In choosing how to blend between actions and
how to execute the animations, it needs to convey and maintain its own personality
and emotions. The important open problem here is how to enable characters to do
this without hand coding all different variations given all different contexts? This is
the central question and problem that this chapter deals with. The aim is to review
ongoing research that may provide readers with a good starting point to tackle this
problem.
To start discussing this problem, we will first define believability, or believable
characters. Believability is gauged by the extent to which a viewer engages and em-

pathizes with an animated character [13–15]. In the context of believability, it is
important to note the theory of the Uncanny Valley. Proposed in 1970 by Japanese
roboticist Masahiro Mori, the theory explains human reactions towards increasing
levels of realism to non-human entities, e.g., robots. The theory suggests that hu-
mans may develop feelings of repulsion or negativity towards non-human entities
as the level of realism increases. In recent years, the application of the Uncanny
Valley theory broadened to areas of animation. In the context of this chapter, we
look towards believability over realism, focusing on broader concepts of nonverbal
communication that may contribute to future efforts to solve the Uncanny Valley
problem.
Character believability can be approached from several perspectives, including
personality theories, movement theories, emotion and cognitive theories. Each of
these perspectives have been studied in different fields of inquiry, including psy-
chology, kinesiology, animation, and acting. This chapter attempts to discuss many
of these perspectives. However, due to space limitations several notable works are
left out, such as the narrative and literature perspective.
In this chapter we study two concepts: non-verbal behavior and their relation to
character attributes. We define non-verbal behavior as a single or pattern of move-
ments and postures that are exhibited in the body, such as hand movement, or leg
movement. For example, the motion of quickly glancing at a character then at the
ground is considered a non-verbal behavior pattern. We use the terms character at-
tributes, character characteristics, and character model to mean a list of parameters
that define a character, including age, physique, personality, behavior tendencies,
quirks, and habits.
500 M.S. El-Nasr et al.
We specifically explore believable character through personality models
(e.g., Factor theories from psychology [16, 17]), nonverbal character behavior
models (e.g., Ekman’s Facial experession [18]), motion analysis models (e.g.,
Laban Movement Analysis [19, 20]), improvisational theatre character models
(e.g., Johnstone’s impro [21, 22]) and animation techniques, including Disney’s an-

imation methods [13]. In discussing these models, we will discuss theories as well
as practical applications of these models within the computer science or believable
agents field. We will conclude by discussing the current state of the art of interactive
believable characters, identifying open problems that need to be addressed in order
for the field to move forward.
We specifically explore believable character through personality models such
as Factor theories from psychology [16, 17], nonverbal character behavior models
such as Ekman’s Facial experession [18], motion analysis models such as Laban
Movement Analysis [19, 20], improvisational theatre character models such as
Johnstone’s impro [21, 22] and animation techniques, including Disney’s animation
methods [13]. In discussing these models, we will discuss theories as well as prac-
tical applications of these models within the computer science or believable agents
field. We will conclude by discussing the current state of the art of interactive be-
lievable characters, identifying open problems that need to be addressed in order for
the field to move forward.
Character Personality
The term personality has its origin in the Ancient Greek literature; it comes from
the role played by actors who wore a mask and read aloud the script from their
characters’ scroll. Character as a synonym of personality comes from this same
theatrical origin; the word persona was used for mask. Personality is a psychological
concept that has been widely used inside and outside this field. Aiken [23] (page xi)
describes that:
The term personality refers to the organized totality of the qualities, traits, and behaviors
that characterize a person’s individuality and by which, together with his or her physical
attributes, the person is recognized as unique.
This definition reached consensus. However, Aiken [23] also warns that this defi-
nition is open and abstract, and hence difficult to operationalize. Different theories
define personality with different terms and emphasize certain characteristics on top
of others. Nevertheless, all of them agree that uniqueness is a key of personality,
and that personality is a combination of variables that composes a unique pattern of

behavior.
Personality is an important concept that is related to character believability. Char-
acters seen in movies, theatre productions, animations, and video games all inhabit
a particular personality. This personality is what makes them distinct and memo-
rable. We expect characters to have personality and subject such personality through
their goals, behaviors, and expressions. But what is personality and how can it be
22 Believable Characters 501
operationalized for computational representation of an adaptive believable charac-
ter? In this section, we will look at previous work from psychology and theatre to
try to conceptualize an appropriate computational representation. We will also dis-
cuss previous computational models of personality within fields such as graphics
and artificial intelligence.
Through the literature, two theoretical perspectives of personality are assumed:
nomothetic and idiographic.Thenomothetic perspective approaches personality
from a general perspective trying to compose personality models which can ex-
plain all different types of personalities. The idiographic perspective emphasizes the
uniqueness of personality structure with the belief that every person has his own sys-
tem, such as in phenomenological theories [24]. In this section, we discuss different
personality models that fall within the nomothetic perspective, because we believe
this approach is most appropriate for the development of computational models.
Body Type Theories
Theories on body-types have typically collected their data from physiological char-
acteristics and made assumptions based on generalizations. Since there are many
exceptions to their classifications, these theories do not enjoy academic popularity,
yet they could offer physical cues for modeling characters. Major body-type re-
searchers and theorists were criminologists, such as Cesare Lombroso [25], Ernst
Kretschmer [26], and William Sheldon and S. S. Stevens [27].
Lombroso studied the body constitution of criminals. He stated that their physio-
logical development is in a lower stage, and that their physiognomy is different from
other people. Lombroso presumed that there was a born-criminal conception; how-

ever, since many other criminals don’t possess such features, his idea didn’t receive
much popularity. Repeated identified atavistic characteristics were: large jaws, high
cheekbones, receding foreheads, handle-shape ears, long arms, and other primitive
physical traits [25].
Ernst Kretschmer [26] created the first scientific theory describing personality
types based on body build. However, his research had low validity and applica-
tions of it were relegated. He collected different data measuring bodily constitution
of mental patients and others, which he used to formulate four categorizations of
body types:
1) Asthenic or Leptosomic physique: tall, thin, lanky, angular body build. People
in this type are characterized as introverts, withdrawing behavior, and ‘schizoid
temperament’. This type of mental patient has schizophrenic symptoms.
2) Pyknic physique: round, stocky body build. People in this type are associated
with emotional instability. Mental patients with this body complex develop bipo-
lar disorder (manic-depressive).
3) Athletic physique: broad shoulders and slim hips. People in this type are prone
to develop either a maniac-depressive disorder or schizophrenia.
502 M.S. El-Nasr et al.
Fig. 2 Sheldom’s body types
4) Dysplastic physique: any other body build that does not fit into any of the three
other categories.
Similar to Kretschmer, Sheldon and Stevens [27] developed a quantitative classifi-
cation of personality along three dimensions of body types: Endomorphy (fatness),
mesomorphy (muscularity), and ectomorphy (thinness), where each dimension was
defined on a score from 1 to 7. For example, a person with 7 for endomorphy,1for
mesomorphy, and 1 for ectomorphy represents an extremely fat person who lacks
other characteristics (see Figure 2). Their classification method is fuzzy as it is hard
to score measures such as fatness or thinness. They found correlations between high
scores of body types to temperament types which also ranged in a 7-point scale. The
temperament types are:

1) Viscerotonics: associated to the endomorphy type. This temperament is
characterized by being gregarious, friendly, and the enjoyment of comfort
and eating.
2) Somatotonics: correlated to the mesomorphy type. This temperament is charac-
terized by enjoying physical exercise, dominance, being loud and assertive, but
not empathetic.
3) Cerebrotonics: linked to ectomorphy. This temperament is characterized by be-
ing quiet and reserved, reacting quickly, but over-sensitive to pain and have
sleeping difficulties.
22 Believable Characters 503
Body type theories are interesting to explore further for believable characters. They
represent an important construct for character designs, concepts, and development.
However, we see this type of approach to be in the hand of a designer rather than a
decision of an adaptive system. The development of this approach could be a useful
tool for designers as they develop their own characters, as well as a method for
training character design artists.
Psychodynamic Theories
Freud, known for pioneering the psychoanalysis approach to therapy, developed a
theory of personality where each individual was described to have three levels of
awareness: unconscious, preconscious and conscious [28]. Personality was then cat-
egorized as three entities: id, ego, and superego. The id is full of animal instincts
and operates by prioritizing pleasure satisfaction, but is purely unconscious. The
ego mediates between the id, the superego, and the external world by evaluating the
consequences of actions. The superego is formed by the mandates that have been in-
ternalized, and the ideal image of oneself. The ego and superego have unconscious,
preconscious, and conscious levels. According to Freud the human personality is a
struggle of power among id, ego, and superego.
Freud also asserts that personality is shaped through the progress of the psycho-
sexual stages [23, 29]. During each stage a part of the body is the primary source
of satisfaction and psychoenergetical (libidinal) stimulation. The first stage is oral,

from birth to eighteen months, in which the mouth brings gratification as in eat-
ing and sucking. The second stage is anal, from the eighteen months to three years
old, in which the anus provides satisfaction by retaining and defecating, and sphinc-
ter control is achieved. The third stage is phallic, from three to six years old, in
which genitals are the focus in a rudimentary and egocentric way. The fourth stage
is latency, from six to twelve years old, and is characterized by a decrease in the con-
centration of the genital organs. Finally is the genital stage, from the age of twelve,
when genitals are the center of gratification, sexual instincts are fully developed,
and sexual maturity is achieved. However, the libidinal energy that flows might get
stuck in certain psychosexual stages. This fixation can be the result of traumatic
or stressful experiences; it might also lead to regression of behaviors. For instance,
fixation in the oral stage could be the cause of verbosity, gorging, smoking, nail
biting, sarcasm or hostility. These characteristics originate from gratification of the
mouth and lips through sucking, eating, and chewing. Whereas fixation in the anal
stage can develop into an obsessive-compulsive personality, stubbornness, stingi-
ness, stuttering, and petulance, having to do with the satisfaction of eliminating and
controlling feces.
There are a lot of different theories that emerged based on psycho-analysis. One
such research was the work of Carl Jung [29–31]. Jung detached from Freud’s work;
he analyzed the personality through different ways of orientation towards the ex-
ternal world. He identified two attitudes and four functions of thoughts shown in