Handbook of Multimedia for Digital Entertainment and Arts- P24 ppt

31 SoundScapes/Artabilitation – Evolution of a Hybrid Human Performance 695
resonance. This aesthetic is an inner resonance recognized from achieved user-
experience. Having experienced a vocation as a stage performer I relate to emotions
achieved through a satisfying entertainment of an audience. Thus, in the therapy
the user entertains him or her-self through the mediating interactive digital media
system. This can be from controlling a game or creating art, i.e. music making or
painting via digital tools, and is evident as FUN resulting from the created situ-
ations. Facilitator knowledge of the potentials of the digital media is essential to
user-experience, successful guidance (targeted as via mediating technology), and
user self-determined entertainment.
The same system that was created specific for the rehabilitation training was
used by me to perform abstract expressionistic stage art and interactive installations
at various large exhibitions. These include ongoing tours in Museums of Modern
Art as well as at International-National Cultural events (e. g., Danish NeWave, New
York 1999; Cultural Olympiad 1996, 2000; and European Cultural Capital events
1996, 2000). Underground events were as informative to the concept as the larger
venues (see next section). Insight was gained from these situations. By combining
self-reflection with collaborative inter-/multi-disciplinary reflective analysis of the
human performance situations a significant input to designing the system, facilitat-
ing sessions, and evaluating outcomes evolved.
Underground Non-formal Learning
An example of how learning was apparent from the entertainment and arts is where
I directed, produced and performed in a one-man-show realized as the inaugural
Aarhus Festival Fringe (‘Festuge’). This 1999 event took place in an emptied storage
rooms that were adjacent to my sizable interactive room installation titled Circle of
Interactive Light (COIL), hosted by the Radisson SAS Scandinavia Hotel, Aarhus,
Denmark. The Festival committee had declined my proposal to bring together East
and West German artists to feature alongside North and South Korean artists as too
politically sensitive. The festival theme was ‘THE WALL, Ten Years After’ – and
my proposal involved artists of distinction from both sides of the Berlin Wall and
the DMZ to discus and showcase how political issues affect art and artists lives.

Understandable that the committee declined!
The eventual “FESTUGE FRINGE” protest performance featured my digital in-
teractive media in the form of the motion sensitive system utilized in various ways
with projections inside and outside the human body. The final section of the perfor-
mance was where I used fourteen infrared sensors to control one sound and a library
of image manipulations. The purpose was to explore non-control and subliminal
performance by enabling the feedback to control me and this was done by mapping
the data to maximum parameter control of a specific aspect of the sound envelope.
In four of the twelve performances I achieved what I refer to as non-control with
subliminal interaction. This was where I, as performer, experienced being as one
with the feedback stimuli. If conscious intent or query was involved, the experience
696 A.L. Brooks
and higher-state of interaction (or inbetweenness – see Kidd) was lost. I relate this
targeted non-control and interaction to the therapy work with profoundly disabled
where often the user’s abilities are of a non-controlled state so my learning experi-
ences as performer/user assist by informing my role as facilitator.
Following the performances the wall would open to exhibit the technology used,
and then audience walked through the wall into the COIL exhibition space where
they could experience the virtual interactive spaces and debate over a drink in the
caf
´
e I had built for the occasion. The piece titled “Behind the Wall” targeted beyond
Berlin and DMZ issues as it reflected on human barriers across ages and cultures –
and especially committees.
The next section introduces the ArtAbilitation
2
workshops from 2007 and 2008.
The ArtAbilitation movement has evolved from the SoundScapes work. It is where
digital interactive media is used to create an entertaining user-experience.
ArtAbilitation Workshops, Casa da Musica, Porto, Portugal

The “Ao Alcance de Todos - Within Everyone’s Reach” festival was hosted at Casa
da M
´
usica (Figure 13), Porto, Portugal in April 2008. Eight one hour workshops
were held with 144 disabled children and adults attending with caregivers. The
workshops were created in a room 238 square meters floor area and approximately
20 meters high. Additionally, a symposium for professionals was hosted immedi-
ately following the workshops for 35 professionals (international, national, regional,
and local // social workers, psychologists, researchers, teachers, and students :::)
- many of whom had attended the workshops. This event was a continuation of
the previous year’s inaugural event where the theme was ‘Music, Technology and
Disability’, and where six workshops were attended by 91 attendees including 61
from special care institutes, of which 39 had profound disability. 30 student music
teachers also attended. A local crew assists in the realization of the workshops (see
Petersson and Brooks 2007; Brooks 2008).
This section does not detail the workshops but rather it exemplifies the design
issues where digital interactive media, entertainment and the arts are combined to
result in various user-experiences that inform the ongoing research and refinement
of the acknowledged international ArtAbilitation movement (Wiederhold 2007).
I begin by giving an overview of commonalities in designing both workshops.
The 2007 workshop is then detailed in more depth followed by the 2008 workshop.
Both workshop designs targeted participant learning from interactive experiences
where active participation formed a context for meaning. Offered were opportuni-
ties for augmenting learning and awareness. Observations were of self-determined
active participation that established goals that in turn engaged interest, curiosity
and play. In this way, motivation to achieve goals is considered intrinsic rather than
extrinsic, which can result from an activity, task or goal introduced by someone else,
2

31 SoundScapes/Artabilitation – Evolution of a Hybrid Human Performance 697

Fig. 13 Casa da Musica, Porto, Portugal (photo with permission Casa da Musica)
and or with possibly over zealous guidance/facilitation. Contextual design thus be-
came a way to configure learning resources and interaction (Kress & van Leeuwen,
2001). Exploration, play and transformation were also targeted in both events such
that each participant had an ‘action-stillness’ profile that evolved through inter-
actions. Each cycle of activity was considered a new creation that contributed to
form patterning of actions resulting from the activity. Each activity was designed
to increment challenges of confrontation. The resultant ‘play’ and created ‘creativ-
ity’ scenarios involved manipulation of provided tools. The manipulation required
a degree of competence that was learnt through exploration of the tool’s traits
(Bruner 1972).
Absence of negative consequences seemingly encouraged participant free ex-
pressive explorations, which in turn, can over time result in development of un-
employed skill (Beach, 1942). Such development is significant even if it is at a
micro scale, i.e. micro-development (Yan and Fischer 2002). Responsive environ-
ment composition and learning process/outcome evolve as the assessment focus
instrument of observers. Realized was new learning spaces and approaches to learn-
ing and rehabilitation through emphasizing user’s creation of meaning via serious
play. The approach is that no aspect of the learning process and outcome is taken for
granted. Rather it is formulated into play and creative activities that are inherent to
e.g., games and art making. The activities of play and creativity conceal the embed-
ded learning and training involved for the user. An emphasis is on a supplementing
tool for traditional practices rather than a replacement. In this sense, learning is at a
‘subliminal’ level for the user as he or she engages in the responsive environment.
Thus, motivation is optimized through action and stillness cycles where the user
698 A.L. Brooks
Fig. 14 Sala de Ensaio 1, Casa da M´usica, Oporto, Portugal (C Dcamera) – (Brooks 2008)
iteratively explores and transforms the feedback media. This process contains
choices and decisions that indicate learning, e.g. in the form of increased repertoire
of expressions, changes of skills, and new patterns of social interaction (Brooks

2008).
In the 2007 workshop a private (2) and a public (1) area were designed – see
Sala de Ensaio diagram (Figure 14). This was inspired from the evaluation consul-
tations mentioned earlier in this chapter where a caregiver experienced that a private
space, without any intervention, was desiredbefore a user felt comfortable enough to
express through the SoundScapes system. My previous research (Brooks et al. 2002)
presented how certain individuals prefer to explore, play and create without any oth-
ers being present (Figures 15–16). Exemplifying the SoundScapes open system and
concept a digital video camera was used to create a feedback loop play-space where
participant gesture distorted image silhouette and color change to a RGB lighting
system that reacted to voice input. Interactions in the private space were video
taped and analyzed as achieving a positive flow state (Csikszentmih
´
alyi 1996). It
was observed in the private space that the chain of exploration-play-creation began
with a curiosity that evolved out of the isolation and initial stillness that was first
encountered within the created environment. Thereby, stillness became part of the
action and vice versa. Indications pointed to how interactive play and creativity that
offered choices between interaction and rest in a silent space enhanced the sense of
control.
A created public space (Figures 17–18) questioned participant perception and
awareness where peer-support and scaffolding of exploration was evident. Over-
head infrared camera tracking was mapped to auditory music making (Figure 17)
31 SoundScapes/Artabilitation – Evolution of a Hybrid Human Performance 699
Figs. 15-16 Private space images of total engagement from ArtAbilitation 2007, (photo credit
Augusto Brazio, Casa da Musica)
and image effects (Figure 18). Many results occurred from this space. One memo-
rable instance was when a woman who was profoundly physically disabled insisted
in exploring the interactive space out of her wheelchair – evident was that crawling
around required immense effort (Figure 18). Her motions were tracked to open

up a digital mask that concealed an image of a famous Portuguese footballer. Her
disabled colleagues were supporting by shouting instructions. Motivation was stag-
gering to achieve the whole erosion of the mask. Another instance was where an
autistic group was in the workshop and one of them began to move and open up the
same mask. The first exposed facial element of the footballer was the chin and im-
mediately another autistic male shouted out the name of the recognized footballer.
In both instances (and others) the public virtual interactive space empowered imme-
diate learning to be exhibited (Petersson and Brooks 2007).
Five physical ‘VIS rooms’ were created in the 2008 workshop. Each was de-
signed to offer differing experiences of empowerment through motion creating
images and auditory feedback. Large sub-woofer speakers positioned on the wooden
floor gave tactile stimulation which was especially important for the attending class
of young deaf children. Findings were that the interactive spaces offered a place
where comprehension of tasks could be shared from the strong to the weak in each
group. Again wheelchairs were discarded to explore by crawling around the spaces.
700 A.L. Brooks
Figs. 17-18 Public interactive space: two designs utilizing overhead camera tracking
Targeted tangible outcome was that each participant received a painting that they
created through gesture. Reports of how many showed these paintings to family,
friends and non-attending staff stated to the powerful effect.
Both workshops exhibited action-stillness cycles, which consisted of iterative
loops of exploring and transforming, constitute one part of a theoretical map for
the purposes of analyzing critical incidents in a non-formal learning process. These
cycles are related to the user’s learning experience. The other part of the theoretical
map concerns design issues in the form of use qualities relative to the user’s interac-
tive experience; transparency, social-action space, user control/autonomy, pliability,
playability and seductivity (L
¨
owgren and Stolterman 2005). During a whole session
31 SoundScapes/Artabilitation – Evolution of a Hybrid Human Performance 701

cycle of action and stillness, facilitators have the possibility to reflect upon the in-
dications of learning that occur during the process. The user profile influences the
facilitator’s decisions on how to set up the attributes of the responsive environments
relative to the desired learning process and the expected outcome of that process.
Once these prerequisites are set, the user is expected to ‘experience from the inter-
action and learn from the experience’.
Hence, the aim of the workshop events was to investigate the user’s performance
in using responsive environments designed to engage participants in experience-
learning through action and stillness cycles. A sub-question concerns the ability of
perception and the associated learning curve of the attendees with cognitive disabil-
ities to be able to easily correlate across dimensions of scale and plane – a matter
which influences the participant’s interactive as well as learning experience.
The final reported event is one that was built upon previous work where third
party performance gesture created visuals to complement classical music situations.
Visualizing classical music
The previous sections present the motion-sensitive environment and its use as a re-
habilitation training supplement within the community of people with a disability,
both profound from birth, and acquired. The use of the created interactive system is
presented in intimate institute settings for individuals and a later section shares two
examples of larger workshop situations where groups attended. This section exem-
plifies how a similar concept and technique of using digital interactive media was
implemented to elicit dynamic performance data from a situation that featured the
‘Orquestra Nacional do Porto’ - regarded by many as The Portuguese National Or-
chestra. This was again in Casa da Musica, Porto, Portugal but this time in the main
1200 seat state-of-the-art auditorium. See publication Interpretation
3
(Brooks 2008).
A goal of the exploratory study was to dynamically complement the music by
offering an experience of inter/multi-sensory stimulation for both audience and per-
formers. This built upon my earlier work in Auckland, New Zealand, and Aarhus,

Denmark where the different situations, one with orchestra and one with choir, both
had a similar stimulation goal. An aim of this work is to offer inclusive access so
that even people who are deaf may have an opportunity to appreciate classical mu-
sic. Important to mention is how in the choir study three sensors mapped to RGB
enabled the conductor to paint the choir backdrop through his gestured interactions
with the singers. The session was in the Danish Radio TV studio and six ‘takes’
were recorded. It was evident that his gestures were expanding as he became used
to the interaction and – in his own words - “how it felt that the air around my body
was activated”. This expansion of gesture relates to the work in the disability sector
where targeted in subliminal motivated augmented motion.
The following Figs.19–26 illustrate the result from the visualization experiment.
Overhead cameras and stage sensors sourced the performance data and this was
3
/>702 A.L. Brooks
Fig. 19 Data from stage performance – conductor gesture, section/musician expressiveness, lights
and music - sourced to dynamically affect the synchronized visuals for environment change in real-
time (photo credit Jo˜ao Messias, Casa da Musica)
Fig. 20 Sourced stage data mixed and matched to digitally mirror performance in abstract – here
camera and music stand lights - creating and effecting projected visuals (photo credit Jo˜ao Messias,
Casa da Musica)
31 SoundScapes/Artabilitation – Evolution of a Hybrid Human Performance 703
Fig. 21 Projected moving image dynamically matched to digitally mirror performance (photo
credit Jo
˜
ao Messias, Casa da Musica)
mapped by created algorithms to effect the environment variables in real-time to
complement the music. A technical deficit proved the biggest hurdle and results
were unsatisfactory as image mixing was via switching instead of faders on a mixer.
Thus, nuance of performance was unable to be matched as I desired. However, as
an experiment to explore the concept it did build my knowledge of the concept. The

work is planned to evolve further locally for the next phase of experiments due to
the problems of accessing the specialized equipment required for technical setup.
The work has been accepted for the Cultural Olympiad 2012 London.
704 A.L. Brooks
Fig. 22 Projected image (center of screen) shows data capture from camera overhead (photo credit
Jo
˜
ao Messias, Casa da Musica)
Conclusions and Future Directions
This article informs of digital interactive media sculpted into a responsive ges-
ture driven environment that is used within rehabilitation training. Entertainment
achieved through a user creating and playing acts as a stimulant towards augmenting
life quality and motivating participation in training that otherwise is boring, tedious
and mundane – often for both patient and therapist/facilitator. Artistic composition
in the form of music-making and digital painting is empowered through an interface
and multi-sensory virtual environment content that is adaptive and flexible enough
to address idiosyncratic needs. However, whilst I create for another’s creativity, my
31 SoundScapes/Artabilitation – Evolution of a Hybrid Human Performance 705
Fig. 23 Musicians can observe effect of their own performance (photo credit Jo
˜
ao Messias, Casa
da Musica)
art is not the ‘product’ of the users’ interaction, i.e. the music or painting, as many
judge, but rather it is the aesthetic resonance that is achieved in and of the user. El-
lis (1996, 2004) describes aesthetic resonance achieved within his ‘Sound Therapy’
body of research. This was extemporized in a 2002 article (Brooks et al. 2002)fol-
lowing findings that explored interactive visuals in the form of animations, virtual
reality, and robotic lighting devices with colored and patterned gobos. This aesthetic
is a representation of a system where affordances were perceived and the targeted
action achieved. Resultant sense of self-agency and inner empowerment is realized

cumulating to signify a state of flow that is identified by the facilitator. Investiga-
tion is of how to optimize user experience toward self-determined development via
the virtual interactive space (VIS) that is created between the user and the system.
706 A.L. Brooks
Fig. 24 Conductor engaged in role as he manipulates responsive image artifacts (photo credit Jo
˜
ao
Messias, Casa da Musica)
Fig. 25 Selective use of dynamic shadow effects are merged into environment (photo credit Jo
˜
ao
Messias, Casa da Musica)
Potential micro-development (Yan and Fischer 2002) is a suggested potential even
for those who are profoundly impaired. Problematic on the other hand is where
a user is fixated on formal therapeutic conditioning and gets tied up in the ‘cor-
31 SoundScapes/Artabilitation – Evolution of a Hybrid Human Performance 707
Fig. 26 View from behind the control station where author interacts (photo credit Paul Sharkey)
rect way of doing’ instead of ‘just doing’ with a focus on the mediating feedback
content rather than on the impaired feed-forward function. User achievements are
commonplace.
The activities of exploring interactive systems in the field of disability are seem-
ingly growing. In Denmark commercial suppliers that sell targeted games for this
market attempt a monopoly with substandard goods and services that tend to dimin-
ish opportunities for end users rather than increase. After two decades of research
it is clear that open flexible and affordable systems that have selectable libraries of
input device as well as selectable libraries of content stimuli are optimal. Systems
that can be adapted, and personalized within customized environments tailored to
708 A.L. Brooks
an individual profile offer maximum opportunities and benefit. That benefit is not
restricted to those with impairment as fun social interactions can motivate in playing

games or making art.
Interface technologies have improved and motion detection devices such as the
Nintendo Wii offer a data device that permits access to data and thus can advance the
field in an affordable flexible package – either as a controller for games or music/art
making. An approach I made to Sony Computer Entertainment Europe (SCEE),
London, after the launch of the EyeToy asking such access to interface data was
rejected. Their representatives stated discriminatingly that they did not wish to be
seen exploiting disabled people. With such technology available it is a shame that
such a commercial profile prevents opportunity inquiry. However, computer vision
and graphical card advances makes affordable non-commercial sensor, single or
multiple camera systems, a reality and our students in MEDIALOGY
4
in Esbjerg
are active exploring both user-centered games and art as controlled mediums in the
SensoramaLab VR complex.
Such diversity in the SoundScapes concept is recognized by Eaglestone &
Bamidis (2008) as elements of a hybrid system consisting of networks of complex
inter-connected subsystems comprising ‘created technical systems’ and ‘natural hu-
man systems’. The MAX software has also undergone huge improvements since I
first used it as version 2.5 in the early 1990s. Integrated capability to manipulate
digital audio signals in real-time from a computer became a reality via the MSP
(Max Signal Processing) add-on, thus there was no longer a requirement for exter-
nal audio hardware. Subsequently another package developed and released for the
MAX software environment in 2003 was Jitter, which enabled control of real-time
video, 3D and matrix processing. Latest MAX activities is a partnering with a loop-
based sequencer software named Ableton Live that offers intuitive live performance
capabilities where MIDI manipulations can slot into a playing song such that it re-
mains ‘musical’. Whilst developed for DJs and VJs the potentials in my research
for such a tool is acknowledged and this is a future direction when the packages
are released. Other explored software includes Touch Designer by Derivative, and

Syntetik’s awesome Studio Artist interfaced with a Wacom Cintiq tablet.
In this sense a tool to supplement traditional training becomes available if the
community of therapists and medical staff are prepared to accept an alternative ap-
proach in the form of a hybrid non-formal learning apparatus and method. If they
do and are prepared to collaborate in evolving the concept with other interested
researchers from the related disciplines then the next-generation of therapists may
well be teamed with digital artists, computer scientists, and fMRI neurologists in
creating, facilitating, and evaluating the human in new ways, with new methods, and
with new apparatus, where FUN is the targeted user-experience and human benefit
the ultimate goal. New virtual interactive spaces for learning are exhibited in this
work that combines digital interactive media with entertainment and the arts.
4

31 SoundScapes/Artabilitation – Evolution of a Hybrid Human Performance 709
References
1. K. E. Misulis. “Essentials of Clinical Neurophysiology.” Butterworth-Heinemann, 1997.
2. A. Lucier. “Statement On:luciermisulis Music for Solo Performer.” In: D. Rosenboom (Ed.),
Biofeedback and the Arts, Results of Early Experiments. Aesthetic Research Center of Canada
Publications, 1976.
3. D. Warner. “Rock ’n roll Science:warner Playing the Body Electric” [online] Available from
/>4. A. L. Brooks, A. Camurri, N. Canagarajah and S. Hasselblad. “Interaction with shapes and
sounds as a therapy for special needs and rehabilitation.” In:brooks International Conference
on Disability, Virtual Reality and Associated Technologies (4th). Veszpr´em, Hungary, 18–20
September. P. Sharkey, C. S. L
´
anyi, and P. Standen, (Eds.). Reading: University of Reading,
pp. 205–212, 2002.
5. A. L. Brooks. Robotic Synchronized to Human Gesture as a Virtual Coach in (Re)habilitation
Therapy. In: 3rd International Workshop on Virtual Rehabilitation (IWVR2004), VRlab,
EPFL, Lausanne, Switzerland, pp. 17–16, 2004.

6. E. Petersson and A. Brooks. Non-formal therapy and learning potentials through human ges-
ture synchronized to robotic gesture. International Journal Universal Access in the Information
Society. Springer 6(2), pp. 167–177, 2007.
7. W. M. Marston. Emotions of Normal People, Routledge, 1928.
8. A. L. Brooks. SoundScapes: A concept of Virtual Interactive Space (V.I.S.) [unpub-
lished]. World Summit for Social Development/NGO Forum 6–12 March, Holmen,
Copenhagen, 1995.
9. A. L. Brooks. Virtual interactive space (V.I.S.) as a movement capture interface tool
giving multimedia feedback for treatment and analysis. In: “Bridging Cultures” – pro-
gram of The 13th International Congress of the World Confederation for Physical Ther-
apy, Yokohama Japan, May 23–28, Science Links Japan: />200110/000020011001A0418015.php, 1999.
10. A. L. Brooks. Virtual interactive space (V.I.S.). In: ‘Pushing the limits: optimising poten-
tial through science and technology’, Congress Program and Abstract Book, 5th Scientific
Congress, Sydney 2000 Paralympic Games, Oct. 11–13, Convention Centre, Darling Harbor,
Sydney, Australia, 2000.
11. A. L. Brooks. HUMANICS 1 – a feasibility study to create a home internet based telehealth
product to supplement acquired brain injury therapy. In: International Conference on Dis-
ability, Virtual Reality and Associated Technologies (5th). Oxford University, UK, 20–22
September 2004. Sharkey, P., McCrindle, R. and Brown, D. (eds.) Reading: University of
Reading, pp. 43–50, 2004.
12. A. L. Brooks and E. Petersson. Recursive reflection and learning in raw data video analysis
of interactive ‘play’ environments for special needs health care. In: Proceedings of 7th Inter-
national Workshop on Enterprise networking and Computing in Healthcare Industry, Korea
HEALTHCOM 2005, IEEE Signal Processing Society, USA, pp. 83–87, 2005.
13. A. L. Brooks and S. Hasselblad. Creating aesthetically resonant environments for the hand-
icapped, elderly and rehabilitation: Sweden. Proceedings of 6th International Conference on
Disability, Virtual Reality and Associated Technologies (ICDVRAT), Esbjerg, Denmark, 18th–
20th September, pp. 191–198, 2004.
14. A. L. Brooks. Enhanced Gesture Capture in Virtual Interactive Space. Computers in Art, De-
sign, and Education (CADE), 29 June–01 July, Copenhagen Business School, Denmark and

Malm¨o University, Sweden, 2004.
15. A. L. Brooks and E. Petersson. Stillness design attributes in non-formal rehabilitation:
CADE2007 - Computers in Art Design and Education. Perth, Curtin University of Technology,
pp. 36–44, 2007.
710 A.L. Brooks
16. G. Goldkuhl and P. J.
˚
Agerfalk. “Actability: A way to understand information systems prag-
matics.” CMTO Research papers No. 2000:13, Link
¨
oping University. Presented at the 3
rd
International Workshop on Organisational Semiotics, 4 July 2000, Stafford, UK, 2000.
17. P. J.
˚
Agerfalk and S. Cronholm. “Usabilityversus Actability: A Conceptual Comparative Anal-
ysis.” Presented at the HCI International conference New Orleans, USA, Lawrence Erlbaum
and Associates, pp. 235–237, 2001.
18. A. N. Leont’ev. “Activity, consciousness and personality.” NJ: Prentice-Hall, Englewood
Cliffs, 1981.
19. R. Davies. “Commentary on P. J. Standen and D. J. Brown – Virtual Reality in Rehabilitation
of People with Intellectual Disabilities: Review.” CyberPsychology and Behavior, 8(3), 2005.
20. A. L. Brooks and E. Petersson. “Play Therapy Utilizing the Sony EyeToy
r
.” In: Annual
International Workshop on Presence (8th), London, 21–23 September 2005. Slater, M. (ed.)
London: University College London, pp. 303–314, 2005.
21. C. Hummels, T. Djajadiningrapt, and K. Overbeeke. “Knowing, doing and feeling:
Communicating with your digital products. Interdisziplin
¨

ares Kolleg Kognitions und Neu-
rowissenschaften, G¨unne am M¨ohnesee, March 2–9, pp. 289–308, 2001.
22. B. Buxton. “Sketching User Experiences: getting the design right and the right design,”
Elsevier, 2007.
23. M. Polanyi. “Personal Knowledge: Towards a Post-Critical Philosophy,” Routledge,
2002 [1958].
24. R. Wollheim. “Art and Its Objects,” 2nd ed., Cambridge University press, 1980.
25. P. Ellis. “Layered analysis: A video-based qualitative research tool to support the development
of a new approach for children with special needs.” The Bulletin for the Council for Research
in Music Education. University of Illinois at Urbana-Champaign, USA, 130, pp.65–74, 1996.
26. P. Ellis. “Caress – ‘an endearing touch’.” In: J. Siraj-Blatchford (ed.) Developing New Tech-
nologies for Young Children. London: Trentham Books, pp. 113–137, 2004.
27. A. Pollini, Experimenting with an Ubiquitous Computing Open Architecture, [PhD Thesis]
2008
28. E. Petersson and A. L. Brooks. ArtAbilitation
r
: An Interactive Installation for the
Study of Action and Stillness Cycles in Responsive Environments. Computers in Art,
Design, and Education (CADE 2007) />cade/pdf/CADE2007Conferenc eprogram&abstracts.pdf, 2007.
38. A. L. Brooks. Towards a platform of alternative and adaptive interactive systems for idiosyn-
cratic special needs, Proc. 7th Intl Conf. on Disability, Virtual Reality and Assoc. Technologies
with ArtAbilitation, pp 319–326, Maia, Portugal, 8–11 Sept. 2008.
30. B. K. Wiederhold. Virtual Healers, International Association for CyberTherapy & Rehabilita-
tion, 2007.
31. G. Kress and T. van Leeuwen. “Reading Images. The Grammar of Visual Design.” London,
Routledge, 1996.
32. J. S. Bruner. “Nature and Uses of Immaturity.” In: J. S. Bruner, A. Jolly and K. Sylva (eds.),
Play – Its Role in Development and Evolution. Basic Books, 1976
33. F. A. Beach. “Comparison of Copulatory Behavior of Male Rats Raised in Isolation, Cohabi-
tation, and Segregation.” Journal of Genetic Psychology, 60, pp. 121–136, 1942.

34. Z. Yan and K. Fischer. “Always Under Construction: Dynamic variations in adult cognitive
development,” Human Development, 45, pp. 141–160, 2002.
35. M. Csikszentmih
´
alyi. Creativity: Flow and the Psychology of Discovery and Invention. New
York: Harper Perennial, 1996.
36. J. L
¨
owgren and E. Stolterman. Thoughtful Interaction Design, Boston, MIT, 2005.
37. B. Eaglestone and P. D. Bamidis. “Music composition for the multi-disabled: A systems per-
spective.” Disability Human Development, 7(1), pp. 19–24, 2008.
31 SoundScapes/Artabilitation – Evolution of a Hybrid Human Performance 711
38. A. L. Brooks. Interpretations: inter-sensory stimulation concept targeting inclusive access of-
fering appreciation of classical music for all ages, standing, & disability, Proc. 7th Intl Conf.
on Disability, Virtual Reality and Assoc. Technologies with ArtAbilitation, pp 15–22, Maia &
Porto, Portugal, 8–11 Sept. 2008.
Images are marked with credit to photographer or permitting body. Otherwise
copyright is with the author.
Additional images are viewed at the author’s online gallery - .
com/anthony.lewis.brooks
Chapter 32
Natural Interaction in Intelligent Spaces:
Designing for Architecture and Entertainment
Flavia Sparacino
Introduction
Designing responsive environments for various venues has become trendy today.
Museums wish to create attractive “hands-on” exhibits that can engage and interest
their visitors. Several research groups are building an “aware home” that can assist
elderly people or chronic patients to have an autonomous life, while still calling for
or providing immediate assistance when needed.

The design of these smart spaces needs to respond to several criteria. Their main
feature is to allow people to freely move in them. Whether they are navigating a
3D world or demanding assistance, we can’t strap users with encumbering sen-
sors and limiting tethers to make them interact with the space. Natural interaction,
based on people’s spontaneous gestures, movements and behaviors is an essential
requirement of intelligent spaces. Capturing the user’s natural input and triggering a
corresponding action is however, in many cases, not sufficient to ensure the appro-
priate response by the system. We need to be able to interpret the users’ actions in
context and communicate to people information that is relevant to them, appropriate
to the situation, and adequately articulated (simple or complex) at the right time.
On the basis of my work and research I will argue that intelligent spaces need to
be supported by three forms of intelligence: perceptual intelligence, which captures
people’s presence and movement in the space in a natural and non-encumbering
way; interpretive intelligence, which “understands” people’s actions and is capable
of making informed guesses about their behavior; and narrative intelligence, which
presents us with information, articulated stories, images, and animations, in the right
place, at the right time, all tailored to our needs and preferences.
All three forms of intelligence need to co-exist and co-operate for an Intelligent
Space to be effective. Narrative intelligence is important so that the space provides
us with relevant information that matches our interests and needs. We need systems
able to select how much information to give, with what detail and composition, and
F. Sparacino (

)
Sensing Places and MIT, Santa Monica, CA, USA
e-mail: fl
B. Furht (ed.), Handbook of Multimedia for Digital Entertainment and Arts,
DOI 10.1007/978-0-387-89024-1 32,
c
Springer Science+Business Media, LLC 2009

713
714 F. Sparacino
how to sequence and articulate various fragments. A lack of interpretive intelligence
will produce applications that are unable to determine the appropriate time to get
our attention on a specific matter or story, and which nag the user about whether
he or she wants this or that. Context modeling and behavior modeling are ways of
approaching interpretive intelligence, the first one from the situation’s perspective
and the second one from the user’s perspective. Perceptual Intelligence is about
endowing spaces witheyes, ears, and sensors that allow them to capture how they are
being used by people. This is, in its full complexity, still an open field of research:
scene interpretation and object recognition are today active and open territories for
scientific investigation.
Augmenting a space with all three forms of intelligence can be seen as endowing
the space with a mind of its own, which transforms it from a simple container of
people and things to an active participant and cooperating agent of our lives. Per-
ceptual intelligence represents the bottom layer of this virtual brain, and processes
sensorial inputs. Interpretive Intelligence is the middle layer whose role is to “make
sense” of the input data: it identifies situations and people’s behaviors. Narrative
Intelligence is the upper layer, a bit like the brain cortex, and it regulates the output
and communication between the intelligent space and us (Fig. 1).
The above description of space intelligence has provided a high level framework
for the author’s research. In the following sections of this paper I will first illustrate
Fig. 1 Layered intelligence model for interactive spaces
32 Designing for Architecture and Entertainment 715
in more detail the three forms of space intelligence. I will then describe incremental
contributions to intelligence modeling for spaces I developed in the past years with
a focus on applications for architecture and entertainment. The main contribution
of this paper is to show that Bayesian networks are an ideal modeling tool for all
three forms of space intelligence and to provide a unifying mathematical description
for robust sensing, user and context modeling, and articulated information delivery

(storytelling).
Related Work
The work here presented is highly interdisciplinary and it draws from various
disciplines.
Smart Spaces
The author began research in Smart Spaces and Natural Interfaces in collaboration
with Alex Pentland at MIT [29, 41, 44, 45]. A variety of research groups today have
taken this field of investigation further, with a main focus on assisted living and on
creating spaces that are useful for ordinary everyday activities. Georgia Tech has
developed the Aware Home, a place embedded with technology that enables older
adults to age in place, and which helps people communicate with distant relatives
and friends [22]. Microsoft’s Easy. Living is an intelligent environment that con-
tains myriad devices that work together to provide users access to information and
services [7]. These devices may be stationary, as with acoustic speakers or ceiling
lights, or they may be mobile, as with laptop computers or mobile telephones. Mi-
crosoft’s goal is to allow typical PC-focused activities to move off of a fixed desktop
and into the environment [24]. The Stanford Interactive Workspaces project is ex-
ploring new possibilities for people to work together in technology-rich spaces with
computing and interaction devices on many different scales [19]. Their work aims
at augmenting a dedicated meeting space with large displays, wireless/multimodal
I/O devices, and seamless integration of mobile and wireless appliances including
handheld PCs. Rather than seeing the space as a pro-active facilitator Stanford’s
room is a reconfigurable platform for running a variety of applications. MIT’s In-
telligent Room is a highly interactive environment that uses embedded computation
to observe and participate in normal, everyday events. Microphone and camera ar-
rays enable it to listen to people and observe what they do. People can speak with,
gesture to, and interact with it in several ways [6, 9]. A robust agent-based soft-
ware infrastructure supports the operation of these tools. The room’s intelligence is
modeled by a multi-agent society that builds a higher level and context based repre-
sentation of the user’s activity. Each activity has an associated software agent, called

behavior agent, which responds to the user’s actions and performs the corresponding
716 F. Sparacino
appropriate reaction [12]. The room detects and reacts to activities such as watching
a movie, meetings, entering the room. When new sensors are added to the space,
the behavior rules need to be manually modified to take advantage of the new input
information. The space intelligence model proposed in this paper uses instead a uni-
fied mathematical framework, Bayesian networks, both for perceptual intelligence
and user modeling. Thanks to the properties of Bayesian networks, context or user
behavior is modeled not as a set of fixed high level rules, but it is grounded on data
observation, and therefore it can adapt or fine tune the room’s response to the actual
behavior of people in it, both instantly and through time. Additionally the Bayesian
network architecture makes it easy to add new sensors or new responses to the room
without having to re-program the entire system to take new elements into account.
As the applications of interest to the author are geared towards entertainment and
to support new forms of interactive architecture, this paper also highlights the im-
portance of and describes narrative intelligence, which allows the intelligent space
to use its resources to articulate an audiovisual narration as a function of the user’s
estimated interests and behaviors in the space.
In a recent paper Emiliani and Stephanidis have examined the requirements that
spaces with ambient-intelligence technologies need to have to support elderly peo-
ple and people with disabilities. According to their work, the main high-level design
features of a system with ambient intelligence are that it be unobtrusive (i.e., many
distributed devices are embedded in the environment, not intruding upon our con-
sciousness unless we need them), personalized (i.e., it can recognize the user, and
its behavior can be tailored to the user’s needs), adaptive (i.e., its behavior can
change in response to a person’s actions and environment), and anticipatory (i.e.,
it anticipates a person’s desires and environment as much as possible without me-
diation) [11]. The work presented in this paper uses similar requirements for an
entertainment space (i.e. a museum) and it offers a layered model of intelligence
with a unified mathematical representation which also includes narrative intelli-

gence. Early work by the author on space intelligence applied some of the ideas
discussed in this paper towards the design of an interactive museum exhibit [36].
Perceptual Intelligence and Natural Interaction
Pavlovic [27] and Wu [47] have explored use of hand gestures in human computer
interaction, with an emphasis on 3D tracking and multi-modal approaches for ges-
ture recognition. Starner [42] has shown one of the first examples of effective gesture
recognition using HMMs. Brand, Oliver, and Pentland [5] have demonstrated the
higher performance of coupled HMMs for tasks which require gesturing with both
hands at the same time. Campbell and others [8] have studied the effects of the
appropriate feature choice for a gesture recognition task, using stereo vision.
The author has developed a variety of natural interfaces for art and entertainment
Installations [33, 37]. The work on natural interfaces has grown from a monocular
real time body tracking technique, called Pfinder (person finder), and the real time
32 Designing for Architecture and Entertainment 717
blob tracking approach associated with it and described in section 32 [46]. Stereo
tracking of pointing and command gestures, and HMM based gesture recognition is
discussed in [40]. Jojic, Brumitt, and Meyers [20] also use stereo cameras to detect
people pointing and estimate the direction of their pointing. As opposed to the blob
tracking approach developed by the author, they use disparity maps which are less
sensitive to lighting changes. In the blob-based approach, light invariance can be
achieved using adaptation, or implementing color invariant classification [4].
Bayesian Networks for User Modeling and Interactive Narrative
The work of Pearl [28] is fundamental to the field of Bayesian networks. Jordan’s
book [21] had the merit of grouping together some of the major advancements since
Pearl’s 1988 book. Jensen [17, 18] has written two thorough introductory books
that provide a very good tutorial, or first reading, in the field. Bayesian networks
have gained popularity in the early nineties, when they were successfully applied to
medical diagnosis [13].
Albrecht et al [1], have been among the first to model the behavior of a par-
ticipant in a computer game using Bayesian networks. They use several network

structures to predict the user’s current goal, next action, and next location in a multi-
user Dungeon adventure game. With respect to their work, the system here presented
performs not just user state estimation, which can also be used to predict the next
action that the user will do, but it also adds a probabilistic mechanism for content
selection. The focus of the narrative intelligence model here presented is to function
as a sensor-driven computational system that uses the same unifying mathematical
framework (Bayesian Networks) for sensor modeling (tracking, gesture recogni-
tion), user estimation, and content selection. Jebara [16], uses CHMMS, which are
a particular case of Dynamic Bayesian Networks, to perform, first analysis, and then
synthesis, of a player’s behavior in a game. Conati et al. [10] have built an intelli-
gent tutoring system able to perform knowledge assessment, plan recognition and
prediction of students’ actions during problem solving using Bayesian networks.
Jameson [15], provides a useful overview of student modeling techniques, and com-
pares the Bayesian network approach with other popular modeling techniques.
The narrative intelligence model here presented is inspired by work in
probabilistic knowledge representation. Koller and Pfeffer have used probabilis-
tic inference techniques that allow most frame bases knowledge representation
systems to annotate their knowledge bases with probabilistic information, and to
use that information to answer probabilistic queries. Their work is relevant to de-
scribe and organize content in any database system so that it can later be selected
either by a typed probabilistic query or by a sensor driven query [23]. Using a
content database, annotated probabilistically, the sto(ry)chastics system selects the
most appropriate content segment at each time step, and it delivers, interactively in
time and space, an audiovisual narration to the museum visitor as a function of the
estimated visitor type.
718 F. Sparacino
Criteria for Intelligent Space Design
Perceptual Intelligence
To transform a space in a computer and sensor-driven narrative space we need to
be able to interact with the digital content on display in ways that are not as lim-

iting as when we interact with the familiar keyboard and mouse. Furthermore, it
is unrealistic to encumber participants with gloves, cables, or heavy virtual reality
glasses: these fail to fully engage people as the technology dominates over the expe-
rience. Touch-based screens or hardware-based sensors tend to wear and break after
use by hundreds of people and therefore require frequent replacement and a high
maintenance load. Ideally we want to endow our interactive spaces with eyes, ears
and perceptual intelligence so that they can interpret people’s natural movement,
gestures, and voice. Our spaces should be aware of visitors approaching an object
of interest, of how they approach it, (speed, direction, pauses): they should be able
to understand pointing gestures as commands, as well as occasionally understand
more sophisticated gestures (zoom-in, zoom-out, rotate, move up, move down, me,
you, etc.) and voice commands. Unencumbering computer vision interfaces or other
wireless sensors, such as infrared sensors, electric field sensors, civil use sonars and
radars, are the ideal input device or communication interface with the space.
The robustness and reliability of a natural interface is also very important. If the
interface breaks often or does not work consistently, the “magic” of involvement
and immersion in the interactive experience vanishes. Therefore ideally more than
one sensor should be used to capture the participant’s input. Cooperation of sen-
sor modalities which have various degrees of redundancy and complementarity can
guarantee robust, accurate perception. We can use the redundancy of the sensors
to register the data they provide with one another. We then use the complementar-
ity of the sensors to resolve ambiguity or to reduce errors when an environmental
perturbation affects the system.
Interpretive Intelligence
To make good use of reliable measurements about the user, we need to be able to
interpret our measurements in the context of what the user is trying to do with the
digital media, or what we, as designers, want people to do, so that they get the most
out of the experiences we wish to offer. The same or similar gesture of the public can
have different meanings according to the context and history of interaction. For ex-
ample, the same pointing gesture of the hand can be interpreted either as a zoom-in

command gesture, or more simply, as a selection gesture. In a similar way, the sys-
tem needs to develop expectations on the likelihood of the user’s responses based on
the specific context of interaction and content shown. These expectations influence
in turn the interpretation of sensory data. Following on the previous example, rather
32 Designing for Architecture and Entertainment 719
than teaching both the user and the system to perform or recognize two slightly
different gestures, one for zoom-in and one for selecting, we can simply teach the
system how to correctly interpret slightly similar gestures, based on the context and
history of interaction, by developing expectations on the probability of the follow-on
gesture. In summary, our systems need to have a user model which characterizes the
behavior and the likelihood of responses of the public. This model also needs to be
flexible and should be adaptively revised by learning the user’s interaction profile.
Together with a user model, the system should build a model of the “situation” in
which the user is involved while interacting with digital media (context modeling).
Narrative Intelligence
In order to turn computers into articulated storytellers that respond to people’s nat-
ural gestures and voice, we cannot simply model interaction as a list of coupled
inputs and outputs. This simply defines a map of causes and effects that associates
an action of the user to a response produced by the interactive space. Systems au-
thored with this method tend to produce applications that are repetitive and shallow.
We need instead narrative machines that are able to orchestrate stories whose com-
position and length can vary as a function of the publics’ interests. Just as a museum
guide adapts his/her explanation of the artwork on display according to the visitors’
base knowledge and curiosity, our narrative engines should be able to take into ac-
count and adapt to the publics’ needs. To accomplish this goal we need to model
the story we wish to narrate so that it takes into account and encompasses the user’s
intentions and the context of interaction. Consequently the story should develop on
the basis of the system’s constant evaluation of how the user’s actions match the
system’s expectations about those actions, and the system’s goals.
Intelligence Modeling

Over the last decade, a method of reasoning using probabilities, variously called
belief networks, Bayesian networks, probabilistic causal networks, influence dia-
grams, knowledge maps, etc., has become popular within the AI community, and
the machine learning, and pattern recognition communities. (Dynamic) Bayesian
networks have been successfully applied to a variety of perceptual modeling tasks
such as multimodal sensing for gesture recognition and sensor fusion [27], speech
recognition [25], and body motion understanding [26]. Research in user and context
modeling applies Bayesian networks to identify the behavior of a participant in a
computer game [1], to interpret a car driver behavior [30], to understand the needs
of a student. More recently the author has investigated Bayesian networks for story
modeling and content selection in sensor-driven interactive narrative spaces [35]. In
this paper I will argue that Bayesian networks are an ideal intelligence modeling
720 F. Sparacino
tool as they can be used effectively to model respectively perceptual, interpretive,
and narrative intelligence for interactive spaces.
Applications
This section describes three examples of spaces, or space-components, the author
developed which each contribute, piece-wise, to the construction of intelligent en-
vironments. In some cases the system only has perceptual intelligence, whereas
in others, the interpretive and narrative intelligence modeling is the focus of the
contribution. Since the aim of this paper is to provide a unified view of space mod-
eling techniques, the following sections will summarize the approach and results
obtained. The reader will find a more accurate description of the implementation
details in the included bibliography.
Perceptual Intelligence: Navigating the Internet City
This section presents a natural interface to navigate inside a 3D Internet city, using
body gestures. This work uses a combination of computer vision and pattern recog-
nition techniques to capture people’s interaction in a natural way. A wide-baseline
stereo pair of cameras is used to obtain 3D body models of the user’s hands and
head. The interface feeds this information to an Hidden Markov Model (HMM) ges-

ture classifier to reliably recognize the user’s browsing commands. With regard to
the intelligence modeling framework here described, Smyth [32] demonstrated that
HMMs are equivalent to dynamic Bayesian networks. To illustrate the features of
this interface I describe its application to a custom built 3D Internet browser which
facilitates the recollection of information by organizing and embedding it inside a
virtual city through which the user navigates [35].
Natural Interfaces: Motivation
Recent technological progress today allows most home users to be able to afford
powerful graphics hardware and computer processors. With this equipment people
can navigate in sophisticated 3D graphical environments and play engaging com-
puter games in highly realistic and fascinating 3D landscapes. Such progress has not
been paralleled by equivalent advances in man-machine interfaces to facilitate ac-
cess and displacement in virtual worlds. People still use quite primitive and limiting
interfaces: the joystick, button-activated game consoles, or the computer keyboard
itself. Full immersion and skillful exploration of 3D graphical environments are
limited by the user’s ability to use these interfaces, and repetitive use often involves