3D Graphics with OpenGL ES and M3G- P14 pptx

114 ANIMATION CHAPTER 4
Morphing is a good technique for producing complex deformations, such as facial
expressions, that are not easily reproduced by simple transformations. The downside
of morphing is that it takes a lot of memory per keyframe, so the number of base
shapes should be kept relatively small for complex meshes. Alternatively, morphing
can be used to define lots of keyframes for very simple meshes to get fairly complex
animation that is still computationally cheap to render in real time.
4.2.2 SKINNING
For complex characters with lots of vertices and more or less arbitrary numbers of possible
poses, morphing quickly becomes inefficient. Another way to deform meshes is to assign
vertices to the joints of an articulated skeleton, animate them, and connect the vertices
with a skin of polygons [Cat72]. However, that still leads to sharp changes at joints. During
the 1990s the gaming and 3D modeling industry generalized this approach and started
calling it skinning [Lan98]. The idea is that each vertex can be associated with sever al
joints or bones, weighted by linear weights. This technique is sometimes referred to as
subspace surface deformation, or linear blend skinning; we simply call it skinning. It is so
commonly used today that we can call it the de facto standard of char acter animation.
The general idea behind skinning is that instead of transforming the whole mesh with
a single transformation matrix, each vertex is individually transformed by a weighted
blend of several matrices as shown in Figure 4.7. By assigning different weights to different
vertices, we can simulate articulated characters with soft flesh around rigid bones.
The skeleton used in skinning stands for a hierarchy of transformations. An example
hierarchy can be seen in Figure 4.7. The pelvis is the root node, and the rest of the body
parts are connected to each other so that the limbs extend deeper into the hierarchy.
Each bone has a transformation relative to the parent node—usually at least translation
and rotation, but scaling can be used, for example, for cartoon-like animation. The hier-
archy also has a rest pose (also known as bind pose) in which the bone transformations
are such that the skeleton is aligned with the untransformed mesh.
Having the skeleton hierarchy, we can compute transformations from the bones to the
common root node. This gives us a transformation matrix T
i

for each bone i. The matrices
for the rest pose are important, and we denote those B
i
.
The relative transformation that takes a rest pose B to a target pose T is TB
−1
.From
this, and allowing a vertex v to have weighted influence w
i
from several bones, we get the
skinning equation for a transformed vertex
v

=

w
i
T
i
B
−1
i
v.
(4.12)
Note that we can either transform the vertex with each matrix, then compute a blend of
the transformed vertices, or compute a blend of the matrices and transform the vertex just
once using the blended matrix. The latter can in some cases be more efficient if the inverse
SECTION 4.2 DEFORMING MESHES 115
Figure 4.7: Left: A skeletally animated, or skinned, character. Each arrow ends in a joint, and each
joint has a bone transformation, usually involving at least translation and rotation.

Right: A close-up
of one animated joint, demonstrating vertex blending. The vertices around the joint are conceptually
transformed with both bone transformations, resulting in the positions denoted by the thin lines and
black dots. The transformed results are then interpolated (dotted line, white dots) to obtain the final
skins (thick lines).
transpose matrix is needed for transforming vertex normals. Also, the modelview matrix
can be premultiplied into each matrix T
i
to avoid doing the camera transformation as a
separate step after the vertex blending.
With hardware-accelerated skinning, using either vertex shaders or the OpenGL matrix
palette extension (Section 10.4.3), the vertices will be transformed each time the mesh is
rendered. With multi-pass rendering in particular, the mesh will therefore be transformed
multiple times. A software implementation can easily perform the calculations only when
necessary and cache the results, but this will still place a considerable burden on the CPU.
As an animated mesh typically changes for each frame, there is usually no gain from using
software skinning if hardware acceleration is available, but it is worth keeping the option
in mind for special cases.
The animation for skinning can come from a number of sources. It is possible to use
keyframe animation to animate the bones of the skeleton, with the keyframes modeled by
hand or extracted from motion capture data. Another possibility is to use physics-based
animation. Rigid body dynamics [Len04] are often used to produce “ragdoll” effects, for
example when a foe is gunned down and does a spectacular fall from height in a shooter
game. Inverse kinematics (IK) [FvFH90, WW92] can also be used to make hands touch
scene objects, align feet with the ground, and so forth. Often a combination of these tech-
niques is used with keyframe animation driving the normal motion, rigid body dynamics
116 ANIMATION CHAPTER 4
stepping in for falling and other special effects, and IK making small corrections to avoid
penetrating scene geometry.
4.2.3 OTHER DYNAMIC DEFORMATIONS

Naturally, dynamic deformation of meshesneed not be limited to morphing and skinning.
As we can apply arbitrary processing to the vertices, either in the application code or, more
commonly, in gr aphics hardware, almost unlimited effects are possible.
One common example of per-vertex animation is water simulation. By applying displace-
ments to each vertex based on a fluid simulation model, a convincing effect can be created.
Different kinds of physics-based deformation effects include soft body modeling, whereby
the mesh deforms upon contact based on, for example, a mass-and-spring simulation. A
variation of this is cloth modeling, where air density plays a more important role. The
details of creating these and other effects are be yond the scope of this book. For further
information, refer to the bibliography ([WW92, EMP
+
02]).
Once the vertex data is dynamically modified by the application, it needs to be fed to the
rendering stage. Most graphics engines prefer static vertex data, which allows for opti-
mizations such as precomputing bounding volumes or optimizing the storage format and
location of the data. Vertex data that is dynamically uploaded from the application pro-
hibits most such optimizations, and it also requires additional memory bandwidth to
transfer the data between application and graphics memory. Therefore, there is almost
always some performance reduction associated with dynamically modifying vert ices. The
magnitude of this performance hit can vary greatly by system and application—for exam-
ple, vertex shaders in modern GPUs can perform vertex computations more efficiently
than application code because there is no need to move the data around in memory, and
it has an instruction set optimized for that particular task. This is also the reason that
modern rendering APIs, including both OpenGL ES and M3G, have built-in support for
the basic vertex deformation cases—to enable the most efficient implementation for the
underlying hardware.
5
CHAPTER
SCENE MANAGEMENT
By dealing with individual triangles, matrices, and disparate pieces of rendering state, you

are in full control of the rendering engine and will get exactly what you ask for. However,
creating and managing 3D content at that level of detail quickly becomes a burden; this
typically happens when cubes and spheres no longer cut it, and graphic artists need to
get involved. Getting their animated object hierarchies and fancy materials out of 3ds
Max or Maya and into your real-time application can be a big challenge. The task is
not made any easier if your runtime API cannot handle complete objects, materials,
characters, and scenes, together with their associated animations. The artists and their
tools deal with higher-level concepts than triangle strips and blending functions, and
your runtime engine should accommodate to that.
Raising the abstraction level of the runtime API closer to that of the modeling tools
facilitates a content-driven approach to development, where designers can work inde-
pendently of programmers, but it has other benefits as well. It flattens the learning
curve, reduces the amount of boilerplate code, eliminates many common sources of
error, and in general increases the productivity of both novice and expert programmers.
A high-level API can also result in better performance, particularly if you are not already
a 3D guru with in-depth knowledge of all the software and hardware configurations
that your application is supposed to be running on.
In this chapter, we take a look at how 3D objects are composed, how the objects can
be organized into a scene graph, and how the scene graph can be efficiently rendered
and updated. Our focus is on how these concepts are expressed in M3G, so we do not
117
118 SCENE MANAGEMENT CHAPTER 5
cover the whole spectrum of data structures that have been used in other systems or
that you could use in your own g ame engine. For the most part, we will use terminology
from M3G.
5.1 TRIANGLE MESHES
A 3D object combines geometric primitives and rendering state into a self-contained visual
entity that is easier to animate and interact with than the low-level bits and pieces are. 3D
objects can be defined in many ways, e.g., with polygons, lines, points, B
´

ezier patches,
NURBS, subdivision surfaces, implicit surfaces, or voxels, but in this chapter we concen-
trate on simple triangle meshes, as they are the only type of geometric primitive supported
by M3G.
A triangle mesh consists of vertices in 3D space, connected into triangles to define a
surface, plus associated rendering state to specify how the surface is to be shaded. The
structure of a triangle mesh in M3G is as shown in Figure 5.1: vertex coordinates, other
per-vertex attributes, and triangle indices are stored in their respective buffers, while
rendering state is aggregated into what we call the appearance of the mesh. Although
this exact organization is specific to M3G, other scene graphs are usually similar. We
will explain the function of each of the mesh components below.
VertexBuffers are used to store per-vertex attributes, which, in the case of M3G,
include vertex coordinates (x, y, z), texture coordinates (s, t, r, q), normal vectors
(n
x
, n
y
, n
z
), and colors (R, G, B, A). Note that the two first texture coordinates (s, t) are
Mesh
VertexBuffer
Texture
Coordinates
Coordinates
Normals
Colors
Texture
Coordinates
VertexArray

VertexArray
VertexArray
VertexArray
VertexArray
Appearance
Appearance
Appearance
IndexBuffer
IndexBuffer
IndexBuffer

Figure 5.1: The components of a triangle mesh in M3G.
SECTION 5.1 TRIANGLE MESHES 119
enough for typical use cases, but three or four can be used for projective texture mapping
and other tricks.
The coordinates and normals of a triangle mesh are given in its local coordinate system—
object coordinates—and are transformed into eye coordinates by the modelview matrix.
The mesh can be animated and instantiated by changing the modelview matr ix between
frames (for animation) or between draw calls (for instantiation). Texture coordinates are
also subject to a 4 × 4 projective transformation. This allows you to scroll or otherwise
animate the texture, or to project it onto the mesh; see Section 3.4.1 for details.
IndexBuffers define the surface of the mesh by connecting vertices into triangles, as
shown in Figure 5.2. OpenGL ES defines three ways to form triangles from consecutive
indices—triangle strips, lists, and fans—but M3G only supports triangle strips. There
may be multiple index buffers per mesh; each buffer then defines a submesh, which is the
basic unit of rendering in M3G. Splitting a mesh into submeshes is necessary if different
parts of the mesh have different rendering state; for example, if one part is translucent
while others are opaque, or if the parts have different texture maps.
The Appearance defines how a mesh or submesh is to be shaded, textured, blended,
and so on. The appearance is typically divided into components that encapsulate coher-

ent subsets of the low-level rendering state: Figure 5.3 shows how this was done for M3G.
The appearance components have fairly self-explanatory names: the Texture2D object,
for instance, contains the texture blending, filtering, and wrapping modes, as well as the
4 × 4 texture coordinate transformation matrix. The texture image is included by refer-
ence, and stored in an Image2D object. Appearances and their component objects can
vertices
coordinates
triangles
texcoords
indices
colors
normals
1234
xxxxyyyyzzzz
rrrrggggbbbb
sssstttt
123424453376
nx nx nx nxny ny ny nynz nz nz nz





T1 T2 T3 T4
2
1
T1
3
T2
T3

4
Figure 5.2: Triangle meshes are formed by indexing a set of vertex arrays. Here the triangles are organized into a triangle
list, i.e., every three indices define a new triangle. For example, triangle T2 is formed by the vertices 2, 4, and 3.
120 SCENE MANAGEMENT CHAPTER 5
Appearance
Material
Fog
Compositing
Mode
Polygon
Mode
Image2D
Image2D
Texture2D
Texture2D
Figure5.3: The appearance components in M3G. Implementations may support an arbitrary number
of texturing units, but the most common choice (two units) is shown in this diagram.
be shared between an arbitrary number of meshes and submeshes in the scene graph. The
appearance components of M3G are discussed in detail in Chapter 14.
5.2 SCENE GRAPHS
Rendering a single 3D object may be useful in a demo or a tutorial, but to create something
more exciting you will need a number of 3D objects in a particular spatial and logical
arrangement—a 3D scene.
3D scenes can be organized into many different data structures that are collectively
referred to as scene graphs. The term is decidedly vague, covering everything from simple
lists of objects up to very sophisticated spatial databases. In this section we aim to char-
acterize the design space of scene graphs, progressively narrowing down our scope to the
small subset of that space that is relevant for M3G.
5.2.1 APPLICATION AREA
When setting out to design a scene graph system, the first thing to decide is what it is for.

Is it for graphics, physics, artificial intelligence, spatial audio, or a combination of these?
Is it designed for real-time or offline use, or both? Is it for a specific game genre, such
as first-person shooters or flight simulators, or maybe just one title? A unified scene rep-
resentation serving all conceivable applications would certainly be ideal, but in practice
we have to specialize to avoid creating a huge monolithic system that runs slowly and is
difficult to use.
SECTION 5.2 SCENE GRAPHS 121
Typical scene graphs strike a balance by specializing in real-time animation and rendering,
but not in any particular application or game genre. This is also the case with M3G.
Physics, artificial intelligence, audio, user interaction, and everything else is left for the
user, although facilitated to some extent by the ability to store metadata and invisi-
ble objects into the main scene graph. Adjunct features such as collision detection are
included in some systems to serve as building blocks for physics simulation, path find-
ing, and so on. M3G does not support collision detection, but it does provide for simple
picking—that is, shooting a ray into the scene to see which object and triangle it first inter-
sects. This can be used as a replacement to proper collision detection in some cases.
5.2.2 SPATIAL DATA STRUCTURE
Having decided to go for a rendering-oriented scene graph, the next step is to pick the
right spatial data structure for our system. The application areas or game genres that we
have in mind play a big role in that decision, because there is no single data structure that
would be a perfect fit for all types of 3D scenes.
The main purpose of a spatial data structure in this context is visibility processing, that
is, quickly determining which parts of the scene will not contribute to the final rendered
image. Objects may be too far away from the viewer, occluded by a wall, or outside the
field of view, and can thus be eliminated from further processing. This is called visibility
culling. In large scenes that do not fit into memory at once, visibility processing includes
paging, i.e., figuring out when to load each part of the scene from the mass storage device,
and which parts to remove to make room for the new things.
Depending on the type of scene, the data structure of choice may be a hierarchical space
partitioning scheme such as a quadtree, octree, BSP tree, or kd-tree. Quadtrees, for exam-

ple, are a good match with terrain rendering. Some scenes might be best handled with
portals or precomputed potentially visible sets (PVS). Specialized data structures are
available for massive terrain scenes, such as those in Google Earth. See Chapter 9 of
Real-Time Rendering [AMH02] for an overview of these and other visibility processing
techniques.
Even though this is only scratching the surface, it becomes clear that having built-in
support for all potentially useful data structures in the runtime engine is impossible.
Their sheer number is overwhelming, not to mention the complexity of implementing
them. Besides, researchers around the world are constantly coming up with new and
improved data structures.
The easy way out, taken by M3G and most other scene graphs, is to not incorporate any
spatial data structures beyond a transformation hierarchy, in which scene graph nodes
are positioned, oriented, and otherw ise transformed with respect to their scene graph
parents. This is a convenient way to organize a 3D scene, as it mirrors the way that things
are often laid out in the real world—and more important, in 3D modeling tools.
122 SCENE MANAGEMENT CHAPTER 5
The solar system is a classic example of hierarchical transformations: the moons orbit
the planets, the planets orbit the sun, and every thing revolves around its own axis. The
solar system is almost trivial to set up and animate with hierarchical transformations, but
extremely difficult without them. The human skeleton is another typical example.
Visibility processing in M3G is limited to view frustum culling that is based on a bounding
volume hierarchy; see Figure 5.4. While the standard does not actually say anything about
bounding volumes or visibility processing, it appears that all widely deployed implemen-
tations have independently adopted similar means of hierarchical view frustum culling.
We will discuss this in more detail in Section 5.3.
Implementing more specialized or more advanced visibility processing is left for the user.
Luckily, this does not mean that you would have to ditch the whole scene graph and start
from scratch if you wanted to use a quadtree, for instance. You can leverage the built-in
scene tree as a basis for any of the tree structures mentioned above. Also, the same triangle
meshes and materials can often be used regardless of the higher-level data structure.

The fact that typical scene graphs are geared toward hier archical view frustum culling and
transformations is also their weakness. There is an underlying assumption that the scene
graph structure is a close match to the spatial layout of the scene. To put it another way,
nodes are assumed to lie close to their siblings, parents, and descendants in world space.
Violating this assumption may degrade performance. If this were not the case, you might
want to arrange your scene such that all nonplayer characters are in the same branch of
the graph, for instance.
The implicit assumption of physical proximity may also cause you trouble when nodes
need to be moved with respect to each other. For instance, characters in a game world
A
AB
BC
C
D
D
Figure 5.4: A bounding volume hierarchy (BVH) consisting of axis-aligned bounding boxes, illustrated in two dimensions
for clarity. The bounding volume of node A encloses the bounding volumes of its children.
SECTION 5.2 SCENE GRAPHS 123
may be wandering freely from one area to another. The seemingly obvious solution is
to relocate the moving objects to the branches that most closely match their physical
locations. However, sometimes it may be difficult to determine where each object should
go. Structural changes to the scene graph may not come for free, either.
5.2.3 CONTENT CREATION
Creating any nontr ivial scene by manually typing in vertices, indices and rendering state
bits is doomed to failure. Ideally, objects and entire scenes would be authored in commer-
cial or proprietary tools, and exported into a format that can be imported by the runtime
engine. M3G defines its own file format to bridge the gap between the runtime engine
and DCC tools such as 3ds Max, Maya, or Softimage; see Figure 5.5. The file format is a
precise match with the capabilities of the runtime API, and supports a reasonable subset
of popular modeling tool features.

From the runtime engine’s point of view, the main problem with DCC tools is that they
are so flexible. The scene graph designer is faced with an abundance of animation and
rendering techniques that the graphics artists would love to use, but only a fraction of
which can be realistically supported in the runtime engine. See Figure 5.6 to get an idea
of the variety of features that are available in a modern authoring tool.
DCC tool
Exporter
Intermediate
Format
(e.g., COLLADA)
Optimizer &
Converter
Delivery
Format
(M3G)
M3G Loader
Runtime Scene Graph
Figure5.5: A typical M3G content production pipeline. None of the publicly available exporters that we are aware of actually
use COLLADA as their intermediate format, but we expect that to change in the future.