Scalable voip mobility intedration and deployment- P36 ppsx

The Future: Video Mobility and Beyond 351
www.newnespress.com
pixels in the device’s display, as there is not much sense in using a video stream that has a
higher resolution than the display.
Each pixel holds the color information for that pixel. This makes a pixel a rectangular area
of uniform color. Computer representations of human-visible color can be measured in
many different ways—which is both a problem and an opportunity. For raw images, which
have no compression, one common way to measure color is to use the RGB method. RGB
relies on how the human eye can see three primary colors—red, green, and blue—and that
all colors are some combination of those three. For this reason, computer and television
displays are made with pixels that are themselves made of the three different colors, each lit
with independent intensity. The perception of color is a rich and detailed subject, but there
are some concepts to it that we should briefly address here to give a sense for what is going
on. Pure light, as electromagnetic radiation, can be thought of as having one frequency. The
colors of the rainbow are all colors of one pure “tone,” or light of one single frequency.
The eye has cones, the receptors that show color, and these cones come in three types, not
surprisingly one for red, one for blue, and one for green. The three cones will respond
differently for each pure, spectral color, giving the eye good coverage over all of the colors
that can exist in the world. When light of one frequency falls upon the cones of the eye,
each type cone responds differently and predictably. From this comes the perception of the
color of that frequency.
Of course, not all colors are made of pure color. White is most certainly not a pure color, as
it is able to be split by a prism into the entire rainbow spectrum. It must be more than just
one frequency, then. In fact, it is a balanced mixture of many frequencies, and comes out
white because it is able to excite the three primary color cones the eye sees with even
intensity. This is no coincidence, as while happens to represent the total mixture of
frequencies the sun radiates down on the planet. Because everything we see is reflected
from normally white light, our most common experiences of colors have to belong to the
subset of white light itself.
We can think of representing color by, then, a triplet of intensities, representing the amount
of activation of the three cones in the eye. This is a good start. As it happens, however,

video displays are not built around the exact excitement of the three cones. The red, blue,
and green are a bit different than what the eye sees, mostly because display screens need to
generate light from something that could be mass produced, and it was sufficient to use a
different red, green, and blue, so long as the entire range of colors could get close to
running the gamut of what the eye can see. (Gamut is the correct term for the space of
colors that a particular way of representing colors can cover.) This means that RGB can
represent essentially all of the colors we know of and use on a regular basis, though often
as only an approximation to the real thing. There are some colors in nature that can never
be seen on a screen—often those that are the most vibrant.
352 Chapter 9
www.newnespress.com
However, we can still get a sense of how the primary RGB colors of light add up. Red and
blue readily mix to form purple, which is easy to describe as a reddish-blue hue. Blue and
green mix to form a blue-green color, such as cyan. The odd one is red and green together,
which produces yellow, a color most people would not describe with the term red-green,
being a pair of colors which seem as opposite as can be imagined. (It is difficult to imagine
that the two colors of a poinsettia can ever combine to form the color of a banana.) All
three together form white, and all three absent form black. Staring at any screen up close
will show exactly how this process works in video displays to blend together to form a
color, as each pixel is made of subpixels, or regions of only one color, that can only vary by
intensity. When viewed from a reasonable distance, the eye does not do a good job seeing
the separate subpixels, and so it blends the three different hues together to form the final,
intended, color.
In computer displays, the widely-used sampling method is to given each primary color an
intensity of eight bits, producing a 24-bit overall color sample per pixel. This lets us step back
and look at the size of video. Compared to raw voice, which must encode only one 16-bit
sample at a given time, raw video must encode hundreds of thousands of 24-bit color samples.
Even though the number of times a second the set of pixels, the picture known as a frame is far
less—the size of a frame quickly dominates. On standard video, the picture (and thus pixel
color intensities) changes up to 30 times a second, far less often than the 8000 times a second

that the voice intensity changes. Multiplying it out, a raw voice stream requires 128,000 bits a
second; a raw 640×480 video at 30 frames per second requires 221,184,000 bits per second for
just the video portion, not including any associated audio streams.
9.1.2 Video Compression
The large size of video clearly begs for compression. Fortunately, much of the detail in
video is wasted on the viewer, and video has a tremendous amount of room for lossy
compression to be employed. Video compression is an area of active research, but the basics
are easy to understand, and are used in modern compression algorithms, such as those for
MPEG video.
9.1.2.1 Still Image Compression
The first area we can look to compress video is with the representation of color. Many of
you who may read this book may also remember how color digital displays evolved, and
thus would have a natural understanding of how excessive 16,777,216 possible colors per
pixel once seemed. However, it does bear repeating. The eye can be challenged, by certain
color transitions, or changes of color from one to the next, to need all 24 bits to capture
most of the range of differences the eye can see. But, ignoring those specific, challenging
situations, the eye can really see only a few thousand colors. Furthermore, if exact color
reproduction is not the point—and it is not, for video—making minor approximations here
The Future: Video Mobility and Beyond 353
www.newnespress.com
or there is quite acceptable. Therefore, the first technique for video compression is to
radically reduce the number of colors, following the usual media compression technique of
focusing the bits to where they are most perceivable by the human observer, and then filling
in the details with bits that may not be kept or can be afforded to be lost.
Although the red, green, and blue representation works quite well for representing what the
video display needs to do, it is not the most obvious choice for representing human-
perceived color. We can take a hint from the development of analog television. Intensity—
the overall intensity of the pixel—matters the most. Thus, black and white works quite well
for representing the subject of the video, so long as people are not dressed with garish
colors that would go missed. Because intensity is so important, representing the intensity is

the best use of the bits of information for a pixel. Once the black-and-white intensity is
known, the remaining bits can be used to give the shade of pixel a tint, or hue. Three
primary colors mean three additional hue values, right? Thankfully not. If we think of color
as a mathematical vector of three dimensions, knowing the intensity is like knowing the
length. We only need to know how far the color extends into two of the three dimensions,
along with the length, to get the intended color back out. (This is embedded in the very
definition of having three dimensions to color.) The two choices that television designers
settled on was to record both how red and how blue the color is, leaving green to be
inferred. It starts with an intensity, also known as a luminance in this representation, such as
highest for white and lowest for black. The tinting, using the two chrominances, each add
or subtract an amount of the primary color they represent. Together, the luminance and two
chrominances create another 24-bit value, often represented by the abbreviation YC
b
C
r

(Y is the standard symbol for luminance, and C is for chrominance). White can be
represented, in percentages, as 100% Y, 0% C
b
, and 0% C
r
, for (100, 0, 0). Black is
(0, 0, 0), and middle gray is (50, 0, 0). From white, we can get cyan, which has full blue
and green but no red. This requires subtracting off the full measure of red. Therefore, a
nearly pure cyan is (100, −100, 0). Similarly, a nearly pure green is (100, −100, −100),
removing both the red and blue components. The qualification of nearly is used only
because the standard weightings for the YC
b
C
r

space require a bit of tweaking to get to the
pure RGB versions of the color. (YP
b
P
r
, the name for component video cables such as used
with HD televisions, refers to the same concept, but for analog signals.)
Given that the eye is most particular to the precision of the intensity, and less so to the
precision of the tinting, video compression will commonly change the ratio of information
rate for each. Specifically, the video compression can require that some of the information
be held the same across multiple pixels. By halving the amount of information for the
chrominance, the 4:2:2 encoding stores twice as much information to luminance than to the
blue and the red chrominances, by requiring that the chrominances change only every other
pixel horizontally. Simply halving the amount of information in one pixel dedicated to
chrominance would result in saving one-third of the bits. The sacrifice is in color fidelity
354 Chapter 9
www.newnespress.com
when the color changes, but for video—and often for still images as well—this may
not matter, as the eye considered to be roughly have as sensitive to the tint as it is to
brightness. There is also a 4:2:0 “ratio,” which is actually more common. 4:2:0 is a
special term that means making squares of two pixels by two pixels share the same
chrominance.
This sort of color compression falls into the category of quantization compression, where
the goal is to represent either a continuous or more precise value with a less precise
quantized value. Quantization compression was also used in voice, for the two logarithmic
encoders in G.711. That compression cut the number of bits in half, but it was smarter than
just dividing the signal by 256, as the quantization steps between two encoded values do not
have to be even, and logarithmic encoding concentrated the slices more towards the smaller
signal values. 4:2:2 compression for video concentrates the bits more to luminance.
Even with reducing the chrominance, video compression will also quantize the luminance,

based on the range and precision that it actually achieves over the area being quantized.
This is a fundamental part of encoding still images and moving videos, as well. Because the
range of intensities within an image varies from part to part, this sort of quantization is best
done in small chunks, or regions, of the image. Once quantized, the encoding must have the
particular parameters used to make the quantization happen.
As a general rule, the hard part of media compression is finding a representation that has
the bits shuffled and applied to categories that matter differently—from there, compression
can be achieved simply by chopping bits from the categories (high-intensity audio samples,
chrominance) that matter the least. This can be clearly seen than with the base for modern
image and video compression, as found in JPEG and MPEG formats. Cutting colors and
rescaling to pack in bits where they matter most is one thing. But the designers of JPEG
thought of the image differently, looking at its frequency components. Just as audio has the
obvious frequency components, representing the set of pitches that are being heard at a
given time, video has two dimensions (horizontal and vertical) of frequencies at any given
point. The thinking behind JPEG is that the higher frequencies represent the presence of
detail, and the lower frequencies represent the slight variations. In audio compression,
converting the signal to frequencies is useful because some frequencies are not as important,
and producing what amounts to a rank ordering of the most important frequencies at a given
time allows the important parts of the signal to be preserved, while the less important
parts—such as the faint but often highly detailed noises in the background of a recording—
can be erased or approximated more easily. The same applies to video.
(If you are thinking, by now, that color, being comprised of multiple pure tones or
frequencies of light, could benefit from being represented this way, then you are on the right
path. Color itself, however, happens to not be a good example for being represented and
then compressed this way, because the eye already does such a good job removing most of
The Future: Video Mobility and Beyond 355
www.newnespress.com
the information out of light by forcing it from an infinite-dimensional space of continuous
functions to a three-dimensional space of primary colors, with enough tolerance for
approximation.)

One method used to convert an image (or any signal) from space-defined pixels to
frequencies is to take the Fourier transform of the image. It’s easier to think of Fourier
transforms first with audio. We know that a sound can be made of one or more pitches. A
chord can contain, for example, the four sounds of middle D flat, E flat, F, and A flat
(producing a Dbadd2 chord), and those four tones will be the four most important
frequencies while the chord is being played. Of course, the instrument or instruments
playing the chord each produce a number of both similar and widely different tones around
the main tone of the note, and losing those would lose the character of the instrument
completely. But, by seeing that some pitches are more important than others, we can begin
to see how the pitches can be ranked. The Fourier transform does not do any ranking itself.
It is purely mathematical—a change in representation (really, linear basis) from time to
frequency and back. The notion is rather simple. Overlap the signal with the one for each
pure tone or pitch. The higher the overlap, the more that pitch is present in the signal. We
can think of the Fourier transform as testing the signal for overlap with each and every pure
tone. Overlap for two signals, or functions, can simply be thought of as the sum over the
entire signal of the product, or multiple, of the signal and the “test” tone. If the tone
overlaps well with the signal, then each value in the pure test tone will multiply with that of
the pure tone embedded in the real signal, and the sum of this will add up to produce a
large number. However, if the pure tone is not present in the original signal, then the sums
will all go out of step, and the result will be small. Figure 9.1 shows this in action.
If two signals are out of phase, but have the same frequency, then the sum of the product
can go to 0, even though there is a match. But using complex numbers (see Chapter 5), the
phase can be captured without ambiguity or mistake. It is this process that produces the
Fourier transform. For math’s sake, we can write it as
F f t e dt
i t
ω
ω
(
)

=
(
)
−
∫
where F is the Fourier-transformed representation of f, based on the angular frequency ω,
which is 2π times the frequency.
What does all this mean? It means that we have a mathematical way of converting a signal
to its frequencies. A continuous signal has a continuous (infinite) number of frequencies.
But, with the digital world, signals are always finite and discrete. We can shift from the
continuous Fourier transform and go to the discrete variant, however. This variant uses the
same math, but replaces the integral with a discrete sum. The result of a discrete Fourier
transform of a signal of a certain number of samples is a new signal with the same number
356 Chapter 9
www.newnespress.com
Intensity
Time
a) Original signal 2 cos(10 · 2 p x) + cos(30 · 2 p x), composed of two signals: one at a frequency of 30Hz, and another,
half as strong, at a frequency of 10Hz.
0
Intensity
Time
c) Overlap for 10Hz test signal in original signal, which does have a 10Hz component. Notice how the running
sum, representing the amount of overlap the test signal has with the original as a running sum of the product of
the two signals at each point, steadily increases. This 10Hz test signal is a match, and there will be a peak in
the Fourier transform at 10Hz.
0
Intensity
Frequency
b) Fourier Transform, or frequency plot, of the same signal. Noice how there are strong peaks at both 10Hz and

30Hz, with the 30Hz peak having twice the intensity. This is the advantage of the Fourier transform, which pulls
out the frequencies present in a signal.
0
0 10 20 30 40 50 60 70 80 90 100
Original signal
10Hz test function
Running sum (integral)
Intensity
Time
d) Overlap for 20Hz test signal in original signal, which does not have a 20Hz component. The running sum
now doesn’t steadily increase, from left to right, but instead vacillates around zero, as expected because the
test signal now does not overlap with the original signal. This 20Hz test signal is not a match, and there will
be no peak in the Fourier transform at 20Hz.
0
Original signal
10Hz test function
Running sum (integral)
Figure 9.1: Fourier Transform
The Future: Video Mobility and Beyond 357
www.newnespress.com
of samples, but with the first sample representing the lowest frequency—the larger the
sample, the more this frequency is present in the original signal—and so on.
The method now begins to become clear. Take the signal, then the discrete Fourier
transform of it. The most important frequencies will have the largest value, and less
important frequencies will have smaller values. Assigning more bits to the larger
frequencies and less bits to the smaller by quantization will compress the signal. Most of
the frequency components will actually get compressed to zero, or completely removed,
when compression is successful.
Coming back to video, the thought is the same. The discrete Fourier transform—and its
variant, the discrete cosine transform (DCT), which works with entirely real numbers (no

imaginary numbers)—can work in both the horizontal and vertical directions, to capture the
frequencies present in the image. Now, it may not seem that a video image obviously has
frequencies. Over the entire image, it probably doesn’t have any ones that can be seen
intuitively. But the trick is to divide the image up into small rectangles—the size depends
on a number of factors, such as how much or little the part of the image in the rectangle
varies—and do the frequency transform and subsequent quantization for each rectangle.
Now, the benefit for frequency information can begin to make sense. A rectangle whose
image barely changes, such as background shading, has very little frequency information,
and so can be compressed greatly. But even areas that represent real shapes can be
compressed rather well, with a lot of loss but preserving the rough character of that part of
the image. As long as each rectangle does not have a lot of irregularity in it, the
compression will be good for each rectangle. From here, the compressors try to figure out
how to make each rectangle as large as possible for the same bits, looking for areas with
lots of similarity.
This adaptive sizing of rectangles, and the rectangles themselves, can be seen fairly easy on
compressed images, but usually happens away from the action (intentionally, as we will
see). Let’s take the example in Figure 9.2.
The lefthand image is the original, and the righthand is with compression set high—as
would happen away from the action. Right away, you can see the outlines of the rectangles,
in this case, all of the same size, as this is a JPEG image. Most of the squares, such as those
for the sky and the solid parts of the building—wherever there is not a lot of detail—got
compressed down to no frequency components in any direction: a solid color. But some
areas had more than one frequency component that wasn’t compressed to zero. If you look
at the right side of the tower, where it meets the sky, you will notice that all of the squares
seem to have vertical bands, and thus are horizontally smooth. This happens because one
frequency component in the horizontal direction got retained, which makes sense for a
mostly vertical shape. Squares with more than one component in each direction can be seen
where people’s heads are.
358 Chapter 9
www.newnespress.com

9.1.2.2 Motion Compression
Once a still frame has been compressed, we can move on to compressing the motion itself.
The simple way of doing this would be to just have a sequence of compressed images, one
compressed image for each frame. This is actually done in a format called Motion JPEG.
However, doing so would not take advantage of the fact that most frames are nearly
identical, with similar backgrounds but different images of the active subjects as they move
around.
We can think of compressing the motion itself by starting out with the first frame, a
compressed still image like any other. But, for the following frame, imagine not storing
another compressed image. Instead, just store which pixels in the image, corresponding to a
moving subject, have moved, and in which direction they have moved. The decoder will
just copy those pixels forward, moving them according to the directions the encoder gives.
The only new pixels the encoder needs to send are those for the background that got
revealed as the moving subject moved away from them. As most objects are pretty solid or
large, the pixels in them move together, and so encoding regions of essentially uniform
motion is fairly simple and highly efficient. From this comes the dual concept of the key
frame (or I frame for intra-coded), the first frame in the sequence with the complete set of
pixels, and the intermediate frames (or P frame for predictive), the following frames that
only carry the motion and the newly revealed pixels.
This would work just fine, except that viewers like to jump around in videos, or
intermediate frames might get lost somewhere. If there were only one key frame, the video
would be ruined for any absence of even one bit of information of the intermediate frames.
To overcome that, the encoder starts off with a new key frame every so often. This effect,
too, is something you can see rather easily. DVDs and digital video recorders use similar
Figure 9.2: Compression Artifacts
The Future: Video Mobility and Beyond 359
www.newnespress.com
compression algorithms, with key frames spaced fairly far apart and a few dozen
intermediate frames in between. When you fast forward or rewind the video, you may see
these key frames go by, one by one, as still pictures. This is quite different than the old

analog VCR days, where the video would just animate faster, and happens because
processing the intermediate frames takes too much time for fast forwarding.
Figure 9.3 illustrates the parts of the intermediate frame that are used for added
compression.
Frame 1 Frame 2
a) The ball moves from Frame 1 to Frame 2.
b) The intermediate frame encoding for Frame 2 stores that
the pixels that make up the ball in Frame 1 need to move
down and to the right to make Frame 2.
c) The intermediate frame encoding for Frame 2 also stores
the newly revealed pixels for Frame 2, shown here.
Movement Revealed Background
Figure 9.3: Motion Compression
360 Chapter 9
www.newnespress.com
Video compression leaves much of the intelligence to the compressor: the decompressor is
required only to execute the instructions. More intelligent compressors can focus on the
subject matter of importance, and the decompressor itself does not need to understand what
matters more in the subject in order to do its job.
There are a few types of video codecs, all of them similar to each other, but with some
differences in the degree of intelligence and the type of formatting. ITU H.262, the codec
used in MPEG-2 video, is the most common codec, used in DVDs, as well as a wide
variety of downloadable Internet video content. The bitrate can go as high as 10Mbps for
standard-definition DVD video.
ITU H.264, for Advanced Video Coding (AVC) and the foundation for MPEG-4 video, was
designed to produce a far better picture at a significantly smaller bit rate—the goal is to be
about half the bit rate of MPEG-2, for the same quality. AVC includes a number of
improvements, including the ability for the decoders to smooth the edges between the
blocks, such as those seen in Figure 9.2. AVC is the foundation of most high-definition
(HD) video, including for Blu-ray video discs and many satellite and cable television

transmissions. AVC is also used in YouTube and other Adobe Flash–based video downloads.
Other, often proprietary, codecs exist for videoconferencing and webinar (web seminar)
broadcasts, which can take advantage of the constrained subject matter—a series of heads or
presentation slides, for example—to compress even better than general-purpose video
compressors.
9.1.3 Video Signaling and Bearer Technologies
Video must be carried in much the same way as voice. The video flow or call may need to
be set up—this is especially true for conferencing—and then the video stream itself must be
transported, along with the related audio streams.
9.1.3.1 Video Bearer
Let’s start with the video transport, as the bearer, first. Because many video downloads are
streaming, rather than conferencing, both real-time and stream-based transports can be
considered.
Real-time video transport is often based on the same RTP mechanism that is used for voice.
When transported this way, each of the frames in the video may span multiple RTP packets.
The opposite also will happen, and multiple frames may meet in any given RTP packet.
However, the RTP mechanism applies the same timestamp and sequence number functions
to the video stream, allowing the video decoder to piece back together the stream when
packets are lost or reordered. The video sender can send separate RTP streams, sharing the
same timestamp clock, for each of the media streams that make up the video. This can be