Joe Celko s SQL for Smarties - Advanced SQL Programming P58 pps

542 CHAPTER 23: STATISTICS IN SQL
This approach works fine for two variables and would produce a table
that could be sent to a report writer program to give a final version. But
where are your column and row totals? This means you also need to write
these two queries:
SELECT race, COUNT(*) FROM Personnel GROUP BY race;
SELECT sex, COUNT(*) FROM Personnel GROUP BY sex;
However, what I wanted was a table with a row for males and a row
for females, with columns for each of the racial groups, just as I drew it.
But let us assume that we want to get this information broken down
within a third variable, such as a job code. I want to see the job_nbr and
the total by sex and race within each job code. Our query set starts to get
bigger and bigger. A crosstab can also include other summary data, such
as total or average salary within each cell of the table.
23.7.1 Crosstabs by Cross Join
A solution proposed by John M. Baird of Datapoint in San Antonio,
Texas involves creating a matrix table for each variable in the crosstab,
thus:
SexMatrix
sex Male Female
==================
'M' 1 0
'F' 0 1
RaceMatrix
race asian black caucasian latino Other
========================================================
asian 1 0 0 0 0
black 0 1 0 0 0
caucasian 0 0 1 0 0
latino 0 0 0 1 0
Other 0 0 0 0 1

The query then constructs the cells by using a CROSS JOIN
(Cartesian product) and summation for each one, thus:
23.7 Cross Tabulations 543
SELECT job_nbr,
SUM(asian * male) AS AsianMale,
SUM(asian * female) AS AsianFemale,
SUM(black * male) AS BlackMale,
SUM(black * female) AS BlackFemale,
SUM(cauc * male) AS CaucMale,
SUM(cauc * female) AS CaucFemale,
SUM(latino * male) AS LatinoMale,
SUM(latino * female) AS LatinoFemale,
SUM(other * male) AS OtherMale,
SUM(other * female) AS OtherFemale
FROM Personnel, SexMatrix, RaceMatrix
WHERE (RaceMatrix.race = Personnel.race)
AND (SexMatrix.sex = Personnel.sex)
GROUP BY job_nbr;
Numeric summary data can be obtained from this table. For example,
the total salary for each cell can be computed by
SUM(<race> *
<sex> * salary) AS <cell name> in place of what we have here.
23.7.2 Crosstabs by Outer Joins
Another method, due to Jim Panttaja, uses a series of temporary tables or
VIEWs and then combines them with OUTER JOINs.
CREATE VIEW Guys (race, maletally)
AS SELECT race, COUNT(*)
FROM Personnel
WHERE sex = 'M'
GROUP BY race;

Correspondingly, you could have written:
CREATE VIEW Dolls (race, femaletally)
AS SELECT race, COUNT(*)
FROM Personnel
WHERE sex = 'F'
GROUP BY race;
But they can be combined for a crosstab, without column and row
totals, like this:
544 CHAPTER 23: STATISTICS IN SQL
SELECT Guys.race, maletally, femaletally
FROM Guys LEFT OUTER JOIN Dolls
ON Guys.race = Dolls.race;
The idea is to build a starting column in the crosstab, then
progressively add columns to it. You use the
LEFT OUTER JOIN to avoid
missing-data problems.
23.7.3 Crosstabs by Subquery
Another method takes advantage of the orthogonality of correlated
subqueries in SQL-92. Think about what each row or column in the
crosstab wants.
SELECT DISTINCT race,
(SELECT COUNT(*)
FROM Personnel AS P1
WHERE P0.race = P1.race
AND sex = 'M') AS MaleTally,
(SELECT COUNT(*)
FROM Personnel AS P2
WHERE P0.race = P2.race
AND sex = 'F') AS FemaleTally
FROM Personnel AS P0;

An advantage of this approach is that you can attach another column
to get the row tally by adding
(SELECT COUNT(*)
FROM Personnel AS P3
WHERE P0.race = P3.race) AS RaceTally
Likewise, to get the column tallies, union the previous query with:
SELECT 'Summary',
(SELECT COUNT(*)
FROM Personnel
WHERE sex = 'M') AS GrandMaleTally,
(SELECT COUNT(*)
FROM Personnel
WHERE sex = 'F') AS GrandFemaleTally,
23.8 Harmonic Mean and Geometric Mean 545
(SELECT COUNT(*)
FROM Personnel) AS GrandTally
FROM Personnel;
23.7.4 Crosstabs by CASE Expression
Probably the best method is to use the CASE expression. If you need to
get the final row of the traditional crosstab, you can add:
SELECT sex,
SUM(CASE race WHEN 'caucasian' THEN 1 ELSE 0 END) AS
caucasian,
SUM(CASE race WHEN 'black' THEN 1 ELSE 0 END) AS black,
SUM(CASE race WHEN 'asian' THEN 1 ELSE 0 END) AS asian,
SUM(CASE race WHEN 'latino' THEN 1 ELSE 0 END) AS latino,
SUM(CASE race WHEN 'other' THEN 1 ELSE 0 END) AS other,
COUNT(*) AS row_total
FROM Personnel
GROUP BY sex

UNION ALL
SELECT ' ',
SUM(CASE race WHEN 'caucasian' THEN 1 ELSE 0 END),
SUM(CASE race WHEN 'black' THEN 1 ELSE 0 END),
SUM(CASE race WHEN 'asian' THEN 1 ELSE 0 END),
SUM(CASE race WHEN 'latino' THEN 1 ELSE 0 END),
SUM(CASE race WHEN 'other' THEN 1 ELSE 0 END),
COUNT(*) AS column_total
FROM Personnel;
23.8 Harmonic Mean and Geometric Mean
The harmonic mean is defined as the reciprocal of the arithmetic mean
of the reciprocals of the values of a set. It is appropriate when dealing
with rates and prices. Of limited use, it is found mostly in averaging
rates.
SELECT COUNT(*)/SUM(1.0/x) AS harmonic_mean
FROM Foobar;
The geometric mean is the exponential of the mean of the logs of the
data items. You can also express it as the nth root of the product of the
(n) data items. This second form is more subject to rounding errors than
546 CHAPTER 23: STATISTICS IN SQL
the first. The geometric mean is sometimes a better measure of central
tendency than the simple arithmetic mean when you are analyzing
change over time.
SELECT EXP (AVG (LOG (nbr))) AS geometric_mean
FROM NumberTable;
If you have negative numbers this will blow up, because the logarithm
is not defined for values less than or equal to zero.
23.9 Multivariable Descriptive Statistics in SQL
More and more SQL products are adding more complicated descriptive
statistics to their aggregate function library. For example, CA-Ingres

comes with a very nice set of such tools.
Many of the single-column aggregate functions for which we just gave
code are built-in functions. If you have that advantage, then use them.
They will have corrections for floating-point rounding errors and be
more accurate.
Descriptive statistics are not all single-column computations. You
often want to know relationships among several variables for prediction
and description. Let’s pick one statistic that is representative of this class
of functions and see what problems we have writing our own aggregate
function for it.
23.9.1 Covariance
The covariance is defined as a measure of the extent to which two
variables move together. Financial analysts use it to determine the degree
to which return on two securities is related over time. A high covariance
indicates similar movements. This code is due to Steve Kass:
CREATE TABLE Samples
(sample_nbr INTEGER NOT NULL PRIMARY KEY,
x FLOAT NOT NULL,
y FLOAT NOT NULL);
INSERT INTO Samples
VALUES (1, 3, 9), (2, 2, 7), (3, 4, 12), (4, 5, 15), (5, 6, 17);
SELECT sample_nbr, x, y,
((1.0/n) * SUM((x - xbar)*(y - ybar))) AS covariance
23.9 Multivariable Descriptive Statistics in SQL 547
FROM Samples
CROSS JOIN
(SELECT COUNT(*), AVG(x), AVG(y) FROM Samples)
AS A (n, xbar, ybar)
GROUP BY n;
23.9.2 Pearson’s r

One of the most useful covariants is Pearson’s r, or the linear correlation
coefficient. It measures the strength of the linear association between two
variables. In English, given a set of observations (x1, y1), (x2, y2), . . . ,
(xn, yn), I want to know: when one variable goes up or down, how well
does the other variable follow it?
The correlation coefficient always takes a value between +1 and -1.
Positive one means that they match to each other exactly. Negative one
means that increasing values in one variable correspond to decreasing
values in the other variable. A correlation value close to zero indicates no
association between the variables. In the real world, you will not see +1
or −1 very often—this would mean that you are looking at a natural law,
and not a statistical relationship. The values in between are much more
realistic, with 0.70 or greater being a strong correlation.
The formula translates into SQL in a straightforward manner.
CREATE TABLE Samples
(sample_name CHAR(3) NOT NULL PRIMARY KEY,
x REAL, y REAL);
INSERT INTO Samples
VALUES ('a', 1.0, 2.0), ('b', 2.0, 5.0), ('c', 3.0, 6.0);
r= 0.9608
SELECT (SUM(x - AVG(x))*(y - AVG(y)))
/ SQRT(SUM((x - AVG(x))^2) * SUM((y - AVG(y))^2))
AS pearson_r
FROM Samples;
SQRT() is the square root function, which is quite common in SQL
today, and
^2 is the square of the number. Some products use
POWER(x, n) instead of the exponent notation. Alternately, or you can
use repeated multiplication.
548 CHAPTER 23: STATISTICS IN SQL

23.9.3 NULLs in Multivariable Descriptive Statistics
If (x, y) = (NULL, NULL), then the query will drop the pair in the
aggregate functions, as per the usual rules of SQL. But what is the correct
(or reasonable) behavior if (x, y) has one and only one
NULL in the pair?
We can make several arguments.
1. Drop the pairs that contain any NULLs. That is quick and easy
with a “
WHERE x IS NOT NULL AND y IS NOT NULL”
clause added to the query. The argument is that if you don’t
know one or both values, how can you know what their rela-
tionship is?
2. Convert
(x, NULL) to (x, AVG(y)) and (NULL, y) to
(AVG(x), y). The idea is to “smooth out” the missing values
with a reasonable replacement that is based on the whole set
from which known values were drawn. There might be better
replacement values in a particular situation, but that idea
would still hold.
3. Replace (
NULL, NULL) with (a, a) for some value to say that the
NULLs are in the same grouping. This kind of “pseudo-
equality” is the basis for putting
NULLs into one group in a
GROUP BY operation. I am not sure what the correct practice
for the
(x, NULL) and (y, NULL) pairs are.
4. First calculate a linear regression with the known pairs, say y =
(a + b*x), and then fill in the expected values. If you forgot
your high school algebra, that would be

y[i] = a + b *
x[i] for the pair (x[i], NULL), and x[i] = (y - a) / b.
5. Catch the
SQLSTATE warning code message (found in
Standard SQL) to show that an aggregate function has dropped
NULLs before doing the computations, and use the message to
report to the user about the missing data.
I can also use COUNT(*) and COUNT(x+y) to determine how much
data is missing. I think we would all agree that if I have a small subset of
non-
NULL pairs, then my correlation is less reliable than if I obtained it
from a large subset of non-
NULL pairs.
There is no right answer to this question. You will need to know the
nature of your data to make a good decision.

CHAPTER

24

Regions, Runs, Gaps, Sequences,
and Series

T

ABLES DO NOT HAVE an ordering to their rows. Yes, the physical storage
of the rows in many SQL products might be ordered if the product is
built on an old file system. More modern implementations might not
construct and materialize the result set rows until the end of the query
execution.

The first rule in a relational database is that all relationships are
shown in tables by values in columns. This means that things
involving an ordering must have a table with at least two columns.
One column, the sequence number, is the primary key; the other
column has the value that holds that position in the sequence.
The sequence column has consecutive unique integers, without any
gaps in the numbering. Examples of this sort of data would be ticket
numbers, time series data taken at fixed intervals, and the like. The
ordering of those identifiers carries some information, such as physical
or temporal location. A subsequence is a set of consecutive unique
identifiers within a larger containing sequence that has some property.
This property is usually consecutive numbering.
For example, given the data

CREATE TABLE List
(seq_nbr INTEGER NOT NULL UNIQUE,
val INTEGER NOT NULL UNIQUE);

550 CHAPTER 24: REGIONS, RUNS, GAPS, SEQUENCES, AND SERIES

INSERT INTO List
VALUES (1, 99), (2, 10), (3, 11), (4, 12), (5, 13), (6, 14), (7,
0);

You can find subsequences of size three that follow the rule—(10, 11,
12), (11, 12, 13), and (12, 13, 14)—but the longest sequence is (10, 11,
12, 13, 14), and it is of size five.
A run is like a sequence, but the numbers do not have to be
consecutive, just increasing and contiguous. For example, given the run
{(1, 1), (2, 2), (3, 12), (4, 15), (5, 23)}, you can find subruns of size

three: (1, 2, 12), (2, 12, 15), and (12, 15, 23).
A region is contiguous, and all the values are the same. For example,
{(1, 1), (2, 0), (3, 0), (4, 0), (5, 25)} has a region of zeros that is three
items long.
In procedural languages, you would simply sort the data and scan it.
In SQL, you have to define everything in terms of sets and nested sets.
Some of these queries can be done with the OLAP addition to SQL-99,
but they are not yet common in SQL products.

24.1 Finding Subregions of Size (

n

)

This example is adapted from

SQL and Its Applications

(Lorie and
Daudenarde 1991). You are given a table of theater seats:

CREATE TABLE Theater
(seat_nbr INTEGER NOT NULL PRIMARY KEY, sequencing number
occupancy_status CHAR(1) NOT NULL values
CONSTRAINT valid_occupancy_status
CHECK (occupancy_status IN ('A', 'S'));

In this table, an occupancy_status code of ‘A’ means available, and ‘S’
means sold. Your problem is to write a query that will return the

subregions of (

n

) consecutive seats still available. Assume that
consecutive seat_nbrs means that the seats are also consecutive for a
moment, ignoring rows of seating where seat_nbr(

n

) and seat_nbr((

n

) +
1) might be on different physical theater rows. For (

n

) = 3, we can write a
self-

JOIN

query, thus:

SELECT T1.seat_nbr, T2.seat_nbr, T3.seat_nbr
FROM Theater AT T1, Theater AT T2, Theater AT T3
WHERE T1.occupancy_status = 'A'

24.2 Numbering Regions 551

AND T2.occupancy_status = 'A'
AND T3.occupancy_status = 'A'
AND T2.seat_nbr = T1.seat_nbr + 1
AND T3.seat_nbr = T2.seat_nbr + 1;

The trouble with this answer is that it works only for (

n

= 3). This
pattern can be extended for any (

n

), but what we really want is a
generalized query where we can use (

n

) as a parameter to the query.
The solution given by Lorie and Daudenarde starts with a given
seat_nbr and looks at all the available seats between it and ((

n

) - 1) seats
further up. The real trick is switching from the English-language
statement “All seats between here and there are available” to the passive-

voice version, “Available is the occupancy_status of all the seats between
here and there,” so that you can see the query.



SELECT seat_nbr, ' thru ', (seat_nbr + (:(n) - 1))
FROM Theater AS T1
WHERE occupancy_status = 'A'
AND 'A' = ALL (SELECT occupancy_status
FROM Theater AS T2
WHERE T2.seat_nbr > T1.seat_nbr
AND T2.seat_nbr <= T1.seat_nbr + (:(n) - 1));

Please notice that this returns subregions. That is, if seats (1, 2, 3, 4,
5) are available, this query will return (1, 2, 3), (2, 3, 4), and (3, 4, 5) as
its result set.

24.2 Numbering Regions

Instead of looking for a region, we want to number the regions in the
order in which they appear. For example, given a view or table with a
payment history, we want to break it into groupings of behavior—for
example, whether or not the payments were on time or late.

CREATE TABLE PaymentHistory
(payment_nbr INTEGER NOT NULL PRIMARY KEY,
paid_on_time CHAR(1) DEFAULT 'Y' NOT NULL
CHECK(paid_on_time IN ('Y', 'N')));
INSERT INTO PaymentHistory
VALUES (1006, 'Y'), (1005, 'Y'),