Joe Celko s SQL for Smarties - Advanced SQL Programming P48 pptx

442 CHAPTER 21: AGGREGATE FUNCTIONS

'Chester' 'A.' 'Arthur' 'R' 1881 1885
'Grover' ' ' 'Cleveland' 'D' 1885 1889
'Benjamin' ' ' 'Harrison' 'R' 1889 1893
'Grover' ' ' 'Cleveland' 'D' 1893 1897
'William' ' ' 'McKinley' 'R' 1897 1901
'Theodore' ' ' 'Roosevelt' 'R' 1901 1909
'William' 'H.' 'Taft' 'R' 1909 1913
'Woodrow' ' ' 'Wilson' 'D' 1913 1921
'Warren' 'G.' 'Harding' 'R' 1921 1923
'Calvin' ' ' 'Coolidge' 'R' 1923 1929
'Herbert' 'C.' 'Hoover' 'R' 1929 1933
'Franklin' 'D.' 'Roosevelt' 'D' 1933 1945
'Harry' 'S.' 'Truman' 'D' 1945 1953
'Dwight' 'D.' 'Eisenhower' 'R' 1953 1961
'John' 'F.' 'Kennedy' 'D' 1961 1963
'Lyndon' 'B.' 'Johnson' 'D' 1963 1969
'Richard' 'M.' 'Nixon' 'R' 1969 1974
'Gerald' 'R.' 'Ford' 'R' 1974 1977
'James' 'E.' 'Carter' 'D' 1977 1981
'Ronald' 'W.' 'Reagan' 'R' 1981 1989
'George' 'H.W.' 'Bush' 'R' 1989 1993
'William' 'J.' 'Clinton' 'D' 1993 2001
'George' 'W. ' 'Bush' 'R' 2001 NULL

Your civics teacher has just asked you to tell her how many people
have been President of the United States. So you write the query as

SELECT

COUNT(*)



FROM

Presidents; and get the wrong answer. For
those of you who have been out of high school too long, more than one
Adams, more than one John, and more than one Roosevelt have served
as president. Many people have had more than one term in office, and
Grover Cleveland served two discontinuous terms. In short, this
database is not a simple one-row, one-person system. What you really
want is not

COUNT(*)

, but something that is able to look at unique
combinations of multiple columns. You cannot do this in one column, so
you need to construct an expression that is unique. The point is that you
need to be very sure that the expression you are using as a parameter is
really what you wanted to count.
The

COUNT([ALL] <value expression>)

returns the number of
members in the

<value expression>

set. The

NULL

s were thrown
away before the counting took place, and an empty set returns zero. The
best way to read this is: “Count the number of known values in this

21.2 SUM() Functions 443

expression,” with stress on the word known. In this example you might
use

COUNT(first_name || ' ' || initial || ' ' ||
last_name)

.



The

COUNT(DISTINCT <value expression>)

returns the
number of unique members in the

<value expression>

set. The

NULL

s were thrown away before the counting took place, and then all
redundant duplicates are removed (i.e., we keep one copy). Again, an
empty set returns a zero, just as with the other counting functions.
Applying this function to a key or a unique column is the same as using
the

COUNT(*)

function, but the optimizer may not be smart enough to
spot it.
Notice that the use of the keywords

ALL

and

DISTINCT

follows the
same pattern here as they did in the [

ALL

|

DISTINCT

] options in the

SELECT

clause of the query expressions.

21.2 SUM() Functions

This function works only with numeric values. You should also consult
your particular product’s manuals to find out the precision of the results
for exact and approximate numeric data types.

SUM([ALL] <value expression>)

returns the numeric total of
all known values. The

NULL

s are removed before the summation takes
place. An empty set returns an empty result set, not a zero. If there are
other columns in the

SELECT

list, then that empty set will be converted
into a

NULL

.

SUM(DISTINCT <value expression>)

returns the numeric total
of all known, unique values. The

NULL

s and all redundant duplicates
were removed before the summation took place. Again, an empty set
returns an empty result set, not a zero.
That last rule is hard for people to understand. If there are other
columns in the

SELECT

list, then that empty result set will be converted
into a

NULL

. This is true for the rest of the Standard aggregate functions:

no rows
SELECT SUM(x)
FROM EmptyTable;

one row with (0, NULL) in it
SELECT COUNT(*), SUM(x)
FROM EmptyTable;

444 CHAPTER 21: AGGREGATE FUNCTIONS

The summation of a set of numbers looks as though it should be
easy, but it is not. Make two tables with the same set of positive and
negative approximate numeric values, but put one in random order
and have the other sorted by absolute value. The sorted table will give
more accurate results. The reason is simple: positive and negative
values of the same magnitude will be added together and will get a
chance to cancel each other out. There is also less chance of an
overflow or underflow error during calculations. Most PC SQL
implementations and a lot of mainframe implementations do not
bother with this trick, because it would require a sort for every

SUM()


statement, which would take a long time.
Whenever an exact or approximate numeric value is assigned to exact
numeric, it may not fit into the storage allowed for it. SQL says that the
database engine will use an approximation that preserves leading
significant digits of the original number after rounding or truncating.
The choice of whether to truncate or round is implementation-defined,
however. This can lead to some surprises when you have to shift data
among SQL implementations, or move storage values from a host
language program into an SQL table. It is probably a good idea to create
the columns with one more decimal place than you think you need.

Truncation is defined as truncation toward zero; this means that 1.5
would truncate to 1, and

−

1.5 would truncate to

−

1. This is not true for
all programming languages; everyone agrees on truncation toward zero
for the positive numbers, but you will find that negative numbers may
truncate away from zero (e.g.,

−

1.5 would truncate to

−

2). SQL is also
wishy-washy on rounding, leaving the implementation free to determine
its method. There are two major types of rounding, the scientific method
and the commercial method, which are discussed in Section 3.2.1 on
rounding and truncation math in SQL.

21.3 AVG() Functions

AVG([ALL] <value expression>)

returns the average of the values
in the value expression set. An empty set returns an empty result set. A
set of all

NULL

s will become an empty set. Remember that in general,

AVG(x)

is not the same as

(SUM(x)/COUNT(*))

; the

SUM(x)

function
has thrown away the

NULL

s, but the

COUNT(*)

has not.
Likewise,

AVG(DISTINCT <value expression>)

returns the
average of the distinct known values in the

<value expression> set.
Applying this function to a key or a unique column is the same as the
using
AVG(<value expression>) function.
21.3 AVG() Functions 445
Remember that in general AVG(DISTINCT x) is not the same as
AVG(x) or (SUM(DISTINCT x)/COUNT(*)). The SUM(DISTINCT x)
function has thrown away the duplicate values and
NULLs, but the
COUNT(*) has not. An empty set returns an empty result set.
The SQL engine is probably using the same code for the totaling in
the
AVG() that it used in the SUM() function. This leads to the same
problems with rounding and truncation, so you should experiment a
little with your particular product to find out what happens.
But even more troublesome than those problems is the problem with
the average itself, because it does not really measure central tendency
and can be very misleading. Consider the chart below, from Darrell
Huff’s superlative little book, How to Lie with Statistics (Huff 1954). The
Sample Company has 25 employees, earning the following salaries:
Number of
Employees Salary Statistic
===================================
12 $2,000 Mode, Minimum
1 $3,000 Median

4 $3,700
3 $5,000
1 $5,700 Average
2 $10,000
1 $15,000
1 $45,000 Maximum
The average salary (or, more properly, the arithmetic mean) is
$5,700. When the boss is trying to look good to the unions, he uses this
figure. When the unions are trying to look impoverished, they use the
mode, which is the most frequently occurring value, to show that the
exploited workers are making $2,000 (which is also the minimum salary
in this case).
A better measure in this case is the median, which will be discussed
later; that is, the employee with just as many cases above him as below
him. That gives us $3,000. The rule for calculating the median is that if
there is no actual entity with that value, you fake it.
Most people take an average of the two values on either side of where
the median would be; others jump to the higher or lower value. The
mode also has a problem, because not every distribution of values has
one mode. Imagine a country in which there are as many very poor
people as there are very rich people, and there is nobody in between.
446 CHAPTER 21: AGGREGATE FUNCTIONS
This would be a bimodal distribution. If there were sharp classes of
incomes, that would be a multimodal distribution.
Some SQL products have median and mode aggregate functions as
extensions, but they are not part of the standard. We will discuss in
detail how to write them in pure SQL in Chapter 23.
21.3.1 Averages with Empty Groups
The query used here is a bit tricky, so this section can be skipped on
your first reading. Sometimes you need to count an empty set as part of

the population when computing an average.
This is easier to explain with an example that was posted on
CompuServe. A fish and game warden is sampling different bodies of
water for fish populations. Each sample falls into one or more groups
(muddy bottoms, clear water, still water, and so on) and she is trying to
find the average of something that is not there. This is neither quite as
strange as it first sounds, nor quite as simple, either. She is collecting
sample data on fish in a table like this:
CREATE TABLE Samples
(sample_id INTEGER NOT NULL,
fish CHAR(20) NOT NULL,
found_cnt INTEGER NOT NULL,
PRIMARY KEY (sample_id, fish));
CREATE TABLE SampleGroups
(group_id INTEGER NOT NULL,
sample_id INTEGER NOT NULL,
PRIMARY KEY (group_id, sample_id);
Assume some of the data looks like this:
Samples
sample_id fish found_cnt
============================
1 'Seabass' 14
1 'Minnow' 18
2 'Seabass' 19
21.3 AVG() Functions 447
SampleGroups
group_id sample_id
=====================
1 1
1 2

2 2
She needs to get the average number of each species of fish in the
sample groups. For example, using sample group 1 as shown, which has
samples 1 and 2, we could use the parameters
:my_fish =‘Minnow’
and
:my_group = 1 to find the average number of minnows in sample
group 1, thus:
SELECT fish, AVG(found_cnt)
FROM Samples
WHERE sample_id
IN (SELECT sample_id
FROM SampleGroups
WHERE group_id = :my_group)
AND fish = :my_fish
GROUP BY fish;
But this query will give us an average of 18 minnows, which is wrong.
There were no minnows for sample_id = 2, so the average is ((18 + 0)/2)
= 9. The other way is to do several steps to get the correct answer—first
use a
SELECT statement to get the number of samples involved, then
another
SELECT to get the sum, and then manually calculate the
average.
The obvious answer is to enter a count of zero for each animal under
each sample_id, instead of letting it be missing, so you can use the
original query. You can create the missing rows with:
INSERT INTO Samples
SELECT M1.sample_id, M2.fish, 0
FROM Samples AS M1, Samples AS M2

WHERE NOT EXISTS (SELECT *
FROM Samples AS M3
WHERE M1.sample_id = M3.sample_id
AND M2.fish = M3.fish);
448 CHAPTER 21: AGGREGATE FUNCTIONS
Unfortunately, it turns out that we have over 100,000 different
species of fish and thousands of samples. This trick will fill up more disk
space than we have on the machine. The best trick is to use this
statement:
SELECT fish, SUM(found_cnt)/
(SELECT COUNT(sample_id)
FROM SampleGroups
WHERE group_id = :my_group)
FROM Samples
WHERE fish = :my_fish
GROUP BY fish;
This query is using the rule that the average is the sum of values
divided by the count of the set. Another way to do this would be to use
an
OUTER JOIN and preserve all the group IDs, but that would create
NULLs for the fish that are not in some of the sample groups, and you
would have to handle them.
21.3.2 Averages across Columns
The sum of several columns can be done with COALESCE() function to
effectively remove the
NULLs by replacing them with zeros:
SELECT (COALESCE(c1, 0.0)
+ COALESCE(c2, 0.0)
+ COALESCE(c3, 0.0)) AS c_total
FROM Foobar;

Likewise, the minimum and maximum values of several columns can
be done with a
CASE expression, or the GREATEST() and LEAST()
functions.
Taking an average across several columns is easy if none of the
columns are
NULL. You simply add the values and divide by the number
of columns. However, getting rid of
NULLs is a bit harder. The first trick
is to count the
NULLs:
SELECT (COALESCE(c1-c1, 1)
+ COALESCE(c2-c2, 1)
+ COALESCE(c3-c3, 1)) AS null_cnt
FROM Foobar;
21.4 Extrema Functions 449
The trick is to watch out for a row with all NULLs in it. This could lead
to a division by zero error.
SELECT CASE WHEN COALESCE(c1, c2, c3) IS NULL
THEN NULL
ELSE (COALESCE(c1, 0.0)
+ COALESCE(c2, 0.0)
+ COALESCE(c3, 0.0))
/ (3 - (COALESCE(c1-c1, 1)
+ COALESCE(c2-c2, 1)
+ COALESCE(c3-c3, 1))
END AS hortizonal_avg
FROM Foobar;
21.4 Extrema Functions
The MIN() and MAX() functions are known as extrema functions in

mathematics. They assume that the elements of the set have an ordering,
so it makes sense to select a first or last element based on its value. SQL
provides two simple extrema functions, and you can write queries to
generalize these to (n) elements.
21.4.1 Simple Extrema Functions
MAX([ALL | DISTINCT] <value expression>) returns the
greatest known value in the
<value expression> set. This function
will work on character and temporal values, as well as numeric values.
An empty set returns an empty result set. Technically, you can write
MAX(DISTINCT <value expression>), but it is the same as
MAX(<value expression>); this form exists only for completeness,
and nobody ever uses it.
MIN([ALL | DISTINCT] <value expression>) returns the
smallest known value in the
<value expression> set. This function
will also work on character and temporal values, as well as numeric
values. An empty set returns a
NULL. Likewise, MIN(DISTINCT
<value expression>) exists, but it is defined only for completeness
and nobody ever uses it.
The
MAX() for a set of numeric values is the largest. The MAX() for a
set of temporal data types is the one closest to
9999-12-31, which is
the final data in the ISO-8601 Standard. The
MAX() for a set of character
strings is the last one in the ascending sort order. Likewise, the
MIN()
for a set of numeric values is the smallest. The

MIN() for a set of
450 CHAPTER 21: AGGREGATE FUNCTIONS
temporal data types is the one furthest from 9999-12-31. The MIN()
for a set of character strings is the first one in the ascending sort order,
but you have to know the collation used.
People have a hard time understanding the
MAX() and MIN()
aggregate functions when they are applied to temporal data types. They
seem to expect the
MAX() to return the date closest to the current date.
Likewise, if the set has no dates before the current date, they seem to
expect the
MIN() function to return the date closest to the current date.
Human psychology wants to use the current time as an origin point for
temporal reasoning.
Consider the predicate “
billing_date < (CURRENT_DATE -
INTERVAL '90' DAY)” as an example. Most people have to stop and
figure out that this is looking for billings that are over 90 days past due.
This same thing happens with
MIN() and MAX() functions.
SQL also has funny rules about comparing
VARCHAR strings, which
can cause problems. When two strings are compared for equality, the
shortest one is right-padded with blanks; then they are compared
position for position. Thus, the strings ‘
John ’ and ‘John ’ are equal.
You will have to check your implementation of SQL to see which string is
returned as the
MAX() and which as the MIN(), or whether there is any

pattern to it at all.

There are some tricks with extrema functions in subqueries that differ
from product to product. For example, to find the current employee
status in a table of Salary Histories, the obvious query is:
SELECT *
FROM SalaryHistory AS S0
WHERE S0.change_date
= (SELECT MAX(S1.change_date)
FROM SalaryHistory AS S1
WHERE S0.emp_id = S1.emp_id);
But you can also write the query as:
SELECT *
FROM SalaryHistory AS S0
WHERE NOT EXISTS
(SELECT *
FROM SalaryHistory AS S1
WHERE S0.emp_id = S1.emp_id
AND S0.change_date < S1.change_date);
21.4 Extrema Functions 451
The correlated subquery with a MAX() will be implemented by going
to the subquery and building a working table, which is grouped by
emp_id. Then for each group you will keep track of the maximum and
save it for the final result.
However, the
NOT EXISTS version will find the first row that meets
the criteria and, when found, return
TRUE. Therefore, the NOT
EXISTS() predicate might run faster.
21.4.2 Generalized Extrema Functions

This is known as the Top (or Bottom) (n) values problem, and it
originally appeared in Explain magazine; it was submitted by Jim
Wankowski of Hawthorne, CA (Wankowski n.d.). You are given a table
of Personnel and their salaries. Write a single SQL query that will display
the three highest salaries from that table. It is easy to find the maximum
salary with the simple query
SELECT MAX(salary) FROM
Personnel; but SQL does not have a maximum function that will
return a group of high values from a column. The trouble with this query
is that the specification is bad, for several reasons.
1. How do we define “best salary” in terms of an ordering? Is it
base pay or does it include commissions? For the rest of this
section, assume that we are using a simple table with a column
that has the salary for each employee.
2. What if we have three or fewer personnel in the company? Do
we report all the personnel we do have? Or do we return a
NULL, empty result set or error message? This is the equivalent
of calling the contest for lack of entries.
3. How do we handle two personnel who tied? Include them all
and allow the result set to be bigger than three? Pick an
arbitrary subset and exclude someone? Or do we return a
NULL, empty result set, or error message?
To make these problems more explicit, consider this table:
Personnel
emp_name salary
==================
'Able' 1000.00
'Baker' 900.00