Joe Celko s SQL for Smarties - Advanced SQL Programming P64 pps

602 CHAPTER 26: SET OPERATIONS
CREATE TABLE B (i INTEGER NOT NULL);
INSERT INTO B VALUES (2), (2), (3), (3);
The UNION and INTERSECT operations have regular behavior in that:
(A UNION B) = SELECT DISTINCT (A UNION ALL B) = ((1), (2), (3))
and
(A INTERSECT B) = SELECT DISTINCT (A INTERSECT ALL B) = (2)
However,
(A EXCEPT B) <> SELECT DISTINCT (A EXCEPT ALL B)
Or, more literally, (1) <> ((1), (2)) for the tables given in the example.
Likewise, we have:
(B EXCEPT A) = SELECT DISTINCT (B EXCEPT ALL A) = (3)
by a coincidence of the particular values used in these tables.
26.4 Equality and Proper Subsets
At one point, when SQL was still in the laboratory at IBM, there was a
CONTAINS operator that would tell you if one table was a subset of
another. It disappeared in later versions of the language and no vendor
picked it up. Set equality was never part of SQL as an operator, so you
would have to have used the two expressions
((A CONTAINS B) AND
(B CONTAINS A)) to find out.
Today, you can use the methods shown in the section on Relational
Division to determine containment or equality. However, Itzik Ben-Gan
came up with a novel approach for finding containment and equality
that is worth a mention.
SELECT SUM(DISTINCT match_col)
FROM (SELECT CASE
WHEN S1.col
IN (SELECT S2.col FROM S2)
THEN 1 ELSE -1 END
FROM S1) AS X(match_col)

HAVING SUM(DISTINCT match_col) = :n;
26.4 Equality and Proper Subsets 603
You can set (:n) to 1, 0, or −1 for each particular test.
When I find a matching row in S1, I get a +1; when I find a
mismatched row in S1, get a −1 and they sum together to give me a zero.
Therefore, S1 is a proper subset of S2. If they sum to +1, then they are
equal. If they sum to −1, they are disjoint.


CHAPTER

27

Subsets

I

AM DEFINING SUBSET operations as queries, which extract a particular
subset from a given set, as opposed to set operations, which work
among sets. The obvious way to extract a subset from a table is just to
use a

WHERE

clause, which will pull out the rows that meet that
criterion. But not all the subsets we want are easily defined by such a
simple predicate. This chapter is a collection of tricks for constructing
useful, but not obvious, subsets from a table.

27.1 Every

n

th Item in a Table

SQL is a set-oriented language, which cannot identify individual rows
by their physical positions in a disk file that holds a table. Instead, a
unique logical key is detected by logical expressions, and a row is
retrieved. If you are given a file of employees in which the ordering of
the file is based on their employee numbers, and you want to pick out
every

n

th employee record for a survey, the job is easy. You write a
procedure that loops through the file and writes every

n

th one to a
second file.
The immediate thought of how this should be done in SQL is to
simply compute

MOD (emp_nbr, :n)

, where

MOD()

is the modulo
function found in most SQL implementations, and save those
employee rows where this function is zero. The trouble is that

606 CHAPTER 27: SUBSETS

employees are not issued consecutive identification numbers. The
identification numbers are unique.
Vendor extensions often include an exposed physical row locator that
gives a sequential numbering to the physical records; this sequential
numbering can be used to perform these functions. This practice is a
complete violation of Dr. Codd’s definition of a relational database, and it
requires that the underlying physical implementation use a contiguous
sequential record for each row. Such things are highly proprietary, but
because these features are so low-level, they will run very fast on that one
particular product.
Row numbers have more problems than being nonstandard. If the
physical storage is rearranged, then the row numbers have to change.
Users logged on and looking at the same base table through different

VIEW

s may or may not get the same row number for the same physical
row. One of the advantages of an RDBMS was supposed to be that the
logical view of the data would be consistent, even when the physical
storage changed.
You can get similar results with a self-

JOIN

on the Personnel table
to partition it into a nested series of grouped tables, just as we did for
the “to top

n

” problem. You then pick out the largest value in each
group. There may be an index or a uniqueness constraint on the
emp_nbr column to ensure uniqueness, so the

EXISTS

predicate will
get a performance boost.



SELECT P1.emp_nbr
FROM Personnel AS P1
WHERE EXISTS
(SELECT MAX(emp_nbr)
FROM Personnel AS P2
WHERE P1.emp_nbr >= P2.emp_nbr
HAVING MOD (COUNT(*), :n) = 0);

A nonnested version of the same query looks like this:

SELECT P1.emp_nbr
FROM Personnel AS P1, Personnel AS P2
WHERE P1.emp_nbr >= P2.emp_nbr

GROUP BY P1.emp_nbr
HAVING MOD (COUNT(*), :n) = 0;

27.2 Picking Random Rows from a Table 607

Both queries count the number of P2 rows with a value less than the
P1 row.

27.2 Picking Random Rows from a Table

The answer is that, basically, you cannot directly pick a set of random
rows from a table in SQL. There is no randomize operator in the
standard, and you don’t often find the same pseudo-random number
generator function in various vendor extensions, either.
Picking random rows from a table for a statistical sample is a handy
thing, and you do it in other languages with a pseudo-random number
generator. There are two kinds of random drawings from a set, with or
without replacement. If SQL had random number functions, I suppose
they would be shown as

RANDOM(x)

and

RANDOM(DISTINCT x)

. But
there is no such function in SQL, and none is planned. Examples from
the real world include dealing a poker hand (a random with no
replacement situation) and shooting craps (a random with replacement

situation). If two players in a poker game get identical cards, you are
using a pinochle deck. In a craps game, each roll of the dice is
independent of the previous one and can repeat it.
The problem is that SQL is a set-oriented language, and wants to do
an operation “all at once” on a well-defined set of rows. Random sets are
defined by a nondeterministic procedure by definition, instead of a
deterministic logic expression.
The SQL/PSM language does have an option to declare or create a
procedure that is

DETERMINISTIC

or

NOT DETERMINISTIC

. The

DETERMINISTIC

option means that the optimizer can compute this
function once for a set of input parameter values and then use that result
everywhere in the current SQL statement that a call to the procedure
with those parameters appears. The

NOT DETERMINISTIC

option
means given the same parameters, you might not get the same results for
each call to the procedure within the same SQL statement.

Unfortunately, most SQL products do not have this feature in their
proprietary procedural languages. Thus, the random number function in
Oracle is nondeterministic and the one in SQL Server is deterministic.
For example,

CREATE TABLE RandomNbrs
(seq_nbr INTEGER NOT NULL PRIMARY KEY,
randomizer FLOAT NOT NULL);

608 CHAPTER 27: SUBSETS

INSERT INTO RandomNbrs
VALUES (1, RANDOM()),
(2, RANDOM()),
(3, RANDOM());

This query will result in the three rows all getting the same value in
the randomizer column in a version of SQL Server, but three different
numbers in a version of Oracle.
While subqueries are not allowed in

DEFAULT

clauses, system-related
functions such as

CURRENT_TIMESTAMP

and

CURRENT_USER

are
allowed. In some SQL implementations, this includes the

RANDOM()


function.

CREATE TABLE RandomNbrs2
(seq_nbr INTEGER PRIMARY KEY,
randomizer FLOAT warning !! not standard SQL
DEFAULT (
(CASE (CAST(RANDOM() + 0.5 AS INTEGER) * -1)
WHEN 0.0 THEN 1.0 ELSE -1.0 END)
* MOD (CAST(RANDOM() * 100000 AS INTEGER), 10000)
* RANDOM())
NOT NULL);
INSERT INTO RandomNbrs2
VALUES (1, DEFAULT);
(2, DEFAULT),
(3, DEFAULT),
(4, DEFAULT),
(5, DEFAULT),
(6, DEFAULT),
(7, DEFAULT),
(8, DEFAULT),
(9, DEFAULT),
(10, DEFAULT);

Here is a sample output from an SQL Server 7.0 implementation.

seq_nbr randomizer
============================
1 -121.89758452446999
2 -425.61113508053933

27.2 Picking Random Rows from a Table 609

3 3918.1554683876675
4 9335.2668286173412
5 54.463890640027664
6 -5.0169085346410522
7 -5430.63417246276
8 915.9835973796487
9 28.109161998753301
10 741.79452047043048

The best way to do this is to add a column to the table to hold a
random number, then use an external language with a good pseudo-
random number generator in its function library to load the new column
with random values with a cursor in a host language. You have to do it
this way, because random number generators work differently from
other function calls. They start with an initial value called a “seed”
(shown as Random[0] in the rest of this discussion) provided by the user
or the system clock. The seed is used to create the first number in the
sequence, Random[1]. Then each call, Random

[

n

], to the function uses
the previous number to generate the next one, Random[

n

+1].
There is no way to do a sequence of actions in SQL without a cursor,
so you are in procedural code.
The term “pseudo-random number generator” is often referred to as a
just “random number generator,” but this is technically wrong. All of the
generators will eventually return a value that appeared in the sequence
earlier and the procedure will hang in a cycle. Procedures are
deterministic, and we are living in a mathematical heresy when we try to
use them to produce truly random results. However, if the sequence has
a very long cycle and meets some other tests for randomness over the
range of the cycle, then we can use it.
There are many kinds of generators. The linear congruence pseudo-
random number generator family has generator formulas of the form:

Random[n+1] := MOD ((x * Random[n] + y), m);

There are restrictions on the relationships among

x

,

y,

and

m

that deal
with their relative primality. Knuth gives a proof that if

Random[0] is not a multiple of 2 or 5
m = 10^e where (e >= 5)
y = 0
MOD (x, 200) is in the set (3, 11, 13, 19, 21, 27, 29, 37, 53,

610 CHAPTER 27: SUBSETS

59, 61, 67, 77, 83, 91, 109, 117, 123, 131, 133, 139, 141, 147,
163, 171, 173, 179, 181, 187, 189, 197)

then the period will be 5 * 10^(

e

-2).
There are old favorites that many C programmers use from this
family, such as:

Random(n+1) := (Random(n) * 1103515245) + 12345;

Random(n+1) := MOD ((16807 * Random(n)), ((2^31) - 1));

The first formula has the advantage of not requiring a

MOD

function,
so it can be written in standard SQL. However, the simplest generator
that can be recommended (Park and Miller) uses:

Random(n+1) := MOD ((48271 * Random(n)), ((2^31) - 1));

Notice that the modulus is a prime number; this is important.
The period of this generator is ((2^31)

−

2), which is 2,147,483,646,
or more than two billion numbers before this generator repeats. You
must determine whether this is long enough for your application.
If you have an

XOR

function in your SQL, then you can also use shift
register algorithms. The

XOR

is the bitwise exclusive

OR

that works on an
integer as it is stored in the hardware; I would assume 32 bits on most
small computers. Some usable shift register algorithms are:

Random(n+1) := Random(n-103) XOR Random(n-250);
Random(n+1) := Random(n-1063) XOR Random(n-1279);

One method for writing a random number generator on the fly when
the vendor’s library does not have one is to pick a seed using one or more
key columns and a call to the system clock’s fractional seconds, such as

RANDOM(keycol + EXTRACT (SECOND FROM CURRENT_TIME)) *
1000

. This avoids problems with patterns in the keys, while the key
column values ensure uniqueness of the seed values.
Another method is to use a

PRIMARY KEY

or

UNIQUE

column(s) and
apply a hashing algorithm. You can pick one of the random number
generator functions already discussed and use the unique value, as if it

were the seed, as a quick way to get a hashing function. Hashing
algorithms try to be uniformly distributed, so if you can find a good one,
you will approach nearly unique random selection. The trick is that the

27.2 Picking Random Rows from a Table 611

hashing algorithm has to be simple enough to be written in the limited
math available in SQL.
Once you have a column of random numbers, you can convert the
random numbers into a randomly ordered sequence with this statement:

UPDATE RandomNbrs
SET randomizer = (SELECT COUNT(*)
FROM Sequence AS S1
WHERE S1.randomizer <= Sequence.seq_nbr);

To get one random row from a table, you can use this approach:

CREATE VIEW LotteryDrawing (keycol, , spin)
AS SELECT LotteryTickets.*,
(RANDOM(<keycol> + <fractional seconds from clock>))
FROM LotteryTickets
GROUP BY spin
HAVING COUNT(*) = 1;

Then simply use this query:

SELECT *
FROM LotteryDrawing
WHERE spin = (SELECT MAX(spin)

FROM LotteryDrawing)

The pseudo-random number function is not standard SQL, but it is
common enough. Using the keycol as the seed

probably

means that you
will get a different value for each row, but we can avoid duplicates with
the

GROUP BY HAVING

. Adding the fractional seconds will change
the result every time, but it might be illegal in some SQL products,
which disallow variable elements in

VIEW

definitions.
Let’s assume you have a function called

RANDOM()

that returns a
random number between 0.00 and 1.00. If you just want one random
row out of the table, and you have a numeric key column, Tom Moreau
proposed that you could find the

MAX()

and

MIN()

, then calculate a
random number between them.

SELECT L1.*
FROM LotteryDrawing AS L1
WHERE col_1