Data Mining and Knowledge Discovery Handbook, 2 Edition part 13 pot

100 Barak Chizi and Oded Maimon
Rokach, L., Decomposition methodology for classification tasks: a meta decomposer frame-
work, Pattern Analysis and Applications, 9(2006):257–271.
Rokach L., Genetic algorithm-based feature set partitioning for classification prob-
lems,Pattern Recognition, 41(5):1676–1700, 2008.
Rokach L., Mining manufacturing data using genetic algorithm-based feature set decompo-
sition, Int. J. Intelligent Systems Technologies and Applications, 4(1):57-78, 2008.
Rokach, L. and Maimon, O., Theory and applications of attribute decomposition, IEEE In-
ternational Conference on Data Mining, IEEE Computer Society Press, pp. 473–480,
2001.
Rokach L. and Maimon O., Feature Set Decomposition for Decision Trees, Journal of Intel-
ligent Data Analysis, Volume 9, Number 2, 2005b, pp 131–158.
Rokach, L. and Maimon, O., Clustering methods, Data Mining and Knowledge Discovery
Handbook, pp. 321–352, 2005, Springer.
Rokach, L. and Maimon, O., Data mining for improving the quality of manufacturing: a
feature set decomposition approach, Journal of Intelligent Manufacturing, 17(3):285–
299, 2006, Springer.
Rokach, L., Maimon, O., Data Mining with Decision Trees: Theory and Applications, World
Scientific Publishing, 2008.
Rokach L., Maimon O. and Lavi I., Space Decomposition In Data Mining: A Clustering Ap-
proach, Proceedings of the 14th International Symposium On Methodologies For Intel-
ligent Systems, Maebashi, Japan, Lecture Notes in Computer Science, Springer-Verlag,
2003, pp. 24–31.
Rokach, L. and Maimon, O. and Averbuch, M., Information Retrieval System for Medical
Narrative Reports, Lecture Notes in Artificial intelligence 3055, page 217-228 Springer-
Verlag, 2004.
Rokach, L. and Maimon, O. and Arbel, R., Selective voting-getting more for less in sensor
fusion, International Journal of Pattern Recognition and Artificial Intelligence 20 (3)
(2006), pp. 329–350.
Scherf, M. and Brauer, W. Feature selection by means of a feature weighting approach. Tech-
nical Report FKI- 221- 97, Technische Universit at Munchen 1997.

Setiono, R. and Liu, H. Chi2: Feature selection and discretization of numeric attributes. In
Proceedings of the Seventh IEEE International Conference on Tools with Artificial In-
telligence, 1995
Singh, M. and Provan, G. M. Efficient learning of selective Bayesian classifiers. In Machine
Learning: Proceedings of the Thirteenth International network Conference on Machine
Learning. Morgan Kaufmann, 1996.
Skalak, B. Prototype and feature selection by sampling and random mutation hill climbing
algorithms. In Machine Learning: Proceedings of the Eleventh International Conference.
Morgan Kaufmann, 1994.
Vafaie, H. and De Jong, K. Genetic algorithms as a tool for restructuring feature space rep-
resentations. In Proceedings of the International Conference on Tools with A. I. IEEE
Computer Society Press, 1995.
Ward, B., What’s Wrong with Economics. New York: Basic Books, 1972.
6
Discretization Methods
Ying Yang
1
, Geoffrey I. Webb
2
, and Xindong Wu
3
1
School of Computer Science and Software Engineering, Monash University, Melbourne,
Australia
2
Faculty of Information Technology
Monash University, Australia

3
Department of Computer Science

University of Vermont, USA

Summary. Data-mining applications often involve quantitative data. However, learning from
quantitative data is often less effective and less efficient than learning from qualitative data.
Discretization addresses this issue by transforming quantitative data into qualitative data. This
chapter presents a comprehensive introduction to discretization. It clarifies the definition of
discretization. It provides a taxonomy of discretization methods together with a survey of
major discretization methods. It also discusses issues that affect the design and application of
discretization methods.
Key words: Discretization, quantitative data, qualitative data.
Introduction
Discretization is a data-processing procedure that transforms quantitative data into
qualitative data.
Data Mining applications often involve quantitative data. However, there exist
many learning algorithms that are primarily oriented to handle qualitative data (Ker-
ber, 1992, Dougherty et al., 1995, Kohavi and Sahami, 1996). Even for algorithms
that can directly deal with quantitative data, learning is often less efficient and less
effective (Catlett, 1991, Kerber, 1992, Richeldi and Rossotto, 1995, Frank and Wit-
ten, 1999). Hence discretization has long been an active topic in Data Mining and
knowledge discovery. Many discretization algorithms have been proposed. Evalua-
tion of these algorithms has frequently shown that discretization helps improve the
performance of learning and helps understand the learning results.
This chapter presents an overview of discretization. Section 6.1 explains the ter-
minology involved in discretization. It clarifies the definition of discretization, which
has been defined in many differing way in previous literature. Section 6.2 presents a
O. Maimon, L. Rokach (eds.), Data Mining and Knowledge Discovery Handbook, 2nd ed.,
DOI 10.1007/978-0-387-09823-4_6, © Springer Science+Business Media, LLC 2010
102 Ying Yang, Geoffrey I. Webb, and Xindong Wu
comprehensive taxonomy of discretization approaches. Section 6.3 introduces typi-
cal discretization algorithms corresponding to the taxonomy. Section 6.4 addresses

the issue that different discretization strategies are appropriate for different learn-
ing problems. Hence designing or applying discretization should not be blind to its
learning context. Section 6.5 provides a summary of this chapter.
6.1 Terminology
Discretization transforms one type of data to another type. In the large amount of
existing literature that addresses discretization, there is considerable variation in the
terminology used to describe these two data types, including ‘quantitative’ vs. ‘qual-
itative’, ‘continuous’ vs. ‘discrete’, ‘ordinal’ vs. ‘nominal’, and ‘numeric’ vs. ‘cat-
egorical’. It is necessary to make clear the difference among the various terms and
accordingly choose the most suitable terminology for discretization.
We adopt the terminology of statistics (Bluman, 1992, Samuels and Witmer,
1999), which provides two parallel ways to classify data into different types. Data
can be classified into either qualitative or quantitative. Data can also be classified
into different levels of measurement scales. Sections 6.1.1 and 6.1.2 summarize this
terminology.
6.1.1 Qualitative vs. quantitative
Qualitative data, also often referred to as categorical data, are data that can be
placed into distinct categories. Qualitative data sometimes can be arrayed in a mean-
ingful order. But no arithmetic operations can be applied to them. Examples of qual-
itative data are: blood type of a person: A, B, AB, O; and assignment evaluation: fail,
pass, good, excellent.
Quantitative data are numeric in nature. They can be ranked in order. They also
admit to meaningful arithmetic operations. Quantitative data can be further classified
into two groups, discrete or continuous.
Discrete data assume values that can be counted. The data cannot assume all val-
ues on the number line within their value range. An example is: number of children
in a family.
Continuous data can assume all values on the number line within their value
range. The values are obtained by measuring. An example is: temperature.
6.1.2 Levels of measurement scales

In addition to being classified into either qualitative or quantitative, data can also be
classified by how they are categorized, counted or measured. This type of classifi-
cation uses measurement scales, and four common levels of scales are: nominal,
ordinal, interval and ratio.
6 Discretization Methods 103
The nominal level of measurement scales classifies data into mutually exclusive
(non-overlapping), exhaustive categories in which no meaningful order or ranking
can be imposed on the data. An example is: blood type of a person: A, B, AB, O
The ordinal level of measurement scales classifies data into categories that can
be ranked. However, the differences between the ranks cannot be calculated by arith-
metic. An example is: assignment evaluation: fail, pass, good, excellent. It is mean-
ingful to say that the assignment evaluation of pass ranks higher than that of fail. It
is not meaningful in the same way to say that the blood type of A ranks higher than
that of B.
The interval level of measurement scales ranks data, and the differences between
units of measure can be calculated by arithmetic. However, zero in the interval level
of measurement does not mean ‘nil’ or ‘nothing’ as zero in arithmetic means. An
example is: Fahrenheit temperature. It has a meaningful difference of one degree
between each unit. But 0 degree Fahrenheit does not mean there is no heat. It is
meaningful to say that 74 degree is two degrees higher than 72 degree. It is not
meaningful in the same way to say that the evaluation of excellent is two degrees
higher than the evaluation of good.
The ratio level of measurement scales possesses all the characteristics of interval
measurement, and there exists a zero that, the same as arithmetic zero, means ‘nil’ or
‘nothing’. In consequence, true ratios exist between different units of measure. An
example is: number of children in a family. It is meaningful to say that family X has
twice as many children as does family Y. It is not meaningful in the same way to say
that 100 degree Fahrenheit is twice as hot as 50 degree Fahrenheit.
The nominal level is the lowest level of measurement scales. It is the least power-
ful in that it provides the least information about the data. The ordinal level is higher,

followed by the interval level. The ratio level is the highest. Any data conversion
from a higher level of measurement scales to a lower level of measurement scales
will lose information. Table 6.1 gives a summary of the characteristics of different
levels of measurement scales.
Table 6.1. Measurement Scales
Level Ranking ? Arithmetic operation ? Arithmetic zero ?
Nominal no no no
Ordinal yes no no
Interval yes yes no
Ratio yes yes yes
6.1.3 Summary
In summary, the following classification of data types applies:
1. qualitative data:
a) nominal;
b) ordinal;
104 Ying Yang, Geoffrey I. Webb, and Xindong Wu
2. quantitative data:
a) interval, either discrete or continuous;
b) ratio, either discrete or continuous.
We believe that ‘discretization’ as it is usually applied in data mining is best de-
fined as the transformation from quantitative data to qualitative data. In consequence,
we will refer to data as either quantitative or qualitative throughout this chapter.
6.2 Taxonomy
There exist diverse taxonomies in the existing literature to classify discretization
methods. Different taxonomies emphasize different aspects of the distinctions among
discretization methods.
Typically, discretization methods can be either primary or composite. Primary
methods accomplish discretization without reference to any other discretization
method. Composite methods are built on top of some primary method(s).
Primary methods can be classified as per the following taxonomies.

1. Supervised vs. Unsupervised (Dougherty et al., 1995). Methods that use the
class information of the training instances to select discretization cut points are
supervised. Methods that do not use the class information are unsupervised.
Supervised discretization can be further characterized as error-based, entropy-
based or statistics-based according to whether intervals are selected using met-
rics based on error on the training data, entropy of the intervals, or some statisti-
cal measure.
2. Parametric vs. Non-parametric. Parametric discretization requires input from
the user, such as the maximum number of discretized intervals. Non-parametric
discretization only uses information from data and does not need input from the
user.
3. Hierarchical vs. Non-hierarchical. Hierarchical discretization selects cut points
in an incremental process, forming an implicit hierarchy over the value range.
The procedure can be split or merge (Kerber, 1992). Split discretization ini-
tially has the whole value range as an interval, then continues splitting it into
sub-intervals until some threshold is met. Merge discretization initially puts
each value into an interval, then continues merging adjacent intervals until some
threshold is met. Some discretization methods utilize both split and merge pro-
cesses. For example, intervals are initially formed by splitting, and then a merge
process is performed to post-process the formed intervals. Non-hierarchical dis-
cretization does not form any hierarchy during discretization. For example, many
methods scan the ordered values only once, sequentially forming the intervals.
4. Univariate vs. Multivariate (Bay, 2000). Methods that discretize each attribute
in isolation are univariate. Methods that take into consideration relationships
among attributes during discretization are multivariate.
5. Disjoint vs. Non-disjoint (Yang and Webb, 2002). Disjoint methods discretize
the value range of the attribute under discretization into disjoint intervals. No
6 Discretization Methods 105
intervals overlap. Non-disjoint methods discretize the value range into intervals
that can overlap.

6. Global vs. Local (Dougherty et al., 1995). Global methods discretize with re-
spect to the whole training data space. They perform discretization once only,
using a single set of intervals throughout a single classification task. Local meth-
ods allow different sets of intervals to be formed for a single attribute, each set
being applied in a different classification context. For example, different dis-
cretizations of a single attribute might be applied at different nodes of a decision
tree (Quinlan, 1993).
7. Eager vs. Lazy (Hsu et al., 2000, Hsu et al., 2003). Eager methods perform
discretization prior to classification time. Lazy methods perform discretization
during the classification time.
8. Time-sensitive vs. Time-insensitive. Under time-sensitive discretization, the
qualitative value associated with a quantitative value can change along the time.
That is, the same quantitative value can be discretized into different values de-
pending on the previous values observed in the time series. Time-insensitive
discretization only uses the stationary pro-perties of the quantitative data.
9. Ordinal vs. Nominal. Ordinal discretization transforms quantitative data into
ordinal qualitative data. It aims at taking advantage of the ordering information
implicit in quantitative attributes, so as not to make values 1 and 2 as dissimi-
lar as values 1 and 10. Nominal discretization transforms quantitative data into
nominal qualitative data. The ordering information is hence discarded.
10. Fuzzy vs. Non-fuzzy (Wu, 1995, Wu, 1999, Ishibuchi et al., 2001). Fuzzy dis-
cretization first discretizes quantitative attribute values into intervals. It then
places some kind of membership function at each cut point as fuzzy borders.
The membership function measures the degree of each value belonging to each
interval. With these fuzzy borders, a value can be discretized into a few different
intervals at the same time, with varying degrees. Non-fuzzy discretization forms
sharp borders without employing any membership function.
Composite methods first choose some primary discretization method to form the
initial cut points. They then focus on how to adjust these initial cut points to achieve
certain goals. The taxonomy of a composite method sometimes is flexible, depending

on the taxonomy of its primary method.
6.3 Typical methods
Corresponding to our taxonomy in the previous section, we here enumerate some
typical discretization methods. There are many other methods that are not reviewed
due to the space limit. For a more comprehensive study on existing discretization
algorithms, Yang (2003) and Wu (1995) offer good sources.
106 Ying Yang, Geoffrey I. Webb, and Xindong Wu
6.3.1 Background and terminology
A term often used for describing a discretization approach is ‘cut point’. Discretiza-
tion forms intervals according to the value range of the quantitative data. It then as-
sociates a qualitative value to each interval. A cut point is a value among the quanti-
tative data where an interval boundary is located by a discretization method. Another
commonly-mentioned term is ‘boundary cut point’, which are values between two
instances with different classes in the sequence of instances sorted by a quantitative
attribute. It has been proved that evaluating only the boundary cut points is sufficient
for finding the minimum class information entropy (Fayyad and Irani, 1993).
We use the following terminology. Data comprises a set or sequence of instances.
Each instance is described by a vector of attribute values. For classification learning,
each instance is also labelled with a class. Each attribute is either qualitative or quan-
titative. Classes are qualitative. Instances from which one learns cut points or other
knowledge are training instances. If a test instance is presented, a learning algo-
rithm is asked to make a prediction about the test instance according to the evidence
provided by the training instances.
6.3.2 Equal-width, equal-frequency and fixed-frequency discretization
We arrange to present these three methods together because they are seemingly sim-
ilar but actually different. They all are typical of unsupervised discretization. They
are also typical of parametric discretization.
When discretizing a quantitative attribute, equal width discretization (EWD)
(Catlett, 1991, Kerber, 1992, Dougherty et al., 1995) predefines k, the number of
intervals. It then divides the number line between v

min
and v
max
into k intervals of
equal width, where v
min
is the minimum observed value, v
max
is the maximum ob-
served value. Thus the intervals have width w =(v
max
−v
min
)/k and the cut points
are at v
min
+ w,v
min
+ 2w,···,v
min
+(k −1)w.
When discretizing a quantitative attribute, equal-frequency discretization (EFD)
(Catlett, 1991, Kerber, 1992,Dougherty et al., 1995) predefines k, the number of in-
tervals. It then divides the sorted values into k intervals so that each interval contains
approximately the same number of training instances. Suppose there are n training
instances, each interval then contains n/k training instances with adjacent (possibly
identical) values. Note that training instances with identical values must be placed
in the same interval. In consequence it is not always possible to generate k equal-
frequency intervals.
When discretizing a quantitative attribute, fixed-frequency discretization

(FFD) (Yang and Webb, 2004) predefines a sufficient interval frequency k. Then
it discretizes the sorted values into intervals so that each interval has approximately
4
the same number k of training instances with adjacent (possibly identical) values.
It is worthwhile contrasting EFD and FFD, both of which form intervals of equal
frequency. EFD fixes the interval number that is usually arbitrarily chosen. FFD fixes
4
Just as for EFD, because of the existence of identical values, some intervals can have in-
stance frequency exceeding k.
6 Discretization Methods 107
the interval frequency that is not arbitrary but to ensure each interval contains suffi-
cient instances to supply information such as for estimating probability.
6.3.3 Multi-interval-entropy-minimization discretization ((MIEMD)
Multi-interval-entropy-minimization discretization (Fayyad and Irani, 1993) is typ-
ical of supervised discretization. It is also typical of non-parametric discretization.
To discretize an attribute, MIEMD evaluates as a candidate cut point the midpoint
between each successive pair of the sorted values. For evaluating each candidate cut
point, the data are discretized into two intervals and the resulting class information
entropy is calculated. A binary discretization is determined by selecting the cut point
for which the entropy is minimal amongst all candidates. The binary discretization
is applied recursively, always selecting the best cut point. A minimum description
length criterion (MDL) is applied to decide when to stop discretization.
6.3.4 ChiMerge, StatDisc and InfoMerge discretization
EWD and EFD are non-hierarchical discretization. MIEMD involves a split proce-
dure and hence is hierarchical discretization. A typical merge approach to hierarchi-
cal discretization is ChiMerge (Kerber, 1992). It uses the
χ
2
(Chi square) statistic
to determine if the relative class frequencies of adjacent intervals are distinctly dif-

ferent or if they are similar enough to justify merging them into a single interval.
The ChiMerge algorithm consists of an initialization process and a bottom-up merg-
ing process. The initialization process contains two steps: (1) ascendingly sort the
training instances according to their values for the attributes being discretized, (2)
construct the initial discretization, in which each instance is put into its own interval.
The interval merging process contains two steps, repeated continuously: (1) compute
the
χ
2
for each pair of adjacent intervals, (2) merge the pair of adjacent intervals with
the lowest
χ
2
value. Merging continues until all pairs of intervals have
χ
2
values ex-
ceeding a predefined
χ
2
-threshold. That is, all intervals are considered significantly
different by the
χ
2
independence test. The recommended
χ
2
-threshold is at the 0.90,
0.95 or 0.99 significant level.
StatDisc discretization (Richeldi and Rossotto, 1995) extends ChiMerge to al-

low any number of intervals to be merged instead of only 2 as ChiMerge does. Both
ChiMerge and StatDisc are based on a statistical measure of dependency. The statis-
tical measures treat an attribute and a class symmetrically. A third merge discretiza-
tion, InfoMerge (Freitas and Lavington, 1996) argues that an attribute and a class
should be asymmetric since one wants to predict the value of the class attribute given
the discretized attribute but not the reverse. Hence InfoMerge uses information loss,
which is calculated as the amount of information necessary to identify the class of an
instance after merging and the amount of information before merging, to direct the
merge procedure.
108 Ying Yang, Geoffrey I. Webb, and Xindong Wu
6.3.5 Cluster-based discretization
The above mentioned methods are all univariate. A typical multivariate discretization
technique is cluster-based discretization (Chmielewski and Grzymala-Busse, 1996).
This method consists of two steps. The first step is cluster formation to determine
initial intervals for the quantitative attributes. The second step is post-processing to
minimize the number of discretized intervals. Instances here are deemed as points in
n-dimensional space which is defined by n attribute values. During cluster formation,
the median cluster analysis method is used. Clusters are initialized by allowing each
instance to be a cluster. New clusters are formed by merging two existing clusters that
exhibit the greatest similarity between each other. The cluster formation continues as
long as the level of consistency of the partition is not less than the level of consistency
of the original data. Once this process is completed, instances that belong to the same
cluster are indiscernible by the subset of quantitative attributes, thus a partition on
the set of training instances is induced. Clusters can be analyzed in terms of all
attributes to find out cut points for each attribute simultaneously. After discretized
intervals are formed, post-processing picks a pair of adjacent intervals among all
quantitative attributes for merging whose resulting class entropy is the smallest. If
the consistency of the dataset after the merge is above a given threshold, the merge
is performed. Otherwise this pair of intervals are marked as non-mergable and the
next candidate is processed. The process stops when each possible pair of adjacent

intervals are marked as non-mergable.
6.3.6 ID3 discretization
ID3 provides a typical example of local discretization. ID3 (Quinlan, 1986) is an
inductive learning program that constructs classification rules in the form of a de-
cision tree. It uses local discretization to deal with quantitative attributes. For each
quantitative attribute, ID3 divides its sorted values into two intervals in all possible
ways. For each division, the resulting information gain of the data is calculated. The
attribute that obtains the maximum information gain is chosen to be the current tree
node. And the data are divided into subsets corresponding to its two value intervals.
In each subset, the same process is recursively conducted to grow the decision tree.
The same attribute can be discretized differently if it appears in different branches of
the decision tree.
6.3.7 Non-disjoint discretization
The above mentioned methods are all disjoint discretization. Non-disjoint discretiza-
tion (NDD) (Yang and Webb, 2002), on the other hand, forms overlapping inter-
vals for a quantitative attribute, always locating a value toward the middle of its
discretized interval. This strategy is desirable since it can efficiently form for each
single quantitative value a most appropriate interval.
When discretizing a quantitative attribute, suppose there are N instances. NDD
identifies among the sorted values t

atomic intervals, (a

1
,b

1
],(a

2

,b

2
], ,(a

t

,b

t

],
6 Discretization Methods 109
each containing s

instances, so that
5
s

=
s
3
s

×t

= N. (6.1)
One interval is formed for each set of three consecutive atomic intervals, such
that the kth (1 ≤ k ≤ t


−2) interval (a
k
,b
k
] satisfies a
k
= a

k
and b
k
= b

k+2
. Each
value v is assigned to interval (a

i
−
1
,b

i
+
1
] where i is the index of the atomic interval
(a

i
,b


i
] such that a

i
< v ≤b

i
, except when i = 1 in which case v is assigned to interval
(a

1
,b

3
] and when i = t

in which case v is assigned to interval (a

t

−2
,b

t

]. Figure 6.1
illustrates the procedure. As a result, except in the case of falling into the first or the
last atomic interval, a numeric value is always toward the middle of its corresponding
interval, and intervals can overlap with each other.

Atomic Interval
In
te
r
va
l
Fig. 6.1. Atomic Intervals Compose Actual Intervals
6.3.8 Lazy discretization
The above mentioned methods are all eager. In comparison, lazy discretization
(LD) (Hsu et al., 2000,Hsu et al., 2003) defers discretization until classification time.
It waits until a test instance is presented to determine the cut points for each quan-
titative attribute of this test instance. When classifying an instance, LD creates only
one interval for each quantitative attribute containing its value from the instance, and
leaves other value regions untouched. In particular, it selects a pair of cut points for
each quantitative attribute such that the value is in the middle of its corresponding in-
terval. Where the cut points locate is decided by LD’s primary discretization method,
such as EWD.
5
Theoretically any odd number k besides 3 is acceptable in (6.1) as long as the same number
k of atomic intervals are grouped together later for the probability estimation. For simplic-
ity, we take k = 3 for demonstration.