a comparison of neural network architectures for

A Comparison of Neural Network Architectures for
Handwritten Digit Recognition

Eman M. El-Sheikh
1
, Bradley A. Swain
1
, and Mohamed A. Khabou
2

1
Department of Computer Science, University of West Florida, Pensacola, FL, USA
2
Department of Electrical & Computer Engineering, University of West Florida, Pensacola, FL, USA


Abstract - In this paper, we describe the development
and use of an artificial neural network architecture for
recognizing handwritten digit data. The feed-forward
neural network, which was implemented in Java, was
designed to be modular and parameterized so that
various parameters and settings of the network could be
easily modified. The network was used to recognize a
large set of data representing handwritten digits.
Various configurations of the network were run on the
data set to determine the best settings for this type of
data set. A correct classification rate of 93.77% was
achieved on a testing set containing 3000 images not
used in the training phase. We present a summary and
analysis of the results.
Keywords: Machine learning, neural network,

handwritten digit recognition, classification, Java
application.

1 Introduction
The focus of this work is the development, use,
and analysis of an artificial neural network architecture
for the recognition of handwritten digit data. Neural
networks have a long track record of successful use for
pattern recognition problems, including character
recognition. The application of recognizing
handwritten digit data was selected for this study due
to the numerous potential applications, including mail
sorting, automated check reading, and data entry for
hand-held devices.
In this paper, we describe the development of a
software neural network architecture. The system,
which was implemented using the Java programming
language, was designed to be modular and
parameterized to allow easy modification of the
network’s size, structure, and parameters. The overall
motivation for our study is the analysis of various
machine learning techniques for real-world
applications with the purpose of determining the
appropriateness and usefulness of each technique.
Specifically, for the study described in this paper, we
focused on the use of neural network learning
techniques for handwritten digit recognition. Our
objective was two-fold: to test the neural network
implementation on a large-scale, real-world data set
using a variety of network parameters and sizes, and to

determine the best network structure and settings for
the handwritten digit data set. The results provide
evidence for the use of neural network learning
techniques for this type of application and data set, and
insight about the appropriate design, use, and
parameterization of the network to achieve acceptable
results.
In the next section, we provide a brief summary of the
literature relevant to neural network techniques and
applications. Section 3 describes the design,
implementation, and use of the neural network. Section
4 describes the methods used for our tests, including a
description of the data set, experiments we conducted,
and results of our study. The last section includes the
conclusions derived and lessons learned from
analyzing the results, as well as plans for future work.

2 Literature Review
The use of neural networks in pattern recognition
problems dates back to the early 1950s when the early
perceptron models were introduced by Rosenblatt. In
the 1960s and 1970s the progress in neural networks
slowed due to limited computing capabilities and the
unfounded perceptions concerning the limitations of
neural networks. The enhancement of computing
capabilities in the 1980s and the emphatic support by
Hopfield, Rumelhart, and other researchers accelerated
the development of new neural network models and
theories. Today numerous neural network models and
algorithms are available including back propagation

learning, competitive learning, Kohonen learning,
Hopfield networks, Boltzman machines, shared-weight
neural networks and self-organizing neural networks
[4].
Neural networks have been used in a variety of
applications ranging from stock price prediction to
automatic target detection and recognition. They are
especially useful as pattern classifiers and/or
recognizers because of their robustness, fault tolerance,
and universal function approximation capability. In
theory, a multi-layer feed-forward neural network with
enough nodes and hidden layers can approximate any
bounded function and its derivative to any arbitrary
accuracy given a “large enough” training set [5].
Results published in [1] state that a network with N
inputs, one hidden layer of H units, and a total of W
weights will require in the order of W/
ε
training
patterns to yield an error less than
ε
on the test set.

3 Neural Network Architecture
The neural network application used for this
paper was written using the Sun Java 2 Runtime
Environment (v. 1.5.0), mainly on Mac OS X 10.5.1,
but has been successfully used on several Linux
distributions as well. Theoretically, the program should
work on any platform for which a Java virtual machine

exists, as it uses no platform-specific features or calls.
The design of the application attempted to make as few
assumptions as possible. All attributes, input, and
output values are stored as real numbers, allowing for
both continuous and discrete attributes. Missing
attribute values, however, are not handled by the
implementation. The application uses a simple feed-
forward back-propagation-trained network, using the
sigmoid activation function, and is designed to allow
the use of a different activation function. The system
was designed to be modular and easily re-configured.
Configurable attributes of the network include the
number of input neurons, the number of hidden layers,
the number of neurons in each hidden layer, the
number of output neurons, the learning rate (α), the
threshold at which the value of the sigmoid function
will cause the neurons to fire, and the number of
epochs for which to train the network, all of which are
passed into the application by way of command-line
parameters. There is a complete connection between
nodes in layers, with every node in the layer being
connected to every node in the following layer, and the
initial weights for these connections are initialized to
random values in the [-1, 1] range.
4 Methods and Results
In this section, we describe the data set that we
used for the study, the configurations of the neural
network architecture that we used for our experiments,
and the result of our test runs.
4.1 Data

The problem of handwritten character recognition
has been studied intensively for over thirty years [7].
This research has been driven by the great number of
potential applications including the recognition of
handwritten addresses for mail sorting purposes.
Unlike the problem of machine-printed character
recognition, the difficulties in the handwritten
character recognition come from the fact that people
write differently, as can be seen in Figure 1.
Numerous methods have been proposed to capture the
distinctive features of handwritten characters [2, 3, 6,
8]. These approaches can be classified into two
categories: global analysis and structural analysis. As
an example of the first category, we find techniques
such as template matching, moment invariant,
mathematical transforms (Fourier, Walsh, Hadamard,
etc). In the second category, the main goal is to capture
the essential shape features of the characters mainly
from their skeletons or contours. Such features include
loops, endpoints, junctions, arcs, concavities,
convexities, and strokes.
The main challenges to create a classifier based on
features are: (1) what type of features to use, (2) how
many features to use, (3) how to select the “best”
features, and (4) how to define criteria for selecting the
“best” features.
The use of feature vectors as input to handwritten digit
classifiers has many advantages over the use of the
two-dimensional digit images. The most important
advantage is the reduction of the dimension of the

input space. The smallest “reasonable” dimensions of a
digit image are 16×16, which corresponds to an input
vector of dimension 256. Such input dimensions would
produce a huge neural network if multiple hidden
layers are used. The use of features in this specific case
would reduce the dimension of the input space
significantly.



Figure 1. Representative samples from the data set.


In our experiments we used a data set that consisted of
6000 unconstrained binary images of handwritten
digits (600 images from each digit class) from actual
USPS mail pieces. The data was collected at the
Environmental Research Institute of Michigan (ERIM)
and was used by many researchers to test their
handwriting recognition systems [3]. All images were
moment-normalized to a size of 24×18 as described in
[2]. A subset of this data set (2000 images) was used
for training our system and a different subset (3500
images) was used for testing. The results we report in
this paper were achieved with the testing data set.
Three sets of features were used to train and test the
neural networks. Each set contains 60 features, which
have values in the [0, 1] range and represent the degree
of correlation between a feature’s template and the
input image. A value of 0 indicates that a feature

template did not match the digit image; a value of 1
indicates a complete match; and values in between
these two extremes indicate a partial match.
These 60 “best” features were selected using three
measures: an information measure, an orthogonality
measure, and a combination of the two. The
information measure is based on Shannon’s entropy. It
is a statistical measurement of the capability of a
feature to separate digit classes. If the information
measure of a feature is high, it indicates that the feature
is able to separate at least some of the classes well
since the conditional probabilities of the classes given
that the feature is either present or not present are
significantly different from each other. The
orthogonality measure is based on the known
mathematical property that the best basis to represent
an n-dimensional space is using n orthogonal vectors
(i.e. their dot product equals 0). The orthogonality
measure yields features that do not respond similarly to
an input image, i.e. the response of one feature would
be high when the other is low on some classes and
vice-versa on all the other classes. Thus the features
selected based on the orthogonality measure provide
discrimination capability since they behave differently
on each class. The third measure we used is a
combination of the information and orthogonality
measures. More details about the measures and
selection process can be found in [3].
4.2 Experiments
We ran the networks on the handwritten digit

data, training on the first 2000 digits in each file and
testing on the final 3500 digits. For a comparison
between architectures, we ran tests using two hidden
layers with 25 and 20 nodes, two hidden layers with 25
and 15 nodes, one hidden layer with 25 nodes, and one
hidden layer with 15 nodes, respectively. The other
parameters to the network were fixed for all runs of all
architectures. After trial runs with different values, a
learning rate of 0.05 was used, as it was small enough
to avoid saturating the network, but large enough to
provide ample learning. The number of training epochs
was set at 500, as experimentation proved this to
provide the best testing results, compared to 200
(under-trained), and 750 and 1000 epochs (over-
trained). Finally, for all tests the firing threshold for the
activation function of the hidden neurons was set at
0.9, which experimentally gave the best results.


Table 1. Summary of performance results for each architecture configuration and each feature data set.

Hidden
Layer(s)
Info
Features
Time
(mm:ss)
Orthogonal
Features
Time

(mm:ss)
Combination
Features
Time
(mm:ss)
25/20 90.63% 04:18 93.77% 04:20 93.74% 04:15
25/15 90.83% 03:56 93.34% 03:59 93.34% 03:59
25 90.54% 03:18 93.57% 03:19 93.43% 03:20
15 89.91% 02:08 92.86% 02:08 92.83% 02:08

4.3 Results
The results for the experiments are summarized in
Table 1. The table shows the percentage of handwritten
digit samples correctly classified for different
configurations of the architecture. For each row, the
first column shows the architecture of the network
used. The first two runs both used two hidden layers;
the last two used only one hidden layer. The remaining
columns consist of data set and time pairs, with the
percentage of the test data correctly classified and the
amount of time that the network took to train and test,
respectively. Each architecture and data set
combination was run 5 times, and the best result
presented for each configuration of the architecture that
was tested.
Based on previous work done on the data sets, the two
hidden layer architecture with 25 nodes in the first
layer and 15 nodes in the second layer was expected to
perform best. For the information features data set, this
held true, but for the other data sets, it actually

performed worse than both the 25/20 two hidden layer
architecture and the single hidden layer with 25 nodes
architecture. For the other data sets, the 25/20
architecture outperformed the other network
architectures, with only a minor increase in time
required over the 25/15 architecture. With a slightly
reduced performance, however, the single hidden layer
with 25 nodes performs nearly as well as the 25/20
architecture, but reduces the running time by as much
as a minute.

5 Conclusions
Overall, the implemented system was able to
successfully classify the test data using each of three
feature sets. The single hidden layer network with 15
nodes performed better on the orthogonal and
combination features data set, achieving a prediction
accuracy of 92.86% and 92.83%, respectively. The
information set did not perform as well due to the fact
that some features were almost identical and hence did
not provide more information to the network. Adding
more nodes to the single hidden layer improved the
accuracy slightly. With 25 nodes in the hidden layer,
the accuracy improved to 90.54% for the information
features, 93.57% for the orthogonal features, and
93.43% for the combination features data set. This
slight improvement in accuracy comes at a price of an
increase in time requirements, since the running time
increased by over a minute. It is interesting to note that
expanding the architecture to include two hidden layers

with 25 nodes in the first layer and 15 nodes in the
second layer increases the running time but does not
improve the classification accuracy for the orthogonal
and combination features data set, and only has a slight
benefit for the information features data set, improving
the accuracy from 90.54% to 90.83%. Increasing the
number of nodes in the second hidden layer from 15 to
20 provides the best classification accuracy for the
orthogonal features data set and the combined features
data set, namely 93.77% and 93.74% respectively, but
does not improve the prediction rate for the
information features data.
The orthogonal features data set and the combined
features data set provided better results for all
configurations of the network than the information
features data set. This was somewhat expected since
the selection process can theoretically yield very
similar information features which do not provide extra
class discrimination to the neural network. This
possibility is not present in the orthogonality and
combined features since the feature selection process
using these two measures inherently discourages
similar features. Both the orthogonal and combination
features data sets provided fairly comparable results, in
terms of both accuracy and running time. The single
hidden layer with 25 nodes performs nearly as well as
the two hidden layer architecture with 25 and 20 nodes
on all three data sets, but reduces the running time by
as much as a minute.
We would like to continue testing the developed

system with other large data sets for handwritten digit
and character recognition. We are also interested in
exploring the combination of different machine
learning techniques for handwritten digit recognition
and similar character recognition problems. More
specifically, we are interested in developing a system
that integrates a decision tree learning technique with a
neural network architecture and testing it with the three
data sets used in this study to compare the performance
of such a system with the results of using a decision
tree or neural network separately. We anticipate that
such a technique that fuses both machine learning
techniques will yield better results than using each
technique individually, and will allow the recognition
of larger, more complex data sets.

6 References
[1] E. Baum and D. Haussler, “What Size Net Gives
Valid Generalization?” Neural Computation, vol. 1,
no. 1, pp. 151-160, 1989.
[2] P. D. Gader, B. Forester, M. Ganzberger, A.
Gillies, B. Mitchell, M. Whalem, and T. Yocum,
“Recognition of Handwritten Digits Using Template
and Model Matching,” Pattern Recognition, vol. 24,
pp. 421-432, 1991.
[3] P. D. Gader and M. A. Khabou, “Automatic
Feature Generation for Handwritten Digit
Recognition,” IEEE Trans. Pattern Analysis Machine
Intelligence, vol. 18, no. 12, pp. 1256-1262, 1996.
[4] S. Haykin, Neural Networks, a Comprehensive

Foundation, MacHillan Publishing Co., 1994.
[5] K. Hornik, M. Stinchrombe and H. White,
“Universal Approximation of an Unknown mapping
and its Derivatives Using Multilayer Feed-forward
Networks,” Neural Networks, vol. 3, pp. 551-560,
1990.
[6] C.Y. Suen, “Distinctive Features in the Automatic
Recognition of Handprinted Characters,” Signal
Processing, vol. 4, pp. 193-207, 1982.
[7] C.Y. Suen , “Character Recognition by Computer
and Application,” Handbook of Pattern Recognition
and Image Processing, Academic Press, pp. 569-586,
1986.
[8] C.Y. Suen, C. Nadal, R. Legault, T.A. Mai, and
L. Lam, “Computer Recognition of Unconstrained
Handwritten Numerals.” Proc. of IEEE, vol. 80, no 7,
pp. 1162-1180, 1992.