IT training numerical linear aproximation in c abdelmalek malek 2008 05 19

Numerical
Linear
Approximation
in C

© 2008 by Taylor & Francis Group, LLC


CHAPMAN & HALL/CRC

Numerical Analysis and Scientific Computing
Aims and scope:

Scientific computing and numerical analysis provide invaluable tools for the sciences and engineering.
This series aims to capture new developments and summarize state-of-the-art methods over the
whole spectrum of these fields. It will include a broad range of textbooks, monographs and
handbooks. Volumes in theory, including discretisation techniques, numerical algorithms, multiscale
techniques, parallel and distributed algorithms, as well as applications of these methods in multidisciplinary fields, are welcome. The inclusion of concrete real-world examples is highly encouraged.
This series is meant to appeal to students and researchers in mathematics, engineering and
computational science.

Editors
Choi-Hong Lai
School of Computing and
Mathematical Sciences
University of Greenwich

Frédéric Magoulès
Applied Mathematics and
Systems Laboratory

Ecole Centrale Paris

Editorial Advisory Board
Mark Ainsworth
Mathematics Department
Strathclyde University

Peter Jimack
School of Computing
University of Leeds

Todd Arbogast
Institute for Computational
Engineering and Sciences
The University of Texas at Austin

Takashi Kako
Department of Computer Science
The University of Electro-Communications

Craig C. Douglas
Computer Science Department
University of Kentucky
Ivan Graham
Department of Mathematical Sciences
University of Bath

Peter Monk
Department of Mathematical Sciences
University of Delaware

Francois-Xavier Roux
ONERA
Arthur E.P. Veldman
Institute of Mathematics and Computing Science
University of Groningen

Proposals for the series should be submitted to one of the series editors above or directly to:
CRC Press, Taylor & Francis Group
24-25 Blades Court
Deodar Road
London SW15 2NU
UK
© 2008 by Taylor & Francis Group, LLC


Numerical
Linear
Approximation
in C

Nabih N. Abdelmalek, Ph.D.
William A. Malek, M.Eng.

© 2008 by Taylor & Francis Group, LLC


Chapman & Hall/CRC
Taylor & Francis Group
6000 Broken Sound Parkway NW, Suite 300
Boca Raton, FL 33487-2742

© 2008 by Taylor & Francis Group, LLC
Chapman & Hall/CRC is an imprint of Taylor & Francis Group, an Informa business
No claim to original U.S. Government works
Printed in the United States of America on acid-free paper
10 9 8 7 6 5 4 3 2 1
International Standard Book Number-13: 978-1-58488-978-6 (Hardcover)
This book contains information obtained from authentic and highly regarded sources Reasonable efforts have been made to publish reliable data and information, but the author and publisher
cannot assume responsibility for the validity of all materials or the consequences of their use. The
Authors and Publishers have attempted to trace the copyright holders of all material reproduced
in this publication and apologize to copyright holders if permission to publish in this form has not
been obtained. If any copyright material has not been acknowledged please write and let us know so
we may rectify in any future reprint
Except as permitted under U.S. Copyright Law, no part of this book may be reprinted, reproduced,
transmitted, or utilized in any form by any electronic, mechanical, or other means, now known or
hereafter invented, including photocopying, microfilming, and recording, or in any information
storage or retrieval system, without written permission from the publishers.
For permission to photocopy or use material electronically from this work, please access www.
copyright.com ( or contact the Copyright Clearance Center, Inc. (CCC)
222 Rosewood Drive, Danvers, MA 01923, 978-750-8400. CCC is a not-for-profit organization that
provides licenses and registration for a variety of users. For organizations that have been granted a
photocopy license by the CCC, a separate system of payment has been arranged.
Trademark Notice: Product or corporate names may be trademarks or registered trademarks, and
are used only for identification and explanation without intent to infringe.
Library of Congress Cataloging-in-Publication Data
Abdelmalek, Nabih N.
Numerical linear approximation in C / Nabih Abdelmalek and William A.
Malek.
p. cm. -- (CRC numerical analysis and scientific computing)
Includes bibliographical references and index.
ISBN 978-1-58488-978-6 (alk. paper)

1. Chebyshev approximation. 2. Numerical analysis. 3. Approximation theory.
I. Malek, William A. II. Title. III. Series.
QA297.A23 2008
518--dc22
Visit the Taylor & Francis Web site at

and the CRC Press Web site at

© 2008 by Taylor & Francis Group, LLC

2008002447


v

Contents

List of figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xvii
Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xix
Acknowledgements. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxiii
Warranties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxiii
About the authors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxv

PART 1
Preliminaries and Tutorials
Chapter 1
1.1
1.2

1.3

1.4

Applications of Linear Approximation

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
Applications to social sciences and economics . . . . . . . . . . . . . . . . 5
1.2.1
Systolic blood pressure and age . . . . . . . . . . . . . . . . . . . 6
1.2.2
Annual teacher salaries . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.2.3
Factors affecting survival of island species . . . . . . . . . . 7
1.2.4
Factors affecting fuel consumption. . . . . . . . . . . . . . . . . 8
1.2.5
Examining factors affecting the mortality rate . . . . . . . . 8
1.2.6
Effects of forecasting . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
1.2.7
Factors affecting gross national products . . . . . . . . . . . . 9
Applications to industry . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
1.3.1
Windmill generating electricity . . . . . . . . . . . . . . . . . . 10
1.3.2
A chemical process . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
Applications to digital images . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
1.4.1
Smoothing of random noise in digital images . . . . . . . 12
1.4.2
Filtering of impulse noise in digital images . . . . . . . . . 14

1.4.3
Applications to pattern classification . . . . . . . . . . . . . . 15

© 2008 by Taylor & Francis Group, LLC


vi

Numerical Linear Approximation in C
1.4.4
1.4.5
1.4.6
1.4.7

Chapter 2
2.1
2.2
2.3

2.4
2.5
2.6

3.2
3.3
3.4
3.5

3.6

3.7

16
18
19
20

Preliminaries

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Discrete linear approximation and solution of overdetermined
linear equations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Comparison between the L1, the L2 and the L∞ norms by a
practical example. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.3.1
Some characteristics of the L1 and the Chebyshev
approximations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Error tolerances in the calculation. . . . . . . . . . . . . . . . . . . . . . . . .
Representation of vectors and matrices in C . . . . . . . . . . . . . . . . .
Outliers and dealing with them . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.6.1
Data editing and residual analysis. . . . . . . . . . . . . . . . .

Chapter 3
3.1

Restoring images with missing high-frequency
components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
De-blurring digital images using the Ridge equation . .
De-blurring images using truncated eigensystem . . . . .

De-blurring images using quadratic programming . . . .

25
26
30
32
32
33
34
36

Linear Programming and the Simplex Algorithm

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.1.1
Exceptional linear programming problems. . . . . . . . . .
Notations and definitions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
The simplex algorithm. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.3.1
Initial basic feasible solution . . . . . . . . . . . . . . . . . . . .
3.3.2
Improving the initial basic feasible solution . . . . . . . . .
The simplex tableau. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
The two-phase method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.5.1
Phase 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.5.2
Phase 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.5.3
Detection of the exceptional cases . . . . . . . . . . . . . . . .

Duality theory in linear programming. . . . . . . . . . . . . . . . . . . . . .
3.6.1
Fundamental properties of the dual problems. . . . . . . .
3.6.2
Dual problems with mixed constraints . . . . . . . . . . . . .
3.6.3
The dual simplex algorithm . . . . . . . . . . . . . . . . . . . . .
Degeneracy in linear programming and its resolution . . . . . . . . .
3.7.1
Degeneracy in the simplex method. . . . . . . . . . . . . . . .
3.7.2
Avoiding initial degeneracy in the simplex algorithm .

© 2008 by Taylor & Francis Group, LLC

39
41
44
45
47
48
51
55
55
56
56
57
59
60
61

62
62
63


Contents

3.8
3.9

3.7.3
Resolving degeneracy resulting from equal θmin . . . . .
3.7.4
Resolving degeneracy in the dual simplex method. . . .
Linear programming and linear approximation. . . . . . . . . . . . . . .
3.8.1
Linear programming and the L1 approximation . . . . . .
3.8.2
Linear programming and Chebyshev approximation . .
Stability of the solution in linear programming . . . . . . . . . . . . . .

Chapter 4
4.1
4.2

4.3
4.4

4.5

4.6
4.7

vii
63
64
64
64
66
67

Efficient Solutions of Linear Equations

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
Vector and matrix norms and relevant theorems. . . . . . . . . . . . . . 72
4.2.1
Vector norms. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
4.2.2
Matrix norms. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
4.2.3
Hermitian matrices and vectors . . . . . . . . . . . . . . . . . . 73
4.2.4
Other matrix norms . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
4.2.5
Euclidean and the spectral matrix norms . . . . . . . . . . . 77
4.2.6
Euclidean norm and the singular values . . . . . . . . . . . . 78
4.2.7
Eigenvalues and the singular values of the sum and
the product of two matrices. . . . . . . . . . . . . . . . . . . . . . 79

4.2.8
Accuracy of the solution of linear equations . . . . . . . . 80
Elementary matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
Gauss LU factorization with complete pivoting . . . . . . . . . . . . . . 84
4.4.1
Importance of pivoting . . . . . . . . . . . . . . . . . . . . . . . . . 85
4.4.2
Using complete pivoting . . . . . . . . . . . . . . . . . . . . . . . . 86
4.4.3
Pivoting and the rank of matrix A. . . . . . . . . . . . . . . . . 88
Orthogonal factorization methods . . . . . . . . . . . . . . . . . . . . . . . . . 90
4.5.1
The elementary orthogonal matrix H . . . . . . . . . . . . . . 90
4.5.2
Householder’s QR factorization with pivoting. . . . . . . 90
4.5.3
Pivoting in Householder’s method . . . . . . . . . . . . . . . . 92
4.5.4
Calculation of the matrix inverse A–1 . . . . . . . . . . . . . . 93
Gauss-Jordan method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
Rounding errors in arithmetic operations . . . . . . . . . . . . . . . . . . . 95
4.7.1
Normalized floating-point representation . . . . . . . . . . . 95
4.7.2
Overflow and underflow in arithmetic operations . . . . 96
4.7.3
Arithmetic operations in a d.p. accumulator . . . . . . . . . 97
4.7.4
Computation of the square root of a s.p. number . . . . 100
4.7.5

Arithmetic operations in a s.p. accumulator . . . . . . . . 100
4.7.6
Arithmetic operations with two d.p. numbers. . . . . . . 101
4.7.7
Extended simple s.p. operations in a d.p. accumulator 101
4.7.8
Alternative expressions for summations and innerproduct operations. . . . . . . . . . . . . . . . . . . . . . . . . . . . 104

© 2008 by Taylor & Francis Group, LLC


viii

Numerical Linear Approximation in C
4.7.9
4.7.10
4.7.11
4.7.12
4.7.13

More conservative error bounds . . . . . . . . . . . . . . . . .
D.p. summations and inner-product operations . . . . .
Rounding error in matrix computation . . . . . . . . . . . .
Forward and backward round-off error analysis. . . . .
Statistical error bounds and concluding remarks . . . .

104
105
106
107

109

PART 2
The L1 Approximation
Chapter 5
5.1
5.2
5.3
5.4
5.5
5.6
5.7
5.8
5.9
5.10
5.11
5.12
5.13
5.14

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.1.1
Characterization of the L1 solution . . . . . . . . . . . . . . .
Linear programming formulation of the problem . . . . . . . . . . . .
Description of the algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . .
The dual simplex method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Modification to the algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . .
Occurrence of degeneracy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A significant property of the L1 approximation . . . . . . . . . . . . .
Triangular decomposition of the basis matrix . . . . . . . . . . . . . . .

Arithmetic operations count . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Numerical results and comments. . . . . . . . . . . . . . . . . . . . . . . . .
DR_L1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
LA_L1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
DR_Lone . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
LA_Lone . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Chapter 6
6.1
6.2
6.3
6.4

Linear L1 Approximation
113
116
116
118
120
124
126
129
130
132
134
140
145
165
170

One-Sided L1 Approximation

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6.1.1
Applications of the algorithm . . . . . . . . . . . . . . . . . . .
6.1.2
Characterization and uniqueness. . . . . . . . . . . . . . . . .
A special problem of a general constrained one . . . . . . . . . . . . .
Linear programming formulation of the problem . . . . . . . . . . . .
Description of the algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6.4.1
Obtaining an initial basic feasible solution . . . . . . . . .
6.4.2
One-sided L1 solution from above . . . . . . . . . . . . . . .

© 2008 by Taylor & Francis Group, LLC

183
186
186
186
187
188
188
191


Contents

6.5

6.6
6.7

6.4.3
The interpolation property . . . . . . . . . . . . . . . . . . . . .
Numerical results and comments. . . . . . . . . . . . . . . . . . . . . . . . .
DR_Loneside. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
LA_Loneside . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Chapter 7
7.1
7.2
7.3
7.4
7.5
7.6
7.7

8.2
8.3
8.4
8.5
8.6
8.7

191
191
195
203

L1 Approximation with Bounded Variables

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
7.1.1
Linear L1 approximation with non-negative
parameters (NNL1) . . . . . . . . . . . . . . . . . . . . . . . . . . .
A special problem of a general constrained one . . . . . . . . . . . . .
Linear programming formulation of the problem . . . . . . . . . . . .
7.3.1
Properties of the matrix of constraints . . . . . . . . . . . .
Description of the algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Numerical results and comments. . . . . . . . . . . . . . . . . . . . . . . . .
DR_Lonebv . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
LA_Lonebv . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Chapter 8
8.1

ix

213
216
216
217
220
221
222
225
233

L1 Polygonal Approximation of Plane Curves

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
8.1.1
Two basic issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
8.1.2
Approaches for polygonal approximation . . . . . . . . .
8.1.3
Other unique approaches. . . . . . . . . . . . . . . . . . . . . . .
8.1.4
Criteria by which error norm is chosen. . . . . . . . . . . .
8.1.5
Direction of error measure . . . . . . . . . . . . . . . . . . . . .
8.1.6
Comparison and stability of polygonal
approximations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
8.1.7
Applications of the algorithm . . . . . . . . . . . . . . . . . . .
The L1 approximation problem . . . . . . . . . . . . . . . . . . . . . . . . . .
Description of the algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Linear programming technique . . . . . . . . . . . . . . . . . . . . . . . . . .
8.4.1
The algorithm using linear programming . . . . . . . . . .
Numerical results and comments. . . . . . . . . . . . . . . . . . . . . . . . .
DR_L1pol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
LA_L1pol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

© 2008 by Taylor & Francis Group, LLC

245

245
246
248
249
249
250
251
251
252
253
255
255
261
267


x

Numerical Linear Approximation in C

Chapter 9
9.1
9.2
9.3
9.4

9.5
9.6
9.7
9.8

9.9

Piecewise L1 Approximation of Plane Curves

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
9.1.1
Applications of piecewise approximation. . . . . . . . . .
Characteristics of the piecewise approximation . . . . . . . . . . . . .
The discrete linear L1 approximation problem . . . . . . . . . . . . . .
Description of the algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . .
9.4.1
Piecewise linear L1 approximation with pre-assigned
tolerance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
9.4.2
Piecewise linear approximation with near-balanced
L1 norms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Numerical results and comments. . . . . . . . . . . . . . . . . . . . . . . . .
DR_L1pw1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
LA_L1pw1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
DR_L1pw2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
LA_L1pw2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

275
276
276
278
279
279
280
281

285
289
294
298

PART 3
The Chebyshev Approximation
Chapter 10
10.1
10.2
10.3
10.4
10.5
10.6
10.7

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
10.1.1
Characterization of the Chebyshev solution . . . . . . . .
Linear programming formulation of the problem . . . . . . . . . . . .
10.2.1
Property of the matrix of constraints . . . . . . . . . . . . .
Description of the algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A significant property of the Chebyshev approximation . . . . . .
10.4.1
The equioscillation property of the Chebyshev norm.
Numerical results and comments. . . . . . . . . . . . . . . . . . . . . . . . .
DR_Linf. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
LA_Linf. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Chapter 11
11.1

Linear Chebyshev Approximation
307
309
310
311
315
320
321
321
325
334

One-Sided Chebyshev Approximation

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 347
11.1.1
Applications of the algorithm . . . . . . . . . . . . . . . . . . . 349

© 2008 by Taylor & Francis Group, LLC


Contents
11.2
11.3
11.4
11.5
11.6

11.7

A special problem of a general constrained one . . . . . . . . . . . . .
Linear programming formulation of the problem . . . . . . . . . . . .
11.3.1
Properties of the matrix of constraints . . . . . . . . . . . .
Description of the algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . .
11.4.1
One-sided Chebyshev solution from below . . . . . . . .
Numerical results and comments. . . . . . . . . . . . . . . . . . . . . . . . .
11.5.1
Simple relationships between the Chebyshev and
one-sided Chebyshev approximations . . . . . . . . . . . .
DR_Linfside . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
LA_Linfside . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Chapter 12
12.1
12.2
12.3
12.4
12.5
12.6
12.7

13.2
13.3
13.4
13.5
13.6

13.7
13.8
13.9

350
351
353
356
360
361
362
365
373

Chebyshev Approximation with Bounded Variables

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
12.1.1
Linear Chebyshev approximation with non-negative
parameters (NNLI) . . . . . . . . . . . . . . . . . . . . . . . . . . .
A special problem of a general constrained one . . . . . . . . . . . . .
Linear programming formulation of the problem . . . . . . . . . . . .
12.3.1
Properties of the matrix of constraints . . . . . . . . . . . .
Description of the algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Numerical results and comments. . . . . . . . . . . . . . . . . . . . . . . . .
DR_Linfbv. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
LA_Linfbv. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Chapter 13

13.1

xi

387
389
389
390
392
394
396
399
408

Restricted Chebyshev Approximation

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
13.1.1
The semi-infinite programming problem . . . . . . . . . .
13.1.2
Special cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
13.1.3
Applications of the restricted Chebyshev algorithm. .
A special problem of general constrained algorithms . . . . . . . . .
Linear programming formulation of the problem . . . . . . . . . . . .
13.3.1
Properties of the matrix of constraints . . . . . . . . . . . .
Description of the algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Triangular decomposition of the basis matrix . . . . . . . . . . . . . . .
Arithmetic operations count . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Numerical results and comments. . . . . . . . . . . . . . . . . . . . . . . . .
DR_Restch. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
LA_Restch. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

© 2008 by Taylor & Francis Group, LLC

419
421
421
422
422
423
425
427
428
429
430
435
446


xii

Numerical Linear Approximation in C

Chapter 14
14.1
14.2
14.3

14.4
14.5
14.6

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
The problem as presented by Descloux. . . . . . . . . . . . . . . . . . . .
Linear programming analysis of the problem . . . . . . . . . . . . . . .
14.3.1
The characteristic set R1 and how to obtain it . . . . . .
14.3.2
Calculating matrix (DT)–1 . . . . . . . . . . . . . . . . . . . . . .
14.3.3
The case of a rank deficient coefficient matrix. . . . . .
Numerical results and comments. . . . . . . . . . . . . . . . . . . . . . . . .
DR_Strict. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
LA_Strict . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Chapter 15
15.1
15.2
15.3
15.4

15.5
15.6
15.7
15.8
15.9

16.2

16.3
16.4
16.5
16.6
16.7
16.8

469
471
472
473
475
476
476
481
489

Piecewise Chebyshev Approximation

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
15.1.1
Applications of piecewise approximation. . . . . . . . . .
Characteristic properties of piecewise approximation . . . . . . . .
The discrete linear Chebyshev approximation problem . . . . . . .
Description of the algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . .
15.4.1
Piecewise linear Chebyshev approximation with
pre-assigned tolerance. . . . . . . . . . . . . . . . . . . . . . . . .
15.4.2
Piecewise linear Chebyshev approximation with

near-balanced Chebyshev norms . . . . . . . . . . . . . . . .
Numerical results and comments. . . . . . . . . . . . . . . . . . . . . . . . .
DR_Linfpw1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
LA_Linfpw1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
DR_Linfpw2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
LA_Linfpw2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Chapter 16
16.1

Strict Chebyshev Approximation

517
519
519
519
520
520
521
521
525
529
534
538

Solution of Linear Inequalities

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
16.1.1
Linear programming techniques . . . . . . . . . . . . . . . . .

Pattern classification problem . . . . . . . . . . . . . . . . . . . . . . . . . . .
Solution of the system of linear inequalities Ca > 0 . . . . . . . . . .
Linear one-sided Chebyshev approximation algorithm . . . . . . .
Linear one-sided L1 approximation algorithm . . . . . . . . . . . . . .
Numerical results and comments. . . . . . . . . . . . . . . . . . . . . . . . .
DR_Chineq . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
DR_L1ineq . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

© 2008 by Taylor & Francis Group, LLC

545
548
549
551
551
552
553
560
566


Contents

xiii

PART 4
The Least Squares Approximation
Chapter 17
17.1
17.2

17.3

17.4
17.5
17.6

17.7

17.8

Least Squares and Pseudo-Inverses of Matrices

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Least squares solution of linear equations. . . . . . . . . . . . . . . . . .
17.2.1
Minimal-length least squares solution . . . . . . . . . . . .
Factorization of matrix A . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
17.3.1
Gauss LU factorization . . . . . . . . . . . . . . . . . . . . . . . .
17.3.2
Householder’s factorization . . . . . . . . . . . . . . . . . . . .
17.3.3
Givens’ transformation (plane rotations) . . . . . . . . . .
17.3.4
Classical and modified Gram-Schmidt methods. . . . .
Explicit expression for the pseudo-inverse . . . . . . . . . . . . . . . . .
17.4.1
A+ in terms of Gauss factorization . . . . . . . . . . . . . . .
17.4.2
A+ in terms of Householder’s factorization . . . . . . . .

The singular value decomposition (SVD) . . . . . . . . . . . . . . . . . .
17.5.1
Spectral condition number of matrix A . . . . . . . . . . .
17.5.2
Main properties of the pseudo-inverse A+ . . . . . . . . .
Practical considerations in computing. . . . . . . . . . . . . . . . . . . . .
17.6.1
Cholesky’s decomposition . . . . . . . . . . . . . . . . . . . . .
17.6.2
Solution of the normal equation . . . . . . . . . . . . . . . . .
17.6.3
Solution via Gauss LU factorization method . . . . . . .
17.6.4
Solution via Householder’s method . . . . . . . . . . . . . .
17.6.5
Calculation of A+ . . . . . . . . . . . . . . . . . . . . . . . . . . . .
17.6.6
Influence of the round-off error . . . . . . . . . . . . . . . . .
Linear spaces and the pseudo-inverses . . . . . . . . . . . . . . . . . . . .
17.7.1
Definitions, notations and related theorems . . . . . . . .
17.7.2
Subspaces and their dimensions . . . . . . . . . . . . . . . . .
17.7.3
Gram-Schmidt orthogonalization . . . . . . . . . . . . . . . .
17.7.4
Range spaces of A and AT and their orthogonal
complements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
17.7.5
Representation of vectors in Vm . . . . . . . . . . . . . . . . .

17.7.6
Orthogonal projection onto range and null spaces . . .
17.7.7
Singular values of the orthogonal projection
matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Multicollinearity, collinearity or the ill-conditioning of
matrix A. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
17.8.1
Sources of multicollinearity . . . . . . . . . . . . . . . . . . . .

© 2008 by Taylor & Francis Group, LLC

575
576
578
579
579
581
582
582
583
584
584
585
587
587
588
588
588
589

589
589
590
591
591
592
594
596
598
599
601
601
603


xiv

17.9
17.10
17.11
17.12
17.13
17.14
17.15
17.16

Numerical Linear Approximation in C
17.8.2
Detection of multicollinearity . . . . . . . . . . . . . . . . . . .
Principal components analysis (PCA) . . . . . . . . . . . . . . . . . . . .

17.9.1
Derivation of the principal components . . . . . . . . . . .
Partial least squares method (PLS) . . . . . . . . . . . . . . . . . . . . . . .
17.10.1 Model building for the PLS method . . . . . . . . . . . . .
Ridge equation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
17.11.1 Estimating the Ridge parameter . . . . . . . . . . . . . . . . .
17.11.2 Ridge equation and variable selection . . . . . . . . . . . .
Numerical results and comments. . . . . . . . . . . . . . . . . . . . . . . . .
DR_Eluls . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
LA_Eluls . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
DR_Hhls . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
LA_Hhls . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Chapter 18

Piecewise Linear Least Squares Approximation

18.1

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
18.1.1
Preliminaries and notation . . . . . . . . . . . . . . . . . . . . .
18.2 Characteristics of the approximation. . . . . . . . . . . . . . . . . . . . . .
18.3 The discrete linear least squares approximation problem . . . . . .
18.4 Description of the algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . .
18.4.1
Piecewise linear L2 approximation with pre-assigned
tolerance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
18.4.2
Piecewise linear L2 approximation with nearbalanced L2 norms . . . . . . . . . . . . . . . . . . . . . . . . . . .

18.5 Numerical results and comments. . . . . . . . . . . . . . . . . . . . . . . . .
18.6 The updating and downdating techniques . . . . . . . . . . . . . . . . . .
18.6.1
The updating algorithm. . . . . . . . . . . . . . . . . . . . . . . .
18.6.2
The downdating algorithm . . . . . . . . . . . . . . . . . . . . .
18.6.3
Updating and downdating in the L1 norm. . . . . . . . . .
18.7 DR_L2pw1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
18.8 LA_L2pw1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
18.9 DR_L2pw2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
18.10 LA_L2pw2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Chapter 19
19.1
19.2
19.3

604
605
605
607
609
612
613
614
614
621
630
645

654

671
673
673
673
674
675
675
675
677
678
679
680
682
686
691
695

Solution of Ill-Posed Linear Systems

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 703
Solution of ill-posed linear systems . . . . . . . . . . . . . . . . . . . . . . 704
Estimation of the free parameter . . . . . . . . . . . . . . . . . . . . . . . . . 708

© 2008 by Taylor & Francis Group, LLC


Contents
19.4

19.5
19.6
19.7
19.8
19.9

xv

Description of the new algorithm . . . . . . . . . . . . . . . . . . . . . . . .
19.4.1
Steps of the algorithm . . . . . . . . . . . . . . . . . . . . . . . . .
Optimum value of the rank . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
19.5.1
The parameters TOLER and EPS . . . . . . . . . . . . . . . .
Use of linear programming techniques . . . . . . . . . . . . . . . . . . . .
Numerical results and comments. . . . . . . . . . . . . . . . . . . . . . . . .
DR_Mls . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
LA_Mls . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

709
709
714
714
715
717
724
730

PART 5
Solution of Underdetermined Systems

Of Linear Equations
Chapter 20
20.1
20.2
20.3
20.4
20.5
20.6

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
20.1.1
Applications of the algorithm . . . . . . . . . . . . . . . . . . .
Linear programming formulation of the problem . . . . . . . . . . . .
20.2.1
Properties of the matrix of constraints . . . . . . . . . . . .
Description of the algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Numerical results and comments. . . . . . . . . . . . . . . . . . . . . . . . .
DR_Fuel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
LA_Fuel. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Chapter 21
21.1
21.2
21.3
21.4
21.5
21.6

L1 Solution of Underdetermined Linear Equations
739

740
741
742
744
747
749
755

Bounded and L1 Bounded Solutions of
Underdetermined Linear Equations

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
21.1.1
Applications of the algorithms . . . . . . . . . . . . . . . . . .
Linear programming formulation of the two problems. . . . . . . .
21.2.1
Properties of the matrix of constraints . . . . . . . . . . . .
Description of the algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . .
21.3.1
Occurrence of degeneracy. . . . . . . . . . . . . . . . . . . . . .
21.3.2
Uniqueness of the solution . . . . . . . . . . . . . . . . . . . . .
Numerical results and comments. . . . . . . . . . . . . . . . . . . . . . . . .
DR_Tmfuel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
LA_Tmfuel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

© 2008 by Taylor & Francis Group, LLC

765
767

767
769
771
771
772
772
776
783


xvi

Numerical Linear Approximation in C

Chapter 22
22.1
22.2
22.3
22.4
22.5
22.6

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
22.1.1
Applications of the algorithm . . . . . . . . . . . . . . . . . . .
The linear programming problem . . . . . . . . . . . . . . . . . . . . . . . .
22.2.1
Properties of the matrix of constraints . . . . . . . . . . . .
Description of the algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . .
22.3.1

The reduced tableaux . . . . . . . . . . . . . . . . . . . . . . . . .
Numerical results and comments. . . . . . . . . . . . . . . . . . . . . . . . .
DR_Effort . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
LA_Effort . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Chapter 23
23.1
23.2
23.3
23.4

23.5
23.6
23.7

Chebyshev Solution of Underdetermined Linear
Equations
799
801
801
803
806
811
812
815
822

Bounded Least Squares Solution of
Underdetermined Linear Equations

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
23.1.1
Applications of the algorithm . . . . . . . . . . . . . . . . . . .
Quadratic programming formulation of the problems. . . . . . . . .
Solution of problem (E0). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Solution of problem (E). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
23.4.1
Asymmetries in the simplex tableau . . . . . . . . . . . . . .
23.4.2
The condensed tableau for problem (E) . . . . . . . . . . .
23.4.3
The dual method of solution . . . . . . . . . . . . . . . . . . . .
23.4.4
The reduced tableau for problem (E) . . . . . . . . . . . . .
23.4.5
The method of solution of problem (E) . . . . . . . . . . .
Numerical results and comments. . . . . . . . . . . . . . . . . . . . . . . . .
DR_Energy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
LA_Energy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

833
834
836
837
840
841
842
843
844
844

845
848
852

Appendices
Appendix A
Appendix B
Appendix C
Appendix D

References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Main Program . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Constants, Types and Function Prototypes . . . . . . . . . . . .
Utilities and Common Functions. . . . . . . . . . . . . . . . . . . .

© 2008 by Taylor & Francis Group, LLC

869
893
897
917


xvii

List of Figures

Figure 1-1:
Figure 1-2:
Figure 2-1:

Relative coordinates of pixels inside a 3 by 3 window . . . . 12
Pixel labels inside the 3 by 3 window . . . . . . . . . . . . . . . . . 12
Curve fitting a set of 8 points, including a wild point,
using the L1, L2 and L∞ norms . . . . . . . . . . . . . . . . . . . . . . 31
Figure 3-1: Feasibility Region . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
Figure 3-2: Unbounded Solution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
Figure 3-3: Inconsistent Constraints. . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
Figure 4-1: General floating-point format . . . . . . . . . . . . . . . . . . . . . . . 95
Figure 6-1: Curve fitting with vertical parabolas of a set of 8 points
using L1 approximation and one-sided L1 approximations 184
Figure 7-1: Curve fitting with vertical parabolas of a set of 8 points
using L1 approximation and L1 approximation with
bounded variables between –1 and 1. . . . . . . . . . . . . . . . . 215
Figure 8-1: L1 polygonal approximation for a waveform. The L1
norm in any segment is ≤ 0.6. . . . . . . . . . . . . . . . . . . . . . . 256
Figure 8-2: L1 polygonal approximation for a contour. The L1
norm in any segment is ≤ 0.4. . . . . . . . . . . . . . . . . . . . . . . 256
Figure 9-1: Disconnected linear L1 piecewise approximation with
vertical parabolas. The L1 residual norm in any segment
≤ 6.2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 282
Figure 9-2: Connected linear L1 piecewise approximation with
vertical parabolas. The L1 residual norm in any segment
≤ 6.2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 282
Figure 9-3: Near-balanced residual norm solution. Disconnected
linear L1 piecewise approximation with vertical
parabolas. Number of segments = 4 . . . . . . . . . . . . . . . . . 283
Figure 11-1: Curve fitting with vertical parabolas of a set of 8
points using Chebyshev and one-sided Chebyshev
approximations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 349

Figure 12-1: Curve fitting a set of 8 points using Chebyshev
approximation and Chebyshev approximation with
bounded variables between –1 and 1. . . . . . . . . . . . . . . . . 389
© 2008 by Taylor & Francis Group, LLC


xviii

Numerical Linear Approximation in C

Figure 13-1: Curve fitting a set of 8 points using restricted
Chebyshev approximations with arbitrary ranges . . . . . . . 422
Figure 15-1: Disconnected linear Chebyshev piecewise approximation
with vertical parabolas. Chebyshev residual norm in any
segment ≤ 1.3. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 522
Figure 15-2: Connected linear Chebyshev piecewise approximation
with vertical parabolas. Chebyshev residual norm in
any segment ≤ 1.3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 522
Figure 15-3: Near-balanced residual norm solution. Disconnected linear
Chebyshev piecewise approximation with vertical parabolas.
Number of segments = 4 . . . . . . . . . . . . . . . . . . . . . . . . . . 523
Figure 16-1: Two curves that each separate the patterns of class A
from the patterns of class B . . . . . . . . . . . . . . . . . . . . . . . . 555
Figure 18-1: Disconnected linear L2 piecewise approximation with
vertical parabolas. The L2 residual norm in any
segment ≤ 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 676
Figure 18-2: Connected linear L2 piecewise approximation with
vertical parabolas. The L2 residual norm in any
segment ≤ 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 676
Figure 18-3: Near-balanced residual norm solution. Disconnected linear

L2 piecewise approximation with vertical parabolas.
Number of segments = 4 . . . . . . . . . . . . . . . . . . . . . . . . . . 677

© 2008 by Taylor & Francis Group, LLC


xix

Preface
Discrete linear approximation is one of the most frequently used
techniques in all areas of science and engineering. Linear
approximation of a continuous function is typically done by digitizing
the function, then applying discrete linear approximation to the
resulting data. Discrete linear approximation is equivalent to the
solution of an overdetermined system of linear equations in an
appropriate measure or norm, with or without some constraints.
This book describes algorithms for the solution of overdetermined
and underdetermined systems of linear equations. Software
implementations of the algorithms and test drivers, all written in the
programming language C, are provided at the end of each chapter.
Also included are piecewise linear approximations to plane curves in
the three main norms, as well as solutions of overdetermined linear
inequalities and of ill-posed linear systems.
It is assumed that the reader has basic knowledge of elementary
functional analysis and the programming language C.
All the algorithms in this book are based on linear programming
techniques, except the least squares ones. In addition, Chapter 23 uses
a quadratic programming technique.
Linear programming proved to be a powerful tool in solving linear
approximation problems. The solution, if it exists, is obtained in a

finite number of iterations, and no conditions are imposed on the
coefficient matrix, such as the Haar condition or the full rank
condition. The issue of the uniqueness of the solution is determined by
methods that use linear programming. Characterization of the solution
may also be deduced from the final tableau of the linear programming
problem.
The algorithms presented in this book are compared with existing
algorithms and are found to be the best or among the best of them.
The first author developed and published almost all the algorithms
presented in this book while he was with the National Research
Council of Canada (NRC), in Ottawa, Canada. While at the NRC, the
algorithms were applied successfully in several engineering projects.
Since then, the software was converted to FORTRAN 77, and then to
C for this book.

© 2008 by Taylor & Francis Group, LLC


xx

Numerical Linear Approximation in C

The second author is the son of the first author. He has been
instrumental in converting the software from FORTRAN 77 to C. He
has worked for over 25 years as a software engineer in the high-tech
industry in Ottawa, and has spent 4 years teaching courses in software
engineering at Carleton University in Ottawa, Canada.
This book may be considered a companion to the books,
Numerical Recipes in C, the Art of Scientific Computing, second
edition, W.H. Press et. al, Cambridge University Press, 1992 and

A Numerical Library in C for Scientists and Engineers, H.T. Lau,
CRC Press, 1994. Except for solving the least squares approximation
by the normal equation and by the Singular Value Decomposition
methods (Section 15.4 in the former and 3.4.2 in the latter), they lack
the subject of Numerical Linear Approximation. Also, the generic
routine on linear programming and the simplex method in the former
reference (Section 10.8) has little or no application to linear
approximation. Standard Subroutine Packages, such as LINPACK,
LAPACK, IMSL and NAG, may contain few linear approximation
routines, but not in C.
This book is divided into 5 main parts. Part 1 includes
introductory chapters 1 and 2, and tutorial chapters 3 and 4. Chapter 1
describes some diverse applications of linear approximation. In
Chapter 2, the relationship between Discrete Linear Approximation in
a certain norm and the solution of overdetermined linear equations in
the same norm is established. The L1, the least squares (L2) and the
Chebyshev (minimax or L∞) norms are considered. A comparison is
made between the approximations in the three norms, using a simple
example. An explanation of single- and double-precision computation, tolerance parameters and the programmatic representation of
vectors and matrices is also given. The chapter concludes with a study
of outliers or odd points in data, how to identify them, and some
methods of dealing with them. Chapter 3 gives an overall description
of the subject of linear programming and the simplex method. Also,
the (ordinary or two-sided) L1 and the (ordinary or two-sided)
Chebyshev approximation problems are formulated as linear
programming problems. Chapter 4 starts with the familiar subject of
vector and matrix norms and some relevant theorems. Then,
elementary matrices, which are used to perform elementary operations
on a matrix equation, are described. The solution of square linear

© 2008 by Taylor & Francis Group, LLC


Preface

xxi

systems using the Gauss LU factorization method with complete
pivoting and the Householder’s QR factorization method with
pivoting follows. A note on the Gauss-Jordan elimination method for
a set of underdetermined system of linear equations is given. This
chapter ends with a presentation of rounding error analysis for simple
and extended arithmetic operations. Chapter 4 serves as an
introduction to Chapter 17 on the pseudo-inverses of matrices and the
solution of linear least squares problems.
Part 2 of this book includes chapters 5 through 9. Chapters 5, 6
and 7 present the ordinary L1 approximation problem, the one-sided
L1 approximation, and the L1 approximation with bounded variables,
respectively. Chapters 8 and 9 present, respectively, the L1 polygonal
approximation and the linear L1 piecewise approximation of plane
curves.
Part 3 contains chapters 10 through 16. Chapters 10 through 15
present, respectively, the ordinary Chebyshev approximation problem,
the one-sided Chebyshev approximation, the Chebyshev approximation with bounded variables, the Restricted Chebyshev approximation, the Strict Chebyshev approximation and the piecewise
Chebyshev approximation of plane curves. Chapter 16 presents the
solution of overdetermined linear inequalities and its application to
pattern classification, using the one-sided Chebyshev and the
one-sided L1 approximation algorithms.
Part 4 contains chapters 17 through 19. Chapter 17 introduces the
pseudo-inverses of matrices and the minimal length least squares

solution. It then describes the least squares approximation by the
Gauss LU factorization method with complete pivoting and by
Householder’s QR factorization method with pivoting. An
introduction of linear spaces and the pseudo-inverses is given next.
The interesting subject of multicollinearity, or the ill-conditioning of
the coefficient matrix, is presented. This leads to the subject of
dealing with multicollinearity via the Principal Components Analysis
(PCA), the Partial Least Squares (PLS) method and the Ridge
equation technique. Chapter 18 presents the piecewise approximation
of plane curves in the L2 norm. Chapter 19 presents the solution of
ill-posed systems such as those arising from the discretization of the
Fredholm integral equation of the first kind.
Part 5 contains Chapters 20 through 23. They present the solution

© 2008 by Taylor & Francis Group, LLC


xxii

Numerical Linear Approximation in C

of underdetermined systems of consistent linear equations subject to
different constraints on the elements of the unknown solution vector
(not on the residuals, since the residuals are zeros). Chapter 20
constitutes the L1 approximation problem for the elements of the
solution vector. Chapter 21 describes (a) the solution of
underdetermined linear equations subject to bounds on the elements
of the solution vector and (b) the L1 approximation problem for these
elements that satisfy bounds on them. Chapter 22 describes the L∞
approximation problem for the elements of the solution vector.

Finally, Chapter 23 describes the bounded L2 approximation problem
for the elements of the solution vector.
Chapters 5 through 23 include the C functions that implement the
linear approximation algorithms, along with sample drivers. Each
driver contains a number of test case examples. The results of one or
more of the test cases are given in the text. Most of these chapters also
include a numerical example solved in detail.
To use the software provided in this book, it is not necessary to
understand the theory behind each one of the functions. The examples
given in each driver act as a guide to their usage. The code was
compiled and tested using MicrosoftTM Visual C++TM 6.0, Standard
Edition. To ensure maximum portability, only the most basic features
of the ANSI C standard were used, and no C++ features were
employed.
It is hoped that this book will be of benefit to scientists, engineers
and university students.

© 2008 by Taylor & Francis Group, LLC


xxiii

Acknowledgments
The first author wishes to thank his friends and colleagues at the
Institute for Informatics of the National Research Council of Canada,
particularly, Stuart Baxter, Brice Wightman, Frank Farrell, Bert
Lewis and Tom Bach for their unfailing help during his stay with the
Institute. From the Institute for Information Technology, he wishes to
thank his friend and colleague Tony Kasvand, who got him interested
in the fields of image processing and computer vision to which

numerical techniques were applied.
We also wish to sincerely thank Robert B. Stern, Executive Editor
at Taylor & Francis Group, Chapman and Hall, and CRC Press for his
support and interest in this book. Our thanks also go to the reviewers
for their constructive comments.
Last but not least, we thank our families for their love and
encouragement.

Warranties
The authors and the publisher of this book do not assume any
responsibility for any damage to person or property caused directly or
indirectly by the use, misuse or any other application of any of the
algorithms contained in this book.
We also make no warranties, in any way or form that the programs
contained in this book are free of errors or are consistent or
inconsistent with any type of machine.
The contents of this book including the software on a CD are
subject to Copyright laws. The individual owner of this book may
copy the contents of the CD to their own Personal Computer for their
own exclusive use, but they should not allow it to be copied by any
third parties.

© 2008 by Taylor & Francis Group, LLC


xxv

About the authors
Nabih N. Abdelmalek
The first author was born in Egypt in 1929. He received a B.Sc. in

electrical engineering in 1951, a B.Sc. in mathematics in 1954, both
from Cairo University, Egypt, and a Ph.D. in mathematical physics in
1958 from Manchester University, England.
From 1959 to 1965 he was an associate professor of mathematics
in the faculty of engineering, Cairo University, Egypt. From 1965 to
1967 he was a member of Scientific Staff at Northern Electric (now
Nortel Networks) in Ottawa, Canada. From 1967 until his retirement
in 1990, he was with the National Research Council of Canada, in
Ottawa, where he was a senior research officer. His interest was in the
application of numerical analysis techniques to image processing,
particularly image restoration, data compression, pattern classification
and segmentation of 3-D range images.
William A. Malek
The second author was born in Egypt in 1960. He received a
B.Eng. in electrical engineering in 1982 and an M.Eng. in computer
engineering in 1984, both from Carleton University in Ottawa
Canada.
From 1984 to 1988 he was a lecturer in the Department of
Systems and Computer Engineering at Carleton University, teaching
undergraduate and graduate courses in software engineering and
real-time systems. Since 1982, he has worked in Ottawa as a software
engineer for networking, avionics and military applications.

© 2008 by Taylor & Francis Group, LLC