perlovsky - neural networks and intellect - using model-based concepts (oxford, 2001)

Neural Networks and Intellect:
Using Model-Based Concepts
Leonid I. Perlovsky
OXFORD UNIVERSITY PRESS
NEURAL NETWORKS AND INTELLECT
This page intentionally left blank
NEURAL NETWORKS AND INTELLECT
Using Model-Based Concepts
Leonid I. Perlovsky
New York • Oxford
OXFORD UNIVERSITY PRESS
2001
Oxford University Press
Oxford New York
Athens Auckland Bangkok Bogot
´
a Buenos Aires Calcutta
Cape Town Chennai Dar es Salaam Delhi Florence Hong Kong Istanbul
Karachi Kuala Lumpur Madrid Melbourne Mexico City Mumbai
Nairobi Paris S
˜
ao Paulo Shanghai Singapore Taipei Tokyo Toronto Warsaw
and associated companies in
Berlin Ibadan
Copyright © 2001 by Oxford University Press, Inc.
Published by Oxford University Press, Inc.,
198 Madison Avenue, New York, New York, 10016

Oxford is a registered trademark of Oxford University Press
All rights reserved. No part of this publication may be reproduced,
stored in a retrieval system, or transmitted, in any form or by any means,

electronic, mechanical, photocopying, recording, or otherwise,
without the prior permission of Oxford University Press.
Library of Congress Cataloging-in-Publication Data
Perlovsky, Leonid I.
Neural networks and intellect : using model-based concepts / Leonid I. Perlovsky.
p. cm.
Includes bibliographical references and index.
ISBN 0-19-511162-1
1. Neural networks (computer science) 2. Mathematical models. I. Title.
QA76.87.P435 2000
006.3'2—dc21 00-026297
Printing (last digit):987654321
Printed in the United States of America
on acid-free paper
CONDENSED TABLE OF CONTENTS
PREFACE xix
PART ONE: OVERVIEW: 2300 YEARS OF PHILOSOPHY, 100 YEARS
OF MATHEMATICAL LOGIC, AND 50 YEARS OF COMPUTATIONAL
INTELLIGENCE
1 Introduction: Concepts of Intelligence 3
2 Mathematical Concepts of Mind 51
3 Mathematical versus Metaphysical Concepts of Mind 125
PART TWO: MODELING FIELD THEORY: NEW MATHEMATICAL
THEORY OF INTELLIGENCE WITH ENGINEERING APPLICATIONS
4 Modeling Field Theory 153
5 MLANS: Maximum Likelihood Adaptive Neural System for Grouping
and Recognition 206
6 Einsteinian Neural Network 263
7 Prediction, Tracking, and Dynamic Models 289
8 Quantum Modeling Field Theory (QMFT) 321

9 Fundamental Limitations on Learning 329
10 Intelligent System Organization: MFT, Genetic Algorithms,
and Kant 356
PART THREE: FUTURISTIC DIRECTIONS: FUN STUFF:
MIND—PHYSICS
+ MATHEMATICS + CONJECTURES
11 Gödel Theorems, Mind, and Machine 383
12 Toward Physics of Consciousness 391
LIST OF SYMBOLS 425
DEFINITIONS 429
BIBLIOGRAPHY 447
INDEX 461
v
This page intentionally left blank
CONTENTS
PREFACE xix
PART ONE: OVERVIEW: 2300 YEARS OF PHILOSOPHY, 100 YEARS
OF MATHEMATICAL LOGIC, AND 50 YEARS OF COMPUTATIONAL
INTELLIGENCE
1 Introduction: Concepts of Intelligence 3
1.1 CONCEPTS OF INTELLIGENCE IN MATHEMATICS, PSYCHOLOGY,
AND PHILOSOPHY 3
1.1.1 What Is Intelligence? 3
1.1.2 Plato, Occam, and Neural Networks 4
1.1.3 Rule-Based Artificial Intelligence, Complexity, and
Aristotle 6
1.1.4 Philosophy vs. Architecture of Intelligent Tracker 8
1.1.5 Summary 12
1.2 PROBABILITY, HYPOTHESIS CHOICE, PATTERN RECOGNITION,
AND COMPLEXITY 13

1.2.1 Prerequisite: Basic Notions of the Theory of Probability 13
1.2.2 Classical Hypotheses Choice Paradigms and Definitions 20
1.2.3 Pattern Recognition 22
1.2.4 A Priori Information and Adaptation 24
1.2.5 Mathematical Formulation of Model-Based Recognition 27
1.2.6 Conundrum of Combinatorial Complexity 29
1.3 PREDICTION, TRACKING, AND DYNAMIC MODELS 29
1.3.1 Linear Regression 30
1.3.2 Regression as an Expectation 32
1.3.3 Autoregression 33
1.3.4 Tracking 35
vii
viii Contents
1.3.5 Association Problem 37
1.4 PREVIEW: INTELLIGENCE, INTERNAL MODEL, SYMBOL, EMOTIONS,
AND CONSCIOUSNESS 42
Notes 45
Bibliographical Notes 46
Problems 47
2 Mathematical Concepts of Mind 51
2.1 COMPLEXITY, ARISTOTLE, AND FUZZY LOGIC 52
2.1.1 Conundrum of Combinatorial Complexity 52
2.1.2 Adaptivity, Apriority, and Complexity 53
2.1.3 Fuzzy Logic and Complexity 55
2.2 NEAREST NEIGHBORS AND DEGENERATE GEOMETRIES 58
2.2.1 The Nearest Neighbor Concept 58
2.2.2 Mathematical Formulation 59
2.2.3 What Constitutes Simple and Complex Classification
Problems? 59
2.2.4 Degenerate Geometry of Classification Spaces 60

2.3 GRADIENT LEARNING, BACK PROPAGATION, AND FEEDFORWARD
NEURAL NETWORKS 62
2.3.1 Concept of Discriminating Surfaces and Gradient
Learning 62
2.3.2 Mathematical Formulation 64
2.3.3 Learning Disability 67
2.4 RULE-BASED ARTIFICIAL INTELLIGENCE 68
2.4.1 Minsky, Apriority, and Adaptivity 68
2.4.2 Soar Production System 70
2.5 CONCEPT OF INTERNAL MODEL 73
2.5.1 Prolegomena: Parametric vs. Nonparametric Estimation 73
2.5.2 Model-Based Vision (MBV) 74
2.5.3 Adaptivity and MBV 75
2.6 ABDUCTIVE REASONING 76
2.6.1 Deduction, Induction, and Abduction 76
2.6.2 Abductive Reasoning Trees and Bayesian Networks 77
2.7 STATISTICAL LEARNING THEORY AND SUPPORT
VECTOR MACHINES 79
2.7.1 Model Complexity: Risk Minimization vs. PDF Estimation 79
Contents ix
2.7.2 Consistency of ERM and VC Dimension 81
2.7.3 Support Vector Machines (SVM) 82
2.8 AI DEBATES PAST AND FUTURE 85
2.8.1 Arguments and Disagreements: An Overview 85
2.8.2 Can a Machine Think? 87
2.8.3 Rule-Based AI vs. Connectivism 90
2.8.4 Emerging Debates 91
2.9 SOCIETY OF MIND 94
2.9.1 Society of Agents 94
2.9.2 Types of Agents 95

2.9.3 Frames and Unity of Apperception 96
2.9.4 Limitations and What Is Next 96
2.10 SENSOR FUSION AND JDL MODEL 97
2.10.1 Sensor Fusion and Origins of JDL Model 97
2.10.2 Definitions, Issues, and Types of Fusion Problems 98
2.10.3 Sensor Fusion Levels 99
2.10.4 Hierarchy of JDL Model Organization 100
2.11 HIERARCHICAL ORGANIZATION 100
2.12 SEMIOTICS 104
2.13 EVOLUTIONARY COMPUTATION, GENETIC ALGORITHMS,
AND CAS 106
2.13.1 Complex Adaptive Systems (CAS) 107
2.13.2 CAS: Complexity vs. Fuzziness 109
2.14 NEURAL FIELD THEORIES 110
2.14.1 Grossberg’s Method: Physics of Mind 110
2.14.2 ART Neural Network 111
2.14.3 Illusions and A Priori Contents of Vision 114
2.14.4 Motor Coordination and Sensorimotor Control 115
2.14.5 Emotions and Learning 116
2.14.6 Quantum Neurodynamics 118
2.14.7 Modeling Field Theory 119
2.15 INTELLIGENCE, LEARNING, AND COMPUTABILITY 120
2.15.1 Computability: Turing vs. Physics 120
2.15.2 Computational Methods of Intelligence: Summary 121
Notes 121
Bibliographical Notes 122
Problems 124
x Contents
3 Mathematical versus Metaphysical Concepts of Mind 125
3.1 PROLEGOMENON: PLATO, ANTISTHENES, AND ARTIFICIAL

INTELLIGENCE 126
3.2 LEARNING FROM ARISTOTLE TO MAIMONIDES 127
3.2.1 The Controversy of Aristotle 127
3.2.2 Finite Angels of Maimonides 128
3.2.3 Nexus of Aquinas 131
3.3 HERESY OF OCCAM AND SCIENTIFIC METHOD 131
3.3.1 Cynics, Occam, and Empiricism 131
3.3.2 Nominalism, Behaviorism, and Cybernetics 132
3.4 MATHEMATICS VS. PHYSICS 135
3.4.1 Pythagoras, Descartes, Newton 135
3.4.2 Computation: Metaphor vs. Physical Model 137
3.4.3 Physics of Mind vs. Physics of Brain 138
3.5 KANT: PURE SPIRIT AND PSYCHOLOGY 138
3.6 FREUD VS. JUNG: PSYCHOLOGY OF PHILOSOPHY 140
3.7 WITHER WE GO FROM HERE? 141
3.7.1 Apriority and Adaptivity 141
3.7.2 Fuzzy Logic, Models, and Neural Fields 143
Notes 145
Bibliographical Notes 147
Problems 148
PART TWO: MODELING FIELD THEORY: NEW MATHEMATICAL
THEORY OF INTELLIGENCE WITH ENGINEERING APPLICATIONS
4 Modeling Field Theory 153
4.1 INTERNAL MODELS, UNCERTAINTIES, AND SIMILARITIES 154
4.1.1 Certainty and Uncertainties 154
4.1.2 Models and Levels 154
4.1.3 Lower Level Models 155
4.1.4 Similarity Measures 156
4.2 MODELING FIELD THEORY DYNAMICS 160
4.2.1 Overview of the MFT System 161

4.2.2 MFT Dynamic Equations 162
4.2.3 Continuous MFT System 163
Contents xi
4.2.4 Heterarchy, Multiple Scales, and Local Maxima 163
4.2.5 MFT, Fuzzy Logic, and Aristotelian Forms 164
4.3 BAYESIAN MFT 165
4.3.1 Bayesian A-Similarity Measure and the Principle of Maximum
Likelihood 165
4.3.2 Bayesian AZ-Similarity Measure 168
4.3.3 MLANS Learning Equations 169
4.4 SHANNON–EINSTEINIAN MFT 172
4.4.1 Einstein, Likelihood, and Electromagnetic Spectrum 172
4.4.2 Einsteinian Gaussian Mixture Model 174
4.4.3 Equilibrium of the Photon Ensemble 176
4.4.4 Einsteinian Likelihood and Shannon’s Mutual Information 177
4.4.5 Information and Alternative Choice States 177
4.4.6 Mutual Model-Data Information 179
4.4.7 Shannon–Einsteinian Similarity 181
4.4.8 Shannon–Einsteinian MFT Dynamics 183
4.4.9 Historical Roots of Maximum Information and Maximum
Entropy Estimation 185
4.4.10 Likelihood, Information, Ergodicity, and Uncertainty 186
4.4.11 Forward and Inverse Problems 186
4.5 MODELING FIELD THEORY NEURAL ARCHITECTURE 187
4.6 CONVERGENCE 189
4.6.1 Aspects of Convergence 189
4.6.2 Proof of Convergence 190
4.7 LEARNING OF STRUCTURES, AIC, AND SLT 192
4.8 INSTINCT OF WORLD MODELING: KNOWLEDGE INSTINCT 194
4.9 SUMMARY 194

Notes 196
Bibliographical Notes 198
Problems 198
5 MLANS: Maximum Likelihood Adaptive Neural System for
Grouping and Recognition 206
5.1 GROUPING, CLASSIFICATION, AND MODELS 206
5.2 GAUSSIAN MIXTURE MODEL: UNSUPERVISED LEARNING OR
GROUPING 208
5.2.1 Architecture and Parameters 208
xii Contents
5.2.2 Likelihood Structure and Learning Algorithm 210
5.2.3 Examples of MLANS Unsupervised Classification 213
5.3 COMBINED SUPERVISED AND UNSUPERVISED LEARNING 225
5.3.1 Supervised and Unsupervised Learning 225
5.3.2 Perfect Teacher 227
5.3.3 Probabilistic or Fuzzy Teacher 227
5.3.4 Partial Supervision 228
5.3.5 Examples 229
5.4 STRUCTURE ESTIMATION 231
5.4.1 Goals and Approaches of Structural Optimization:
Models vs. Decisions 231
5.4.2 Maximum Likelihood Estimation of Structure 233
5.4.3 Minimum Classification Entropy 234
5.4.4 Other Structural Issues 236
5.5 WISHART AND RICIAN MIXTURE MODELS FOR RADAR IMAGE
CLASSIFICATION 238
5.5.1 Synthetic Aperture Radar 238
5.5.2 Data Description 239
5.5.3 Physically Based Clutter and Target Models 241
5.5.4 NASA Data Examples 245

5.5.5 Stockbridge Data Examples 246
5.5.6 Summary of SAR Models 250
5.6 CONVERGENCE 250
5.6.1 Convergence and Learning 250
5.6.2 The ML Equations 252
5.6.3 Local Convergence and EM Algorithm 254
5.6.4 Global Convergence 256
5.7 MLANS, PHYSICS, BIOLOGY, AND OTHER NEURAL
NETWORKS 257
Note 260
Bibliographical Notes 260
Problems 261
6 Einsteinian Neural Network 263
6.1 IMAGES, SIGNALS, AND SPECTRA 263
6.1.1 Definitions, Notations, and Simple Signal Models 263
6.1.2 Frequency Components, Spectrum, and Spectral Models 265
6.1.3 Model-Based Spectrum Estimation 269
Contents xiii
6.2 SPECTRAL MODELS 269
6.3 NEURAL DYNAMICS OF ENN 271
6.3.1 Shannon’s Similarity Dynamics of Einsteinian
Spectral Models 271
6.3.2 Two-Dimensional Time–Frequency ENN 272
6.4 APPLICATIONS TO ACOUSTIC TRANSIENT SIGNALS AND SPEECH
RECOGNITION 274
6.4.1 Transient Signals 274
6.4.2 Examples of One-Dimensional Spectrum Estimation 274
6.4.3 Two-Dimensional Time–Frequency Models 278
6.4.4 Hierarchical ENN + MLANS Architecture for
Signal Recognition 279

6.5 APPLICATIONS TO ELECTROMAGNETIC WAVE PROPAGATION IN
THE IONOSPHERE 280
6.5.1 Over-the-Horizon Radar Spectra 280
6.5.2 Spectral Models 284
6.6 SUMMARY 284
6.7 APPENDIX 285
Notes 286
Bibliographical Notes 286
Problems 287
7 Prediction, Tracking, and Dynamic Models 289
7.1 PREDICTION, ASSOCIATION, AND NONLINEAR
REGRESSION 290
7.1.1 Multidimensional Linear Regression 290
7.1.2 Multidimensional Autoregression 291
7.1.3 Nonlinear General Fuzzy Regression ANS (GFRANS) 292
7.1.4 Nonlinear Autoregression 294
7.1.5 Example: Data Mining and Revenue Prediction 295
7.1.6 Summary of Section 7.1 296
7.2 ASSOCIATION AND TRACKING USING BAYESIAN MFT 297
7.2.1 Concurrent Association and Tracking (CAT) 297
7.2.2 Linear Model for Tracking 299
7.2.3 Second-Order Model for Tracking 300
7.2.4 Link-Track Model 300
7.2.5 Random Noise and Clutter Model 301
7.2.6 Active Sensor and Doppler Track Models 301
xiv Contents
7.2.7 Autoregression Model for Tracking 302
7.2.8 Models for Tracking Resolved Objects 303
7.2.9 Object-Track Declaration 303
7.3 ASSOCIATION AND TRACKING USING SHANNON–EINSTEINIAN

MFT (SE-CAT) 304
7.3.1 Association and Tracking in Radar Spectral Data 304
7.3.2 Association and Tracking of Spatiotemporal Patterns 306
7.3.3 CAT of Spatiotemporal Patterns Described by General PDE
Models 307
7.3.4 Examples of Concurrent Association and Tracking in Radar
Data 308
7.4 SENSOR FUSION MFT 312
7.4.1 Information Fusion Problem 312
7.4.2 Mathematical Formulation 312
7.4.3 Can Sensor Fusion Degrade Performance? 313
7.5 ATTENTION 314
Notes 316
Bibliographical Notes 316
Problems 316
8 Quantum Modeling Field Theory (QMFT) 321
8.1 QUANTUM COMPUTING AND QUANTUM PHYSICS
NOTATIONS 321
8.1.1 Quantum vs. Classical Computers 321
8.1.2 Quantum Physics Notations and the QMF System 322
8.2 GIBBS QUANTUM MODELING FIELD SYSTEM 324
8.3 HAMILTONIAN QUANTUM MODELING FIELD SYSTEM 326
Bibliographical Notes 327
Problem 328
9 Fundamental Limitations on Learning 329
9.1 THE CRAMER–RAO BOUND ON SPEED OF LEARNING 329
9.1.1 CRB, Neural Networks, and Learning 329
9.1.2 Classical CRB for the Gaussian Means 330
9.1.3 CR Theorem 331
9.1.4 CRB for General MLANS Concurrent Association and

Estimation 333
Contents xv
9.2 OVERLAP BETWEEN CLASSES 335
9.2.1 Overlap Matrix 335
9.2.2 Overlapping Parts of Means 336
9.2.3 Overlapping Parts of Covariance Matrices 336
9.3 CRB FOR MLANS 339
9.3.1 CRB for Prior Rates 339
9.3.2 CRB for Means 341
9.3.3 CRB for Covariances 341
9.3.4 MLANS Performance vs. CRB: Example 3 Continuation 342
9.4 CRB FOR CONCURRENT ASSOCIATION AND TRACKING
(CAT) 344
9.4.1 CRB for Linear Tracks 344
9.4.2 Rule-of-Thumb CRB for CAT 345
9.5 SUMMARY: CRB FOR INTELLECT AND EVOLUTION? 348
9.6 APPENDIX: CRB RULE OF THUMB FOR TRACKING 349
Notes 353
Bibliographical Notes 354
Problems 354
10 Intelligent System Organization: MFT, Genetic Algorithms,
and Kant 356
10.1 KANT, MFT, AND INTELLIGENT SYSTEMS 357
10.1.1 Understanding Is Based on Internal Models 357
10.1.2 Judgment Is Based on Similarity Measures 359
10.1.3 Reason Is Based on Similarity Maximization 361
10.1.4 Hierarchical Organization of Intelligent Systems 361
10.1.5 Aristotle, Kant, Zadeh, MFT, Anaconda, and Frog 363
10.2 EMOTIONAL MACHINE (TOWARD MATHEMATICS OF
BEAUTY) 366

10.2.1 Cyberaesthetics or Intellectual Emotions 366
10.2.2 Purposiveness, Beauty, and Mathematics 367
10.2.3 Instincts, “Lower Emotions,” and Psychological Types 368
10.3 LEARNING: GENETIC ALGORITHMS, MFT, AND SEMIOSIS 370
10.3.1 The Origin of A Priori Models 370
10.3.2 Genetic Algorithms of Structural Evolution 371
10.3.3 MFT, CAS, and Evolution of Complex Structures 372
10.3.4 Semiosis: Dynamic Symbol 375
Notes 378
xvi Contents
Bibliographical Notes 378
Problems 379
PART THREE: FUTURISTIC DIRECTIONS: FUN STUFF:
MIND—PHYSICS
+ MATHEMATICS + CONJECTURES
11 Gödel Theorems, Mind, and Machine 383
11.1 PENROSE AND COMPUTABILITY OF MATHEMATICAL
UNDERSTANDING 383
11.2 LOGIC AND MIND 385
11.3 G
¨
ODEL, TURING, PENROSE, AND PUTNAM 387
11.4 G
¨
ODEL THEOREM VS. PHYSICS OF MIND 388
Note 390
Bibliographical Notes 390
12 Toward Physics of Consciousness 391
12.1 PHENOMENOLOGY OF CONSCIOUSNESS 392
12.1.1 Popular Conceptions and Misconceptions about

Consciousness 392
12.1.2 What Is Consciousness? 393
12.1.3 Consciousness of Bodhisattvas 395
12.1.4 Consciousness versus Unconscious 396
12.1.5 Consciousness versus Emotions 397
12.1.6 Why Is Consciousness Needed? 400
12.1.7 Collective and Individual Consciousness 400
12.1.8 Consciousness, Time, and Space 403
12.1.9 MFT and Searle Revisited 404
12.1.10 Neural Structures of Consciousness 409
12.2 PHYSICS OF SPIRITUAL SUBSTANCE: FUTURE DIRECTIONS 412
12.2.1 Path to Understanding 412
12.2.2 Physical Nature of Symbol and the Emergence of
Consciousness 414
12.2.3 Nature of Free Will and Creativity 415
12.2.4 Mysteries of Physics and Consciousness: New Physical
Phenomena? 418
Contents xvii
12.3 EPILOGUE 419
Notes 422
Bibliographical Notes 423
LIST OF SYMBOLS 425
DEFINITIONS 429
BIBLIOGRAPHY 447
INDEX 461
This page intentionally left blank
PREFACE
This book describes a new mathematical concept called modeling field theory; demonstrates
applications of neural networks based on this theory to a variety of problems; and analyzes
relationships among mathematics, computational concepts in neural networks, and concepts

of mind in psychology and philosophy. Deep philosophical questions are discussed and
related in detail to mathematics and the engineering of intelligence. The book is directed
toward a diverse audience of students, teachers, researchers, and engineers working in the
areas of neural networks, artificial intelligence, cognitive science, fuzzy systems, pattern
recognition and machine/computer vision, data mining, robotics navigation and recogni-
tion, target tracking, sensor fusion, spectrum analysis, time series analysis, and financial
market forecast. Mathematically inclined philosophers, semioticians, and psychologists
will find many issues of interest discussed. Although graduate level is assumed, interested
undergraduates will find that most of the material is readily accessible.
Architectures and learning mechanisms of modeling field neural networks utilize a
concept of an internal “world” model. The concept of internal models of the mind originated
in artificial intelligence and cognitive psychology, but its roots date back to Plato and
Aristotle. Intelligent systems based on rules utilize models (rules) in their final conceptual
forms. Like the Eide (Ideas) of Plato, rules lack adaptivity. In modeling field theory, the
adaptive models are similar to the Forms of Aristotle and serve as the basis for learning.
By combining the a priori knowledge of models with adaptive learning, the new mathe-
matical concept addresses the most perplexing problems in the field of neural networks
and intelligent systems: fast learning and robust generalization. An important aspect of this
mathematical and engineering advancement isthediscovery of a new type of instinct, a basic
instinct to learn, and the role of the related affective signals in general learning. Modeling
field theory serves as a stepping stone toward mathematical description of the general
phenomena of mind identified by Kant: Understanding (pure reason), Judgment (including
higher emotions, beautiful, and sublime) and Will (practical reason and freedom). The
combination of intuition with a mathematically unified paradigm provides the foundation
of a physical theory of mind.
The book is based on a number of conference presentations and journal publications.
It summarizes results of a large research and development effort: during the past 12 years
I have been leading a large and successful government-funded neural network program at
Nichols Research Corporation. It was expanded to commercial applications, most notably
data mining in several areas. In 2000, new commercial companies were formed including

InnoVerity (for developing applications in the areas of internet and bioinformatics) and
xix
xx Preface
Ascent Capital Management (for financial predition and investment management). The
book describes applications to a number of complicated, real-world problems that have not
been solved in the past by other approaches. These applications address pattern and image
recognition, data mining, nonlinear time series prediction and spectrum estimation, tracking
of patterns in data and imagery sequences, using a variety of sensors and information
sources, and the problems of sensor and information fusion.
The first three chapters review mathematical and philosophical concepts of intelli-
gence and mind. Chapter 1, the introduction, begins with the discussion of mathematical
approaches to intelligence during the past 50 years and their relationships to philosophical
concepts of mind during the 2300 years since Plato. Classical mathematical concepts
of hypothesis choice, pattern recognition, prediction, association, tracking, and sensor
fusion are reviewed in a concise, mathematically unified framework. This original, unified
mathematical framework is presented with an eye toward modeling field theory, which is
gradually developed throughout the book.
Chapters 2 and 3 review concepts of the mind in mathematics, engineering, philosophy,
psychology, and linguistics and analyze fundamental computational concepts of major
algorithmic and neural network paradigms. This analysis provides continuity to a large
variety of seemingly disparate techniques and establishes relationships between contem-
porary computational concepts of modeling intellect and concepts of mind discussed over
2300 years. I found this interrelationship to be much closer than currently thought among
scientists and philosophers of today. From the contemporary point of view, the questions
about mind posed by ancient philosophers are astonishingly scientific. Contemporary math-
ematical concepts of intellect are traced as a continuous line through the entire history of
psychology and philosophy to the concepts of mind developed in Buddhism, Judaism,
Islam, Christianity, and ancient Greece. This interrelationship is emphasized throughout
the book. I discuss specific mathematical reasons that lead to a conclusion that knowledge
has to be given to us a priori, that is inborn. I show that this knowledge cannot be given as

expert rules similar to the Ideas of Plato, but has to be given in a different representation,
as in the Aristotelian Forms of mind, which correspond to modeling fields in my theory.
The origin of Aristotelian mathematics is traced in Grossberg’s ART neural network, in
the concept of neural field theory, and in similar concepts of other neural networks. It is
a striking conclusion that philosophers of the past have been closer to the computational
concepts emerging today than pattern recognition and AI experts of just few years ago.
Chapter 2 analyzes learning requirements for each fundamental computational concept and
considers relationships between learning requirements, computational complexity, Turing,
and physical computability. Chapter 3 relates mathematical and engineering analysis to
philosophical analysis. It turns out that fuzzy logic, introduced by Zadeh 2300 years after
Aristotelian logic, is an essential ingredient for developing mathematical concepts of the
mind based on the Aristotelian theory of Forms.
Chapters 4 through 10 present the new mathematical apparatus for modeling intel-
ligence, with examples of engineering applications. The modeling field theory (MFT)
is introduced in Chapter 4. Its three main components are internal models, measures of
similarity between the models and the world, and adaptation laws. Deterministic, stochastic,
and fuzzy variabilities in data are discussed, followed by an introduction of the concept of
modeling fields. A general theory of similarity between a set of models and the world is de-
veloped. Aristotelian, fuzzy, and adaptive-fuzzy similarities are considered. Maximization
Preface xxi
of adaptive-fuzzy similarity leads to dynamic learning equations of modeling field theory.
Two types of adaptive-fuzzy similarity are formulated based on two fundamental concepts
of statistics and information theory: Bayesian likelihood and Shannon’s information. The
principle of similarity maximization, and in particular maximization of likelihood and in-
formation, is discussed as an internal drive or instinct to improve the internal representation
of the world, that is, an instinct to learn.
Chapters 5 through 7 develop several specific model-based neural networks for vari-
ous applications of increasing complexity. Chapter 5 discusses the Maximum Likelihood
Adaptive Neural System (MLANS), based on Bayesian similarity, for pattern and image
recognition applications. Chapter 6 considers Shannon–Einsteinian similarity and discusses

Modeling-field Einsteinian ANS (MEANS) for spectrum estimation of transient signals
in the frequency domain and in the two-dimensional time-frequency domain. Chapter 7
discusses dynamic temporal and spatiotemporal models for prediction,association, tracking,
and recognition of objects and spatiotemporal patterns. Tracking multiple patterns is related
to nonlinear time series prediction and tracking applications are discussed along with
financial market prediction. Association models are extended to multiple sensor fusion
and related to mechanisms of attention. These chapters contain numerous examples of
applications to complex real-world problems, many of which could not have been previ-
ously solved.
Chapter 8 addresses a possibility that biological neurons may perform quantum
computations. A quantum computation algorithm for MFT is described. Chapter 9 con-
siders general limitations on learning for any intelligent system, algorithm, or neural net-
work. Fundamental bounds on learning (the Cramer–Rao bounds) are discussed and new
types of bounds are presented for clustering, association, tracking, and nonlinear predic-
tion. Is it possible to compute the fundamental mathematical bound on the entire evolu-
tion process?
Chapter 10 discusses the architecture and organization of an intelligent system. The
three-component mathematical structure of the modeling field theory is related to the three
main components of intelligence identified by Kant: Understanding, Judgment, and Will.
Hierarchical and heterarchical organization of Kant–MFT intelligent systems is related to
genetic algorithms, complex adaptive systems, and semiotics. A dynamic nature of symbol
is discussed. What are the relationships between emotions and thinking? Is a mathematical
theory of emotional intellect possible? What kinds of internal models are needed for higher
emotional feelings and ethics? Learning behavior leads to improving the internal model,
and its mechanisms are related to Kantian reflective judgment—a foundation of higher
intellectual abilities. The mathematics of the learning instinct is related to the concept
of beauty.
The last two chapters, 11 through 12, contain fun stuff: philosophy and psychology
are combined with conjectures based on physical and mathematical intuition about mind.
Chapter 11 considers general limitations of logic, computational complexity, Turing com-

putability, and Gödel theorems. Are Gödel theoremsrelevant to the problems of recognition?
Are difficulties encountered by algorithms and neural networks of mathematical intelligence
related to Gödel theorems? Does it explain the difference between a human and a machine
mind? the nature of free will and creativity? Chapter 12 discusses a possibility of the
physical theory of consciousness based on modeling field theory. I discuss the differentiated
phenomenology of consciousness and creativity within a framework of modeling field
xxii Preface
theory. The Epilogue presents a fresh view on the main discussions of this book: concepts
of computational intelligence versus concepts of mind in philosophy, psychology, and
linguistics. Can our contemporary mathematical concepts throw light on ancient philo-
sophical problems? Can the thoughts of ancient philosophers guide us in constructing
mathematical theories of mind? This book gives affirmative answers to both questions.
However, mathematicians and engineers should not be too cavalier about mysteries of the
mind, and contemporary philosophers should not bow to mathematical fashions of the day.
I attempt to delineate a fuzzy boundary separating questions that today are beyond the
scientific method. The book ends with a consideration of the future directions of research
in the physical theory of mind.
HOW TO READ THIS BOOK
The bookis self-contained in that the concepts ofphilosophy and mathematics are introduced
from the basics. Detailed references are provided for further exploration of individual
topics. The Definitions section at the end of the book summarizes all the important con-
cepts used throughout the book in alphabetical order for easy reference. The following
table provides guidance for several types of readers, who might prefer to read this book
selectively.
Concepts Chapters and Sections
General Philosophical Concepts of Intellect and Their Relationships to
Mathematical Concepts
Ch. 1, Sec. 1.1
Ch. 2, Sec. 2.1, 2.8
Ch. 3

Ch. 9, Sec. 9.5
Ch. 10 through 12
Overview of Basic Mathematical Concepts of Modeling Intelligence and Their
Relationships to Philosophical Concepts
Ch. 1, Sec. 1.2, 1.3, 1.4
Ch. 2
Ch. 9 through 12
Mathematical Concepts and Techniques Related to Specific Applications with
Intermittent Discussions of Philosophical Connections
Ch. 1, except Sec. 1.1
Ch. 2, except Sec. 2.1
Ch. 4 through 9
Bibliographical references as well as cross-references among the book chapters were
kept to the minimum within the main text. These are contained in Notes and Bibliographical
Notes at the end of each chapter, as well as in the Bibliography section at the end of
the book.
SEMESTER COURSES:SUGGESTED OUTLINES
Several semester courses can be designed using this book. The following table outlines a
few suggestions.
Preface xxiii
Course Title and Description Book Chapters
1. Introduction to Modern Pattern Recognition, Prediction, Tracking, and Fusion.
A general unified mathematical formulation of problems and solution methods in
several areas of statistics and signal processing.
Prerequisites: probability
Desirable: signal processing
Level: graduate or advanced undergraduate
Chapter 1 (Sec. 1.1 is optional), plus any of the
examples from Chapters 5 through 7. Or, use
your favorite problems.

2. Mathematical Concepts of Intelligence.
The course reviews classical mathematical concepts of intelligent algorithms,
symbolic AI, and neural networks. After analysis of successes and deficiencies
of the classical techniques, new emergent concepts are introduced: evolutionary
computation, hierarchical organization, and neural fields.
Prerequisites: probability
Desirable: a course in neural networks or AI
Level: graduate or advanced undergraduate
Chapter 2 (Sec. 2.1 is optional), plus any of the
examples from Chapters 5 through 7. Or, use
your favorite problems.
3. Model-Based Neural Networks: Statistical Models.
Internal models of the world are considered an essential part of intelligence in AI,
cognitive sciences, and psychology. The course describes how to design neural
networks with internal models. Model-based neural networks combine domain
knowledge with learning and adaptivity of neural networks.
Prerequisites: probability
Level: graduate or advanced undergraduate
Chapter 5.
4. Model-Based Neural Networks: Dynamic Models.
Internal models of the world are considered an essential part of intelligence in AI,
cognitive sciences, and psychology. The course describes how to design neural
networks with internal models. Model-based neural networks combine domain
knowledge with learning and adaptivity of neural networks.
Prerequisites: probability and signal processing
Level: graduate or advanced undergraduate
Chapters 6 and 7.
5. Relationships between Philosophical and Mathematical Concepts of Mind (for
students with hard-science background).
Relationships between contemporary mathematical concepts of intelligence and

2300-year-old philosophical concepts of mind are much closer than is generally
recognized. Specific mathematical concepts and debates are related to specific
philosophical ones.
Prerequisites: a course in AI, neural networks, pattern recognition, signal processing,
or control
Level: graduate or undergraduate
Chapter 2, Chapter 3, Sec. 3.1, Chapters 10
through 12.
6. Relationships between Philosophical and Mathematical Concepts of Mind (for
students without hard-science background).
Relationships between contemporary mathematical concepts of intelligence and
2300-year-old philosophical concepts of mind are much closer than is generally
recognized. Specific mathematical concepts and debates are related to specific
philosophical ones.
Prerequisites: a course in classical or contemporary philosophy
Level: graduate or undergraduate
Chapter 2, Chapter 3, Sec. 3.1, Chapters 10
and 11 (with mathematical contents being
optional), Chapter 12.
xxiv Preface
ACKNOWLEDGMENTS
A number of people contributed to this book in various ways: through discussions, en-
couragement, and support. The ideas within this book were gradually taking shape while,
together with my coworkers, we were solving our everyday research and development
problems at Nichols Research. My sponsors were interested in these ideas to the extent that
they provided financial support for the research, on which this book is founded, and some of
them have been actively working on similar ideas. Some read my papers and manuscripts
and gave valuable advice. Many issues were clarified in the heated Internet discussions
among the subscribers to the Architectures for Intelligent Control Systems list. My friends
attending Friday night gatherings at Jordan road provided much discussion and thought.

And my wife inspired me to think profoundly.
It is my pleasure to thank the people whose thoughts, ideas, encouragement, and
support shaped this book and made it possible: M. Akhundov, J. Albus, U. Aleshkovsky,
R. Berg, R. Brockett, B. Burdick, G. Carpenter, W. Chang, D. Choi, R. Deming, V. Dmitriev,
W. Freeman, K. Fukunaga, L. Garvin, R. Gudwin, M. Gouzie, S. Greineder, S. Grossberg,
M. Karpovsky, M. Kreps, L. Levitin, A. Lieberman, T. Luginbuhl, A. Meystel, K. Moore,
V. Oytser, D. Radyshevsky, C. Plum, A. Samarov, W. Schoendorf, D. Skatrud, R. Streit,
E. Taborsky, E. Tichovolsky, B. Veijers, D. Vinkovetsky, Y. Vinkovetsky, V. Webb, M.
Xiarhos, L. Zadeh, and G. Zainiev.