Notes on neural networks and deep learning

Introduction to Neural Networks and Deep Learning
Introduction to the Convolutional Network
Andres Mendez-Vazquez

March 28, 2021

1 / 148


Outline
1

Introduction
The Long Path
The Problem of Image Processing
Multilayer Neural Network Classification
Drawbacks
Possible Solution

2

Convolutional Networks
History
Local Connectivity
Sharing Parameters

3

Layers
Convolutional Layer
Convolutional Architectures

A Little Bit of Notation
Deconvolution Layer
Alternating Minimization
Non-Linearity Layer
Fixing the Problem, ReLu function
Back to the Non-Linearity Layer
Rectification Layer
Local Contrast Normalization Layer
Sub-sampling and Pooling
Strides
Normalization Layer AKA Batch Normalization
Finally, The Fully Connected Layer

4

An Example of CNN
The Proposed Architecture
Backpropagation
Deriving wr,s,k
Deriving the Kernel Filters
2 / 148


Outline
1

Introduction
The Long Path
The Problem of Image Processing
Multilayer Neural Network Classification

Drawbacks
Possible Solution

2

Convolutional Networks
History
Local Connectivity
Sharing Parameters

3

Layers
Convolutional Layer
Convolutional Architectures
A Little Bit of Notation
Deconvolution Layer
Alternating Minimization
Non-Linearity Layer
Fixing the Problem, ReLu function
Back to the Non-Linearity Layer
Rectification Layer
Local Contrast Normalization Layer
Sub-sampling and Pooling
Strides
Normalization Layer AKA Batch Normalization
Finally, The Fully Connected Layer

4

An Example of CNN
The Proposed Architecture
Backpropagation
Deriving wr,s,k
Deriving the Kernel Filters
3 / 148


The Long Path [1]
2018
Channel
Boosting

Beyond

2018

Attention

Channel Boosted CNN

CBAM
Residual Attention
Module

2018

Feature Map
Explotation

A Small History
of a
Revolution

Complex Architectures and
The Attention Revolution

PolyNet
WideResNext

2017

ResNext

The Beginnig of Atention?

ZfNet

Parameter
Optimization

Residual and Multipath
Architectures

2016

Multi-Path
Connectivity

FractalNet

Dense Net
ResNet

2015
Skip Connections

Highway Net

Depth
Revolution

VGG
2014

Effective Reception Filed
(Small Size Filters)

Inception-ResNet
Factorization

Spatial
Explotation

Depth
Explotation

Spatial
Explotation

2010 ImageNet

2006 GPU

2007 NVIDIA
PROGRAMMING

The Revolution

Inception-V2

Parallelism

2014

GoogleNet

Inception
Block

First Results

Inception-V4
Inception-V3

Bottleneck

2006 Maxpooling

6.5

Early 2000
CNN Stagnation

5

9

Feature
Visualization

2017

4.5

SE Net
Transformers-CNN

PyramidalNet

Width
Explotation

4 5 1 1
2 6 2 6
3 5 7 3
1 9 2 1

CMPE-SE

2013

2012

3D CNN's
AlexNet

1998

LeNet
1989

Early Attempts
1979

ConvNet

Neurocognition

4 / 148


Outline
1

Introduction
The Long Path
The Problem of Image Processing
Multilayer Neural Network Classification
Drawbacks
Possible Solution

2

Convolutional Networks
History
Local Connectivity
Sharing Parameters

3

Layers
Convolutional Layer
Convolutional Architectures
A Little Bit of Notation
Deconvolution Layer
Alternating Minimization
Non-Linearity Layer
Fixing the Problem, ReLu function
Back to the Non-Linearity Layer
Rectification Layer
Local Contrast Normalization Layer
Sub-sampling and Pooling
Strides
Normalization Layer AKA Batch Normalization
Finally, The Fully Connected Layer

4

An Example of CNN
The Proposed Architecture

Backpropagation
Deriving wr,s,k
Deriving the Kernel Filters
5 / 148


Digital Images as pixels in a digitized matrix [2]
Ilumination
Source

Output
Ilumination
Source

6 / 148


Further [2]

Pixel values typically represent
Gray levels, colors, heights, opacities etc

Something Notable
Remember digitization implies that a digital image is an
approximation of a real scene

7 / 148


Further [2]

Pixel values typically represent
Gray levels, colors, heights, opacities etc

Something Notable
Remember digitization implies that a digital image is an
approximation of a real scene

7 / 148


Images

Common image formats include
On sample/pixel per point (B&W or Grayscale)
Three samples/pixel per point (Red, Green, and Blue)
Four samples/pixel per point (Red, Green, Blue, and “Alpha”)

8 / 148


Therefore, we have the following process

Low Level Process
Imagen

Noise
Removal

Sharpening

9 / 148


Example
Edge Detection

10 / 148


Then

Mid Level Process

Input

Processes
Output
Object
Image Recognition Attributes
Segmentation

11 / 148


Example
Object Recognition

12 / 148

Therefore

It would be nice to automatize all these processes
We would solve a lot of headaches when setting up such process

Why not to use the data sets
By using a Neural Networks that replicates the process.

13 / 148


Therefore

It would be nice to automatize all these processes
We would solve a lot of headaches when setting up such process

Why not to use the data sets
By using a Neural Networks that replicates the process.

13 / 148


Outline
1

Introduction
The Long Path
The Problem of Image Processing
Multilayer Neural Network Classification

Drawbacks
Possible Solution

2

Convolutional Networks
History
Local Connectivity
Sharing Parameters

3

Layers
Convolutional Layer
Convolutional Architectures
A Little Bit of Notation
Deconvolution Layer
Alternating Minimization
Non-Linearity Layer
Fixing the Problem, ReLu function
Back to the Non-Linearity Layer
Rectification Layer
Local Contrast Normalization Layer
Sub-sampling and Pooling
Strides
Normalization Layer AKA Batch Normalization
Finally, The Fully Connected Layer

4

An Example of CNN
The Proposed Architecture
Backpropagation
Deriving wr,s,k
Deriving the Kernel Filters
14 / 148


Multilayer Neural Network Classification
We have the following classification [3]

15 / 148


Outline
1

Introduction
The Long Path
The Problem of Image Processing
Multilayer Neural Network Classification
Drawbacks
Possible Solution

2

Convolutional Networks
History
Local Connectivity
Sharing Parameters

3

Layers
Convolutional Layer
Convolutional Architectures
A Little Bit of Notation
Deconvolution Layer
Alternating Minimization
Non-Linearity Layer
Fixing the Problem, ReLu function
Back to the Non-Linearity Layer
Rectification Layer
Local Contrast Normalization Layer
Sub-sampling and Pooling
Strides
Normalization Layer AKA Batch Normalization
Finally, The Fully Connected Layer

4

An Example of CNN
The Proposed Architecture
Backpropagation
Deriving wr,s,k
Deriving the Kernel Filters
16 / 148


Drawbacks of previous neural networks

The number of trainable parameters becomes extremely large
Large N

A
Z

17 / 148


Drawbacks of previous neural networks
In addition, little or no invariance to shifting, scaling, and other forms
of distortion
Large N

A
Z

18 / 148


Drawbacks of previous neural networks
In addition, little or no invariance to shifting, scaling, and other forms
of distortion
Large N
Shift to the Left

A
Z

19 / 148

Drawbacks of previous neural networks

The topology of the input data is completely ignored

20 / 148


For Example

We have
Black and white patterns: 232×32 = 21024
Gray scale patterns: 25632×32 = 2561024

21 / 148


For Example

If we have an element that the network has never seen

22 / 148


Possible Solution

We can minimize this drawbacks by getting
Fully connected network of sufficient size can produce outputs that
are invariant with respect to such variations.

Problem!!!
Training time
Network size
Free parameters

23 / 148