Deep learning with keras by antonio gulli

TitlePage
DeepLearningwithKeras








ImplementneuralnetworkswithKerasonTheanoandTensorFlow









AntonioGulli
SujitPal

BIRMINGHAM-MUMBAI



DeepLearningwithKeras

Copyright©2017PacktPublishing
Allrightsreserved.Nopartofthisbookmaybereproduced,storedinaretrievalsystem,ortransmittedin
anyformorbyanymeans,withoutthepriorwrittenpermissionofthepublisher,exceptinthecaseofbrief
quotationsembeddedincriticalarticlesorreviews.
Everyefforthasbeenmadeinthepreparationofthisbooktoensuretheaccuracyoftheinformation
presented.However,theinformationcontainedinthisbookissoldwithoutwarranty,eitherexpressor
implied.Neithertheauthors,norPacktPublishing,anditsdealersanddistributorswillbeheldliablefor
anydamagescausedorallegedtobecauseddirectlyorindirectlybythisbook.
PacktPublishinghasendeavoredtoprovidetrademarkinformationaboutallofthecompaniesand
productsmentionedinthisbookbytheappropriateuseofcapitals.However,PacktPublishingcannot
guaranteetheaccuracyofthisinformation.
Firstpublished:April2017
Productionreference:1240417

PublishedbyPacktPublishingLtd.
LiveryPlace
35LiveryStreet
Birmingham
B32PB,UK.
ISBN978-1-78712-842-2
www.packtpub.com



Credits

Authors
AntonioGulli

CopyEditor

VikrantPhadkay

SujitPal

Reviewers
MikeDahlin

ProjectCoordinator
NidhiJoshi

NickMcClure
CorradoZocollo

CommissioningEditor
AmeyVarangaonkar

Proofreader
SafisEditing


AcquisitionEditor
DivyaPoojari

Indexer
FrancyPuthiry

ContentDevelopmentEditor
CherylDsa

Graphics

TaniaDutta

TechnicalEditor
DineshPawar

ProductionCoordinator
ArvindkumarGupta



AbouttheAuthors
AntonioGulliisasoftwareexecutiveandbusinessleaderwithapassionforestablishingandmanaging
globaltechnologicaltalent,innovation,andexecution.Heisanexpertinsearchengines,onlineservices,
machinelearning,informationretrieval,analytics,andcloudcomputing.Sofar,hehasbeenluckyenough
togainprofessionalexperienceinfourdifferentcountriesinEuropeandmanagedpeopleinsixdifferent
countriesinEuropeandAmerica.AntonioservedasCEO,GM,CTO,VP,director,andsiteleadin
multiplefieldsspanningfrompublishing(Elsevier)toconsumerinternet(Ask.comandTiscali)andhightechR&D(MicrosoftandGoogle).
Iwouldliketothankmycoauthor,SujitPal,forbeingasuchtalentedcolleague,alwayswillingto
helpwithahumblespirit.Iconstantlyappreciatehisdedicationtoteamwork,whichmadethisbooka
realthing.
IwouldliketothankFrancoisChollet(andthemanyKerascontributors)fortakingthetimeandeffort
tobuildanawesomedeeplearningtoolkitthatiseasytousewithoutsacrificingtoomuchpower.
IwouldalsoliketothankoureditorsfromPackt,DivyaPoojari,CherylDsa,andDineshPawar,and
ourreviewersfromPacktandGoogle,fortheirsupportandvaluablesuggestions.Thisbookwouldnot
havebeenpossiblewithoutyou.
Iwouldliketothankmymanager,Brad,andmycolleaguesMikeandCorradoatGooglefor
encouragingmetowritethisbook,andfortheirconstanthelpinreviewingthecontent.
IwouldliketothankSameFusy,HerbaciarniaiKawiarniainWarsaw.Igottheinitialinspirationto
writethisbookinfrontofacupofteachosenamonghundredsofdifferentoffers.Thisplaceismagic
andIstronglyrecommendvisitingitifyouareinsearchofaplacetostimulatecreativeness(http://www.s

amefusy.pl/).
ThenIwouldliketothankHRBPatGoogleforsupportingmywishtodonateallofthisbook's
royaltiesinfavorofaminority/diversityscholarship.
IwouldliketothankmyfriendsEric,Laura,Francesco,Ettore,andAntonellaforsupportingmewhen
Iwasinneed.Long-termfriendshipisarealthing,andyouaretruefriendstome.
IwouldliketothankmysonLorenzoforencouragingmetojoinGoogle,mysonLeonardoforhis
constantpassiontodiscovernewthings,andmydaughterAuroraformakingmesmileeverydayofmy
life.FinallythankstomyfatherElioandmymotherMariafortheirlove.
SujitPalisatechnologyresearchdirectoratElsevierLabs,workingonbuildingintelligentsystems
aroundresearchcontentandmetadata.Hisprimaryinterestsareinformationretrieval,ontologies,natural
languageprocessing,machinelearning,anddistributedprocessing.Heiscurrentlyworkingonimage
classificationandsimilarityusingdeeplearningmodels.Priortothis,heworkedintheconsumer
healthcareindustry,wherehehelpedbuildontology-backedsemanticsearch,contextualadvertising,and
EMRdataprocessingplatforms.HewritesabouttechnologyonhisblogatSalmonRun.
Iwouldliketothankmycoauthor,AntonioGulli,foraskingmetojoinhiminwritingthisbook.This
wasanincredibleopportunityandagreatlearningexperienceforme.Besides,hadhenotdoneso,I
quiteliterallywouldn'thavebeenheretoday.
IwouldliketothankRonDaniel,thedirectorofElsevierLabs,andBradleyPAllen,chiefarchitectat
Elsevier,forintroducingmetodeeplearningandmakingmeabelieverinitscapabilities.


IwouldalsoliketothankFrancoisChollet(andthemanyKerascontributors)fortakingthetimeand
efforttobuildanawesomedeeplearningtoolkitthatiseasytousewithoutsacrificingtoomuch
power.
ThankstooureditorsfromPackt,DivyaPoojari,CherylDsa,andDineshPawar,andourreviewers
fromPacktandGoogle,fortheirsupportandvaluablesuggestions.Thisbookwouldnothavebeen
possiblewithoutyou.
Iwouldliketothankmycolleaguesandmanagersovertheyears,especiallytheoneswhotooktheir
chanceswithmeandhelpedmemakediscontinuouschangesinmycareer.
Finally,IwouldliketothankmyfamilyforputtingupwithmethesepastfewmonthsasIjuggled

work,thisbook,andfamily,inthatorder.Ihopeyouwillagreethatitwasallworthit.





AbouttheReviewer

NickMcClureiscurrentlyaseniordatascientistatPayScaleInc.inSeattle,Washington,USA.Priorto
that,heworkedatZillowandCaesarsEntertainment.Hegothisdegreesinappliedmathematicsfromthe
UniversityofMontanaandtheCollegeofSaintBenedictandSaintJohn'sUniversity.Nickhasalso
authoredTensorFlowMachineLearningCookbookbyPacktPublishing.
Hehasapassionforlearningandadvocatingforanalytics,machinelearning,andartificialintelligence.
Nickoccasionallyputshisthoughtsandmusingonhisblog,fromdata.org,orthroughhisTwitteraccountat
@nfmcclure.





www.PacktPub.com
Forsupportfilesanddownloadsrelatedtoyourbook,pleasevisitwww.PacktPub.com.
DidyouknowthatPacktofferseBookversionsofeverybookpublished,withPDFandePubfiles
available?YoucanupgradetotheeBookversionatwww.PacktPub.comandasaprintbookcustomer,youare
entitledtoadiscountontheeBookcopy.Getintouchwithusatformoredetails.
Atwww.PacktPub.com,youcanalsoreadacollectionoffreetechnicalarticles,signupforarangeoffree
newslettersandreceiveexclusivediscountsandoffersonPacktbooksandeBooks.

/>
Getthemostin-demandsoftwareskillswithMapt.MaptgivesyoufullaccesstoallPacktbooksand

videocourses,aswellasindustry-leadingtoolstohelpyouplanyourpersonaldevelopmentandadvance
yourcareer.


Whysubscribe?
FullysearchableacrosseverybookpublishedbyPackt
Copyandpaste,print,andbookmarkcontent
Ondemandandaccessibleviaawebbrowser





CustomerFeedback

ThanksforpurchasingthisPacktbook.AtPackt,qualityisattheheartofoureditorialprocess.Tohelpus
improve,pleaseleaveusanhonestreviewonthisbook'sAmazonpageat />Ifyou'dliketojoinourteamofregularreviewers,youcane-mailusatWe
awardourregularreviewerswithfreeeBooksandvideosinexchangefortheirvaluablefeedback.Help
usberelentlessinimprovingourproducts!





TableofContents

Preface
Mission
Howdeeplearningisdifferentfrommachinelearningandartificialintelligence
Whatthisbookcovers

Whatyouneedforthisbook
Whothisbookisfor
Conventions
Readerfeedback
Customersupport
Downloadingtheexamplecode
Downloadingthecolorimagesofthisbook
Errata
Piracy
Questions

1.

NeuralNetworksFoundations
Perceptron
ThefirstexampleofKerascode
Multilayerperceptron—thefirstexampleofanetwork
Problemsintrainingtheperceptronandasolution
Activationfunction—sigmoid
Activationfunction—ReLU
Activationfunctions
Arealexample—recognizinghandwrittendigits
One-hotencoding—OHE
DefiningasimpleneuralnetinKeras
RunningasimpleKerasnetandestablishingabaseline
ImprovingthesimplenetinKeraswithhiddenlayers
FurtherimprovingthesimplenetinKeraswithdropout
TestingdifferentoptimizersinKeras

Increasingthenumberofepochs
Controllingtheoptimizerlearningrate
Increasingthenumberofinternalhiddenneurons
Increasingthesizeofbatchcomputation
Summarizingtheexperimentsrunforrecognizinghandwrittencharts
Adoptingregularizationforavoidingoverfitting
Hyperparameterstuning
Predictingoutput
Apracticaloverviewofbackpropagation
Towardsadeeplearningapproach
Summary

2.

KerasInstallationandAPI
InstallingKeras
Step1—installsomeusefuldependencies
Step2—installTheano
Step3—installTensorFlow
Step4—installKeras
Step5—testingTheano,TensorFlow,andKeras
ConfiguringKeras
InstallingKerasonDocker
InstallingKerasonGoogleCloudML
InstallingKerasonAmazonAWS
InstallingKerasonMicrosoftAzure
KerasAPI
GettingstartedwithKerasarchitecture
Whatisatensor?
ComposingmodelsinKeras

Sequentialcomposition
Functionalcomposition
Anoverviewofpredefinedneuralnetworklayers
Regulardense
Recurrentneuralnetworks—simple,LSTM,andGRU


Convolutionalandpoolinglayers
Regularization
Batchnormalization
Anoverviewofpredefinedactivationfunctions
Anoverviewoflossesfunctions
Anoverviewofmetrics
Anoverviewofoptimizers
Someusefuloperations
Savingandloadingtheweightsandthearchitectureofamodel
Callbacksforcustomizingthetrainingprocess
Checkpointing
UsingTensorBoardandKeras
UsingQuiverandKeras
Summary

3.

DeepLearningwithConvNets
Deepconvolutionalneuralnetwork—DCNN
Localreceptivefields
Sharedweightsandbias
Poolinglayers
Max-pooling

Averagepooling
ConvNetssummary
AnexampleofDCNN—LeNet
LeNetcodeinKeras
Understandingthepowerofdeeplearning
RecognizingCIFAR-10imageswithdeeplearning
ImprovingtheCIFAR-10performancewithdeeperanetwork
ImprovingtheCIFAR-10performancewithdataaugmentation
PredictingwithCIFAR-10
Verydeepconvolutionalnetworksforlarge-scaleimagerecognition
RecognizingcatswithaVGG-16net
UtilizingKerasbuilt-inVGG-16netmodule


Recyclingpre-builtdeeplearningmodelsforextractingfeatures
Verydeepinception-v3netusedfortransferlearning
Summary

4.

GenerativeAdversarialNetworksandWaveNet
WhatisaGAN?
SomeGANapplications
Deepconvolutionalgenerativeadversarialnetworks
KerasadversarialGANsforforgingMNIST
KerasadversarialGANsforforgingCIFAR
WaveNet—agenerativemodelforlearninghowtoproduceaudio
Summary

5.

WordEmbeddings
Distributedrepresentations
word2vec
Theskip-gramword2vecmodel
TheCBOWword2vecmodel
Extractingword2vecembeddingsfromthemodel
Usingthird-partyimplementationsofword2vec
ExploringGloVe
Usingpre-trainedembeddings
Learnembeddingsfromscratch
Fine-tuninglearnedembeddingsfromword2vec
Fine-tunelearnedembeddingsfromGloVe
Lookupembeddings
Summary

6.

RecurrentNeuralNetwork—RNN
SimpleRNNcells
SimpleRNNwithKeras—generatingtext
RNNtopologies
Vanishingandexplodinggradients
Longshorttermmemory—LSTM
LSTMwithKeras—sentimentanalysis


Gatedrecurrentunit—GRU
GRUwithKeras—POStagging
BidirectionalRNNs

StatefulRNNs
StatefulLSTMwithKeras—predictingelectricityconsumption
OtherRNNvariants
Summary

7.

AdditionalDeepLearningModels
KerasfunctionalAPI
Regressionnetworks
Kerasregressionexample—predictingbenzenelevelsintheair
Unsupervisedlearning—autoencoders
Kerasautoencoderexample—sentencevectors
Composingdeepnetworks
Kerasexample—memorynetworkforquestionanswering
CustomizingKeras
Kerasexample—usingthelambdalayer
Kerasexample—buildingacustomnormalizationlayer
Generativemodels
Kerasexample—deepdreaming
Kerasexample—styletransfer
Summary

8.

AIGamePlaying
Reinforcementlearning
Maximizingfuturerewards
Q-learning
ThedeepQ-networkasaQ-function

Balancingexplorationwithexploitation
Experiencereplay,orthevalueofexperience
Example-KerasdeepQ-networkforcatch
Theroadahead
Summary

9.

Conclusion
Keras2.0—whatisnew
InstallingKeras2.0


APIchanges