Learning tensorflow a guide to building deep learning systems

LearningTensorFlow
AGuidetoBuildingDeepLearningSystems

TomHope,YehezkelS.Resheff,andItayLieder


LearningTensorFlow
byTomHope,YehezkelS.Resheff,andItayLiederCopyright©2017TomHope,ItayLieder,and
YehezkelS.Resheff.Allrightsreserved.
PrintedintheUnitedStatesofAmerica.
PublishedbyO’ReillyMedia,Inc.,1005GravensteinHighwayNorth,Sebastopol,CA95472.
O’Reillybooksmaybepurchasedforeducational,business,orsalespromotionaluse.Onlineeditionsare
alsoavailableformosttitles(Formoreinformation,contactour
corporate/institutionalsalesdepartment:800-998-9938or
Editor:NicoleTache
ProductionEditor:ShinyKalapurakkel
Copyeditor:RachelHead
Proofreader:SharonWilkey
Indexer:JudithMcConville
InteriorDesigner:DavidFutato
CoverDesigner:KarenMontgomery
Illustrator:RebeccaDemarest
August2017:FirstEdition


RevisionHistoryfortheFirstEdition
2017-08-04:FirstRelease
TheO’ReillylogoisaregisteredtrademarkofO’ReillyMedia,Inc.LearningTensorFlow,thecover
image,andrelatedtradedressaretrademarksofO’ReillyMedia,Inc.

Whilethepublisherandtheauthorshaveusedgoodfaitheffortstoensurethattheinformationand
instructionscontainedinthisworkareaccurate,thepublisherandtheauthorsdisclaimallresponsibility
forerrorsoromissions,includingwithoutlimitationresponsibilityfordamagesresultingfromtheuseof
orrelianceonthiswork.Useoftheinformationandinstructionscontainedinthisworkisatyourown
risk.Ifanycodesamplesorothertechnologythisworkcontainsordescribesissubjecttoopensource
licensesortheintellectualpropertyrightsofothers,itisyourresponsibilitytoensurethatyourusethereof
complieswithsuchlicensesand/orrights.
978-1-491-97851-1
[M]


Preface
Deeplearninghasemergedinthelastfewyearsasapremiertechnologyforbuildingintelligentsystems
thatlearnfromdata.Deepneuralnetworks,originallyroughlyinspiredbyhowthehumanbrainlearns,
aretrainedwithlargeamountsofdatatosolvecomplextaskswithunprecedentedaccuracy.Withopen
sourceframeworksmakingthistechnologywidelyavailable,itisbecomingamust-knowforanybody
involvedwithbigdataandmachinelearning.
TensorFlowiscurrentlytheleadingopensourcesoftwarefordeeplearning,usedbyarapidlygrowing
numberofpractitionersworkingoncomputervision,naturallanguageprocessing(NLP),speech
recognition,andgeneralpredictiveanalytics.
Thisbookisanend-to-endguidetoTensorFlowdesignedfordatascientists,engineers,students,and
researchers.Thebookadoptsahands-onapproachsuitableforabroadtechnicalaudience,allowing
beginnersagentlestartwhiledivingdeepintoadvancedtopicsandshowinghowtobuildproductionreadysystems.
Inthisbookyouwilllearnhowto:
1. GetupandrunningwithTensorFlow,rapidlyandpainlessly.
2. UseTensorFlowtobuildmodelsfromthegroundup.
3. TrainandunderstandpopulardeeplearningmodelsforcomputervisionandNLP.
4. Useextensiveabstractionlibrariestomakedevelopmenteasierandfaster.
5. ScaleupTensorFlowwithqueuingandmultithreading,trainingonclusters,andservingoutputin
production.

6. Andmuchmore!
ThisbookiswrittenbydatascientistswithextensiveR&Dexperienceinbothindustryandacademic
research.Theauthorstakeahands-onapproach,combiningpracticalandintuitiveexamples,illustrations,
andinsightssuitableforpractitionersseekingtobuildproduction-readysystems,aswellasreaders
lookingtolearntounderstandandbuildflexibleandpowerfulmodels.


Prerequisites
ThisbookassumessomebasicPythonprogrammingknow-how,includingbasicfamiliaritywiththe
scientificlibraryNumPy.
Machinelearningconceptsaretoucheduponandintuitivelyexplainedthroughoutthebook.Forreaders
whowanttogainadeeperunderstanding,areasonablelevelofknowledgeinmachinelearning,linear
algebra,calculus,probability,andstatisticsisrecommended.


ConventionsUsedinThisBook
Thefollowingtypographicalconventionsareusedinthisbook:
Italic
Indicatesnewterms,URLs,emailaddresses,filenames,andfileextensions.
Constantwidth

Usedforprogramlistings,aswellaswithinparagraphstorefertoprogramelementssuchas
variableorfunctionnames,databases,datatypes,environmentvariables,statements,andkeywords.
Constantwidthbold

Showscommandsorothertextthatshouldbetypedliterallybytheuser.
Constantwidthitalic

Showstextthatshouldbereplacedwithuser-suppliedvaluesorbyvaluesdeterminedbycontext.

UsingCodeExamples
Supplementalmaterial(codeexamples,exercises,etc.)isavailablefordownloadat
/>Thisbookisheretohelpyougetyourjobdone.Ingeneral,ifexamplecodeisofferedwiththisbook,you
mayuseitinyourprogramsanddocumentation.Youdonotneedtocontactusforpermissionunless
you’rereproducingasignificantportionofthecode.Forexample,writingaprogramthatusesseveral
chunksofcodefromthisbookdoesnotrequirepermission.SellingordistributingaCD-ROMof
examplesfromO’Reillybooksdoesrequirepermission.Answeringaquestionbycitingthisbookand
quotingexamplecodedoesnotrequirepermission.Incorporatingasignificantamountofexamplecode
fromthisbookintoyourproduct’sdocumentationdoesrequirepermission.
Weappreciate,butdonotrequire,attribution.Anattributionusuallyincludesthetitle,author,publisher,
andISBN.Forexample:“LearningTensorFlowbyTomHope,YehezkelS.Resheff,andItayLieder
(O’Reilly).Copyright2017TomHope,ItayLieder,andYehezkelS.Resheff,978-1-491-97851-1.”
Ifyoufeelyouruseofcodeexamplesfallsoutsidefairuseorthepermissiongivenabove,feelfreeto
contactusat


O’ReillySafari
NOTE
Safari(formerlySafariBooksOnline)isamembership-basedtrainingandreferenceplatformfor
enterprise,government,educators,andindividuals.
Membershaveaccesstothousandsofbooks,trainingvideos,LearningPaths,interactivetutorials,and
curatedplaylistsfromover250publishers,includingO’ReillyMedia,HarvardBusinessReview,
PrenticeHallProfessional,Addison-WesleyProfessional,MicrosoftPress,Sams,Que,PeachpitPress,
Adobe,FocalPress,CiscoPress,JohnWiley&Sons,Syngress,MorganKaufmann,IBMRedbooks,
Packt,AdobePress,FTPress,Apress,Manning,NewRiders,McGraw-Hill,Jones&Bartlett,and
CourseTechnology,amongothers.
Formoreinformation,pleasevisit />

HowtoContactUs

Pleaseaddresscommentsandquestionsconcerningthisbooktothepublisher:
O’ReillyMedia,Inc.
1005GravensteinHighwayNorth
Sebastopol,CA95472
800-998-9938(intheUnitedStatesorCanada)
707-829-0515(internationalorlocal)
707-829-0104(fax)
Wehaveawebpageforthisbook,wherewelisterrata,examples,andanyadditionalinformation.You
canaccessthispageat />Tocommentorasktechnicalquestionsaboutthisbook,sendemailto
Formoreinformationaboutourbooks,courses,conferences,andnews,seeourwebsiteat
.
FindusonFacebook: />FollowusonTwitter: />WatchusonYouTube: />

Acknowledgments
Theauthorswouldliketothankthereviewerswhoofferedfeedbackonthisbook:ChrisFregly,Marvin
Bertin,OrenSarShalom,andYoniLavi.WewouldalsoliketothankNicoleTacheandtheO’Reillyteam
formakingitapleasuretowritethebook.
Ofcourse,thankstoallthepeopleatGooglewithoutwhomTensorFlowwouldnotexist.


Chapter1.Introduction
Thischapterprovidesahigh-leveloverviewofTensorFlowanditsprimaryuse:implementingand
deployingdeeplearningsystems.Webeginwithaverybriefintroductorylookatdeeplearning.Wethen
presentTensorFlow,showcasingsomeofitsexcitingusesforbuildingmachineintelligence,andthenlay
outitskeyfeaturesandproperties.


GoingDeep
Fromlargecorporationstobuddingstartups,engineersanddatascientistsarecollectinghugeamountsof
dataandusingmachinelearningalgorithmstoanswercomplexquestionsandbuildintelligentsystems.

Whereveronelooksinthislandscape,theclassofalgorithmsassociatedwithdeeplearninghaverecently
seengreatsuccess,oftenleavingtraditionalmethodsinthedust.Deeplearningisusedtodayto
understandthecontentofimages,naturallanguage,andspeech,insystemsrangingfrommobileappsto
autonomousvehicles.Developmentsinthisfieldaretakingplaceatbreakneckspeed,withdeeplearning
beingextendedtootherdomainsandtypesofdata,likecomplexchemicalandgeneticstructuresfordrug
discoveryandhigh-dimensionalmedicalrecordsinpublichealthcare.
Deeplearningmethods—whichalsogobythenameofdeepneuralnetworks—wereoriginallyroughly
inspiredbythehumanbrain’svastnetworkofinterconnectedneurons.Indeeplearning,wefeedmillions
ofdatainstancesintoanetworkofneurons,teachingthemtorecognizepatternsfromrawinputs.Thedeep
neuralnetworkstakerawinputs(suchaspixelvaluesinanimage)andtransformthemintouseful
representations,extractinghigher-levelfeatures(suchasshapesandedgesinimages)thatcapture
complexconceptsbycombiningsmallerandsmallerpiecesofinformationtosolvechallengingtaskssuch
asimageclassification(Figure1-1).Thenetworksautomaticallylearntobuildabstractrepresentations
byadaptingandcorrectingthemselves,fittingpatternsobservedinthedata.Theabilitytoautomatically
constructdatarepresentationsisakeyadvantageofdeepneuralnetsoverconventionalmachinelearning,
whichtypicallyrequiresdomainexpertiseandmanualfeatureengineeringbeforeany“learning”can
occur.


Figure1-1.Anillustrationofimageclassificationwithdeepneuralnetworks.Thenetworktakesrawinputs(pixelvaluesinanimage)
andlearnstotransformthemintousefulrepresentations,inordertoobtainanaccurateimageclassification.

ThisbookisaboutGoogle’sframeworkfordeeplearning,TensorFlow.Deeplearningalgorithmshave
beenusedforseveralyearsacrossmanyproductsandareasatGoogle,suchassearch,translation,
advertising,computervision,andspeechrecognition.TensorFlowis,infact,asecond-generationsystem
forimplementinganddeployingdeepneuralnetworksatGoogle,succeedingtheDistBeliefprojectthat
startedin2011.
TensorFlowwasreleasedtothepublicasanopensourceframeworkwithanApache2.0licensein
November2015andhasalreadytakentheindustrybystorm,withadoptiongoingfarbeyondinternal
Googleprojects.Itsscalabilityandflexibility,combinedwiththeformidableforceofGoogleengineers

whocontinuetomaintainanddevelopit,havemadeTensorFlowtheleadingsystemfordoingdeep
learning.


UsingTensorFlowforAISystems
BeforegoingintomoredepthaboutwhatTensorFlowisanditskeyfeatures,wewillbrieflygivesome
excitingexamplesofhowTensorFlowisusedinsomecutting-edgereal-worldapplications,atGoogle
andbeyond.
Pre-trainedmodels:state-of-the-artcomputervisionforall
Oneprimaryareawheredeeplearningistrulyshiningiscomputervision.Afundamentaltaskincomputer
visionisimageclassification—buildingalgorithmsandsystemsthatreceiveimagesasinput,andreturn
asetofcategoriesthatbestdescribethem.Researchers,datascientists,andengineershavedesigned
advanceddeepneuralnetworksthatobtainhighlyaccurateresultsinunderstandingvisualcontent.These
deepnetworksaretypicallytrainedonlargeamountsofimagedata,takingmuchtime,resources,and
effort.However,inagrowingtrend,researchersarepubliclyreleasingpre-trainedmodels—deepneural
netsthatarealreadytrainedandthatuserscandownloadandapplytotheirdata(Figure1-2).

Figure1-2.Advancedcomputervisionwithpre-trainedTensorFlowmodels.

TensorFlowcomeswithusefulutilitiesallowinguserstoobtainandapplycutting-edgepretrained
models.Wewillseeseveralpracticalexamplesanddiveintothedetailsthroughoutthisbook.


Generatingrichnaturallanguagedescriptionsforimages
Oneexcitingareaofdeeplearningresearchforbuildingmachineintelligencesystemsisfocusedon
generatingnaturallanguagedescriptionsforvisualcontent(Figure1-3).Akeytaskinthisareaisimage
captioning—teachingthemodeltooutputsuccinctandaccuratecaptionsforimages.Heretoo,advanced
pre-trainedTensorFlowmodelsthatcombinenaturallanguageunderstandingwithcomputervisionare
available.

Figure1-3.Goingfromimagestotextwithimagecaptioning(illustrativeexample).

Textsummarization
Naturallanguageunderstanding(NLU)isakeycapabilityforbuildingAIsystems.Tremendousamounts
oftextaregeneratedeveryday:webcontent,socialmedia,news,emails,internalcorporate


correspondences,andmanymore.Oneofthemostsought-afterabilitiesistosummarizetext,takinglong
documentsandgeneratingsuccinctandcoherentsentencesthatextractthekeyinformationfromthe
originaltexts(Figure1-4).Aswewillseelaterinthisbook,TensorFlowcomeswithpowerfulfeatures
fortrainingdeepNLUnetworks,whichcanalsobeusedforautomatictextsummarization.

Figure1-4.Anillustrationofsmarttextsummarization.


TensorFlow:What’sinaName?
Deepneuralnetworks,asthetermandtheillustrationswe’veshownimply,areallaboutnetworksof
neurons,witheachneuronlearningtodoitsownoperationaspartofalargerpicture.Datasuchasimages
entersthisnetworkasinput,andflowsthroughthenetworkasitadaptsitselfattrainingtimeorpredicts
outputsinadeployedsystem.
Tensorsarethestandardwayofrepresentingdataindeeplearning.Simplyput,tensorsarejust
multidimensionalarrays,anextensionoftwo-dimensionaltables(matrices)todatawithhigher
dimensionality.Justasablack-and-white(grayscale)imagesarerepresentedas“tables”ofpixelvalues,
RGBimagesarerepresentedastensors(three-dimensionalarrays),witheachpixelhavingthreevalues
correspondingtored,green,andbluecomponents.
InTensorFlow,computationisapproachedasadataflowgraph(Figure1-5).Broadlyspeaking,inthis
graph,nodesrepresentoperations(suchasadditionormultiplication),andedgesrepresentdata(tensors)
flowingaroundthesystem.Inthenextchapters,wewilldivedeeperintotheseconceptsandlearnto
understandthemwithmanyexamples.

Figure1-5.Adataflowcomputationgraph.Dataintheformoftensorsflowsthroughagraphofcomputationaloperationsthatmake
upourdeepneuralnetworks.


AHigh-LevelOverview
TensorFlow,inthemostgeneralterms,isasoftwareframeworkfornumericalcomputationsbasedon
dataflowgraphs.Itisdesignedprimarily,however,asaninterfaceforexpressingandimplementing
machinelearningalgorithms,chiefamongthemdeepneuralnetworks.
TensorFlowwasdesignedwithportabilityinmind,enablingthesecomputationgraphstobeexecuted
acrossawidevarietyofenvironmentsandhardwareplatforms.Withessentiallyidenticalcode,thesame
TensorFlowneuralnetcould,forinstance,betrainedinthecloud,distributedoveraclusterofmany
machinesoronasinglelaptop.Itcanbedeployedforservingpredictionsonadedicatedserveroron
mobiledeviceplatformssuchasAndroidoriOS,orRaspberryPisingle-boardcomputers.TensorFlowis
alsocompatible,ofcourse,withLinux,macOS,andWindowsoperatingsystems.
ThecoreofTensorFlowisinC++,andithastwoprimaryhigh-levelfrontendlanguagesandinterfaces
forexpressingandexecutingthecomputationgraphs.ThemostdevelopedfrontendisinPython,usedby
mostresearchersanddatascientists.TheC++frontendprovidesquitealow-levelAPI,usefulfor
efficientexecutioninembeddedsystemsandotherscenarios.
Asidefromitsportability,anotherkeyaspectofTensorFlowisitsflexibility,allowingresearchersand
datascientiststoexpressmodelswithrelativeease.Itissometimesrevealingtothinkofmoderndeep
learningresearchandpracticeasplayingwith“LEGO-like”bricks,replacingblocksofthenetworkwith
othersandseeingwhathappens,andattimesdesigningnewblocks.Asweshallseethroughoutthisbook,
TensorFlowprovideshelpfultoolstousethesemodularblocks,combinedwithaflexibleAPIthat
enablesthewritingofnewones.Indeeplearning,networksaretrainedwithafeedbackprocesscalled
backpropagationbasedongradientdescentoptimization.TensorFlowflexiblysupportsmanyoptimization
algorithms,allwithautomaticdifferentiation—theuserdoesnotneedtospecifyanygradientsin
advance,sinceTensorFlowderivesthemautomaticallybasedonthecomputationgraphandlossfunction
providedbytheuser.Tomonitor,debug,andvisualizethetrainingprocess,andtostreamline
experiments,TensorFlowcomeswithTensorBoard(Figure1-6),asimplevisualizationtoolthatrunsin
thebrowser,whichwewillusethroughoutthisbook.

Figure1-6.TensorFlow’svisualizationtool,TensorBoard,formonitoring,debugging,andanalyzingthetrainingprocessand
experiments.

KeyenablersofTensorFlow’sflexibilityfordatascientistsandresearchersarehigh-levelabstraction
libraries.Instate-of-the-artdeepneuralnetsforcomputervisionorNLU,writingTensorFlowcodecan
takeatoll—itcanbecomeacomplex,lengthy,andcumbersomeendeavor.Abstractionlibrariessuchas
KerasandTF-Slimoffersimplifiedhigh-levelaccesstothe“LEGObricks”inthelower-levellibrary,
helpingtostreamlinetheconstructionofthedataflowgraphs,trainingthem,andrunninginference.
AnotherkeyenablerfordatascientistsandengineersisthepretrainedmodelsthatcomewithTF-Slimand
TensorFlow.Thesemodelsweretrainedonmassiveamountsofdatawithgreatcomputationalresources,
whichareoftenhardtocomebyandinanycaserequiremuchefforttoacquireandsetup.UsingKerasor
TF-Slim,forexample,withjustafewlinesofcodeitispossibletousetheseadvancedmodelsfor
inferenceonincomingdata,andalsotofine-tunethemodelstoadapttonewdata.
TheflexibilityandportabilityofTensorFlowhelpmaketheflowfromresearchtoproductionsmooth,
cuttingthetimeandeffortittakesfordatascientiststopushtheirmodelstodeploymentinproductsand
forengineerstotranslatealgorithmicideasintorobustcode.



TENSORFLOWABSTRACTIONS
TensorFlowcomeswithabstractionlibrariessuchasKerasandTF-Slim,offeringsimplifiedhigh-levelaccesstoTensorFlow.
Theseabstractions,whichwewillseelaterinthisbook,helpstreamlinetheconstructionofthedataflowgraphsandenableusto
trainthemandruninferencewithmanyfewerlinesofcode.

Butbeyondflexibilityandportability,TensorFlowhasasuiteofpropertiesandtoolsthatmakeit
attractiveforengineerswhobuildreal-worldAIsystems.Ithasnaturalsupportfordistributedtraining—
indeed,itisusedatGoogleandotherlargeindustryplayerstotrainmassivenetworksonhugeamountsof
data,overclustersofmanymachines.Inlocalimplementations,trainingonmultiplehardwaredevices

requiresfewchangestocodeusedforsingledevices.Codealsoremainsrelativelyunchangedwhen
goingfromlocaltodistributed,whichmakesusingTensorFlowinthecloud,onAmazonWebServices
(AWS)orGoogleCloud,particularlyattractive.Additionally,aswewillseefurtheralonginthisbook,
TensorFlowcomeswithmanymorefeaturesaimedatboostingscalability.Theseincludesupportfor
asynchronouscomputationwiththreadingandqueues,efficientI/Oanddataformats,andmuchmore.
Deeplearningcontinuestorapidlyevolve,andsodoesTensorFlow,withfrequentnewandexciting
additions,bringingbetterusability,performance,andvalue.


Summary
Withthesetoftoolsandfeaturesdescribedinthischapter,itbecomesclearwhyTensorFlowhas
attractedsomuchattentioninlittlemorethanayear.Thisbookaimsatfirstrapidlygettingyouacquainted
withthebasicsandreadytowork,andthenwewilldivedeeperintotheworldofTensorFlowwith
excitingandpracticalexamples.


Chapter2.GowiththeFlow:UpandRunning
withTensorFlow
InthischapterwestartourjourneywithtwoworkingTensorFlowexamples.Thefirst(thetraditional
“helloworld”program),whileshortandsimple,includesmanyoftheimportantelementswediscussin
depthinlaterchapters.Withthesecond,afirstend-to-endmachinelearningmodel,youwillembarkon
yourjourneytowardstate-of-the-artmachinelearningwithTensorFlow.
Beforegettingstarted,webrieflywalkthroughtheinstallationofTensorFlow.Inordertofacilitatea
quickandpainlessstart,weinstalltheCPUversiononly,anddefertheGPUinstallationtolater.1(Ifyou
don’tknowwhatthismeans,that’sOKforthetimebeing!)IfyoualreadyhaveTensorFlowinstalled,skip
tothesecondsection.