Learning apache kafka, second edition by nishant garg

LearningApacheKafkaSecondEdition


TableofContents
LearningApacheKafkaSecondEdition
Credits
AbouttheAuthor
AbouttheReviewers
www.PacktPub.com
Supportfiles,eBooks,discountoffers,andmore
Whysubscribe?
FreeaccessforPacktaccountholders
Preface
Whatthisbookcovers
Whatyouneedforthisbook
Whothisbookisfor
Conventions
Readerfeedback
Customersupport
Errata
Piracy
Questions
1.IntroducingKafka
WelcometotheworldofApacheKafka
WhydoweneedKafka?
Kafkausecases
InstallingKafka
Installingprerequisites
InstallingJava1.7orhigher

DownloadingKafka
BuildingKafka
Summary
2.SettingUpaKafkaCluster


Asinglenode–asinglebrokercluster
StartingtheZooKeeperserver
StartingtheKafkabroker
CreatingaKafkatopic
Startingaproducertosendmessages
Startingaconsumertoconsumemessages
Asinglenode–multiplebrokerclusters
StartingZooKeeper
StartingtheKafkabroker
CreatingaKafkatopicusingthecommandline
Startingaproducertosendmessages
Startingaconsumertoconsumemessages
Multiplenodes–multiplebrokerclusters
TheKafkabrokerpropertylist
Summary
3.KafkaDesign
Kafkadesignfundamentals
Logcompaction
MessagecompressioninKafka
ReplicationinKafka
Summary
4.WritingProducers
TheJavaproducerAPI
SimpleJavaproducers

Importingclasses
Definingproperties
Buildingthemessageandsendingit
CreatingaJavaproducerwithcustompartitioning
Importingclasses
Definingproperties
ImplementingthePartitionerclass


Buildingthemessageandsendingit
TheKafkaproducerpropertylist
Summary
5.WritingConsumers
KafkaconsumerAPIs
Thehigh-levelconsumerAPI
Thelow-levelconsumerAPI
SimpleJavaconsumers
Importingclasses
Definingproperties
Readingmessagesfromatopicandprintingthem
MultithreadedJavaconsumers
Importingclasses
Definingproperties
Readingthemessagefromthreadsandprintingit
TheKafkaconsumerpropertylist
Summary
6.KafkaIntegrations
KafkaintegrationwithStorm
IntroducingStorm
IntegratingStorm

KafkaintegrationwithHadoop
IntroducingHadoop
IntegratingHadoop
Hadoopproducers
Hadoopconsumers
Summary
7.OperationalizingKafka
Kafkaadministrationtools
Kafkaclustertools
Addingservers


Kafkatopictools
Kafkaclustermirroring
Integrationwithothertools
Summary
Index



LearningApacheKafkaSecondEdition



LearningApacheKafkaSecondEdition
Copyright©2015PacktPublishing
Allrightsreserved.Nopartofthisbookmaybereproduced,storedinaretrievalsystem,
ortransmittedinanyformorbyanymeans,withoutthepriorwrittenpermissionofthe
publisher,exceptinthecaseofbriefquotationsembeddedincriticalarticlesorreviews.
Everyefforthasbeenmadeinthepreparationofthisbooktoensuretheaccuracyofthe

informationpresented.However,theinformationcontainedinthisbookissoldwithout
warranty,eitherexpressorimplied.Neithertheauthor,norPacktPublishing,andits
dealersanddistributorswillbeheldliableforanydamagescausedorallegedtobecaused
directlyorindirectlybythisbook.
PacktPublishinghasendeavoredtoprovidetrademarkinformationaboutallofthe
companiesandproductsmentionedinthisbookbytheappropriateuseofcapitals.
However,PacktPublishingcannotguaranteetheaccuracyofthisinformation.
Firstpublished:October2013
Secondedition:February2015
Productionreference:1210215
PublishedbyPacktPublishingLtd.
LiveryPlace
35LiveryStreet
BirminghamB32PB,UK.
ISBN978-1-78439-309-0
www.packtpub.com



Credits
Author
NishantGarg
Reviewers
SandeepKhurana
SaurabhMinni
SupreetSethi
CommissioningEditor
UshaIyer
AcquisitionEditor
MeetaRajani

ContentDevelopmentEditor
ShubhangiDhamgaye
TechnicalEditors
ManalPednekar
ChinmayS.Puranik
CopyEditors
MerilynPereira
AartiSaldanha
ProjectCoordinator
HarshalVed
Proofreaders
StephenCopestake
PaulHindle
Indexer
RekhaNair
Graphics
SheetalAute
ProductionCoordinator
NileshR.Mohite
CoverWork


NileshR.Mohite



AbouttheAuthor
NishantGarghasover14yearsofsoftwarearchitectureanddevelopmentexperiencein
varioustechnologies,suchasJavaEnterpriseEdition,SOA,Spring,Hadoop,Hive,Flume,
Sqoop,Oozie,Spark,Shark,YARN,Impala,Kafka,Storm,Solr/Lucene,NoSQL

databases(suchasHBase,Cassandra,andMongoDB),andMPPdatabases(suchas
GreenPlum).
HereceivedhisMSinsoftwaresystemsfromtheBirlaInstituteofTechnologyand
Science,Pilani,India,andiscurrentlyworkingasatechnicalarchitectfortheBigData
R&DGroupwithImpetusInfotechPvt.Ltd.Previously,Nishanthasenjoyedworking
withsomeofthemostrecognizablenamesinITservicesandfinancialindustries,
employingfullsoftwarelifecyclemethodologiessuchasAgileandSCRUM.
Nishanthasalsoundertakenmanyspeakingengagementsonbigdatatechnologiesandis
alsotheauthorofHBaseEssestials,PacktPublishing.
Iwouldliketothankmyparents(Mr.VishnuMurtiGargandMrs.VimlaGarg)fortheir
continuousencouragementandmotivationthroughoutmylife.Iwouldalsoliketothank
mywife(Himani)andmykids(NitigyaandDarsh)fortheirnever-endingsupport,which
keepsmegoing.
Finally,IwouldliketothankVineetTyagi,CTOandHeadofInnovationLabs,Impetus,
andDr.Vijay,DirectorofTechnology,InnovationLabs,Impetus,forencouragingmeto
write.



AbouttheReviewers
SandeepKhurana,an18yearsveteran,comeswithanextensiveexperienceinthe
SoftwareandITindustry.Beinganearlyentrantinthedomain,hehasworkedinall
aspectsofJava-/JEE-basedtechnologiesandframeworkssuchasSpring,Hibernate,JPA,
EJB,security,Struts,andsoon.Forthelastfewprofessionalengagementsinhiscareer
andalsopartlyduetohispersonalinterestinconsumer-facinganalytics,hehasbeen
treadinginthebigdatarealmandhasextensiveexperienceonbigdatatechnologiessuch
asHadoop,Pig,Hive,ZooKeeper,Flume,Oozie,HBaseandsoon.
Hehasdesigned,developed,anddeliveredmultipleenterprise-level,highlyscalable,
distributedsystemsduringthecourseofhiscareer.Inhislongandfruitfulprofessional
life,hehasbeenwithsomeofthebiggestnamesoftheindustrysuchasIBM,Oracle,

Yahoo!,andNokia.
SaurabhMinniiscurrentlyworkingasatechnicalarchitectatAdNear.Hecompletedhis
BEincomputerscienceattheGlobalAcademyofTechnology,Bangalore.Heis
passionateaboutprogrammingandlovesgettinghishandswetwithdifferenttechnologies.
AtAdNear,hedeployedKafka.Thisenabledsmoothconsumptionofdatatobeprocessed
byStormandHadoopclusters.PriortoAdNear,heworkedwithAdobeandIntuit,where
hedabbledwithC++,Delphi,Android,andJavawhileworkingondesktopandmobile
products.
SupreetSethiisaseasonedtechnologyleaderwithaneyefordetail.Hehasproven
expertiseinchartingoutgrowthstrategiesfortechnologyplatforms.Hecurrentlysteers
theplatformteamtocreatetoolsthatdrivetheinfrastructureatJabong.Heoftenreviews
thecodebasefromaperformancepointofview.Theseaspectsalsoputhimatthehelmof
backendsystems,APIsthatdrivemobileapps,mobilewebapps,anddesktopsites.
TheJabongtechteamhasbeenextremelyhelpfulduringthereviewprocess.They
providedacreativeenvironmentwhereSupreetwasabletoexploresomeofcutting-edge
technologieslikeApacheKafka.
Iwouldliketothankmydaughter,Seher,andmywife,Smriti,forbeingpatientobservers
whileIspentafewhourseverydayreviewingthisbook.



www.PacktPub.com


Supportfiles,eBooks,discountoffers,and
more
Forsupportfilesanddownloadsrelatedtoyourbook,pleasevisitwww.PacktPub.com.
DidyouknowthatPacktofferseBookversionsofeverybookpublished,withPDFand
ePubfilesavailable?YoucanupgradetotheeBookversionatwww.PacktPub.comandas
aprintbookcustomer,youareentitledtoadiscountontheeBookcopy.Getintouchwith

usat<>formoredetails.
Atwww.PacktPub.com,youcanalsoreadacollectionoffreetechnicalarticles,signup
forarangeoffreenewslettersandreceiveexclusivediscountsandoffersonPacktbooks
andeBooks.

/>DoyouneedinstantsolutionstoyourITquestions?PacktLibisPackt’sonlinedigital
booklibrary.Here,youcansearch,access,andreadPackt’sentirelibraryofbooks.


Whysubscribe?
FullysearchableacrosseverybookpublishedbyPackt
Copyandpaste,print,andbookmarkcontent
Ondemandandaccessibleviaawebbrowser


FreeaccessforPacktaccountholders
IfyouhaveanaccountwithPacktatwww.PacktPub.com,youcanusethistoaccess
PacktLibtodayandview9entirelyfreebooks.Simplyuseyourlogincredentialsfor
immediateaccess.



Preface
ThisbookisheretohelpyougetfamiliarwithApacheKafkaandtosolveyourchallenges
relatedtotheconsumptionofmillionsofmessagesinpublisher-subscriberarchitectures.It
isaimedatgettingyoustartedprogrammingwithKafkasothatyouwillhaveasolid
foundationtodivedeepintodifferenttypesofimplementationsandintegrationsforKafka
producersandconsumers.
InadditiontoanexplanationofApacheKafka,wealsospendachapterexploringKafka
integrationwithothertechnologiessuchasApacheHadoopandApacheStorm.Ourgoal

istogiveyouanunderstandingnotjustofwhatApacheKafkais,butalsohowtouseitas
apartofyourbroadertechnicalinfrastructure.Intheend,wewillwalkyouthrough
operationalizingKafkawherewewillalsotalkaboutadministration.