Packt Machine Learning with Spark

MachineLearningwithSpark Copyrighto2015PacktPublishi ing Allrightsreserved.Nopartofthisbookmaybereproduced,storedinaretrieval system,ortransmittedinanyformorbyanymeans,withoutthepriorwritten permissionofthepublisher,exceptinthecaseofbriefquotationsembeddedin criticalarticlesorreviews rthasbeenmadeinthepreparationofthisbooktoensuretheaccuracy oftheinformationpresentedHowevertheinformationcontainedinthisbookis soldwithoutwarranty,citherexpressorimplied.NeithertheauthornorPacl Publishinganditsdealersanddistributorswillbeheldliableforanydamages causedorallegedtobecauseddirectlyorindirectlybythisbook PacktPublishinghasendeavoredtoprovidetrademarkinformationaboutallofthe companiesandproductsmentionedinthisbookbytheappropriateuseofcapitals However,PacktPublishingcannotguaranteetheaccuracyofthisinfori Firstpublished:February2015 Productionreference:1170215 PublishedbyPacktPublishingltd Liveryplace 35Liverystreet Birminghamb32PBUK ISBN978-178328-8519 www.packtpub.com CoverimagebyAkshayPaunikar(akshaypaunikar4@gmail.com) Credits Author ProjectCoordinator NickPentreath Miltondsouza Reviewers Proofreaders Andreamostosi SimranBhogal Haoren Mariagould Krishnasankar AmeeshaGreen Paulhindle CommissioningEditor Rebeccayoue Indexer PriyaSane AcquisitionEditor Rebeccayoue Graphics SheetalAute ContentDevelopmentEditor Abhinashsahu Susmitasabat Productioncoordinator Technicaleditors NiteshThakur Vivekarora PankajKadam CoverWork NiteshThakur CopyEdit KarunaNarayanan Abouttheauthor lickPentreathhasabackgroundinfinancialmarkets,machinelearning,and softwaredevelopment.HehasworkedatGoldmanSachsGroup,Inc;asaresearch scientistattheonlineadtargetingstart-upCognitiveMatchLimited,London;and ledthedataScienceandanalyticsteamatmxit,Africaslargestsocialnetwork Heisacofounderofgraphflow,abigdataandmachinelearningcompanyfocused onuser-centricrecommendationsandcustomerintelligence.Heispassionateabout combiningcommercialfocuswithmachinelearningandcutting-edgetechnologyto buildintelligentsystemsthatlearnfromdatatoaddvaluetothebottomline NickisamemberoftheApacheSparkProjectManagementCommittee Acknowledgments Writingthisbookhasbeenquitearollercoasterrideoverthepastyear,withmany upsanddowns,latenights,andworkingweekends.Ithasalsobeenextremely rewardingtocombinemypassionformachinelearningwithmyloveoftheApache sparkproject,andIhopetobringsomeofthisoutinthisbook Iwouldliketothankthepacktpublishingteamforalltheirassistancethroughout thewritingandeditingprocess:Rebecca,Susmita,Sudhir,Amey,Neil,Vivek, Pankaj,andeveryonewhoworkedonthebook ThanksalsogotoDeboradonatoatstumbleUponforassistancewithdata-and legal-relatedqueries Writingabooklikethiscanbeasomewhatlonelyprocess,soitisincrediblyhelpful togetthefeedbackofreviewerstounderstandwhetheroneisheadedintheright direction(andwhatcourseadjustmentsneedtobemade).I'mdeeplygrateful AndreaMostosi,HaoRen,andKrishnaSankarfortakingthetimetoprovidesuch detailedandcriticalfeedback Icouldnothavegottenthroughthisprojectwithouttheunwaveringsupportofall myfamilyandfriendsespeciallymywonderfulwifeTammywhowillbegladto havemebackintheeveningsandonweekendsonceagain.Thankyouall! Finally,thankstoallofyoureadingthis;Ihopeyoufindituseful Aboutthereviewers AndreaMostosiisatechnologyenthusiast.Aninnovationloversincehewasa child,hestartedaprofessionaljobin2003andworkedonseveralprojects,playing almosteveryroleinthecomputerscienceenvironment.HeiscurrentlytheCtoat TheFool,acompanythattriestomakesenseofwebandsocialdata.Duringhisfree time,helikestraveling,running,cooking,biking,andcoding iwouldliketothankmygeekfriends:Simonem,Danielev,lucat, LuigiP,Michelen,lucaO,Lucab,DiegoCandFabiob.theyare thesmartestpeopleiknow,andcomparingmyselfwiththemhas alwayspushedmetobebetter Haorenisasoftwaredeveloperwhoispassionateaboutscala,distributed systems,machinelearning,andApacheSpark.HewasanexchangestudentatEPFL whenhelearnedaboutScalain2012.HeiscurrentlyworkinginParisasabackend anddataengineerforclaravista-acompanythatfocusesonhigh-performance marketing.HisworkresponsibilityistobuildaSpark-basedplatformforpurchase predictionandanewrecommendersystem Besidesprogrammingheenjoysrunning,swimming,andplayingbasketballand badmintonYoucanlearnmoreathisbloghttp://www.invkrh.me Krishnasankarisachiefdatascientistatblackarrowwhereheisfocusin ng onenhancinguserexperienceviainference,intelligence,andinterfaces.earlier stintsincludeworkingasaprincipalarchitectanddatascientistatTataAmerica InternationalCorporation,directorofdatascienceatabioinformaticsstart-up company,andasadistinguishedengineeratCiscoSystems,Inc.Hehasspokenat variousconferencesaboutdatascience(http://goo.gl/9pyjmh),machinelearning (http://goo.gl/ssem2r),andsocialmediaanalysis(http://goo.g1/d9ypvq).Ile hasalsobeenaguestlecturerattheNavalPostgraduateSchool.Hehaswrittenafew booksonJava,wirelessLansecurity,Web2.0,andnowonSpark.Hisotherpassion iSLEGOrobotics.EarlierinApril,hewasattheStLouisFlLWorldCompetitionas arobotsdesignjudge Www.Packtpub.com Supportfiles,eBooks,discountoffers,andmore Forsupportfilesanddownloadsrelatedtoyourbook,pleasevisit wwwpacktpub.com DidyouknowthatpacktoffersebookversionsofeverybookpublishedwithPDF andepuBfilesavailableYoucanupgradetotheebookversionatwww.packtpub comandasaprintbookcustomer,youareentitledtoadiscountontheebookcopy Getintouchwithusatservice@packtpubcomformoredetails Atwww.Packtpub.comyoucanalsoreadacollectionoffreetechnicalarticles signupforarangeoffreenewslettersandreceiveexclusivediscountsandoffers onpacktbooksandebooks PACKTLIB https://www2.packtpub.ccm/books/subscription/packtlib DoyouneedinstantsolutionstoyourITquestions?PacktLibisPackt'sonlinedigital booklibrary.IIere,youcansearch,access,andreadPackt'sentirelibraryofbooks Whysubscribe? FullysearchableacrosseverybookpublishedbyPackt Copyandpaste,print,andbookmarkcontent Ondemandandaccessibleviaawebbrowser Freeaccessforpacktaccountholders IfyouhaveanaccountwithPacktatwww.packtpub.comyoucanusethistoaccess PacktLibtodayandviewgentirelyfreebooksSimplyuseyourlogincredentialsfor immediateaccess Tableofcontents Preface Chapter1:GettingUpandRunningwithSpark 7 InstallingandsettingupSparklocally 8 Sparkclusters 10 TheSparkprogrammingmodel SparkContextandSparkConf TheSparkshell 12 Resilientdistributeddatasets 14 CreatingRDds 15 Sparkoperations 15 CachingRDDs 18 Broadcastvariablesandaccumulators 19 Thefirststeptoasparkprograminscala 21 ThefirststeptoasparkprograminJava 24 ThefirststeptoaSparkprograminPython 28 GettingSparkrunningonAmazonEC2 30 Launchinganec2Sparkcluster 31 Summary 35 Chapter2:DesigningaMachineLearningSystem 37 IntroducingMovieStream 38 Businessusecasesforamachinelearningsystem 39 Personalization 40 Targetedmarketingandcustomersegmentation 40 Predictivemodelingandanalytics 41 Typesofmachinelearningmodels 41 Thecomponentsofadata-drivenmachinelearningsystem 42 Dataingestionandstorage 42 Datacleansingandtransformation 43