Spark高级数据分析

ForewordPreface1. Analyzing Big DataThe Challenges of Data ScienceIntroducing Apache SparkAbout This Book2. Introduction to Data Analysis with Scala and SparkScala for Data ScientistsThe Spark Programming ModelRecord LinkageGetting Started: The Spark Shell and SparkContextBringing Data from the Cluster to the ClientShipping Code from the Client to the ClusterStructuring Data with Tuples and Case ClassesAggregationsCreating HistogramsSummary Statistics for Continuous VariablesCreating Reusable Code for Computing Summary StatisticsSimple Variable Selection and ScoringWhere to Go from Here3. Recommending Music and the Audioscrobbler Data SetData SetThe Alternating Least Squares Recommender AlgorithmPreparing the DataBuilding a First ModelSpot Checking RecommendationsEvaluating Recommendation QualityComputing AUCHyperparameter SelectionMaking RecommendationsWhere to Go from Here4. Predicting Forest Cover with Decision TreesFast Forward to RegressionVectors and FeaturesTraining ExamplesDecision Trees and ForestsCovtype Data SetPreparing the DataA First Decision TreeDecision Tree HyperparametersTuning Decision TreesCategorical Features RevisitedRandom Decision ForestsMaking PredictionsWhere to Go from Here5. Anomaly Detection in Network Traffic with K-means ClusteringAnomaly DetectionK-means ClusteringNetwork IntrusionKDD Cup 1999 Data SetA First Take on ClusteringChoosing kVisualization in RFeature NormalizationCategorical VariablesUsing Labels with EntropyClustering in ActionWhere to Go from Here6. Understanding Wikipedia with Latent Semantic AnalysisThe Term-Document MatrixGetting the DataParsing and Preparing the DataLemmatizationComputing the TF-IDFsSingular Value DecompositionFinding Important ConceptsQuerying and Scoring with the Low-Dimensional RepresentationTerm-Term RelevanceDocument-Document RelevanceTerm-Document RelevanceMultiple-Term QueriesWhere to Go from Here7. Analyzing Co-occurrence Networks with GraphXThe MEDLINE Citation Index: A Network AnalysisGetting the DataParsing XML Documents with Scala's XML LibraryAnalyzing the MeSH Major Topics and Their Co-occurrencesConstructing a Co-occurrence Network with GraphXUnderstanding the Structure of NetworksConnected ComponentsDegree DistributionFiltering Out Noisy EdgesProcessing EdgeTripletsAnalyzing the Filtered GraphSmall-World NetworksCliques and Clustering CoefficientsComputing Average Path Length with PregelWhere to Go from Here8. 6eospatial and Temporal Data Analysis on the New York City Taxi Trip DataGetting the DataWorking with Temporal and Geospatial Data in SparkTemporal Data with JodaTime and NScalaTimeGeospatial Data with the Esri Geometry API and SprayExploring the Esri Geometry APIIntro to GeoJSONPreparing the New York City Taxi Trip DataHandling Invalid Records at ScaleGeospatial AnalysisSessionization in SparkBuilding Sessions: Secondary Sorts in SparkWhere to Go from Here9. Estimating Financial Risk through Monte Carlo SimulationTerminologyMethods for Calculating VaRVariance-CovarianceHistorical SimulationMonte Carlo SimulationOur ModelGetting the DataPreprocessingDetermining the Factor WeightsSamplingThe Multivariate Normal DistributionRunning the TrialsVisualizing the Distribution of ReturnsEvaluating Our ResultsWhere to Go from Here10. Analyzing Genomics Data and the BDG ProjectDecoupling Storage from ModelingIngesting Genomics Data with the ADAM CLIParquet Format and Columnar StoragePredicting Transcription Factor Binding Sites from ENCODE DataQuerying Genotypes from the 1000 Genomes ProjectWhere to Go from Here11. Analyzing Neuroimaging Data with PySpark and ThunderOverview of PySparkPySpark InternalsOverview and Installation of the Thunder LibraryLoading Data with ThunderThunder Core Data TypesCategorizing Neuron Types with ThunderWhere to Go from HereA.Deeper into SparkB.Upcoming MLlib Pipelines APIIndex