{"product_id":"designing-big-data-platforms-9781119690924","title":"Designing Big Data Platforms","description":"\u003cb\u003eBook Synopsis\u003c\/b\u003e\u003cbr\u003e\u003cb\u003eDESIGNING BIG DATA PLATFORMS\u003c\/b\u003e \u003cp\u003e\u003cb\u003eProvides expert guidance and valuable insights on getting the most out of Big Data systems\u003c\/b\u003e\u003c\/p\u003e\u003cp\u003eAn array of tools are currently available for managing and processing datasome are ready-to-go solutions that can be immediately deployed, while others require complex and time-intensive setups. With such a vast range of options, choosing the right tool to build a solution can be complicated, as can determining which tools work well with each other. \u003ci\u003eDesigning Big Data Platforms\u003c\/i\u003e provides clear and authoritative guidance on the critical decisions necessary for successfully deploying, operating, and maintaining Big Data systems.\u003c\/p\u003e\u003cp\u003eThis highly practical guide helps readers understand how to process large amounts of data with well-known Linux tools and database solutions, use effective techniques to collect and manage data from multiple sources, transform data into meaningful business insights, and much more. Author Yusuf Aytas, a software enginee\u003cbr\u003e\u003cbr\u003e\u003cb\u003eTable of Contents\u003c\/b\u003e\u003cbr\u003e\u003c\/p\u003e\u003cp\u003eList of Contributors \u003ci\u003ex\u003c\/i\u003evii\u003c\/p\u003e \u003cp\u003ePreface xix\u003c\/p\u003e \u003cp\u003eAcknowledgments xxi\u003c\/p\u003e \u003cp\u003eAcronyms xxiii\u003c\/p\u003e \u003cp\u003eIntroduction xxv\u003c\/p\u003e \u003cp\u003e\u003cb\u003e1 An Introduction: What’s a Modern Big Data Platform \u003c\/b\u003e\u003cb\u003e1\u003c\/b\u003e\u003c\/p\u003e \u003cp\u003e1.1 Defining Modern Big Data Platform 1\u003c\/p\u003e \u003cp\u003e1.2 Fundamentals of a Modern Big Data Platform 2\u003c\/p\u003e \u003cp\u003e1.2.1 Expectations from Data 2\u003c\/p\u003e \u003cp\u003e1.2.1.1 Ease of Access 2\u003c\/p\u003e \u003cp\u003e1.2.1.2 Security 2\u003c\/p\u003e \u003cp\u003e1.2.1.3 Quality 3\u003c\/p\u003e \u003cp\u003e1.2.1.4 Extensibility 3\u003c\/p\u003e \u003cp\u003e1.2.2 Expectations from Platform 3\u003c\/p\u003e \u003cp\u003e1.2.2.1 Storage Layer 4\u003c\/p\u003e \u003cp\u003e1.2.2.2 Resource Management 4\u003c\/p\u003e \u003cp\u003e1.2.2.3 ETL 5\u003c\/p\u003e \u003cp\u003e1.2.2.4 Discovery 6\u003c\/p\u003e \u003cp\u003e1.2.2.5 Reporting 7\u003c\/p\u003e \u003cp\u003e1.2.2.6 Monitoring 7\u003c\/p\u003e \u003cp\u003e1.2.2.7 Testing 8\u003c\/p\u003e \u003cp\u003e1.2.2.8 Lifecycle Management 9\u003c\/p\u003e \u003cp\u003e\u003cb\u003e2 A Bird’s Eye View on Big Data \u003c\/b\u003e\u003cb\u003e11\u003c\/b\u003e\u003c\/p\u003e \u003cp\u003e2.1 A Bit of History 11\u003c\/p\u003e \u003cp\u003e2.1.1 Early Uses of Big Data Term 11\u003c\/p\u003e \u003cp\u003e2.1.2 A New Era 12\u003c\/p\u003e \u003cp\u003e2.1.2.1 Word Count Problem 12\u003c\/p\u003e \u003cp\u003e2.1.2.2 Execution Steps 13\u003c\/p\u003e \u003cp\u003e2.1.3 An Open-Source Alternative 15\u003c\/p\u003e \u003cp\u003e2.1.3.1 Hadoop Distributed File System 15\u003c\/p\u003e \u003cp\u003e2.1.3.2 HadoopMapReduce 17\u003c\/p\u003e \u003cp\u003e2.2 What Makes Big Data 20\u003c\/p\u003e \u003cp\u003e2.2.1 Volume 20\u003c\/p\u003e \u003cp\u003e2.2.2 Velocity 21\u003c\/p\u003e \u003cp\u003e2.2.3 Variety 21\u003c\/p\u003e \u003cp\u003e2.2.4 Complexity 21\u003c\/p\u003e \u003cp\u003e2.3 Components of Big Data Architecture 22\u003c\/p\u003e \u003cp\u003e2.3.1 Ingestion 22\u003c\/p\u003e \u003cp\u003e2.3.2 Storage 23\u003c\/p\u003e \u003cp\u003e2.3.3 Computation 23\u003c\/p\u003e \u003cp\u003e2.3.4 Presentation 24\u003c\/p\u003e \u003cp\u003e2.4 Making Use of Big Data 24\u003c\/p\u003e \u003cp\u003e2.4.1 Querying 24\u003c\/p\u003e \u003cp\u003e2.4.2 Reporting 25\u003c\/p\u003e \u003cp\u003e2.4.3 Alerting 25\u003c\/p\u003e \u003cp\u003e2.4.4 Searching 25\u003c\/p\u003e \u003cp\u003e2.4.5 Exploring 25\u003c\/p\u003e \u003cp\u003e2.4.6 Mining 25\u003c\/p\u003e \u003cp\u003e2.4.7 Modeling 26\u003c\/p\u003e \u003cp\u003e\u003cb\u003e3 A Minimal Data Processing and Management System \u003c\/b\u003e\u003cb\u003e27\u003c\/b\u003e\u003c\/p\u003e \u003cp\u003e3.1 Problem Definition 27\u003c\/p\u003e \u003cp\u003e3.1.1 Online Book Store 27\u003c\/p\u003e \u003cp\u003e3.1.2 User Flow Optimization 28\u003c\/p\u003e \u003cp\u003e3.2 Processing Large Data with Linux Commands 28\u003c\/p\u003e \u003cp\u003e3.2.1 Understand the Data 28\u003c\/p\u003e \u003cp\u003e3.2.2 Sample the Data 28\u003c\/p\u003e \u003cp\u003e3.2.3 Building the Shell Command 29\u003c\/p\u003e \u003cp\u003e3.2.4 Executing the Shell Command 30\u003c\/p\u003e \u003cp\u003e3.2.5 Analyzing the Results 31\u003c\/p\u003e \u003cp\u003e3.2.6 Reporting the Findings 32\u003c\/p\u003e \u003cp\u003e3.2.7 Automating the Process 33\u003c\/p\u003e \u003cp\u003e3.2.8 A Brief Review 33\u003c\/p\u003e \u003cp\u003e3.3 Processing Large Data with PostgreSQL 34\u003c\/p\u003e \u003cp\u003e3.3.1 Data Modeling 34\u003c\/p\u003e \u003cp\u003e3.3.2 Copying Data 35\u003c\/p\u003e \u003cp\u003e3.3.3 Sharding in PostgreSQL 37\u003c\/p\u003e \u003cp\u003e3.3.3.1 Setting up Foreign Data Wrapper 37\u003c\/p\u003e \u003cp\u003e3.3.3.2 Sharding Data over Multiple Nodes 38\u003c\/p\u003e \u003cp\u003e3.4 Cost of Big Data 39\u003c\/p\u003e \u003cp\u003e\u003cb\u003e4 Big Data Storage \u003c\/b\u003e\u003cb\u003e41\u003c\/b\u003e\u003c\/p\u003e \u003cp\u003e4.1 Big Data Storage Patterns 41\u003c\/p\u003e \u003cp\u003e4.1.1 Data Lakes 41\u003c\/p\u003e \u003cp\u003e4.1.2 Data Warehouses 42\u003c\/p\u003e \u003cp\u003e4.1.3 Data Marts 43\u003c\/p\u003e \u003cp\u003e4.1.4 Comparison of Storage Patterns 43\u003c\/p\u003e \u003cp\u003e4.2 On-Premise Storage Solutions 44\u003c\/p\u003e \u003cp\u003e4.2.1 Choosing Hardware 44\u003c\/p\u003e \u003cp\u003e4.2.1.1 DataNodes 44\u003c\/p\u003e \u003cp\u003e4.2.1.2 NameNodes 45\u003c\/p\u003e \u003cp\u003e4.2.1.3 Resource Managers 45\u003c\/p\u003e \u003cp\u003e4.2.1.4 Network Equipment 45\u003c\/p\u003e \u003cp\u003e4.2.2 Capacity Planning 46\u003c\/p\u003e \u003cp\u003e4.2.2.1 Overall Cluster 46\u003c\/p\u003e \u003cp\u003e4.2.2.2 Resource Sharing 47\u003c\/p\u003e \u003cp\u003e4.2.2.3 Doing the Math 47\u003c\/p\u003e \u003cp\u003e4.2.3 Deploying Hadoop Cluster 48\u003c\/p\u003e \u003cp\u003e4.2.3.1 Networking 48\u003c\/p\u003e \u003cp\u003e4.2.3.2 Operating System 48\u003c\/p\u003e \u003cp\u003e4.2.3.3 Management Tools 49\u003c\/p\u003e \u003cp\u003e4.2.3.4 Hadoop Ecosystem 49\u003c\/p\u003e \u003cp\u003e4.2.3.5 A Humble Deployment 49\u003c\/p\u003e \u003cp\u003e4.3 Cloud Storage Solutions 53\u003c\/p\u003e \u003cp\u003e4.3.1 Object Storage 54\u003c\/p\u003e \u003cp\u003e4.3.2 Data Warehouses 55\u003c\/p\u003e \u003cp\u003e4.3.2.1 Columnar Storage 55\u003c\/p\u003e \u003cp\u003e4.3.2.2 Provisioned Data Warehouses 56\u003c\/p\u003e \u003cp\u003e4.3.2.3 Serverless Data Warehouses 56\u003c\/p\u003e \u003cp\u003e4.3.2.4 Virtual Data Warehouses 57\u003c\/p\u003e \u003cp\u003e4.3.3 Archiving 58\u003c\/p\u003e \u003cp\u003e4.4 Hybrid Storage Solutions 59\u003c\/p\u003e \u003cp\u003e4.4.1 Making Use of Object Store 59\u003c\/p\u003e \u003cp\u003e4.4.1.1 Additional Capacity 59\u003c\/p\u003e \u003cp\u003e4.4.1.2 Batch Processing 59\u003c\/p\u003e \u003cp\u003e4.4.1.3 Hot Backup 60\u003c\/p\u003e \u003cp\u003e4.4.2 Making Use of Data Warehouse 60\u003c\/p\u003e \u003cp\u003e4.4.2.1 Primary Data Warehouse 60\u003c\/p\u003e \u003cp\u003e4.4.2.2 Shared Data Mart 61\u003c\/p\u003e \u003cp\u003e4.4.3 Making Use of Archiving 61\u003c\/p\u003e \u003cp\u003e\u003cb\u003e5 Offline Big Data Processing \u003c\/b\u003e\u003cb\u003e63\u003c\/b\u003e\u003c\/p\u003e \u003cp\u003e5.1 Defining Offline Data Processing 63\u003c\/p\u003e \u003cp\u003e5.2 MapReduce Technologies 65\u003c\/p\u003e \u003cp\u003e5.2.1 Apache Pig 65\u003c\/p\u003e \u003cp\u003e5.2.1.1 Pig Latin Overview 66\u003c\/p\u003e \u003cp\u003e5.2.1.2 Compilation To MapReduce 66\u003c\/p\u003e \u003cp\u003e5.2.2 Apache Hive 67\u003c\/p\u003e \u003cp\u003e5.2.2.1 Hive Database 68\u003c\/p\u003e \u003cp\u003e5.2.2.2 Hive Architecture 69\u003c\/p\u003e \u003cp\u003e5.3 Apache Spark 70\u003c\/p\u003e \u003cp\u003e5.3.1 What’s Spark 71\u003c\/p\u003e \u003cp\u003e5.3.2 Spark Constructs and Components 71\u003c\/p\u003e \u003cp\u003e5.3.2.1 Resilient Distributed Datasets 71\u003c\/p\u003e \u003cp\u003e5.3.2.2 Distributed Shared Variables 73\u003c\/p\u003e \u003cp\u003e5.3.2.3 Datasets and DataFrames 74\u003c\/p\u003e \u003cp\u003e5.3.2.4 Spark Libraries and Connectors 75\u003c\/p\u003e \u003cp\u003e5.3.3 Execution Plan 76\u003c\/p\u003e \u003cp\u003e5.3.3.1 The Logical Plan 77\u003c\/p\u003e \u003cp\u003e5.3.3.2 The Physical Plan 77\u003c\/p\u003e \u003cp\u003e5.3.4 Spark Architecture 77\u003c\/p\u003e \u003cp\u003e5.3.4.1 Inside of Spark Application 78\u003c\/p\u003e \u003cp\u003e5.3.4.2 Outside of Spark Application 79\u003c\/p\u003e \u003cp\u003e5.4 Apache Flink 81\u003c\/p\u003e \u003cp\u003e5.5 Presto 83\u003c\/p\u003e \u003cp\u003e5.5.1 Presto Architecture 83\u003c\/p\u003e \u003cp\u003e5.5.2 Presto System Design 84\u003c\/p\u003e \u003cp\u003e5.5.2.1 Execution Plan 84\u003c\/p\u003e \u003cp\u003e5.5.2.2 Scheduling 86\u003c\/p\u003e \u003cp\u003e5.5.2.3 Resource Management 86\u003c\/p\u003e \u003cp\u003e5.5.2.4 Fault Tolerance 87\u003c\/p\u003e \u003cp\u003e\u003cb\u003e6 Stream Big Data Processing \u003c\/b\u003e\u003cb\u003e89\u003c\/b\u003e\u003c\/p\u003e \u003cp\u003e6.1 The Need for Stream Processing 89\u003c\/p\u003e \u003cp\u003e6.2 Defining Stream Data Processing 90\u003c\/p\u003e \u003cp\u003e6.3 Streams via Message Brokers 92\u003c\/p\u003e \u003cp\u003e6.3.1 Apache Kafka 92\u003c\/p\u003e \u003cp\u003e6.3.1.1 Apache Samza 93\u003c\/p\u003e \u003cp\u003e6.3.1.2 Kafka Streams 98\u003c\/p\u003e \u003cp\u003e6.3.2 Apache Pulsar 100\u003c\/p\u003e \u003cp\u003e6.3.2.1 Pulsar Functions 102\u003c\/p\u003e \u003cp\u003e6.3.3 AMQP Based Brokers 105\u003c\/p\u003e \u003cp\u003e6.4 Streams via Stream Engines 106\u003c\/p\u003e \u003cp\u003e6.4.1 Apache Flink 106\u003c\/p\u003e \u003cp\u003e6.4.1.1 Flink Architecture 107\u003c\/p\u003e \u003cp\u003e6.4.1.2 System Design 109\u003c\/p\u003e \u003cp\u003e6.4.2 Apache Storm 111\u003c\/p\u003e \u003cp\u003e6.4.2.1 Storm Architecture 114\u003c\/p\u003e \u003cp\u003e6.4.2.2 System Design 115\u003c\/p\u003e \u003cp\u003e6.4.3 Apache Heron 116\u003c\/p\u003e \u003cp\u003e6.4.3.1 Storm Limitations 116\u003c\/p\u003e \u003cp\u003e6.4.3.2 Heron Architecture 117\u003c\/p\u003e \u003cp\u003e6.4.4 Spark Streaming 118\u003c\/p\u003e \u003cp\u003e6.4.4.1 Discretized Streams 119\u003c\/p\u003e \u003cp\u003e6.4.4.2 Fault-tolerance 120\u003c\/p\u003e \u003cp\u003e\u003cb\u003e7 Data Analytics \u003c\/b\u003e\u003cb\u003e121\u003c\/b\u003e\u003c\/p\u003e \u003cp\u003e7.1 Log Collection 121\u003c\/p\u003e \u003cp\u003e7.1.1 Apache Flume 122\u003c\/p\u003e \u003cp\u003e7.1.2 Fluentd 122\u003c\/p\u003e \u003cp\u003e7.1.2.1 Data Pipeline 123\u003c\/p\u003e \u003cp\u003e7.1.2.2 Fluent Bit 124\u003c\/p\u003e \u003cp\u003e7.1.2.3 Fluentd Deployment 124\u003c\/p\u003e \u003cp\u003e7.2 Transferring Big Data Sets 125\u003c\/p\u003e \u003cp\u003e7.2.1 Reloading 126\u003c\/p\u003e \u003cp\u003e7.2.2 Partition Loading 126\u003c\/p\u003e \u003cp\u003e7.2.3 Streaming 127\u003c\/p\u003e \u003cp\u003e7.2.4 Timestamping 127\u003c\/p\u003e \u003cp\u003e7.2.5 Tools 128\u003c\/p\u003e \u003cp\u003e7.2.5.1 Sqoop 128\u003c\/p\u003e \u003cp\u003e7.2.5.2 Embulk 128\u003c\/p\u003e \u003cp\u003e7.2.5.3 Spark 129\u003c\/p\u003e \u003cp\u003e7.2.5.4 Apache Gobblin 130\u003c\/p\u003e \u003cp\u003e7.3 Aggregating Big Data Sets 132\u003c\/p\u003e \u003cp\u003e7.3.1 Data Cleansing 132\u003c\/p\u003e \u003cp\u003e7.3.2 Data Transformation 134\u003c\/p\u003e \u003cp\u003e7.3.2.1 Transformation Functions 134\u003c\/p\u003e \u003cp\u003e7.3.2.2 Transformation Stages 135\u003c\/p\u003e \u003cp\u003e7.3.3 Data Retention 135\u003c\/p\u003e \u003cp\u003e7.3.4 Data Reconciliation 136\u003c\/p\u003e \u003cp\u003e7.4 Data Pipeline Scheduler 136\u003c\/p\u003e \u003cp\u003e7.4.1 Jenkins 137\u003c\/p\u003e \u003cp\u003e7.4.2 Azkaban 138\u003c\/p\u003e \u003cp\u003e7.4.2.1 Projects 139\u003c\/p\u003e \u003cp\u003e7.4.2.2 Execution Modes 139\u003c\/p\u003e \u003cp\u003e7.4.3 Airflow 139\u003c\/p\u003e \u003cp\u003e7.4.3.1 Task Execution 140\u003c\/p\u003e \u003cp\u003e7.4.3.2 Scheduling 141\u003c\/p\u003e \u003cp\u003e7.4.3.3 Executor 141\u003c\/p\u003e \u003cp\u003e7.4.3.4 Security and Monitoring 142\u003c\/p\u003e \u003cp\u003e7.4.4 Cloud 143\u003c\/p\u003e \u003cp\u003e7.5 Patterns and Practices 143\u003c\/p\u003e \u003cp\u003e7.5.1 Patterns 143\u003c\/p\u003e \u003cp\u003e7.5.1.1 Data Centralization 143\u003c\/p\u003e \u003cp\u003e7.5.1.2 Singe Source of Truth 144\u003c\/p\u003e \u003cp\u003e7.5.1.3 Domain Driven Data Sets 145\u003c\/p\u003e \u003cp\u003e7.5.2 Anti-Patterns 146\u003c\/p\u003e \u003cp\u003e7.5.2.1 Data Monolith 146\u003c\/p\u003e \u003cp\u003e7.5.2.2 Data Swamp 147\u003c\/p\u003e \u003cp\u003e7.5.2.3 Technology Pollution 147\u003c\/p\u003e \u003cp\u003e7.5.3 Best Practices 148\u003c\/p\u003e \u003cp\u003e7.5.3.1 Business-Driven Approach 148\u003c\/p\u003e \u003cp\u003e7.5.3.2 Cost of Maintenance 148\u003c\/p\u003e \u003cp\u003e7.5.3.3 Avoiding Modeling Mistakes 149\u003c\/p\u003e \u003cp\u003e7.5.3.4 Choosing Right Tool for The Job 150\u003c\/p\u003e \u003cp\u003e7.5.4 Detecting Anomalies 150\u003c\/p\u003e \u003cp\u003e7.5.4.1 Manual Anomaly Detection 151\u003c\/p\u003e \u003cp\u003e7.5.4.2 Automated Anomaly Detection 151\u003c\/p\u003e \u003cp\u003e7.6 Exploring Data Visually 152\u003c\/p\u003e \u003cp\u003e7.6.1 Metabase 152\u003c\/p\u003e \u003cp\u003e7.6.2 Apache Superset 153\u003c\/p\u003e \u003cp\u003e\u003cb\u003e8 Data Science \u003c\/b\u003e\u003cb\u003e155\u003c\/b\u003e\u003c\/p\u003e \u003cp\u003e8.1 Data Science Applications 155\u003c\/p\u003e \u003cp\u003e8.1.1 Recommendation 156\u003c\/p\u003e \u003cp\u003e8.1.2 Predictive Analytics 156\u003c\/p\u003e \u003cp\u003e8.1.3 Pattern Discovery 157\u003c\/p\u003e \u003cp\u003e8.2 Data Science Life Cycle 158\u003c\/p\u003e \u003cp\u003e8.2.1 Business Objective 158\u003c\/p\u003e \u003cp\u003e8.2.2 Data Understanding 159\u003c\/p\u003e \u003cp\u003e8.2.3 Data Ingestion 159\u003c\/p\u003e \u003cp\u003e8.2.4 Data Preparation 160\u003c\/p\u003e \u003cp\u003e8.2.5 Data Exploration 160\u003c\/p\u003e \u003cp\u003e8.2.6 Feature Engineering 161\u003c\/p\u003e \u003cp\u003e8.2.7 Modeling 161\u003c\/p\u003e \u003cp\u003e8.2.8 Model Evaluation 162\u003c\/p\u003e \u003cp\u003e8.2.9 Model Deployment 163\u003c\/p\u003e \u003cp\u003e8.2.10 Operationalizing 163\u003c\/p\u003e \u003cp\u003e8.3 Data Science Toolbox 164\u003c\/p\u003e \u003cp\u003e8.3.1 R 164\u003c\/p\u003e \u003cp\u003e8.3.2 Python 165\u003c\/p\u003e \u003cp\u003e8.3.3 SQL 167\u003c\/p\u003e \u003cp\u003e8.3.4 TensorFlow 167\u003c\/p\u003e \u003cp\u003e8.3.4.1 Execution Model 169\u003c\/p\u003e \u003cp\u003e8.3.4.2 Architecture 170\u003c\/p\u003e \u003cp\u003e8.3.5 Spark MLlib 171\u003c\/p\u003e \u003cp\u003e8.4 Productionalizing Data Science 173\u003c\/p\u003e \u003cp\u003e8.4.1 Apache PredictionIO 173\u003c\/p\u003e \u003cp\u003e8.4.1.1 Architecture Overview 174\u003c\/p\u003e \u003cp\u003e8.4.1.2 Machine Learning Templates 174\u003c\/p\u003e \u003cp\u003e8.4.2 Seldon 175\u003c\/p\u003e \u003cp\u003e8.4.3 MLflow 175\u003c\/p\u003e \u003cp\u003e8.4.3.1 MLflow Tracking 175\u003c\/p\u003e \u003cp\u003e8.4.3.2 MLflow Projects 176\u003c\/p\u003e \u003cp\u003e8.4.3.3 MLflow Models 176\u003c\/p\u003e \u003cp\u003e8.4.3.4 MLflow Model Registry 177\u003c\/p\u003e \u003cp\u003e8.4.4 Kubeflow 177\u003c\/p\u003e \u003cp\u003e8.4.4.1 Kubeflow Pipelines 177\u003c\/p\u003e \u003cp\u003e8.4.4.2 Kubeflow Metadata 178\u003c\/p\u003e \u003cp\u003e8.4.4.3 Kubeflow Katib 178\u003c\/p\u003e \u003cp\u003e\u003cb\u003e9 Data Discovery \u003c\/b\u003e\u003cb\u003e179\u003c\/b\u003e\u003c\/p\u003e \u003cp\u003e9.1 Need for Data Discovery 179\u003c\/p\u003e \u003cp\u003e9.1.1 Single Source of Metadata 180\u003c\/p\u003e \u003cp\u003e9.1.2 Searching 181\u003c\/p\u003e \u003cp\u003e9.1.3 Data Lineage 182\u003c\/p\u003e \u003cp\u003e9.1.4 Data Ownership 182\u003c\/p\u003e \u003cp\u003e9.1.5 Data Metrics 183\u003c\/p\u003e \u003cp\u003e9.1.6 Data Grouping 183\u003c\/p\u003e \u003cp\u003e9.1.7 Data Clustering 184\u003c\/p\u003e \u003cp\u003e9.1.8 Data Classification 184\u003c\/p\u003e \u003cp\u003e9.1.9 Data Glossary 185\u003c\/p\u003e \u003cp\u003e9.1.10 Data Update Notification 185\u003c\/p\u003e \u003cp\u003e9.1.11 Data Presentation 186\u003c\/p\u003e \u003cp\u003e9.2 Data Governance 186\u003c\/p\u003e \u003cp\u003e9.2.1 Data Governance Overview 186\u003c\/p\u003e \u003cp\u003e9.2.1.1 Data Quality 186\u003c\/p\u003e \u003cp\u003e9.2.1.2 Metadata 187\u003c\/p\u003e \u003cp\u003e9.2.1.3 Data Access 187\u003c\/p\u003e \u003cp\u003e9.2.1.4 Data Life Cycle 188\u003c\/p\u003e \u003cp\u003e9.2.2 Big Data Governance 188\u003c\/p\u003e \u003cp\u003e9.2.2.1 Data Architecture 188\u003c\/p\u003e \u003cp\u003e9.2.2.2 Data Source Integration 189\u003c\/p\u003e \u003cp\u003e9.3 Data Discovery Tools 189\u003c\/p\u003e \u003cp\u003e9.3.1 Metacat 189\u003c\/p\u003e \u003cp\u003e9.3.1.1 Data Abstraction and Interoperability 190\u003c\/p\u003e \u003cp\u003e9.3.1.2 Metadata Enrichment 190\u003c\/p\u003e \u003cp\u003e9.3.1.3 Searching and Indexing 190\u003c\/p\u003e \u003cp\u003e9.3.1.4 Update Notifications 191\u003c\/p\u003e \u003cp\u003e9.3.2 Amundsen 191\u003c\/p\u003e \u003cp\u003e9.3.2.1 Discovery Capabilities 191\u003c\/p\u003e \u003cp\u003e9.3.2.2 Integration Points 192\u003c\/p\u003e \u003cp\u003e9.3.2.3 Architecture Overview 192\u003c\/p\u003e \u003cp\u003e9.3.3 Apache Atlas 193\u003c\/p\u003e \u003cp\u003e9.3.3.1 Searching 194\u003c\/p\u003e \u003cp\u003e9.3.3.2 Glossary 194\u003c\/p\u003e \u003cp\u003e9.3.3.3 Type System 194\u003c\/p\u003e \u003cp\u003e9.3.3.4 Lineage 195\u003c\/p\u003e \u003cp\u003e9.3.3.5 Notifications 196\u003c\/p\u003e \u003cp\u003e9.3.3.6 Bridges and Hooks 196\u003c\/p\u003e \u003cp\u003e9.3.3.7 Architecture Overview 196\u003c\/p\u003e \u003cp\u003e\u003cb\u003e10 Data Security \u003c\/b\u003e\u003cb\u003e199\u003c\/b\u003e\u003c\/p\u003e \u003cp\u003e10.1 Infrastructure Security 199\u003c\/p\u003e \u003cp\u003e10.1.1 Computing 200\u003c\/p\u003e \u003cp\u003e10.1.1.1 Auditing 200\u003c\/p\u003e \u003cp\u003e10.1.1.2 Operating System 200\u003c\/p\u003e \u003cp\u003e10.1.1.3 Network 200\u003c\/p\u003e \u003cp\u003e10.1.2 Identity and Access Management 201\u003c\/p\u003e \u003cp\u003e10.1.2.1 Authentication 201\u003c\/p\u003e \u003cp\u003e10.1.2.2 Authorization 201\u003c\/p\u003e \u003cp\u003e10.1.3 Data Transfers 202\u003c\/p\u003e \u003cp\u003e10.2 Data Privacy 202\u003c\/p\u003e \u003cp\u003e10.2.1 Data Encryption 202\u003c\/p\u003e \u003cp\u003e10.2.1.1 File System Layer Encryption 203\u003c\/p\u003e \u003cp\u003e10.2.1.2 Database Layer Encryption 203\u003c\/p\u003e \u003cp\u003e10.2.1.3 Transport Layer Encryption 203\u003c\/p\u003e \u003cp\u003e10.2.1.4 Application Layer Encryption 203\u003c\/p\u003e \u003cp\u003e10.2.2 Data Anonymization 204\u003c\/p\u003e \u003cp\u003e10.2.2.1 \u003ci\u003ek\u003c\/i\u003e-Anonymity 204\u003c\/p\u003e \u003cp\u003e10.2.2.2 \u003ci\u003el\u003c\/i\u003e-Diversity 204\u003c\/p\u003e \u003cp\u003e10.2.3 Data Perturbation 204\u003c\/p\u003e \u003cp\u003e10.3 Law Enforcement 205\u003c\/p\u003e \u003cp\u003e10.3.1 PII 205\u003c\/p\u003e \u003cp\u003e10.3.1.1 Identifying PII Tables\/Columns 205\u003c\/p\u003e \u003cp\u003e10.3.1.2 Segregating Tables Containing PII 205\u003c\/p\u003e \u003cp\u003e10.3.1.3 Protecting PII Tables via Access Control 206\u003c\/p\u003e \u003cp\u003e10.3.1.4 Masking and Anonymizing PII Data 206\u003c\/p\u003e \u003cp\u003e10.3.2 Privacy Regulations\/Acts 207\u003c\/p\u003e \u003cp\u003e10.3.2.1 Collecting Data 207\u003c\/p\u003e \u003cp\u003e10.3.2.2 Erasing Data 207\u003c\/p\u003e \u003cp\u003e10.4 Data Security Tools 208\u003c\/p\u003e \u003cp\u003e10.4.1 Apache Ranger 208\u003c\/p\u003e \u003cp\u003e10.4.1.1 Ranger Policies 208\u003c\/p\u003e \u003cp\u003e10.4.1.2 Managed Components 209\u003c\/p\u003e \u003cp\u003e10.4.1.3 Architecture Overview 211\u003c\/p\u003e \u003cp\u003e10.4.2 Apache Sentry 212\u003c\/p\u003e \u003cp\u003e10.4.2.1 Managed Components 212\u003c\/p\u003e \u003cp\u003e10.4.2.2 Architecture Overview 213\u003c\/p\u003e \u003cp\u003e10.4.3 Apache Knox 214\u003c\/p\u003e \u003cp\u003e10.4.3.1 Authentication and Authorization 215\u003c\/p\u003e \u003cp\u003e10.4.3.2 Supported Hadoop Services 215\u003c\/p\u003e \u003cp\u003e10.4.3.3 Client Services 216\u003c\/p\u003e \u003cp\u003e10.4.3.4 Architecture Overview 217\u003c\/p\u003e \u003cp\u003e10.4.3.5 Audit 218\u003c\/p\u003e \u003cp\u003e\u003cb\u003e11 Putting All Together \u003c\/b\u003e\u003cb\u003e219\u003c\/b\u003e\u003c\/p\u003e \u003cp\u003e11.1 Platforms 219\u003c\/p\u003e \u003cp\u003e11.1.1 In-house Solutions 220\u003c\/p\u003e \u003cp\u003e11.1.1.1 Cloud Provisioning 220\u003c\/p\u003e \u003cp\u003e11.1.1.2 On-premise Provisioning 221\u003c\/p\u003e \u003cp\u003e11.1.2 Cloud Providers 221\u003c\/p\u003e \u003cp\u003e11.1.2.1 Vendor Lock-in 222\u003c\/p\u003e \u003cp\u003e11.1.2.2 Outages 223\u003c\/p\u003e \u003cp\u003e11.1.3 Hybrid Solutions 223\u003c\/p\u003e \u003cp\u003e11.1.3.1 Kubernetes 224\u003c\/p\u003e \u003cp\u003e11.2 Big Data Systems and Tools 224\u003c\/p\u003e \u003cp\u003e11.2.1 Storage 224\u003c\/p\u003e \u003cp\u003e11.2.1.1 File-Based Storage 225\u003c\/p\u003e \u003cp\u003e11.2.1.2 NoSQL 225\u003c\/p\u003e \u003cp\u003e11.2.2 Processing 226\u003c\/p\u003e \u003cp\u003e11.2.2.1 Batch Processing 226\u003c\/p\u003e \u003cp\u003e11.2.2.2 Stream Processing 227\u003c\/p\u003e \u003cp\u003e11.2.2.3 Combining Batch and Streaming 227\u003c\/p\u003e \u003cp\u003e11.2.3 Model Training 228\u003c\/p\u003e \u003cp\u003e11.2.4 A Holistic View 228\u003c\/p\u003e \u003cp\u003e11.3 Challenges 229\u003c\/p\u003e \u003cp\u003e11.3.1 Growth 229\u003c\/p\u003e \u003cp\u003e11.3.2 SLA 230\u003c\/p\u003e \u003cp\u003e11.3.3 Versioning 231\u003c\/p\u003e \u003cp\u003e11.3.4 Maintenance 232\u003c\/p\u003e \u003cp\u003e11.3.5 Deprecation 233\u003c\/p\u003e \u003cp\u003e11.3.6 Monitoring 233\u003c\/p\u003e \u003cp\u003e11.3.7 Trends 234\u003c\/p\u003e \u003cp\u003e11.3.8 Security 235\u003c\/p\u003e \u003cp\u003e11.3.9 Testing 235\u003c\/p\u003e \u003cp\u003e11.3.10 Organization 236\u003c\/p\u003e \u003cp\u003e11.3.11 Talent 237\u003c\/p\u003e \u003cp\u003e11.3.12 Budget 238\u003c\/p\u003e \u003cp\u003e\u003cb\u003e12 An Ideal Platform \u003c\/b\u003e\u003cb\u003e239\u003c\/b\u003e\u003c\/p\u003e \u003cp\u003e12.1 Event Sourcing 240\u003c\/p\u003e \u003cp\u003e12.1.1 How It Works 240\u003c\/p\u003e \u003cp\u003e12.1.2 Messaging Middleware 241\u003c\/p\u003e \u003cp\u003e12.1.3 Why Use Event Sourcing 242\u003c\/p\u003e \u003cp\u003e12.2 Kappa Architecture 242\u003c\/p\u003e \u003cp\u003e12.2.1 How It Works 243\u003c\/p\u003e \u003cp\u003e12.2.2 Limitations 244\u003c\/p\u003e \u003cp\u003e12.3 Data Mesh 245\u003c\/p\u003e \u003cp\u003e12.3.1 Domain-Driven Design 245\u003c\/p\u003e \u003cp\u003e12.3.2 Self-Serving Infrastructure 247\u003c\/p\u003e \u003cp\u003e12.3.3 Data as Product Approach 247\u003c\/p\u003e \u003cp\u003e12.4 Data Reservoirs 248\u003c\/p\u003e \u003cp\u003e12.4.1 Data Organization 248\u003c\/p\u003e \u003cp\u003e12.4.2 Data Standards 249\u003c\/p\u003e \u003cp\u003e12.4.3 Data Policies 249\u003c\/p\u003e \u003cp\u003e12.4.4 Multiple Data Reservoirs 250\u003c\/p\u003e \u003cp\u003e12.5 Data Catalog 250\u003c\/p\u003e \u003cp\u003e12.5.1 Data Feedback Loop 251\u003c\/p\u003e \u003cp\u003e12.5.2 Data Synthesis 252\u003c\/p\u003e \u003cp\u003e12.6 Self-service Platform 252\u003c\/p\u003e \u003cp\u003e12.6.1 Data Publishing 253\u003c\/p\u003e \u003cp\u003e12.6.2 Data Processing 254\u003c\/p\u003e \u003cp\u003e12.6.3 Data Monitoring 254\u003c\/p\u003e \u003cp\u003e12.7 Abstraction 254\u003c\/p\u003e \u003cp\u003e12.7.1 Abstractions via User Interface 255\u003c\/p\u003e \u003cp\u003e12.7.2 Abstractions via Wrappers 256\u003c\/p\u003e \u003cp\u003e12.8 Data Guild 256\u003c\/p\u003e \u003cp\u003e12.9 Trade-offs 257\u003c\/p\u003e \u003cp\u003e12.9.1 Quality vs Efficiency 258\u003c\/p\u003e \u003cp\u003e12.9.2 Real time vs Offline 258\u003c\/p\u003e \u003cp\u003e12.9.3 Performance vs Cost 258\u003c\/p\u003e \u003cp\u003e12.9.4 Consistency vs Availability 259\u003c\/p\u003e \u003cp\u003e12.9.5 Disruption vs Reliability 259\u003c\/p\u003e \u003cp\u003e12.9.6 Build vs Buy 259\u003c\/p\u003e \u003cp\u003e12.10 Data Ethics 260\u003c\/p\u003e \u003cp\u003e\u003cb\u003eAppendix A Further Systems and Patterns \u003c\/b\u003e\u003cb\u003e261\u003c\/b\u003e\u003c\/p\u003e \u003cp\u003eA.1 Lambda Architecture 261\u003c\/p\u003e \u003cp\u003eA.1.1 Batch Layer 262\u003c\/p\u003e \u003cp\u003eA.1.2 Speed Layer 262\u003c\/p\u003e \u003cp\u003eA.1.3 Serving Layer 262\u003c\/p\u003e \u003cp\u003eA.2 Apache Cassandra 263\u003c\/p\u003e \u003cp\u003eA.2.1 Cassandra Data Modeling 263\u003c\/p\u003e \u003cp\u003eA.2.2 Cassandra Architecture 265\u003c\/p\u003e \u003cp\u003eA.2.2.1 Cassandra Components 265\u003c\/p\u003e \u003cp\u003eA.2.2.2 Storage Engine 267\u003c\/p\u003e \u003cp\u003eA.3 Apache Beam 267\u003c\/p\u003e \u003cp\u003eA.3.1 Programming Overview 268\u003c\/p\u003e \u003cp\u003eA.3.2 Execution Model 270\u003c\/p\u003e \u003cp\u003e\u003cb\u003eAppendix B Recipes \u003c\/b\u003e\u003cb\u003e271\u003c\/b\u003e\u003c\/p\u003e \u003cp\u003eB.1 Activity Tracking Recipe 271\u003c\/p\u003e \u003cp\u003eB.1.1 Problem Statement 271\u003c\/p\u003e \u003cp\u003eB.1.2 Design Approach 271\u003c\/p\u003e \u003cp\u003eB.1.2.1 Data Ingestion 271\u003c\/p\u003e \u003cp\u003eB.1.2.2 Computation 272\u003c\/p\u003e \u003cp\u003eB.2 Data Quality Assurance 273\u003c\/p\u003e \u003cp\u003eB.2.1 Problem Statement 273\u003c\/p\u003e \u003cp\u003eB.2.2 Design Approach 273\u003c\/p\u003e \u003cp\u003eB.2.2.1 Ingredients 273\u003c\/p\u003e \u003cp\u003eB.2.2.2 Preparation 275\u003c\/p\u003e \u003cp\u003eB.2.2.3 The Menu 277\u003c\/p\u003e \u003cp\u003eB.3 Estimating Time to Delivery 277\u003c\/p\u003e \u003cp\u003eB.3.1 Problem Definition 278\u003c\/p\u003e \u003cp\u003eB.3.2 Design Approach 278\u003c\/p\u003e \u003cp\u003eB.3.2.1 Streaming Reference Architecture 278\u003c\/p\u003e \u003cp\u003eB.4 Incident Response Recipe 283\u003c\/p\u003e \u003cp\u003eB.4.1 Problem Definition 283\u003c\/p\u003e \u003cp\u003eB.4.2 Design Approach 283\u003c\/p\u003e \u003cp\u003eB.4.2.1 Minimizing Disruptions 284\u003c\/p\u003e \u003cp\u003eB.4.2.2 Categorizing Disruptions 284\u003c\/p\u003e \u003cp\u003eB.4.2.3 Handling Disruptions 284\u003c\/p\u003e \u003cp\u003eB.5 Leveraging Spark SQL Metrics 286\u003c\/p\u003e \u003cp\u003eB.5.1 Problem Statement 286\u003c\/p\u003e \u003cp\u003eB.5.2 Design Approach 286\u003c\/p\u003e \u003cp\u003eB.5.2.1 Spark SQL Metrics Pipeline 286\u003c\/p\u003e \u003cp\u003eB.5.2.2 Integrating SQL Metrics 288\u003c\/p\u003e \u003cp\u003eB.5.2.3 The Menu 289\u003c\/p\u003e \u003cp\u003eB.6 Airbnb Price Prediction 289\u003c\/p\u003e \u003cp\u003eB.6.1 Problem Statement 290\u003c\/p\u003e \u003cp\u003eB.6.2 Design Approach 290\u003c\/p\u003e \u003cp\u003eB.6.2.1 Tech Stack 290\u003c\/p\u003e \u003cp\u003eB.6.2.2 Data Understanding and Ingestion 291\u003c\/p\u003e \u003cp\u003eB.6.2.3 Data Preparation 291\u003c\/p\u003e \u003cp\u003eB.6.2.4 Modeling and Evaluation 293\u003c\/p\u003e \u003cp\u003eB.6.2.5 Monitor and Iterate 294\u003c\/p\u003e \u003cp\u003eBibliography 295\u003c\/p\u003e \u003cp\u003eIndex 301\u003c\/p\u003e","brand":"John Wiley \u0026 Sons Inc","offers":[{"title":"Default Title","offer_id":49407121293655,"sku":"9781119690924","price":100.76,"currency_code":"GBP","in_stock":true}],"thumbnail_url":"\/\/cdn.shopify.com\/s\/files\/1\/0817\/1739\/5799\/files\/9781119690924.jpg?v=1730498252","url":"https:\/\/bookcurl.com\/products\/designing-big-data-platforms-9781119690924","provider":"Book Curl","version":"1.0","type":"link"}