Data warehousing Books

Book SynopsisData pipelines are the foundation for success in data analytics. Moving data from numerous diverse sources and transforming it to provide context is the difference between having data and actually gaining value from it. This pocket reference defines data pipelines and explains how they work in today's modern data stack.

Read more less

Book Synopsis

Read more less

Book SynopsisDaniel Chen is a graduate student in the Interdisciplinary PhD program in Genetics, Bioinformatics & Computational Biology (GBCB) at Virginia Polytechnic Institute and State University (Virginia Tech). He is involved with Software Carpentry as an instructor, Mentoring Committee Member, and currently serves as the Assessment Committee Chair. He completed his Masters in Public Health at Columbia University Mailman School of Public Health in Epidemiology with a certificate in Advanced Epidemiology and currently extending his Master's thesis work in the Social and Decision Analytics Laboratory under the Virginia Bioinformatics Institute on attitude diffusion in social networks.Table of ContentsForeword by Anne M. Brown xxiii Foreword by Jared Lander xxv Preface xxvii Changes in the Second Edition xxxix Part I: Introduction 1 Chapter 1. Pandas DataFrame Basics 3 Learning Objectives 3 1.1 Introduction 3 1.2 Load Your First Data Set 4 1.3 Look at Columns, Rows, and Cells 6 1.4 Grouped and Aggregated Calculations 23 1.5 Basic Plot 27 Conclusion 28 Chapter 2. Pandas Data Structures Basics 31 Learning Objectives 31 2.1 Create Your Own Data 31 2.2 The Series 33 2.3 The DataFrame 42 2.4 Making Changes to Series and DataFrames 45 2.5 Exporting and Importing Data 52 Conclusion 63 Chapter 3. Plotting Basics 65 Learning Objectives 65 3.1 Why Visualize Data? 65 3.2 Matplotlib Basics 66 3.3 Statistical Graphics Using matplotlib 72 3.4 Seaborn 78 3.5 Pandas Plotting Method 111 Conclusion 115 Chapter 4. Tidy Data 117 Learning Objectives 117 Note About This Chapter 117 4.1 Columns Contain Values, Not Variables 118 4.2 Columns Contain Multiple Variables 122 4.3 Variables in Both Rows and Columns 126 Conclusion 129 Chapter 5. Apply Functions 131 Learning Objectives 131 Note About This Chapter 131 5.1 Primer on Functions 131 5.2 Apply (Basics) 133 5.3 Vectorized Functions 138 5.4 Lambda Functions (Anonymous Functions) 141 Conclusion 142 Part II: Data Processing 143 Chapter 6. Data Assembly 145 Learning Objectives 145 6.1 Combine Data Sets 145 6.2 Concatenation 146 6.3 Observational Units Across Multiple Tables 154 6.4 Merge Multiple Data Sets 160 Conclusion 167 Chapter 7. Data Normalization 169 Learning Objectives 169 7.1 Multiple Observational Units in a Table (Normalization) 169 Conclusion 173 Chapter 8. Groupby Operations: Split-Apply-Combine 175 Learning Objectives 175 8.1 Aggregate 176 8.2 Transform 184 8.3 Filter 188 8.4 The pandas.core.groupby.DataFrameGroupBy object 190 8.5 Working with a MultiIndex 195 Conclusion 199 Part III: Data Types 203 Chapter 9. Missing Data 203 Learning Objectives 203 9.1 What Is a NaN Value? 203 9.2 Where Do Missing Values Come From? 205 9.3 Working with Missing Data 210 9.4 Pandas Built-In NA Missing 216 Conclusion 218 Chapter 10. Data Types 219 Learning Objectives 219 10.1 Data Types 219 10.2 Converting Types 220 10.3 Categorical Data 225 Conclusion 227 Chapter 11. Strings and Text Data 229 Introduction 229 Learning Objectives 229 11.1 Strings 229 11.2 String Methods 233 11.3 More String Methods 234 11.4 String Formatting (F-Strings) 236 11.5 Regular Expressions (RegEx) 239 11.6 The regex Library 247 Conclusion 247 Chapter 12. Dates and Times 249 Learning Objectives 249 12.1 Python's datetime Object 249 12.2 Converting to datetime 250 12.3 Loading Data That Include Dates 253 12.4 Extracting Date Components 254 12.5 Date Calculations and Timedeltas 257 12.6 Datetime Methods 259 12.7 Getting Stock Data 261 12.8 Subsetting Data Based on Dates 263 12.9 Date Ranges 266 12.10 Shifting Values 270 12.11 Resampling 276 12.12 Time Zones 278 12.13 Arrow for Better Dates and Times 280 Conclusion 280 Part IV: Data Modeling 281 Chapter 13. Linear Regression (Continuous Outcome Variable) 283 13.1 Simple Linear Regression 283 13.2 Multiple Regression 287 13.3 Models with Categorical Variables 289 13.4 One-Hot Encoding in scikit-learn with Transformer Pipelines 294 Conclusion 296 Chapter 14. Generalized Linear Models 297 About This Chapter 297 14.1 Logistic Regression (Binary Outcome Variable) 297 14.2 Poisson Regression (Count Outcome Variable) 304 14.3 More Generalized Linear Models 308 Conclusion 309 Chapter 15. Survival Analysis 311 15.1 Survival Data 311 15.2 Kaplan Meier Curves 312 15.3 Cox Proportional Hazard Model 314 Conclusion 317 Chapter 16. Model Diagnostics 319 16.1 Residuals 319 16.2 Comparing Multiple Models 324 16.3 k-Fold Cross-Validation 329 Conclusion 334 Chapter 17. Regularization 335 17.1 Why Regularize? 335 17.2 LASSO Regression 337 17.3 Ridge Regression 338 17.4 Elastic Net 340 17.5 Cross-Validation 341 Conclusion 343 Chapter 18. Clustering 345 18.1 k-Means 345 18.2 Hierarchical Clustering 351 Conclusion 356 Part V. Conclusion 357 Chapter 19. Life Outside of Pandas 359 19.1 The (Scientific) Computing Stack 359 19.2 Performance 360 19.3 Dask 360 19.4 Siuba 360 19.5 Ibis 361 19.6 Polars 361 19.7 PyJanitor 361 19.8 Pandera 361 19.9 Machine Learning 361 19.10 Publishing 362 19.11 Dashboards 362 Conclusion 362 Chapter 20. It's Dangerous To Go Alone! 363 20.1 Local Meetups 363 20.2 Conferences 363 20.3 The Carpentries 364 20.4 Podcasts 364 20.5 Other Resources 365 Conclusion 365 Appendices 367 A. Concept Maps 369B. Installation and Setup 373C. Command Line 377D. Project Templates 379E. Using Python 381F. Working Directories 383G. Environments 385H. Install Packages 389I. Importing Libraries 391J. Code Style 393K. Containers: Lists, Tuples, and Dictionaries 395L. Slice Values 399M. Loops 401N. Comprehensions 403O. Functions 405P. Ranges and Generators 409Q. Multiple Assignment 413R. Numpy ndarray 415S. Classes 417T. SettingWithCopyWarning 419U. Method Chaining 423V. Timing Code 427W. String Formatting 429X. Conditionals (if-elif-else) 433Y. New York ACS Logistic Regression Example 435Z. Replicating Results in R 443 Index 451

Read more less

Book Synopsis

Read more less

Book Synopsis

Read more less

Book SynopsisHow do you model and design your database application in consideration of new technology or new business needs? This title is loaded with design rules and case studies that are applicable to any SQL, UML, or XML-based system. It is useful to those tasked with the creation of data models for the integration of large-scale enterprise data.Trade Review"Database Modeling and Design is one of the best books that I have seen for explaining how to build database applications. The book is informative, well-written, and concise." --Michael Blaha, DSc., Consultant, Modelsoft Consulting Corp"This book book is by far the best book available on classic database design. Topics like normalization and many-to-many and n-ary association semantics are without peer in teaching you how to model real-world complexities. This latest edition extends the classic material with extensive discussion of modern tools and other aspects of logical database design. Every database architect should have this book at hand." --Bob Muller, Data Analyst, Poesys Associates“The book is not only good for beginners, but it also provides greater insight for experienced learners. Perhaps this is why it has evolved into its fifth edition. The book is generally well organized. It starts with the first step in the database life cycle, and progresses in a chronological order to more advanced concepts such as object relational design, Extensible Markup Language (XML), and Web databases. The writing style of the book is simple and straightforward, and the use of database terminology is very concise…In my opinion, the book could be used as a course text, with some help from other sources to cover SQL query-related concepts. However, I would have liked a chapter on SQL that covered simple and complex query design, as well as optimization." --Computing ReviewsTable of Contents1. Introduction2. The Entity-Relationship Model3. Unified Modeling Language (UML)4. Requirements Analysis and Conceptual Modeling5. Transforming the Conceptual Data Model to SQL6. Normalization7. An Example of Logical Database Design8. Object Relational Design9. XML and Web Databases10. Business Intelligence11. CASE ToolsAppendix: The Basics of SQL

Read more less

Book Synopsis Kennedy Behrman is a veteran software and data engineer. He first used Python writing asset management systems in the Visual Effects industry. He then moved into the startup world, using Python at startups using machine learning to characterize videos and predict the social media power of athletes. Table of ContentsPreface xiii I: Learning Python in a Notebook Environment 1 1 Introduction to Notebooks 3 2 Fundamentals of Python 13 3 Sequences 25 4 Other Data Structures 37 5 Execution Control 55 6 Functions 67 II: Data Science Libraries 83 7 NumPy 85 8 SciPy 103 9 Pandas 113 10 Visualization Libraries 135 11 Machine Learning Libraries 153 12 Natural Language Toolkit 159 III: Intermediate Python 171 13 Functional Programming 173 14 Object-Oriented Programming 187 15 Other Topics 201 A Answers to End-of-Chapter Questions 215 Index 221

Read more less

Book SynopsisAmateur astronomers are always on the lookout for new observing challenges. This is a practical guide to locating and viewing the most impressive of Herschel’s star clusters, nebulae and galaxies, cataloging more than 600 of the brightest objects, and offering detailed descriptions and images of 150 to 200 of the best.Trade ReviewFrom the reviews: "Mullaney packs an incredible amount of information into this 166-page book. … All in all, The Herschel Objects, and how to observe them is engaging, challenging, well-written, and comprehensive. So, if you love deep-sky observing – and even if you’ve observed the Astronomical League’s Herschel 400 – Mullaney’s book offers a new list with several hundred additional objects you’ll enjoy." (Michael Bakich, Astronomy Magazine, October, 2007) "The Herschel Objects and How to Observe Them is a fine addition to the Springer series of observing guides. Mullaney has been observing the Herschel objects for many years and his passion for them clearly comes across. … Overall though, this is a book that will be a useful addition to any deep-sky observer’s library." (Paul Money, BBC Sky at Night, February, 2008) "Mullaney begins with a well-written biographical sketch of Herschel and his family, and explains the significance of the work of this great observational astronomer. … the objects are illustrated with excellent images obtained using a modern charge-coupled device (CCD) system. The book concludes with a list of 618 targets that would provide for a lifetime of study. The book will be of greatest interest to experienced observers who wish to push on to the most challenging deep sky objects. … Summing Up: Recommended. General readers." (D. E. Hogg, CHOICE, Vol. 45 (6), February, 2008) "The book opens with a few short chapters on Herschel himself together with a brief introduction to observing techniques … . rounded out with some objects that the author regards as showpieces that were not discovered by Herschel. Any collection of these will of course be very subjective. … I found the book’s reproductions to be a cut above the usual Springer ones and the book does offers something sufficiently different … and the Astronomical League guides to make it worth adding to your collection." (Owen Brazell, The Observatory, Vol. 128 (1203), 2008)Table of ContentsWilliam Herschel's Life, Telescopes and Catalogs.- Herschel's Telescopes.- Herschel's Catalogs and Classes.- Observing Techniques.- Exploring The Herschel Showpieces.- Showpieces of Class I.- Showpieces of Class IV.- Showpieces of Class V.- Showpieces of Class VI.- Showpieces of Class VII.- Showpieces of Class VIII.- Samples of Classes II & III.- Showpieces Missed by Herschel.- The “Missing” Herschel Objects.- Conclusion.

Read more less

Book SynopsisThe techniques pioneered by the Kimball Group have become the industry standard for data warehouse design, development, and management. In this new edition of the Microsoft Data Warehouse Toolkit, the authors share best practices for using these techniques in SQL Server 2008 R2 and Office 2010.Table of ContentsForeword xxvii Introduction xxix Part 1 Requirements, Realities, and Architecture 1 Chapter 1 Defining Business Requirements 3 The Most Important Determinant of Long-Term Success 5 Adventure Works Cycles Introduction 6 Uncovering Business Value 6 Obtaining Sponsorship 7 Defining Enterprise-Level Business Requirements 8 Prioritizing the Business Requirements 22 Revisiting the Project Planning 25 Gathering Project-Level Requirements 26 Summary 28 Chapter 2 Designing the Business Process Dimensional Model 29 Dimensional Modeling Concepts and Terminology 30 Facts 31 Dimensions 33 Bringing Facts and Dimensions Together 34 The Bus Matrix, Conformed Dimensions, and Drill Across 36 Additional Design Concepts and Techniques 38 Surrogate Keys 38 Slowly Changing Dimensions 39 Dates 42 Degenerate Dimensions 43 Snowflaking 43 Many-to-Many or Multivalued Dimensions 44 Hierarchies 47 Aggregate Dimensions 49 Junk Dimensions 51 The Three Fact Table Types 52 Aggregates 53 The Dimensional Modeling Process 54 Preparation 55 Data Profiling and Research 60 Building Dimensional Models 63 Developing the Detailed Dimensional Model 66 Testing and Refining the Model 68 Reviewing and Validating the Model 68 Case Study: The Adventure Works Cycles Orders Dimensional Model 69 The Orders Fact Table 69 The Dimensions 69 Identifying Dimension Attributes and Facts for the Orders Business Process 72 The Final Draft of the Initial Orders Model 74 Detailed Orders Dimensional Model Development 75 Final Dimensional Model 77 Summary 77 Chapter 3 The Toolset 79 The Microsoft DW/BI Toolset 80 Why Use the Microsoft Toolset? 82 Architecture of a Microsoft DW/BI System 83 Why Analysis Services? 84 Why a Relational Store? 86 ETL Is Not Optional 86 The Role of Master Data Services 88 Delivering BI Applications 88 Overview of the Microsoft Tools 89 Which Products Do You Need? 90 SQL Server Development and Management Tools 92 Summary 97 Chapter 4 System Setup 99 System Sizing Considerations 100 Calculating Data Volumes 101 Determining Usage Complexity 102 Estimating Simultaneous Users 104 Assessing System Availability Requirements 105 How Big Will It Be? 105 System Configuration Considerations 105 Memory 106 Monolithic or Distributed? 106 Storage System Considerations 110 Processors 113 Setting Up for High Availability 114 Software Installation and Configuration 115 Development Environment Software Requirements 116 Test and Production Software Requirements 120 Operating Systems 122 SQL Server Relational Database Setup 122 Analysis Services Setup 126 Integration Services Setup 129 Reporting Services Setup 130 Summary 131 Part 2 Building and Populating the Databases 133 Chapter 5 Creating the Relational Data Warehouse 135 Getting Started 136 Complete the Physical Design 137 Surrogate Keys 138 String Columns 138 To Null, or Not to Null? 140 Housekeeping Columns 140 Table and Column Extended Properties 142 Define Storage and Create Constraints and Supporting Objects 142 Create Files and Filegroups 142 Data Compression 144 Entity and Referential Integrity Constraints 145 Initial Indexing and Database Statistics 147 Aggregate Tables 150 Create Table Views 151 Insert an Unknown Member Row 152 Example CREATE TABLE Statement 152 Partitioned Tables 153 Finishing Up 163 Staging Tables 163 Metadata Setup 163 Summary 164 Chapter 6 Master Data Management 165 Managing Master Reference Data 166 Incomplete Attributes 167 Data Integration 168 Systems Integration 170 Master Data Management Systems and the Data Warehouse 171 Introducing SQL Server Master Data Services 171 Model Definition Features 172 Data Management Features 174 User Interface: Exploring and Managing the Master Data 174 Importing and Updating Data 176 Exporting Data 177 Full Versioning of All Attributes 179 Creating a Simple Application 179 Summary 186 Chapter 7 Designing and Developing the ETL System 187 Round Up the Requirements 188 Develop the ETL Plan 191 Introducing SQL Server Integration Services 192 Control Flow and Data Flow 194 SSIS Package Architecture 197 The Major Subsystems of ETL 198 Extracting Data 199 Subsystem 1: Data Profiling 199 Subsystem 2: Change Data Capture System 200 Subsystem 3: Extract System 202 Cleaning and Conforming Data 206 Subsystem 4: Data Cleaning System 206 Subsystem 5: Error Event Schema 214 Subsystem 6: Audit Dimension Assembler 215 Subsystem 7: Deduplication System 216 Subsystem 8: Conforming System 217 Delivering Data for Presentation 218 Subsystem 9: Slowly Changing Dimension Manager 218 Subsystem 10: Surrogate Key Generator 223 Subsystem 11: Hierarchy Manager 223 Subsystem 12: Special Dimensions Manager 224 Subsystem 13: Fact Table Builders 225 Subsystem 14: Surrogate Key Pipeline 229 Subsystem 15: Multi-Valued Dimension Bridge Table Builder 235 Subsystem 16: Late Arriving Data Handler 235 Subsystem 17: Dimension Manager 238 Subsystem 18: Fact Provider System 238 Subsystem 19: Aggregate Builder 239 Subsystem 20: OLAP Cube Builder 239 Subsystem 21: Data Propagation Manager 240 Managing the ETL Environment 240 Summary 243 Chapter 8 The Core Analysis Services OLAP Database 245 Overview of Analysis Services OLAP 247 Why Use Analysis Services? 247 Why Not Analysis Services? 249 Designing the OLAP Structure 250 Planning 251 Getting Started 253 Create a Project and a Data Source View 255 Dimension Designs 257 Creating and Editing Dimensions 261 Creating and Editing the Cube 274 Physical Design Considerations 291 Understanding Storage Modes 293 Developing the Partitioning Plan 294 Designing Performance Aggregations 296 Planning for Deployment 298 Processing the Full Cube 299 Developing the Incremental Processing Plan 299 Summary 304 Chapter 9 Design Requirements for Real-Time BI 305 Real-Time Triage 306 What Does Real-Time Mean? 306 Who Needs Real Time? 307 Real-Time Tradeoffs 308 Scenarios and Solutions 311 Executing Reports in Real Time 313 Serving Reports from a Cache 313 Creating an ODS with Mirrors and Snapshots 314 Creating an ODS with Replication 314 Building a BizTalk Application 315 Building a Real-Time Relational Partition 315 Querying Real-Time Data in the Relational Database 317 Using Analysis Services to Query Real-Time Data 318 Summary 319 Part 3 Developing the BI Applications 321 Chapter 10 Building BI Applications in Reporting Services 323 A Brief Overview of BI Applications 324 Types of BI Applications 325 The Value of Business Intelligence Applications 326 A High-Level Architecture for Reporting 328 Reviewing Business Requirements for Reporting 328 Examining the Reporting Services Architecture 330 Using Reporting Services as a Standard Reporting Tool 332 Reporting Services Assessment 339 The Reporting System Design and Development Process 340 Reporting System Design 341 Reporting System Development 348 Building and Delivering Reports 351 Planning and Preparation 351 Creating Reports 354 Reporting Operations 368 Ad Hoc Reporting Options 369 The Report Model 370 Shared Datasets 371 Report Parts 371 Summary 372 Chapter 11 PowerPivot and Excel 375 Using Excel for Analysis and Reporting 376 The PowerPivot Architecture: Excel on Steroids 378 Creating and Using PowerPivot Databases 380 Getting Started 381 PowerPivot Table Design 381 Creating Analytics with PowerPivot 385 Observations and Guidelines on PowerPivot for Excel 392 PowerPivot for SharePoint 394 The PowerPivot SharePoint User Experience 394 Server-Level Resources 397 PowerPivot Monitoring and Management 397 PowerPivot’s Role in a Managed DW/BI Environment 400 Summary 401 Chapter 12 The BI Portal and SharePoint 403 The BI Portal 404 Planning the BI Portal 405 Impact on Design 406 Business Process Categories 407 Additional Functions 408 Building the BI Portal 409 Using SharePoint as the BI Portal 411 Architecture and Concepts 412 Setting Up SharePoint 417 Summary 426 Chapter 13 Incorporating Data Mining 429 Defining Data Mining 430 Basic Data Mining Terminology 432 Business Uses of Data Mining 433 Roles and Responsibilities 440 SQL Server Data Mining Architecture Overview 440 The Data Mining Design Environment 442 Build, Deploy, and Process 442 Accessing the Mining Models 443 Integration Services and Data Mining 443 Additional Features 444 Architecture Summary 445 Microsoft Data Mining Algorithms 445 Decision Trees 446 Naïve Bayes 447 Clustering 448 Sequence Clustering 448 Time Series 449 Association 449 Neural Network 449 The Data Mining Process 450 The Business Phase 451 The Data Mining Phase 453 The Operations Phase 460 Metadata 462 Data Mining Examples 463 Case Study: Categorizing Cities 463 Case Study: Product Recommendations 472 Summary 488 Part 4 Deploying and Managing the DW/BI System 491 Chapter 14 Designing and Implementing Security 493 Identifying the Security Manager 494 Securing the Hardware and Operating System 495 Securing the Operating System 495 Using Windows Integrated Security 496 Securing the Development Environment 497 Securing the Data 498 Providing Open Access for Internal Users 498 Itemizing Sensitive Data 500 Securing Various Types of Data Access 500 Securing the Components of the DW/BI System 502 Reporting Services Security 502 Analysis Services Security 505 Relational DW Security 514 Integration Services Security 520 Usage Monitoring 521 Summary 521 Chapter 15 Metadata Plan 523 Metadata Basics 524 The Purpose of Metadata 524 Metadata Categories 525 The Metadata Repository 526 Metadata Standards 526 SQL Server 2008 R2 Metadata 527 Cross-Tool Components 528 Relational Engine Metadata 532 Analysis Services 532 Integration Services 533 Reporting Services 533 Master Data Services 534 SharePoint 534 External Metadata Sources 534 Looking to the Future 535 A Practical Metadata Approach 535 Creating the Metadata Strategy 536 Business Metadata Reporting 538 Process Metadata Reporting 541 Technical Metadata Reporting 542 Ongoing Metadata Management 543 Summary 543 Chapter 16 Deployment 545 Setting Up the Environments 546 Testing 550 Development Testing 551 System Testing 555 Data Quality Assurance Testing 557 Performance Testing 559 Usability Testing 562 Testing Summary 563 Deploying to Production 564 Relational Database Deployment 565 Integration Services Package Deployment 567 Analysis Services Database Deployment 568 Reporting Services Report Deployment 571 Master Data Services Deployment 572 Data Warehouse and BI Documentation 573 Core Descriptions 573 Additional Documentation 575 User Training 576 User Support 579 Desktop Readiness and Configuration 580 Summary 581 Chapter 17 Operations and Maintenance 583 Providing User Support 584 Maintaining the BI Portal 585 Extending the BI Applications 586 System Management 587 Governing the DW/BI System 588 Performance Monitoring 593 Usage Monitoring 600 Managing Disk Space 602 Service and Availability Management 603 Performance Tuning the DW/BI System 604 Backup and Recovery 606 Executing the ETL Packages 611 Summary 611 Chapter 18 Present Imperatives and Future Outlook 613 Growing the DW/BI System 613 Lifecycle Review with Common Problems 615 Phase I — ​Requirements, Realities, Plans, and Designs 616 Phase II — ​Developing the Databases 616 Phase III — ​Developing the BI Applications and Portal Environment 617 Phase IV — ​Deploying and Managing the DW/BI System 618 Iteration and Growth 618 What We Like in the Microsoft BI Toolset 619 Future Directions: Room for Improvement 620 Conclusion 623 Index 625

Read more less

Book SynopsisTrade Review"[A] penetrating study of missing (‘dark’) data and its impacts on decisions—skewing stats, enabling fraud, embedding inequity and triggering preventable catastrophes. Advocating ‘data science judo,’ Hand offers expert training, from recognizing when facts are being cherry-picked to designing randomized trials. A book illuminating shadowed corners in science, medicine and policy."---Barbara Kiser, Nature"A tour de force. . . . Hand is a good and able guide to take us through the many aspects of dark data that are potentially skewing our understanding of real world observations and potential scientific breakthroughs. He writes in an accessible and understandable way too."---Simon Cocking, Irish Tech News"Well-written and accessible."---Tim Harford, Undercover Economist"You need to read [Dark Data], and be convinced by David’s reasoning and his examples of cases in which unseen or unreported data play a critical and sometimes even a fatal role. You are likely to walk away with the feeling that the term dark data is indeed a very effective one to arouse both curiosity and suspicion, mixed with happiness that finally a great term was coined by a statistician—and sadness that the statistician is not you."---Xiao-Li Meng, IMS Bulletin"An exploration of a major problem in data analysis with an attempt of classification, analysing causes, mechanisms, and to some extent also suggest mitigations."---Adhemar Bultheel, European Mathematical Society"An excellent guide to the many reasons for caution in interpreting data."---Diane Coyle, Enlightened Economist

Read more less

Book SynopsisTrade Review"[A] penetrating study of missing (‘dark’) data and its impacts on decisions—skewing stats, enabling fraud, embedding inequity and triggering preventable catastrophes. Advocating ‘data science judo,’ Hand offers expert training, from recognizing when facts are being cherry-picked to designing randomized trials. A book illuminating shadowed corners in science, medicine and policy."---Barbara Kiser, Nature"A tour de force. . . . Hand is a good and able guide to take us through the many aspects of dark data that are potentially skewing our understanding of real world observations and potential scientific breakthroughs. He writes in an accessible and understandable way too."---Simon Cocking, Irish Tech News"Well-written and accessible."---Tim Harford, Undercover Economist"You need to read [Dark Data], and be convinced by David’s reasoning and his examples of cases in which unseen or unreported data play a critical and sometimes even a fatal role. You are likely to walk away with the feeling that the term dark data is indeed a very effective one to arouse both curiosity and suspicion, mixed with happiness that finally a great term was coined by a statistician—and sadness that the statistician is not you."---Xiao-Li Meng, IMS Bulletin"An exploration of a major problem in data analysis with an attempt of classification, analysing causes, mechanisms, and to some extent also suggest mitigations."---Adhemar Bultheel, European Mathematical Society"An excellent guide to the many reasons for caution in interpreting data."---Diane Coyle, Enlightened Economist

Read more less

Book SynopsisThis practical book bridges the gap by teaching software engineers mid-level MySQL knowledge beyond the fundamentals, but well shy of deep-level internals required by database administrators (DBAs).

Read more less

Book SynopsisAuthor Mark Edmondson, Google Developer Expert for Google Analytics and Google Cloud, provides a concise yet comprehensive overview of GA4 and its cloud integrations.

Read more less

Book SynopsisAuthor Rukmani Gopalan, a product management leader and data enthusiast, guides data architects and engineers through the major aspects of working with a cloud data lake, from design considerations and best practices to data format optimizations, performance optimization, cost management, and governance.

Read more less

Book SynopsisThis practical guide thoroughly examines this managed service and demonstrates how you can use it to extract value from your data immediately, rather than go through the heavy lifting required to run a typical data warehouse.

Read more less

Book SynopsisThis book explains in a clear and thoughtful voice why systems perform the way they do. It's for anybody who's curious about how computer programs and other processes use their time and about what you can do to improve them.

Read more less

Book SynopsisTable of ContentsIntroduction xix 1 Everything You Ever Needed to Know About Spreadsheets but Were Too Afraid to Ask 1 Some Sample Data 2 Accessing Quick Descriptive Statistics 3 Excel Tables 4 Filtering and Sorting 5 Table Formatting 7 Structured References 7 Adding Table Columns 10 Lookup Formulas 11 VLOOKUP 11 INDEX/MATCH 13 XLOOKUP 15 PivotTables 16 Using Array Formulas 19 Solving Stuff with Solver 20 2 Set It and Forget It: An Introduction to Power Query 27 What Is Power Query? 27 Sample Data 28 Starting Power Query 29 Filtering Rows 32 Removing Columns 33 Find & Replace 34 Close & Load to Table 35 3 Naïve Bayes and the Incredible Lightness of Being an Idiot 39 The World's Fastest Intro to Probability Theory 39 Totaling Conditional Probabilities 40 Joint Probability, the Chain Rule, and Independence 40 What Happens in a Dependent Situation? 41 Bayes Rule 42 Separating the Signal and the Noise 43 Using the Bayes Rule to Create an AI Model 44 High-Level Class Probabilities Are Often Assumed to Be Equal 45 A Couple More Odds and Ends 46 Let's Get This Excel Party Started 47 Cleaning the Data with Power Query 48 Splitting on Spaces: Giving Each Word Its Due 50 Counting Tokens and Calculating Probabilities 55 We Have a Model! Let's Use It 58 4 Cluster Analysis Part 1: Using K-Means to Segment Your Customer Base 65 Dances at Summer Camp 65 Getting Real: K-Means Clustering Subscribers in Email Marketing 70 The Initial Dataset 71 Determining What to Measure 72 Start with Four Clusters 75 Euclidean Distance: Measuring Distances as the Crow Flies 76 Solving for the Cluster Centers 80 Making Sense of the Results 82 Getting the Top Deals by Cluster 83 The Silhouette: A Good Way to Let Different K Values Duke It Out 86 How About Five Clusters? 95 Solving for Five Clusters 96 Getting the Top Deals for All Five Clusters 96 Computing the Silhouette for 5-Means Clustering 99 K-Medians Clustering and Asymmetric Distance Measurements 100 Using K-Medians Clustering 100 Getting a More Appropriate Distance Metric 100 Putting It All in Excel 102 The Top Deals for the 5-Medians Clusters 104 5 Cluster Analysis Part II: Network Graphs and Community Detection 109 What Is a Network Graph? 110 Visualizing a Simple Graph 110 Beyond GiGraph and Adjacency Lists 115 Building a Graph from the Wholesale Wine Data 117 Creating a Cosine Similarity Matrix 118 Producing an R-Neighborhood Graph 121 Introduction to Gephi 123 Creating a Static Adjacency Matrix 124 Bringing in Your R-Neighborhood Adjacency Matrix into Gephi 124 Node Degree 128 Touching the Graph Data 130 How Much Is an Edge Worth? Points and Penalties in Graph Modularity 132 What's a Point, and What's a Penalty? 133 Setting Up the Score Sheet 136 Let's Get Clustering! 138 Split Number 1 138 Split 2: Electric Boogaloo 143 And. . .Split3: Split with a Vengeance 145 Encoding and Analyzing the Communities 146 There and Back Again: A Gephi Tale 151 6 Regression: The Granddaddy of Supervised Artificial Intelligence 157 Predicting Pregnant Customers at RetailMart Using Linear Regression 158 The Feature Set 159 Assembling the Training Data 161 Creating Dummy Variables 163 Let's Bake Our Own Linear Regression 165 Linear Regression Statistics: R-Squared, F-Tests, t-Tests 173 Making Predictions on Some New Data and Measuring Performance 182 Predicting Pregnant Customers at RetailMart Using Logistic Regression 192 First You Need a Link Function 192 Hooking Up the Logistic Function and Reoptimizing 193 Baking an Actual Logistic Regression 196 7 Ensemble Models: A Whole Lot of Bad Pizza 203 Getting Started Using the Data from Chapter 6 203 Bagging: Randomize, Train, Repeat 204 Decision Stump is Another Name for a Weak Learner 204 Doesn't Seem So Weak to Me! 204 You Need More Power! 207 Let's Train It 208 Evaluating the Bagged Model 220 Boosting: If You Get It Wrong, Just Boost and Try Again 223 Training the Model—Every Feature Gets a Shot 224 Evaluating the Boosted Model 231 8 Forecasting: Breathe Easy: You Can't Win 235 The Sword Trade Is Hopping 236 Getting Acquainted with Time-Series Data 236 Starting Slow with Simple Exponential Smoothing 238 Setting Up the Simple Exponential Smoothing Forecast 240 You Might Have a Trend 249 Holt's Trend-Corrected Exponential Smoothing 250 Setting Up Holt's Trend-Corrected Smoothing in a Spreadsheet 252 So Are You Done? Looking at Autocorrelations 258 Multiplicative Holt-Winters Exponential Smoothing 266 Setting the Initial Values for Level, Trend, and Seasonality 268 Getting Rolling on the Forecast 274 And. . .Optimize! 280 Putting a Prediction Interval Around the Forecast 283 Creating a Fan Chart for Effect 287 Forecast Sheets in Excel 289 9 Optimization Modeling: Because That "Fresh-Squeezed" Orange Juice Ain't Gonna Blend Itself 293 Wait Is This Data Science? 294 Starting with a Simple Trade-Off 295 Representing the Problem as a Polytope 296 Solving by Sliding the Level Set 297 The Simplex Method: Rooting Around the Corners 298 Working in Excel 300 Fresh from the Grove to Your Glass with a Pit Stop Through a Blending Model 305 Let's Start with Some Specs 307 Coming Back to Consistency 308 Putting the Data into Excel 309 Setting Up the Problem in Solver 311 Lowering Your Standards 314 Dead Squirrel Removal: the Minimax Formulation 317 If-Then and the "Big M" Constraint 320 Multiplying Variables: Cranking Up the Volume to 11,000 324 Modeling Risk 330 Normally Distributed Data 331 10 Outlier Detection: Just Because They're Odd Doesn't Mean They're Unimportant 339 Outliers Are (Bad?) People, Too 340 The Fascinating Case of Hadlum v Hadlum 340 Tukey's Fences 341 Applying Tukey's Fences in a Spreadsheet 342 The Limitations of This Simple Approach 345 Terrible at Nothing, Bad at Everything 346 Preparing Data for Graphing 347 Creating a Graph 350 Getting the k-Nearest Neighbors 351 Graph Outlier Detection Method 1: Just Use the Indegree 352 Graph Outlier Detection Method 2: Getting Nuanced with k-Distance 355 Graph Outlier Detection Method 3: Local Outlier Factors Are Where It's At 358 11 Moving on From Spreadsheets 363 Getting Up and Running with R 364 A Crash Course in R-ing 366 Show Me the Numbers! Vector Math and Factoring 367 The Best Data Type of Them All: the Dataframe 370 How to Ask for Help in R 371 It Gets Even Better Beyond Base R 372 Doing Some Actual Data Science 374 Reading Data into R 374 Spherical K-Means on Wine Data in Just a Few Lines 375 Building AI Models on the Pregnancy Data 381 Forecasting in R 389 Looking at Outlier Detection 393 12 Conclusion 397 Where Am I? What Just Happened? 397 Before You Go-Go 397 Get to Know the Problem 398 We Need More Translators 398 Beware the Three-Headed Geek-Monster: Tools, Performance, and Mathematical Perfection 399 You Are Not the Most Important Function of Your Organization 401 Get Creative and Keep in Touch! 402 Index 403

Read more less

Book SynopsisJan-Willem Middelburg is a Dutch entrepreneur and author with a passion for technology and innovation. He is the CEO and co-founder of Cybiant, a global technology that company that helps to create a more sustainable world through analytics, big data and automation. He is also President and Chief Examiner of the Enterprise Big Data Framework, an independent organization dedicated to upskilling individuals with expertise in Big Data. In partnership with APMG-International, the Enterprise Big Data Framework offers vendor-neutral certifications for individuals.Trade Review"The Enterprise Big Data Framework is relevant for everybody within an organisation engaged in driving maximum benefits from data. There is something for everybody; from the board considering governance and ethical behaviour to individuals within the organisation knowing where they fit and the value they can get from better use of their organisation's data. If you are considering a transformation project, this is an excellent guide for your project team." * Richard Pharro, CEO, The APM Group Limited *"If you are looking for a good guide to empower your knowledge on big data and to find a framework to help you on your big data journey, then this book is for you. From learning what big data is to defining a big data strategy, Jan-Willem has built a book to empower the learner on the topic of big data." * Jordan Morrow, Chief Strategy & Transformation Officer, DataPrime and Author of Be Data Literate *"This book is a master piece for those who are familiar and those who discover the world of data. It provides an "a la carte framework" starting with a (big) data strategy and the supporting aspects such as big data functions, architecture and algorithms. It covers in depth data platforms architectures, its management as well as data governance, data catalogue and all the required security considerations associated to the various data classifications. You will find details of data life cycle management, of various machine learning algorithms and an important chapter covering AI ethics when building and deploying sophisticated algorithms using data. The concepts covered in this book apply to on-premises and in the (public) cloud environments, making this book a must read." * Jean-Michel Coeur, APAC Technology Practice Lead, Data Analytics, Google Cloud *Table of Contents Section - ONE: Introduction to Big Data; Chapter - 01: Introduction to Big Data; Chapter - 02: The Big Data framework; Chapter - 03: Big Data strategy; Chapter - 04: Big Data architecture; Chapter - 05: Big Data algorithms; Chapter - 06: Big Data processes; Chapter - 07: Big Data functions; Chapter - 08: Artificial intelligence; Section - TWO: Enterprise Big Data analysis; Chapter - 09: Introduction to Big Data analysis; Chapter - 10: Defining the business objective; Chapter - 11: Data ingestion – importing and reading data sets; Chapter - 12: Data preparation – cleaning and wrangling data; Chapter - 13: Data analysis – model building; Chapter - 14: Data presentation; Section - THREE: Enterprise Big Data engineering; Chapter - 15: Introduction to Big Data engineering; Chapter - 16: Data modelling; Chapter - 17: Constructing the data lake; Chapter - 18: Building an enterprise Big Data warehouse; Chapter - 19: Design and structure of Big Data pipelines; Chapter - 20: Managing data pipelines; Chapter - 21: Cluster technology; Section - FOUR: enterprise Big Data algorithm design; Chapter - 22: Introduction to Big Data algorithm design; Chapter - 23: Algorithm design – fundamental concepts; Chapter - 24: Statistical machine learning algorithms; Chapter - 25: The data science roadmap; Chapter - 26: Programming languages 26 visualization and simple metrics; Chapter - 27: Advanced machine learning algorithms; Chapter - 28: Advanced machine learning classification algorithms; Chapter - 29: Technical communication and documentation; Section - FIVE: Enterprise Big Data architecture; Chapter - 30: Introduction to the Big Data architecture; Chapter - 31: Strength and resilience – the Big Data platform; Chapter - 32: Design principles for Big Data architecture; Chapter - 33: Big Data infrastructure; Chapter - 34: Big Data platforms; Chapter - 35: The Big Data application provider; Chapter - 36: System orchestration in Big Data

Read more less

Book SynopsisManaging and Mining Graph Data is a comprehensive survey book in graph management and mining. It contains extensive surveys on a variety of important graph topics such as graph languages, indexing, clustering, data generation, pattern mining, classification, keyword search, pattern matching, and privacy.Trade ReviewFrom the reviews:“This book provides a survey of some recent advances in graph mining. It contains chapters on graph languages, indexing, clustering, pattern mining, keyword search, and pattern matching. … The book is targeted at advanced undergraduate or graduate students, faculty members, and researchers from both industry and academia. … I highly recommend this book to someone who is starting to explore the field of graph mining or wants to delve deeper into this exciting field.” (Dimitrios Katsaros, ACM Computing Reviews, December, 2010)Table of ContentsAn Introduction to Graph Data.- Graph Data Management and Mining: A Survey of Algorithms and Applications.- Graph Mining: Laws and Generators.- Query Language and Access Methods for Graph Databases.- Graph Indexing.- Graph Reachability Queries: A Survey.- Exact and Inexact Graph Matching: Methodology and Applications.- A Survey of Algorithms for Keyword Search on Graph Data.- A Survey of Clustering Algorithms for Graph Data.- A Survey of Algorithms for Dense Subgraph Discovery.- Graph Classification.- Mining Graph Patterns.- A Survey on Streaming Algorithms for Massive Graphs.- A Survey of Privacy-Preservation of Graphs and Social Networks.- A Survey of Graph Mining for Web Applications.- Graph Mining Applications to Social Network Analysis.- Software-Bug Localization with Graph Mining.- A Survey of Graph Mining Techniques for Biological Datasets.- Trends in Chemical Graph Data Mining.

Read more less

Book SynopsisIntroduction to Data Mining.- Data for Data Mining.- Introduction to Classification: Naïve Bayes and Nearest Neighbour.- Using Decision Trees for Classification.- Decision Tree Induction: Using Entropy for Attribute Selection.- Decision Tree Induction: Using Frequency Tables for Attribute Selection.- Estimating the Predictive Accuracy of a Classifier.- Continuous Attributes.- Avoiding Overfitting of Decision Trees.- More About Entropy.- Inducing Modular Rules for Classification.- Measuring the Performance of a Classifier.- Dealing with Large Volumes of Data.- Ensemble Classification.- Comparing Classifiers.- Associate Rule Mining I.- Associate Rule Mining II.- Associate Rule Mining III.- Clustering.- Mining.- Classifying Streaming Data.- Classifying Streaming Data II: Time-dependent Data.- An Introduction to Neural Networks.- Appendix A Essential Mathematics.- Appendix B Datasets.- Appendix C Sources of Further Information.- Appendix D Glossary and Notation.- Appendix E SolutioTable of ContentsIntroduction to Data Mining.- Data for Data Mining.- Introduction to Classification: Naïve Bayes and Nearest Neighbour.- Using Decision Trees for Classification.- Decision Tree Induction: Using Entropy for Attribute Selection.- Decision Tree Induction: Using Frequency Tables for Attribute Selection.- Estimating the Predictive Accuracy of a Classifier.- Continuous Attributes.- Avoiding Overfitting of Decision Trees.- More About Entropy.- Inducing Modular Rules for Classification.- Measuring the Performance of a Classifier.- Dealing with Large Volumes of Data.- Ensemble Classification.- Comparing Classifiers.- Associate Rule Mining I.- Associate Rule Mining II.- Associate Rule Mining III.- Clustering.- Mining.- Classifying Streaming Data.- Classifying Streaming Data II: Time-dependent Data.- An Introduction to Neural Networks.- Appendix A – Essential Mathematics.- Appendix B – Datasets.- Appendix C – Sources of Further Information.- Appendix D – Glossary and Notation.- Appendix E – Solutions to Self-assessment Exercises.- Index.

Read more less

Book Synopsis Learn all of the available upgrade and migration methods in detail to move to Oracle Database version 12c. You will become familiar with database upgrade best practices to complete the upgrade in an effective manner and understand the Oracle Database 12c patching process. So it''s time to upgrade Oracle Database to version 12c and you need to choose the appropriate method while considering issues such as downtime. This book explains all of the available upgrade and migration methods so you can choose the one that suits your environment. You will be aware of the practical issues and proactive measures to take to upgrade successfully and reduce unexpected issues.  With every release of Oracle Database there are new features and fixes to bugs identified in previous versions. As each release becomes obsolete, existing databases need to be upgraded. Oracle Database Upgrade and Migration Methods explains each method along Table of Contents PART I: packetC Background 1 CHAPTER 1: Getting Started 3 Introduction to Database upgrade Necessities of Database upgrade Benefits of Database upgrade Hurdles that affect Database upgrade decision Types of Database upgrade Things to consider before upgrade Engineers involved in upgrade activity Upgrade compatibility matrix Best practices of Database upgrade Database Migration Situations demand Migration Things to consider before migration summary PART II: Language Reference 53 n CHAPTER 2: Database Upgrade methods 55 DBUA Manual/Command line upgrade Export/Import Datapump Transportable Tablespace Golden Gate Create Table as Select (CTAS) Transient Logical Standby Full Transportable Tablespace Summary n CHAPTER 3: Comparison between upgrade methods 151 Comparison between methods 9 Real Application testing (RAT) 10 How to choose best Upgrade method 11 Summary n CHAPTER 4: Upgrade using Database backup 159 Cold backup Hot backup (User-Managed) Logical backup (expdp/impdp) RMAN backup (using duplicate option) Summary n CHAPTER 5: Database Migration methods 171 Export/Import Datapump Transportable Tablespace (TTS) Golden Gate Copy table as select (CTAS) Transport Database Heterogenous Standby database Oracle Streams Summary n CHAPTER 6: Migration of Oracle database from Non-ASM to ASM 175 Introduction Moving Datafiles Online from NON-ASM < to ASM Migrating Oracle 12c CDB with PDBs from NON ASM to ASM using EM Cloud Control 13c ............... Migrating Oracle 12c CDB with PDBs from NON ASM to ASM using RMAN Summary n CHAPTER 7: GI and DB upgrade in RAC environment 205 Introduction CVU Pre-Upgrade Check tool Execution Steps for ORAchk Rolling upgrade for Oracle GI Upgrading 11g RAC to 12c RAC using DBUA Upgrading 11g RAC to 12c RAC Manual Upgrading 11g RAC to 12c RAC using EM 13c Summary PART III: Developing Applications 215 n CHAPTER 8: Database upgrade in DG environment 217 Dummy Text Dummy Text Virtual Dummy Text Dummy Text Dummy Text Dummy Text Flow Dummy Text Summary n CHAPTER 9: Database upgrade in EBS environment 223 Prerequisite steps Preupgrade steps Upgrade steps Post upgrade steps Summary CHAPTER 10: Database upgrade in 12c Multitenant environment Migrate lower version database to Multitenant architecture Container database upgrade Pluggable database upgrade Summary n CHAPTER 11: Databases migrate in Multitenant environment 237 Pluggable database migrate Need for Migrate Migration steps Summary n CHAPTER 12: Oracle Database Patching Stratergies 245 Patching Introduction Opatch tool Types of patches Patch apply stratergies (online and offline patching).... PSU and SPU patching Patch apply in RAC and DG environment Datapatch Queryable patch inventory Summary n CHAPTER 13: Database Downgrade 263 Introduction Limitations of Oracle database downgrade Database downgrade steps Downgrade using database flashback Summary n CHAPTER 14: Database upgrade in 12.2 281 Preupgrade checks Upgrade Emulation DBUA Manual Database upgrade Pluggable database upgrade Downgrade 12.2 database to earlier version............... Summary n n APPENDIX A: Reference Tables 383 n APPENDIX B: Dummy Text 395 INDEX 433

Read more less

Book SynopsisToday, big data is taken seriously, and data science is considered downright sexy. With this anthology of reports from award-winning journalist Mike Barlow, you'll appreciate how data science is fundamentally altering our world, for better and for worse.

Read more less

Book SynopsisWith this in-depth book, current and aspiring engineers will learn powerful, real-world best practices for managing data big and small. Contributors from notable companies including Twitter, Google, StitchFix, Microsoft and Capital One share their experiences and lessons learned for overcoming a variety of specific and often nagging challenges.

Read more less

Book SynopsisWith this practical guide, author Sev Leonard provides a holistic approach to designing scalable data pipelines in the cloud. Intermediate data engineers, software developers, and architects will learn how to navigate cost/performance trade-offs and how to choose and configure compute and storage.

Read more less

Book Synopsis

Read more less

Book SynopsisA practical guide for today’s chief data officers to define and manage data governance programs The relatively new role of chief data officer (CDO) has been created to address the issue of managing a company’s data as a strategic asset, but the problem is that there is no universally accepted “playbook” for this role. Magnifying the challenge is the rapidly increasing volume and complexity of data, as well as regulatory compliance as it relates to data. In this book, Sunil Soares provides a practical guide for today’s chief data officers to manage data as an asset while delivering the trusted data required to power business initiatives, from the tactical to the transformative. The guide describes the relationship between the CDO and the data governance team, whose task is the formulation of policy to optimize, secure, and leverage information as an enterprise asset by aligning the objectives of multiple functions. Soares provides unique insight into the role of the CDO and presents a blueprint for implementing data governance successfully within the context of the position. With practical advice CDOs need, this book helps establish new data governance practices or mature existing practices.

Read more less

Book SynopsisGraph Databases in Action teaches readers everything they need to know to begin building and running applications powered by graph databases. Right off the bat, seasoned graph database experts introduce readers to just enough graph theory, the graph database ecosystem, and a variety of datastores. They also explore modelling basics in action with real-world examples, then go hands-on with querying, coding traversals, parsing results, and other essential tasks as readers build their own graph-backed social network app complete with a recommendation engine! Key Features · Graph database fundamentals · An overview of the graph database ecosystem · Relational vs. graph database modelling · Querying graphs using Gremlin · Real-world common graph use cases For readers with basic Java and application development skills building in RDBMS systems such as Oracle, SQL Server, MySQL, and Postgres. No experience with graph databases is required. About the technology Graph databases store interconnected data in a more natural form, making them superior tools for representing data with rich relationships. Unlike in relational database management systems (RDBMS), where a more rigid view of data connections results in the loss of valuable insights, in graph databases, data connections are first priority. Dave Bechberger has extensive experience using graph databases as a product architect and a consultant. He’s spent his career leveraging cutting-edge technologies to build software in complex data domains such as bioinformatics, oil and gas, and supply chain management. He’s an active member of the graph community and has presented on a wide variety of graph-related topics at national and international conferences. Josh Perryman is technologist with over two decades of diverse experience building and maintaining complex systems, including high performance computing (HPC) environments. Since 2014 he has focused on graph databases, especially in distributed or big data environments, and he regularly blogs and speaks at conferences about graph databases.

Read more less

Book SynopsisMassive modern datasets make traditional data structures and algorithms grind to a halt. This fun and practical guide introduces cutting-edge techniques that can reliably handle even the largest distributed datasets.In Algorithms and Data Structures for Massive Datasets you will learn: Probabilistic sketching data structures for practical problems Choosing the right database engine for your application Evaluating and designing efficient on-disk data structures and algorithms Understanding the algorithmic trade-offs involved in massive-scale systems Deriving basic statistics from streaming data Correctly sampling streaming data Computing percentiles with limited space resources Algorithms and Data Structures for Massive Datasets reveals a toolbox of new methods that are perfect for handling modern big data applications. You’ll explore the novel data structures and algorithms that underpin Google, Facebook, and other enterprise applications that work with truly massive amounts of data. These effective techniques can be applied to any discipline, from finance to text analysis. Graphics, illustrations, and hands-on industry examples make complex ideas practical to implement in your projects—and there’s no mathematical proofs to puzzle over. Work through this one-of-a-kind guide, and you’ll find the sweet spot of saving space without sacrificing your data’s accuracy. Purchase of the print book includes a free eBook in PDF, Kindle, and ePub formats from Manning Publications. About the technology Standard algorithms and data structures may become slow—or fail altogether—when applied to large distributed datasets. Choosing algorithms designed for big data saves time, increases accuracy, and reduces processing cost. This unique book distills cutting-edge research papers into practical techniques for sketching, streaming, and organizing massive datasets on-disk and in the cloud. About the book Algorithms and Data Structures for Massive Datasets introduces processing and analytics techniques for large distributed data. Packed with industry stories and entertaining illustrations, this friendly guide makes even complex concepts easy to understand. You’ll explore real-world examples as you learn to map powerful algorithms like Bloom filters, Count-min sketch, HyperLogLog, and LSM-trees to your own use cases. What's inside Probabilistic sketching data structures Choosing the right database engine Designing efficient on-disk data structures and algorithms Algorithmic tradeoffs in massive-scale systems Computing percentiles with limited space resources About the reader Examples in Python, R, and pseudocode. About the author Dzejla Medjedovic earned her PhD in the Applied Algorithms Lab at Stony Brook University, New York. Emin Tahirovic earned his PhD in biostatistics from University of Pennsylvania. Illustrator Ines Dedovic earned her PhD at the Institute for Imaging and Computer Vision at RWTH Aachen University, Germany.   Table of Contents 1 Introduction PART 1 HASH-BASED SKETCHES 2 Review of hash tables and modern hashing 3 Approximate membership: Bloom and quotient filters 4 Frequency estimation and count-min sketch 5 Cardinality estimation and HyperLogLog PART 2 REAL-TIME ANALYTICS 6 Streaming data: Bringing everything together 7 Sampling from data streams 8 Approximate quantiles on data streams PART 3 DATA STRUCTURES FOR DATABASES AND EXTERNAL MEMORY ALGORITHMS 9 Introducing the external memory model 10 Data structures for databases: B-trees, Bε-trees, and LSM-trees 11 External memory sorting

Read more less

Book SynopsisA comprehensive guide to everything scientists need to know about data management, this book is essential for researchers who need to learn how to organize, document and take care of their own data. Researchers in all disciplines are faced with the challenge of managing the growing amounts of digital data that are the foundation of their research. Kristin Briney offers practical advice and clearly explains policies and principles, in an accessible and in-depth text that will allow researchers to understand and achieve the goal of better research data management. Data Management for Researchers includes sections on: * The data problem – an introduction to the growing importance and challenges of using digital data in research. Covers both the inherent problems with managing digital information, as well as how the research landscape is changing to give more value to research datasets and code. * The data lifecycle – a framework for data’s place within the research process and how data’s role is changing. Greater emphasis on data sharing and data reuse will not only change the way we conduct research but also how we manage research data. * Planning for data management – covers the many aspects of data management and how to put them together in a data management plan. This section also includes sample data management plans. * Documenting your data – an often overlooked part of the data management process, but one that is critical to good management; data without documentation are frequently unusable. * Organizing your data – explains how to keep your data in order using organizational systems and file naming conventions. This section also covers using a database to organize and analyze content. * Improving data analysis – covers managing information through the analysis process. This section starts by comparing the management of raw and analyzed data and then describes ways to make analysis easier, such as spreadsheet best practices. It also examines practices for research code, including version control systems. * Managing secure and private data – many researchers are dealing with data that require extra security. This section outlines what data falls into this category and some of the policies that apply, before addressing the best practices for keeping data secure. * Short-term storage – deals with the practical matters of storage and backup and covers the many options available. This section also goes through the best practices to insure that data are not lost. * Preserving and archiving your data – digital data can have a long life if properly cared for. This section covers managing data in the long term including choosing good file formats and media, as well as determining who will manage the data after the end of the project. * Sharing/publishing your data – addresses how to make data sharing across research groups easier, as well as how and why to publicly share data. This section covers intellectual property and licenses for datasets, before ending with the altmetrics that measure the impact of publicly shared data. * Reusing data – as more data are shared, it becomes possible to use outside data in your research. This chapter discusses strategies for finding datasets and lays out how to cite data once you have found it. This book is designed for active scientific researchers but it is useful for anyone who wants to get more from their data: academics, educators, professionals or anyone who teaches data management, sharing and preservation. "An excellent practical treatise on the art and practice of data management, this book is essential to any researcher, regardless of subject or discipline." —Robert Buntrock, Chemical Information BulletinTrade ReviewApparently, NASA lost much of the early data from space exploration, including high quality video footage of the first moon landing. All the more reason to do as it says in the sub-title to the book. -- Alan Crowden * BES Bulletin *For researchers and consumers of data who are often fraught with managing excess information, Briney's book offers valuable techniques, strategies and standards to help achieve proficient data management and successful outcomes. This book can be useful to both novice researchers and well-established scientists alike. -- Mary F. Miles * Medical Reference Services Quarterly *... recommended as a textbook for graduate-level research techniques courses. It's an important resource for academic and special library shelves and a vital reference for anyone working with data. -- Kristen LaBonte * Issues in Science and Technology Librarianship *Briney has written a useful primer on data management for researchers which provides practical advice throughout on managing data. It is easy to read and clearly structured. http://www.ariadne.ac.uk/issue75/cole -- Gareth Cole, Loughborough University Library * Ariadne *Kristin Briney’s Data Management for Researchers is a book that should be on the shelf (physical or virtual) of every librarian, researcher and research administrator. Scientists, engineers, social scientists, humanists — anyone who’s work involves generating and keeping track of digital data. This is the book for you. .... I recommend this book without hesitation for all academic libraries. Individual researchers, research administrators, funding agency employees and academic librarians would all find much useful information. Simply giving a copy to new graduate students is probably a worthwhile investment at any institution. http://scienceblogs.com/confessions/2016/01/11/reading-diary-data-management-for-researchers-organize-maintain-and-share-your-data-for-research-success-by-kristin-briney/ -- John Dupuis, York University Library, Toronto * ScienceBlogs *Briney takes the reader through a pragmatic and sensible route through the activities of data management. -- David Bawden, AlexandriaThis practical handbook can help bring new researchers quickly up-to-speed on the topic, as well as serve as a reference to meet specific data management needs they encounter throughout the data life cycle. * Enid Karr, Journal of eScience Librarianship *I cannot recommend this slender, seemingly innocent looking book enough – it will literally change how you think about data management. -- CBatt[I] intend to give a copy of this book to each graduate student / trainee that joins my lab -- Christie Bahlai, Practical Data Management for Bug Counters

Read more less

Book SynopsisDevelop modern solutions with Snowflake's unique architecture and integration capabilities; process bulk and real-time data into a data lake; and leverage time travel, cloning, and data-sharing features to optimize data operationsKey Features Build and scale modern data solutions using the all-in-one Snowflake platform Perform advanced cloud analytics for implementing big data and data science solutions Make quicker and better-informed business decisions by uncovering key insights from your data Book DescriptionSnowflake is a unique cloud-based data warehousing platform built from scratch to perform data management on the cloud. This book introduces you to Snowflake's unique architecture, which places it at the forefront of cloud data warehouses.You'll explore the compute model available with Snowflake, and find out how Snowflake allows extensive scaling through the virtual warehouses. You will then learn how to configure a virtual warehouse for optimizing cost and performance. Moving on, you'll get to grips with the data ecosystem and discover how Snowflake integrates with other technologies for staging and loading data.As you progress through the chapters, you will leverage Snowflake's capabilities to process a series of SQL statements using tasks to build data pipelines and find out how you can create modern data solutions and pipelines designed to provide high performance and scalability. You will also get to grips with creating role hierarchies, adding custom roles, and setting default roles for users before covering advanced topics such as data sharing, cloning, and performance optimization.By the end of this Snowflake book, you will be well-versed in Snowflake's architecture for building modern analytical solutions and understand best practices for solving commonly faced problems using practical recipes.What you will learn Get to grips with data warehousing techniques aligned with Snowflake's cloud architecture Broaden your skills as a data warehouse designer to cover the Snowflake ecosystem Transfer skills from on-premise data warehousing to the Snowflake cloud analytics platform Optimize performance and costs associated with a Snowflake solution Stage data on object stores and load it into Snowflake Secure data and share it efficiently for access Manage transactions and extend Snowflake using stored procedures Extend cloud data applications using Spark Connector Who this book is forThis book is for data warehouse developers, data analysts, database administrators, and anyone involved in designing, implementing, and optimizing a Snowflake data warehouse. Knowledge of data warehousing and database and cloud concepts will be useful. Basic familiarity with Snowflake is beneficial, but not necessary.Table of ContentsTable of Contents Getting Started with Snowflake Managing the Data Life Cycle Loading and Extracting Data into and out of Snowflake Building Data Pipelines in Snowflake Data Protection and Security in Snowflake Performance and Cost Optimization Secure Data Sharing Back to the Future with Time Travel Advanced SQL Techniques Extending Snowflake's Capabilities

Read more less

Book SynopsisDocumentation of our cultural heritage is experiencing an explosion of innovation. New tools have appeared in recent decades including laser scanning, rapid prototyping, high dynamic range spherical and infrared imagery, drone photography, augmented and virtual reality and computer rendering in multiple dimensions. These give us visualisations and data that are at once interesting, intriguing and yet sometimes deceptive. This text provides an objective and integrated approach to the subject, bringing together the techniques of conservation with management, photographic methods, various modelling techniques and the use of unmanned aerial systems. This interdisciplinary approach addresses the need for knowledge about deploying advanced digital technologies and the materials and methods for the assessment, conservation, rehabilitation and maintenance of the sustainability of existing structures and designated historic buildings. Furthermore, this book actively provides the knowhow to facilitate the creation of heritage inventories, assessing risk, and addressing the need for sustainability.In so doing it becomes more feasible to mitigate the threats from inherent and external causes, not only for the built heritage but also for moveable objects and intangible heritage that suffer abandonment and negligence as well as looting and illegal trafficking. The book is written by a team of international experts based upon their practical experience and expertise. It therefore creates a unique book that encapsulates the knowledge of this discipline required by anyone working in this field.Trade Review`...this new publication is a welcome addition, highlighting how these 3D techniques can be utilised... ...this well-illustrated volume represents a useful contribution for scholars wishing to gain a better understanding of the underpinnings of 3D recording and documentation’. Medieval Archaeology -------------------- `...I found this book very valuable. It can reach an eclectic audience in providing a broad spectrum of the subject. This book is of major importance for Cultural Heritage 3D recording and management and...an important resource handbook’. International Institute for Conservation of Historic and Artistic Works -------------------- '...this new, richly illustrated reference publication on recording and documenting cultural heritage. ... For anyone considering a digital camera for survey purposes ... this chapter [4] is essential reading, and is rightfully one of the best references currently available on the science behind imaging. ...manages to provide what is probably the most up-to-date reference book on 3D recording, documentation and management of cultural heritage. For any heritage professional, academic, student or interested individual considering applying, acquiring, undertaking or researching digital imaging, photogrammetry, Structure-from-Motion, laser scanning, GIS, BIM or RPAS/UAV within a conservation context, this book should be essential reading before embarking down any one of these rapidly developing technological routes'. Conservation and Management of Archaeological Sites -------------------- '...the images in this book, both in colour and high-resolution, play a critical role along with the text. This is a well produced book that is wonderful to read and view. ...I find this book exceptional for its publishing quality, content and production. It clearly includes cutting-edge knowledge, awareness and experience from many contributors involved in cultural heritage processes around the globe...would be very useful to anyone involved in cultural heritage, documentation of history and site preservation and conservation. It can readily serve as a course text in addition to being a reference text. ... I've nothing but positive things to say about this book - I think you will too'. 3D Visualization WorldTable of ContentsIntroduction - current trends in cultural heritage and documentation; Conservation techniques in cultural heritage; Cultural heritage management tools: The role of GIS and BIM; Basics of photography for cultural heritage imaging; Basics of image-based modelling techniques in cultural heritage 3D recording; Basics of range-based modelling techniques in cultural heritage 3D recording; Cultural heritage documentation with RPAS/UAV

Read more less

Book SynopsisThis book constitutes the refereed proceedings of the 8th EAI International Conference on Game Theory for Networks, GameNets 2019, held in Paris, France, in April 2019. The 8 full and 3 short papers presented were carefully reviewed and selected from 17 submissions. They are organized in the following topical sections: Game Theory for Wireless Networks; Games for Economy and Resource Allocation; and Game Theory for Social Networks.Table of ContentsGame Theory for Wireless Networks.- Games for Economy and Resource Allocation.- Game Theory for Social Networks.

Read more less

Book SynopsisThis book constitutes the thoroughly refereed proceedings of the 16th Italian Research Conference on Digital Libraries, IRCDL 2020, held in Bari, Italy, in January 2020.The 12 full papers and 6 short papers presented were carefully selected from 26 submissions. The papers are organized in topical sections on information retrieval, bid data and data science in DL; cultural heritage; open science. Table of ContentsInformation Retrieval.- Bid Data and Data Science in DL.- Cultural Heritage.- Open Science.

Read more less