Table of Contents
Introduction xxvii
Part I What Happened? 1
Chapter 1 Preliminaries 3
Basic Concepts in Data Analysis 3
What Is a Random Variable? 9
Excel Calculations 13
Chapter 2 Was the 1969 Draft Lottery Fair? 17
The Data 17
The Analysis 18
Excel Calculations 20
Chapter 3 Who Won the 2000 Election: Bush or Gore? 23
Projecting the Undervotes 24
What Happened with the Overvotes? 25
The Butterfl y Did It! 25
Excel Calculations 28
Chapter 4 Was Liverpool Over Barcelona the Greatest Upset in Sports History? 31
How Should We Rank Upsets? 31
Leicester Wins the 2015–2016 Premier League 32
#16 Seed UMBC Beats #1 Seed Virginia 33
The Jets Win Super Bowl III 33
Other Big Upsets 34
Chapter 5 How Did Bernie Madoff Keep His Fund Going? 35
The Mathematics of Ponzi Schemes 36
Madoff’s Purported Strategy 37
The Sharpe Ratio Proves Madoff Was a Fraud 39
Benford’s Law and Madoff’s Fraud 40
Excel Calculations 41
Chapter 6 Is the Lot of the American Worker Improving? 45
Is U.S. Family Income Skewed? 45
Median Income and Politics 46
Causes of Increasing U.S. Income Inequality 48
Money Isn’t Everything: The Human
Development Index 50
Create Your Own Ranking of Well-Being 50
Are Other Countries Catching Up to the U.S.? 51
Excel Calculations 52
Chapter 7 Measuring Income Inequality with the Gini, Palm, and Atkinson Indices 53
The Gini Index 53
The Palma Index 56
The Atkinson Index 57
Excel Calculations 59
Chapter 8 Modeling Relationships Between Two Variables 61
Examples of Relationships Between Two Variables 61
Finding the Best-Fitting (Least Squares) Line 62
Computing the Beta of a Stock 63
What Is a Good R2? 64
Correlation and R2 65
We are Not Living in a Linear World 67
Excel Calculations 69
Chapter 9 Intergenerational Mobility 73
Absolute Intergenerational Mobility 74
Intergenerational Elasticity 74
Rank-Rank Mobility 75
Comparing IGE and Rank-Rank Mobility 75
Measuring Mobility with Quintiles 78
The Great Gatsby Curve 80
Excel Calculations 82
Chapter 10 Is Anderson Elementary School a Bad School? 85
How Can We Adjust for Family Income? 86
Estimating the Least Squares Line 86
Can We Compare Standardized Test Performance for Students in Different States? 86
Excel Calculations 87
Chapter 11 Value-Added Assessments of Teacher Effectiveness 89
Simple Gain Score Assessment 90
Covariate Adjustment Assessment 91
Layered Assessment Model 91
Cross-Classified Constant Growth Assessment 91
Problems with VAA 93
How Much Is a Good Teacher Worth? 94
Excel Calculations 95
Chapter 12 Berkeley, Buses, Cars, and Planes 97
Simpson’s Paradox and College Admissions 98
The Waiting Time Paradox 100
When Is the Average of 40 and 80 Not 60? 100
Why Pre COVID Were There Never Empty
Seats on My Flight? 101
Excel Calculations 101
Chapter 13 Is Carmelo Anthony a Hall of Famer? 103
What Metric Defines Basketball Ability? 104
Wins Above Replacement Player (WARP) 105
Manu, Melo, Dirk, and Dwayne 106
How Do 25,000 Points Lead to So Few Wins? 106
Chapter 14 Was Derek Jeter a Great Fielder? 109
Fielding Statistics: The First Hundred Years 109
Range Factor 110
The Fielding Bible: A Great Leap Forward 111
The Next Frontier 112
Chapter 15 “Drive for Show and Putt for Dough?” 115
Strokes Gained 115
The Myth Exposed 116
Chapter 16 What’s Wrong with the NFL QB Rating? 117
NFL Quarterback Rating 117
ESPN’s Total Quarterback Rating 124
Excel Calculations 125
Chapter 17 Some Sports Have All the Luck 127
Skill vs. Luck: The Key Idea 127
The Results 129
Chapter 18 Gerrymandering 131
A Stylized Example 132
The Mathematics of Gerrymandering 136
Chapter 19 Evidence-Based Medicine 143
James Lind and Scurvy: The Birth of Evidence-Based Medicine 143
The Randomized Streptomycin Tuberculosis Trial 145
Excel Calculations 146
Hormone Replacement: Good or Bad? 148
Chapter 20 How Do We Compare Hospitals? 151
Ratings Criteria 152
Conclusion 156
Excel Calculations 157
Chapter 21 What Is the Worst Health Care Problem in My Country? 159
Disability-Adjusted Life Years 159
Determination of Disability Weights 160
To Age Weight or Discount, That Is the Question 162
Key Facts About World Health 163
Part II What Will Happen? 167
Chapter 22 Does a Mutual Fund’s Past Performance Predict Future Performance? 169
Mutual Fund Basics 170
Morningstar Ratings 170
Risk-Adjusting Fund Returns 171
How Well Do Morningstar Star Ratings
Predict a Fund’s Future Performance? 175
The Effect of Expense Ratio on Long-Term Performance 177
Excel Calculations 178
Chapter 23 Is Vegas Good at Picking NFL Games? 181
How NFL Betting Works 181
Bias and Accuracy 184
Vegas Forecasts are Unbiased 185
Totals Predictions and Money Line Predictions are Unbiased 188
NFL Accuracy: The Line vs. the Computers 188
A System Works Until It Doesn’t 189
Chapter 24 Will My New Hires Be Good Employees? 191
What Data Do We Need to Determine Attributes That Best Predict Employee Performance? 192
Besides GMA, Not Much Affects Job Performance 196
Excel Calculations 197
Chapter 25 Should I Go to State U or Princeton? 199
Analyzing Princeton vs. Penn State 200
Excel Calculations 202
Chapter 26 Will My Favorite Sports Team Be Great Next Year? 203
Francis Galton and Regression to the Mean 203
Regression to the Mean in the NFL and the NBA 204
Excel Calculations 207
Chapter 27 How Did Central Bankers Fail to Predict the 2008 Recession? 209
The Inverted Yield Curve 210
The Sahm Rule: Early Warning Signal for Recession 211
Control Charts and the Housing Price/Rent Ratio 211
Excel Calculations 215
Chapter 28 How Does Target Know If You’re Pregnant? 219
What Available Data Can Be Used
to Identify Pregnant Women? 220
Problems Arise 220
An Example of a Pregnancy Prediction Score 221
Chapter 29 How Does Netflix Recommend Movies and TV Shows? 225
User-Based Collaborative Filtering 226
Item-Based Filtering 229
Chapter 30 Can We Predict Heart Attacks in Real Time? 233
Posterior Probability 234
Sensitivity and Specifi city 235
ROC Curve 235
Back to the Apple Heart Study 237
AliveCor and KardiaBand 239
Chapter 31 Is Proactive Policing Effective? 241
Hot Spots Policing 242
Predictive Policing 243
CCTV 244
Stop and Frisk 244
Broken Windows 246
Excel Calculations 247
Chapter 32 Guess How Many are Coming to Dinner? 249
Which Parameters Must Be Estimated? 250
The Data 252
The Results 253
Which Factor Really Matters? 254
Excel Calculations 254
Chapter 33 Can Prediction Markets Predict the Future? 259
Examples of Trade Contracts 260
Prediction Market Trading Mechanisms 261
Accuracy of Prediction Markets and Wisdom of Crowds 262
Chapter 34 The ABCs of Polling 265
Why are 1,112 People Enough to Represent U.S. Voters? 265
Why Doesn’t a Larger Population Require a Larger Sample Size? 267
So, What Can Go Wrong? 268
Rating Polls 271
Chapter 35 How Did Buzzfeed Make the Dress Go Viral? 273
Measuring Instagram Engagement 274
Tweets Do Not Always Go Viral Immediately 274
Do the First Few Days Predict the Future of a Meme? 275
Chapter 36 Predicting Game of Thrones TV Ratings 277
What Does Google Trends Tell Us? 277
Predicting the Present with Google Trends 278
Using Google Trends to Forecast GOT Ratings 279
Excel Calculations 281
Part III Why Did It Happened? 283
Chapter 37 Does Smoking Cause Lung Cancer? 285
Correlation and Causation Redux 285
The Key Evidence 286
Could Air Pollution Have Caused Lung Cancer? 287
The Cigarette Companies Hit Back 287
Excel Calculations 288
Chapter 38 Why are the Houston Rockets a Good Basketball Team? 291
NBA Shooting Math 101 292
Zach LaVine Battles the Bulls’ Analytics Department 295
Conclusion 296
Excel Calculations 296
Chapter 39 Why Have Sacrifice Bunts and Intentional Walks Nearly Disappeared? 297
The Case Against Bunting 298
Bunting Against the Shift 299
Why are Intentional Walks on the Decline? 300
Chapter 40 Do NFL Teams Pass Too Much and Go for It Often Enough on Fourth Down? 301
The Ascent of Passing 301
Fourth Down Strategy 303
New Data Partially Vindicates the Coaches 304
Teams Should Go for Two More Often 306
Chapter 41 What Caused the 1854 London Cholera Outbreak? 307
Cholera 307
Snow and the Broad Street Pump 308
Snow’s Randomized Controlled Trial 310
Conclusion 311
Excel Calculations 312
Chapter 42 What Affects the Sales of a Retail Product? 313
Painter’s Tape 313
Estimating the Model Parameters 315
Excel Calculations 316
Chapter 43 Why Does the Pareto Principle Explain So Many Things? 319
Power Laws 320
Why Do Incomes Follow the Pareto Principle? 322
Why Do a Few Websites Get Most of the Hits? 323
Excel Calculations 324
Chapter 44 Does Where You Grow Up Matter? 327
Quasi-Experimental Design vs. Randomized Controlled Trials 328
What Drives Neighborhood Differences in Upward Mobility? 329
How Can We Make Things Better? 330
Chapter 45 The Waiting is the Hardest Part 333
Which Factors Influence the Performance of a Queueing System? 334
Operating Characteristics of a Queueing System 334
How Does Variability Degrade the Performance of a Queueing System? 335
Calculating the Operating Characteristics of a Queueing System 336
Excel Calculations 338
Chapter 46 are Roundabouts a Good Idea? 339
What Is a Roundabout? 340
History of Roundabouts 340
Benefi ts of Roundabouts 341
Disadvantages of Roundabouts 343
Roundabout Capacity 344
Roundabouts and Revolutions 345
Chapter 47 Red Light, Green Light, or No Light? 347
What Causes Traffic Jams? 347
How Should We Set the Lights? 348
Ramp Meters and Equity 349
Measuring the Impact of Ramp Meters 350
The Twin Cities Metering Holiday 350
Part IV How Do I Make Good Things Happen? 351
Chapter 48 How Can We Improve K–12 Education? 353
Tennessee’s STAR Study on K–2 Class Size 355
Cost–Benefi t Analysis 356
Can Predictive Analytics Increase Enrollment and Performance in Eighth-Grade Algebra I? 360
Excel Calculations 360
Chapter 49 Can A/B Testing Improve My Website’s Performance? 363
Improving Obama’s Fundraising in 2008 364
The Mechanics of Resampling 365
Excel Calculations 366
Chapter 50 How Should I Allocate My Retirement Portfolio? 369
The Basic Portfolio Optimization Model 369
The Effi cient Frontier 372
Diffi culties in Implementing the Markowitz Model 374
Excel Calculations 374
Chapter 51 How Do Hedge Funds Work? 377
Growth in Hedge Funds and Hedge Fund Fee Structure 378
Shorting a Stock 378
Long/Short and Market-Neutral Strategies 378
Convertible Arbitrage 380
Merger Arbitrage 382
Global Macro Strategy 383
Hedge Fund Performance 384
The George Costanza Portfolio 384
Excel Calculations 385
Chapter 52 How Much Should We Order and When Should We Order? 389
The Economic Order Quantity Model 389
Reorder Points, Service Levels, and Safety Stock 392
Excel Calculations 393
Chapter 53 How Does the UPS Driver Know the Order to Deliver Packages? 397
Why Is the Traveling Salesperson Problem So Hard? 398
Solving the Traveling Salesperson Problem 399
The Traveling Salesperson Problem in the Real World 400
Excel Calculations 401
Chapter 54 Can Data Win a Presidential Election? 405
Democratic Presidential Analytics 405
The GOP Strikes Back 409
Cambridge Analytica and the 2016 Election 411
Excel Calculations 412
Chapter 55 Can Analytics Save Our Republic? 415
Arrow’s Impossibility Theorem 416
It’s Not Easy to Pick a Winner! 417
Ranked-Choice Voting 419
Approval Voting 420
Quadratic Voting 420
Excel Calculations 421
Chapter 56 Why Do I Pay Too Much on eBay? 423
How Many Pennies in the Jar? 423
The Importance of Asymmetric Information 424
The Winner’s Curse and Offshore Oil Leases 424
Sports Free Agents and the Winner’s Curse 425
Can You Avoid the Winner’s Curse? 425
Excel Calculations 427
Chapter 57 Can Analytics Recognize, Predict, or Write a Hit Song? 429
How Does Shazam Know What Song You are Listening To? 430
How Did Hit Song Science Know Norah Jones’s Album Would Be a Smash? 431
Can Artifi cial Intelligence Write a Good Song? 433
Chapter 58 Can an Algorithm Improve Parole Decisions? 437
An Example of Risk Scores 438
ProPublica Criticizes Risk Scores 441
Skeem and Lowenkamp and PCRA 443
Machine Learning and Parole Decisions 444
Chapter 59 How Do Baseball Teams Decide Where to Shift Fielders? 449
The Debut of the Shift 449
The Return of the Shift 450
Empirical Evidence on the Shift 452
Why Not Just Beat the Shift? 452
Excel Calculations 453
Chapter 60 Did Analytics Help the Mavericks Win the 2011 NBA Title? 457
How Can You Evaluate a Basketball Player? 457
From Player Ratings to Lineup Ratings 459
Chapter 61 Who Gets the House in the Hamptons? 463
The Basic Idea 464
What Asset Division Is Best? 465
Excel Calculations 466
Index 469
