Description
Book SynopsisBUSINESS EXPERIMENTS with R A unique text that simplifies experimental business design and is dedicated to the R language Business Experiments with R offers a guide to, and explores the fundamentals of experimental business designs. The book fills a gap in the literature to provide a text on the topic of business statistics that addresses issues such as small samples, lack of normality, and data confounding. The authora noted expert on the topicputs the focus on the A/B tests (and their variants) that are widely used in industry, but not typically covered in business statistics textbooks. The text contains the tools needed to design and analyze two-treatment experiments (i.e., A/B tests) to answer business questions. The author highlights the strategic and technical issues involved in designing experiments that will truly affect organizations. The book then builds on the foundation in Part I and expands the multivariable testing. Since today's companies are using experiments to solve a
Table of ContentsPreface xiii
Acknowledgments xvii
Bruce McCullough xix
About the Companion Website xxi
1 Why Experiment? 1
1.1 Case: Life Expectancy and Newspapers 2
Exercises 6
1.2 Case: Credit Card Defaults 7
1.2.1 Lurking Variables 9
1.2.2 Sample Selection Bias 11
Exercises 13
1.3 Case: Salk Polio Vaccine Trials 14
Exercises 17
1.4 What Is a Business Experiment? 17
1.4.1 Four Steps of an Experiment 21
1.4.2 Big Three of Causality 22
1.4.3 Most Experiments Fail 23
Exercises 24
1.5 Improving Website Designs 24
Exercises 30
1.6 A Brief History of Experiments 31
1.7 Chapter Exercises 34
1.8 Learning More 34
1.9 Statistics Refresher 38
2 Analyzing A/B Tests: Basics 43
2.1 Case: Improving Response to Sales Calls (Two-Sample Test of Means) 44
2.1.1 Initial Analysis and Visualization 45
2.1.2 Confidence Interval for Difference Between Means 47
2.1.3 Reporting Results 52
2.1.4 Hypothesis Test for Comparing Means 52
2.1.5 Power and Sample Size for Tests of Difference of Means 56
2.1.6 Considering Costs 60
Exercises 62
2.2 Case: Email Response Test (Two-Sample Test of Proportions) 64
2.2.1 Confidence Interval and Hypothesis Test for Comparing Two Proportions 66
2.2.2 Better Confidence Intervals for Comparing Two Proportions 66
2.2.3 Power and Sample Size for Tests of Difference of Two Proportions 68
Exercises 70
2.3 Case: Comparing Landing Pages (Two-Sample Test of Means, Again) 71
Exercises 74
2.4 Case: Display Ad Clickthrough Rate 75
2.4.1 Beta-Binomial Model 75
2.4.2 Comparing Two Proportions Using the Beta-Binomial Model 78
Exercises 80
2.5 Case: Hotel Ad Test 81
2.5.1 Tips on Presenting Experimental Findings 83
Exercises 84
2.6 Chapter Exercises 84
2.7 Learning More 86
3 Designing A/B Tests with Large Samples 91
3.1 The Average Treatment Effect 92
Exercises 93
3.2 Internal and External Validity 93
3.2.1 Threats to Internal Validity 93
3.2.2 Threats to External Validity 95
Exercises 96
3.3 Designing Conclusive Experiments 96
Exercises 102
3.4 The Lady Tasting Tea 103
Exercises 103
3.5 Testing a New Checkout Button 104
Exercises 104
3.6 Chapter Exercises 104
3.7 Learning More 104
4 Analyzing A/B Tests: Advanced Techniques 107
4.1 Case: Audio/Video Test Reprise (One-Sided Tests) 108
4.1.1 One-Sided Confidence Intervals 109
4.1.2 One-Sided Power 112
Exercises 113
4.2 Case: Typing Test (Paired t-Test) 114
4.2.1 Matched Pairs 115
Exercises 119
4.3 A/B/n Tests 121
Exercises 126
4.4 Minimum Detectable Effect 126
Exercises 128
4.5 Subgroup Analysis 129
4.5.1 Deficiencies of Subgroup Analysis 132
4.5.2 Subgroup Analysis of Bank Data 133
Exercises 135
4.6 Simpson’s Paradox 136
4.6.1 Sex Discrimination at UC Berkeley 137
4.6.2 Do You Want Kidney Stone Treatment A or Treatment B? 138
4.6.3 When the Subgroup Is Misleading 140
Exercises 143
4.7 Test and Roll 143
Exercises 145
4.8 Chapter Exercises 146
4.9 Learning More 146
4.10 Appendix on One-Sided CIs, Tests, and Sample Sizes 151
5 Designing Tests with Small Samples 159
5.1 Case: Call Center Scripts (ANOVA) 160
5.1.1 Blocking 161
Exercises 165
5.2 Case: Facebook Geo-Testing (Latin Square Design) 166
5.2.1 More on Latin Square Designs 169
5.2.2 Latin Squares and Degrees of Freedom 172
Exercises 175
5.3 Dealing with Covariate Imbalance 177
5.3.1 Matching 179
5.3.2 Rerandomization 180
5.3.3 Propensity Score 181
5.3.4 Optimal Matching 184
5.3.5 Sophisticated Matching: Selling Slushies 184
Exercises 185
5.4 Chapter Exercises 187
5.5 Learning More 188
6 Analyzing Designs via Regression 193
6.1 Experiments and Linear Regression 193
Exercises 198
6.2 Dummies, Effect Coding, and Orthogonality 198
Exercises 203
6.3 Case: Loan Experiment Revisited (Interactions) 203
6.3.1 Interactions 203
6.3.2 Loan Experiment 208
Exercises 215
6.4 Case: Direct Mail (Three-Way Interactions) 215
Exercises 224
6.5 Pretreatment Covariates in Regression 224
Exercises 225
6.6 Chapter Exercises 226
6.7 Learning More 228
6.8 Appendix: The Covariance Matrix of the Regression Coefficients 233
7 Two-Level Full Factorial Experiments 237
7.1 Case: The Postcard Example 238
Exercises 247
7.2 Case: Email Campaign 247
Exercises 250
7.3 The Determinant of a Matrix 252
Exercises 257
7.4 Aliasing 258
Exercises 264
7.5 Blocking (Again) 265
Exercises 269
7.6 Mee’s Blunders 269
7.7 Chapter Exercises 270
7.8 Learning More 271
7.9 Appendix on aliasMatrix and colorMap 273
8 Two-Level Screening Designs 279
8.1 Preliminaries 280
Exercises 286
8.2 Case: Puncture Resistance (Small Screening Experiment) 287
Exercises 288
8.3 Case: College Giving (Big Screening Experiment) 289
Exercises 292
8.4 How to Set Up a Screening Experiment 294
Exercises 295
8.5 Creating a Screening Design 295
Exercises 298
8.6 Chapter Exercises 299
8.7 Learning More 300
9 Custom Design of Experiments 305
9.1 Case: Selling Used Cars at Auction I (Small Custom Screening) 306
9.1.1 Create the Design 307
9.1.2 Evaluate the Design 312
9.1.3 Use the Design 316
Exercises 319
9.2 Case: Selling Used Cars at Auction II (Custom Experiment) 319
Exercises 322
9.3 Custom Experiment with Blocking 322
Exercises 324
9.4 Custom Screening Experiments 326
Exercises 331
9.5 More Than Two Levels 332
Exercises 337
9.6 Chapter Exercises 338
9.7 Learning More 338
10 Epilogue 341
10.1 The Sequential Nature of Experimentation 342
10.2 Approaches to Sequential Experimentation 345
References 347
Index 357
