Description
Book SynopsisA thought-provoking and startlingly insightful reworking of the science of prediction In Prediction Revisited: The Importance of Observation, a team of renowned experts in the field of data-driven investing delivers a ground-breaking reassessment of the delicate science of prediction for anyone who relies on data to contemplate the future. The book reveals why standard approaches to prediction based on classical statistics fail to address the complexities of social dynamics, and it provides an alternative method based on the intuitive notion of relevance. The authors describe, both conceptually and with mathematical precision, how relevance plays a central role in forming predictions from observed experience. Moreover, they propose a new and more nuanced measure of a prediction's reliability. Prediction Revisited also offers: Clarifications of commonly accepted but less commonly understood notions of statisticsInsight into the efficacy of traditional prediction models in a variety of fieldsColorful biographical sketches of some of the key prediction scientists throughout historyMutually supporting conceptual and mathematical descriptions of the key insights and methods discussed within With its strikingly fresh perspective grounded in scientific rigor, Prediction Revisited is sure to earn its place as an indispensable resource for data scientists, researchers, investors, and anyone else who aspires to predict the future from the data-driven lessons of the past.
Table of ContentsTimeline of Innovations ix
Essential Concepts xi
Preface xv
1 Introduction 1
Relevance 2
Informativeness 3
Similarity 4
Roadmap 4
2 Observing Information 7
Observing Information Conceptually 7
Central Tendency 8
Spread 9
Information Theory 10
The Strong Pull of Normality 14
A Constant of Convenience 17
Key Takeaways 18
Observing Information Mathematically 20
Average 20
Spread 21
Information Distance 24
Observing Information Applied 26
Appendix 2.1: On the Inflection Point of the Normal Distribution 32
References 39
3 Co-occurrence 41
Co-occurrence Conceptually 41
Correlation as an Information-Weighted Average of Co-occurrence 46
Pairs of Pairs 49
Across Many Attributes 50
Key Takeaways 52
Co-occurrence Mathematically 54
The Covariance Matrix 58
Co-occurrence Applied 59
References 66
4 Relevance 67
Relevance Conceptually 67
Informativeness 68
Similarity 72
Relevance and Prediction 73
How Much Have You Regressed? 74
Partial Sample Regression 76
Asymmetry 80
Sensitivity 86
Memory and Bias 87
Key Takeaways 88
Relevance Mathematically 90
Prediction 95
Equivalence to Linear Regression 97
Partial Sample Regression 100
Asymmetry 102
Relevance Applied 107
Appendix 4.1: Predicting Binary Outcomes 114
Predicting Binary Outcomes Conceptually 114
Predicting Binary Outcomes Mathematically 116
References 121
5 Fit 123
Fit Conceptually 123
Failing Gracefully 125
Why Fit Varies 126
Avoiding Bias 129
Precision 130
Focus 133
Key Takeaways 134
Fit Mathematically 136
Components of Fit 138
Precision 139
Fit Applied 143
6 Reliability 149
Reliability Conceptually 149
Key Takeaways 153
Reliability Mathematically 155
Reliability Applied 163
References 168
7 Toward Complexity 169
Toward Complexity Conceptually 169
Learning by Example 170
Expanding on Relevance 171
Key Takeaways 175
Toward Complexity Mathematically 177
Complexity Applied 183
References 183
8 Foundations of Relevance 185
Observations and Relevance: A Brief Review of the Main Insights 186
Spread 187
Co-occurrence 187
Relevance 188
Asymmetry 188
Fit and Reliability 189
Partial Sample Regression and Machine Learning Algorithms 189
Abraham de Moivre (1667–1754) 190
Pierre-Simon Laplace (1749–1827) 192
Carl Friedrich Gauss (1777–1853) 193
Francis Galton (1822–1911) 195
Karl Pearson (1857–1936) 197
Ronald Fisher (1890–1962) 199
Prasanta Chandra Mahalanobis (1893–1972) 200
Claude Shannon (1916–2001) 202
References 206
Concluding Thoughts 209
Perspective 209
Insights 210
Prescriptions 210
Index 211
