Former quant Bob Henderson tells a cautionary tale about how real life does not always hew to even the most well-constructed models. From Pocket:
“The point of the story,” Rebonato continued, “is that you always come to data with a structural model behind”—meaning some preconception of causes and effects, and therefore some prejudice for how to interpret the data.
I understood immediately, having spent countless hours over many years struggling to extract probabilities and other forward-looking metrics from historical data. The biggest challenge was always that the answers depended so much on which timeframe you focused on. The past month? Year? Five years? And if you want to use data to estimate the probability of a very rare event, then you need a lot of data—the more rare, the more data—which means taking data from further back in the past, and the further back you go, the less relevant it probably is.
But what’s the alternative to estimating probabilities? If we can’t say anything about what the probabilities are, then we really are forced to worry about the absolute worst possible scenario. A trader would never sell options or short stocks, since his losses could be infinite. People would avoid cars, planes, marriages … everything really, for fear of the worst possible result. We’d all be paralyzed.
Which is why Rebonato told me that professed ignorance of probabilities is “a form of tyranny that’s an excuse for inaction.” And why his favored approach to stress testing (a technique called Bayesian nets) doesn’t involve trying to derive probabilities at all, but rather asks the user to just “produce your best-informed guess.”