7 March 2021

Predictions

“I told you so”. Even if we may not be saying it out loud, we don’t think it any less. That phrase, or its variant, “I knew it”, is however often easy to formulate and just as often difficult to verify. That is, unless the actual prediction was written down. I’m not interested in examining here whether there’s psychological value in telling anyone “I told you so”. Instead, I’d like to look at measuring myself in my ability to make predictions, a topic which got me interested after reading one of Scott Alexander’s posts on that very topic.

To be clear, I’m not trying to boil the ocean in how I make those predictions: I don’t do any specific research and instead merely use my instinct to estimate the probability of something happening by a certain date. Topics cover the whole gamut of what you could imagine: personal health (e.g. weight below a certain threshold), family (e.g. brother winning X lawsuits against his ex-wife), stock market (e.g. bitcoin below a certain value), projects (e.g. a particular project above specific traffic levels), job (e.g. likelihood to still work at Google by end of year), world affairs (e.g. China invading Taiwan).

I’m not trying to prove anything to anyone. I’m simply curious to see how good, or how bad, my prediction gauge is. One way of calculating that gauge is called the Brier score: in mathematical terms, it’s simply the mean squared error. So a prediction with 50% probability will always end up with a Brier score of 0.25 (whether the prediction verifies itself i.e. (1-50%)^2 or not i.e. (0-50%)^2). So I really hope to do better than a coin flip but we shall see. To make it easier to understand, I avoid making predictions with probabilities below 50%: instead, I write the contrary of that prediction with a probability of 100% minus the initial probability. In mathematical terms, I avoid making prediction X with a probability of P if P is below 50%, but instead write it as [-X] with a probability of [100%-P].

Predictions are not targets nor personal goals. They are what I think will realistically happen, usually by the end of the year. I may find as much fun in making additional predictions expiring by the following month, but my initial intention was to write down those year-long predictions and not think about them until I’d verify what happened a year later, and then calculate my Brier score. Because they’re written down in an online document, I cannot tamper the timestamp of when I made each prediction!

I was not completely surprised to discover marketplaces where money can be gambled on similar predictions. Have a look at PredicIt or Metaculus if you’re curious. Actually, for those platforms where real money is traded, it could provide a good indication of what people are thinking.

Before you think that predictions on “important” topics by non-experts are worthless, let me share an excerpt from David Epstein’s book Range: Why Generalists Triumph in a Specialized World:

“Each year, every bank [among the most prestigious international banks] predicted the end-of-year exchange rate. Gigerenzer’s simple conclusion about those projections, from some of the world’s most prominent specialists: “Forecasts of dollar-euro exchange rates are worthless.” In six of the ten years, the true exchange rate fell outside the entire range of all twenty-two bank forecasts.”

Other experiments have been likewise conducted, like the Good Judgement Project. In that case, predictions were made every day for 4 years by various research teams. The questions were tough. What is the chance that a member will withdraw from the European Union by a target date? Will the Nikkei close above 9,500? What is the likelihood of a naval clash claiming more than ten lives in the East China Sea? One of those teams was made up of people selected not because of their particular expertise but because of reduced cognitive bias. That team performed 30% better than intelligence officers with access to actual classified information.

In light of recent events at work, it also dawned on me that predictions could also be used as one way to partially demonstrate some claims. “Mark my words” is what I recently wrote to an HR representative. I should have also written those predictions in a document, with access enabled by a certain date or after I leave. I estimate the probability of me actually doing this by the end of the year at 20%.

PS: for what it’s worth, I thought about writing this article while at my basketball workout.