HBR ran a great article about improving the forecasting abilities of teams this week, (Superforecasting: How to Upgrade Your Company’s Judgement) I highly recommend it, and one of the most interesting tools discussed was something called the Brier Score. It’s an easy way to quantify how well you are doing at accurately forecasting the outcomes of your actions.  It’s pretty simple.  I kept track of my predictions at work and home over this past week and calculated my score, I’ll be excited to see if I can improve it week over week.

I scored a .78 this week over 4 predictions.  You want to get as close to zero as possible. I was wrong about one thing and it really dinged me.

Here’s how you do it.  Write down a forecast about something you can be either right or wrong about, and a degree of confidence you have about your forecast.  For example, I predicted that I was probably going to be invited to join my wife at dinner last night after an event she’s participating in.  We’d discussed whether that would be the case, and we left it open ended – but I had a 60% level of confidence that’s what was going to happen.

And I was right!  So when you’re right with a 60% confidence level, you calculate your score like this: (.60-1)^2 = .16

Now I also predicted this week that a certain woman I admire a lot on the internet was going to be lukewarm about a suggestion we collaborate on a project.  In part just to experiment, I gave that prediction a 70% probability!

And I was wrong!  She was pretty open to it and we’re doing a little experiment together that’s super cool.   I’m really glad I was wrong – but that dinged my Brier Score badly.  When you’re wrong with a 70% confidence level it’s (.70-0)^2=.49.  And we’re looking for as close to zero as possible.  Ouch.

So this week I tracked 4 predictions with confidence levels ranging from 60% to 80% and I was right about the other two, so I added them up and my total score for the week was .78.  We’ll see if I can get it below that next week.

I gave myself some feedback on them where I could, and next week I’m going to think a little harder before committing to predictions.  I’d like to see if there are variations of the Brier Score, or if I should adapt it, to take into consideration the significance of the predictions.  Some of the things I made forecasts this week were much more important than others.

A few other thoughts:

• Putting more thought into predictions so I’m more confident in them will make my score better when I’m right.
• Without some normalization, every prediction you make impacts your score negatively.  I want to be thoughtful and keep track of many things throughout the week, so maybe I should say my score was .195 across 4 predictions.
• There’s more to this but I haven’t drank enough coffee this Saturday morning yet to go much more in depth
• The HBR article suggested you do this kind of thing with groups of people and figure out who’s best at forecasting.  It also suggested that groups collaborate and receive as little as an hour of structured training on avoiding faulty thinking patterns.  The authors found that those conditions dramatically improve success.
• I love models like this – they are so powerful and useful!