Marshall Kirkpatrick's Blog Discovering awesome new things on the Internet since 2005.

Newsletter - About this blog - I love to Tweet - My work history on LinkedIn - Press clippings

How good are you at predicting things? Here’s my Brier Score for the week

Filed under: Knowledge Management,Uncategorized — Marshall Kirkpatrick @

HBR ran a great article about improving the forecasting abilities of teams this week, (Superforecasting: How to Upgrade Your Company’s Judgement) I highly recommend it, and one of the most interesting tools discussed was something called the Brier Score. It’s an easy way to quantify how well you are doing at accurately forecasting the outcomes of your actions.  It’s pretty simple.  I kept track of my predictions at work and home over this past week and calculated my score, I’ll be excited to see if I can improve it week over week.

I scored a .78 this week over 4 predictions.  You want to get as close to zero as possible. I was wrong about one thing and it really dinged me.

Here’s how you do it.  Write down a forecast about something you can be either right or wrong about, and a degree of confidence you have about your forecast.  For example, I predicted that I was probably going to be invited to join my wife at dinner last night after an event she’s participating in.  We’d discussed whether that would be the case, and we left it open ended – but I had a 60% level of confidence that’s what was going to happen.

And I was right!  So when you’re right with a 60% confidence level, you calculate your score like this: (.60-1)^2 = .16

Now I also predicted this week that a certain woman I admire a lot on the internet was going to be lukewarm about a suggestion we collaborate on a project.  In part just to experiment, I gave that prediction a 70% probability!

And I was wrong!  She was pretty open to it and we’re doing a little experiment together that’s super cool.   I’m really glad I was wrong – but that dinged my Brier Score badly.  When you’re wrong with a 70% confidence level it’s (.70-0)^2=.49.  And we’re looking for as close to zero as possible.  Ouch.

So this week I tracked 4 predictions with confidence levels ranging from 60% to 80% and I was right about the other two, so I added them up and my total score for the week was .78.  We’ll see if I can get it below that next week.

I gave myself some feedback on them where I could, and next week I’m going to think a little harder before committing to predictions.  I’d like to see if there are variations of the Brier Score, or if I should adapt it, to take into consideration the significance of the predictions.  Some of the things I made forecasts this week were much more important than others.

A few other thoughts:

  • Putting more thought into predictions so I’m more confident in them will make my score better when I’m right.
  • Without some normalization, every prediction you make impacts your score negatively.  I want to be thoughtful and keep track of many things throughout the week, so maybe I should say my score was .195 across 4 predictions.
  • There’s more to this but I haven’t drank enough coffee this Saturday morning yet to go much more in depth
  • The HBR article suggested you do this kind of thing with groups of people and figure out who’s best at forecasting.  It also suggested that groups collaborate and receive as little as an hour of structured training on avoiding faulty thinking patterns.  The authors found that those conditions dramatically improve success.
  • I love models like this – they are so powerful and useful!

Social listening tips for B2B marketers

Filed under: Uncategorized — Marshall Kirkpatrick @

I’ve been nominated for the awesome B2B News Network’s Social Listening Influencers Index (nominate someone you love, I swear I don’t know who nominated me but now I love them). When asked questions by the press, there’s always an opportunity for a blog post, too!  (See Dave Winer’s Why I Don’t Do Interviews)

Here are my answers to the questions they asked, I hope you find them useful and check out B2B News Network – they’re a good site.  And I don’t say that just because they gave Little Bird a glowing review!

  1. How can professionals become better social listeners?

A few tips from my experience:

Find credible experts on social media and listen to what they talk about, don’t just listen for the mention of your brand’s own name.

Second, try making a habit of reading 3 or 5 messages in your stream for every 1 you post.

Third, turn on mobile push notifications for selected influencers on Twitter – you’ll really get to know them and the cutting edge of your market that way!

  1. How did you become proficient at social listening? 

The method I’ve developed over the years is: (1) oversubscribe, (2) then create a filtered feed of just the highest-priority sources. Then (3) create a system that makes it easy to see almost everything from the high priority list (mobile push notifications, browser bookmarks). Then (4) dip into the general non-priority feed from time to time to see what serendipity delivers.  Repeat! Engage!

  1. What’s different about listening for brands vs b2b? 

There’s a lot of overlap, but in B2B – a single conversation can include insight that informs a company’s whole strategy or opens up a big line of business.  A mention is less likely to drive a lot of direct sales (I don’t know how much that really happens in B2C either, directly) but is more likely to help inform would-be-buyers when they see favorable mentions showing up in search.  So listen, engage, and earn supportive engagement in return.  Also, B2B influencers may be more interested in two way dialogue, whereas B2C listening is more closely associated with immediately transactional relationships.

  1. How does social intelligence help YOUR business? What do you use it for? 

We use social intelligence to look for opportunities; opportunities to find new customers, opportunities to be informed early about important trends, and opportunities to help the customers we already have with their efforts.  By systematically watching the social web for a combination of source-based, validated, popular, and keyword-laden conversations, we discover all kinds of opportunities.

  1. How is social intelligence related to content marketing?

Social intelligence surfaces opportunities to curate content and build relationships upstream and downstream.  It surfaces inspiration of topics, trends, keywords, and collaborators for all kinds of content marketing efforts.

  1. What is the best way to reach/follow you? (Twitter handle, email, etc)

@Marshallk, marshall@getlittlebird.com and 1-503-703-1815

 

How’s that look to you? Disagree with any of the above? Thanks for stopping by!

Powered by WordPress