I am at the San Francisco airport headed home and super excited about the Twitter/Mediasift announcement today. I posted a couple of cryptic tweets that I thought I’d flesh out a little bit here, inspired by the news.

* I believe we’ll look back at these days when social media search is primarily for brand monitoring like geocities “under construction”

By this I mean that there are so many more things to search for on the social web than just brand reputation monitoring – people talking about you. That’s clearly what most people are interested in today but I think in the future we’ll see that there’s far more value to glean from discovery of communities of interest, the nature of their interest, patterns and correlations, changing connections over time, weak bonds and weak signals, early hints and wise ruminations. Maybe thinking the world will find wise ruminations of value is a stretch. 😉

But my point is, there are so many more opportunities online beyond covering-your-ass that I think someday the focus on that will seem silly. Give me an introduction to a new voice articulating where the world might be tomorrow and I’ll trade a chance to hear ten utterances by people about me today. Right?

* I believe that someday soon, creation of keyword lists for group discovery will be a hot part of our industry

Finding or ranking influencers is getting hot already, but determining what keywords can help outline a relevant community of interest when found in their discussion remains a big unmet challenge. There’s too much content being produced every moment and producers are so disinterested in structuring their own communication that the creation or discovery of the structure of a community and its communication is most likely to happen from the outside in, I think. At least in large part.

What language do people use that designates them as belonging to a particular community? Lately I’ve been hunting for Twitter Lists of employees of certain companies. Sometimes those lists are called Team, Staff or Work. Sometimes not. In less structured environments, what language gets used casually that can be used to draw a line between a group being watched vs not, between people relevant to a particular query or not.

That Mediasift allows developers to filter the Twitter fire hose for the presence of 10k keywords for the entry level price makes me picture a query like “who are the community of people who use these 10k keywords on Twitter?”. Creation of that list of keywords, to discover a small group inside a group of hundreds of millions of Twitter users just by the language they use, that’s a technology opportunity if you ask me.

* I believe someday NLP (natural language processing) parsing of business from social messages will make both and the stream much more valuable

How much personal communication is appropriate in a business context? That’s what people always want to know in a new medium. Both are valuable and important though. I think someday we’ll have technologies that will be able to tell the difference and give me control over how to view it all. Let’s say I just discover someone who is business relevant to me. I my say “show me a profile of their interests and circumstances on a personal level, but interrupt me during working hours if they post business-relevant messages.”. Or vice-versa.

Such analysis would mke the personal more valuable because we could appreciate it. It would make the business more valuable because the signal to noise ratio would change and allow us to capture more of the must-read content, depending on the nature of our relationship. Both put separately and summarized would mke awesome context.

Plane boarding, just some initial articulation of some thoughts!

Related: See my recent RWW post on How to Track the Future of the Music Industry (Or Almost Anything Else)

Time Magazine posted a list of 140 of their favorite Twitter accounts to follow, on a variety of topics. They didn’t offer it as one Twitter List you could follow, though. They said they would make it available as one the next day, but that didn’t make much sense to me and I haven’t seen it yet. So I asked Fancy Hands to assemble a list like that. Here it is: Time’s Top 140 People on Twitter.

I had bigger ambitions based on some robots in the cyborg womb, but it appears they’re not ready to emerge yet. More on that later.

Evan Williams announced his departure from Twitter tonight and though I’ve only spoken to him for about ten seconds ever, he’s profoundly changed my life for the better. I thought now might be a good time to thank him, publicly, since I know I’m not alone in feeling this way. I’ve also got a thing or two I’d like to say about the situation.

For those unfamiliar with the details, Mr. Williams was a co-founder of both Blogger.com and Twitter. He is a founding father of both blogging and what some people call micro-blogging.

In case you don’t know me, I blog for a living. I get a lot of the story ideas that I blog about from Twitter. I get a lot of my readership from Twitter. That’s just the most direct and personal way that Evan Williams has effected my personal life and the lives of my family members. There are many more.

It wasn’t so long ago that the media available to any of us was very limited. That’s no longer the case. Williams has been vital to the creation of this post-scarcity media world and to a new world of choices, learning, support for previously silenced or marginalized voices and much more. It’s incredible.

I’m very fortunate to be able to make a living thanks to blogging and Twitter – I wrote 3 years ago that Twitter is paying my rent, only to revise that post later to say Twitter was helping pay my first mortgage. From story discovery to access to experts to community collaboration with things like copy editing, Twitter is great for journalism.

If you’ll forgive me for pointing it out, it’s said that there’s no media organization today that gets more value out of Twitter than Mashable and Mashable learned to use Twitter for journalism from me.

I thanked Williams for all that in the only conversation I’ve ever had with him. I was walking into a bathroom while he was walking out, at some tech conference. I stopped him in the doorway, introduced myself and told him I wanted to thank him – that Twitter had been paying my rent for some time. And that it had changed my life in other ways as well. He said something like “cool, nice to meet you” and that’s the last we’ve ever spoken. That’s fine.

Gmail says I’ve had 63 threads with Twitter’s 3 primary communications people. I’ve only had one with Williams. Two years ago I wrote a blog post titled How Twitter’s Staff Uses Twitter (And Why It Could Cause Problems). It was about how Twitter’s leadership didn’t use their own service very much – they didn’t tweet very often, they didn’t follow very many people and they didn’t follow the top developers in the ecosystem. I argued that an abscense of Twitter power users on Twitter’s own staff could lead the company to make decisions that made life less wonderful for power users and developers. Williams told me by email that my concerns were unfounded. Two years later I think those concerns have been validated, though thankfully less than they might have been. We freakish Twitter users have mostly been left alone. (I believe I’ve been right about that, but I’ve certainly been wrong about other things – like the #fixreplies controversy. I was wrong about that.)

At ReadWriteWeb, we write more blog posts before lunch than Williams has written in the last two years. Blogging is decentralized and no decisions Blogger.com could make today would slow us down elsewhere. Twitter, on the other hand, is not decentralized.

But I love Twitter. I love the clean URL structure and public nature of the data. I love the way it lets me find and curate lists of people around a common topic, like people who work in a particular field or at a particular company. I love the way I can subscribe to those lists in interfaces like Tweetdeck or Flipboard.

I’m following almost nine thousand people on Twitter, but it’s just serendipty that leads me to read messages from most of them. A select few hundred I get in a column in Tweetdeck that pops-up high-priority messages whenever that column gets a new Tweet. So I watch news-makers closely, I develop close relationships with tech analysts by replying to their column quickly and I see inside the minds of companies of interest by putting their staff lists in columns.

A few weeks ago I told a man who worked at a big company about watching Foursquare employees chatter in a column on Twitter. He said, “I wish my competitors talked so freely on Twitter, but I’m sure they don’t.” I sent an email to the virtual assistant program FancyHands and asked them to look up engineers at said company on LinkedIn, then search for their names on Twitter and send me those usernames. I got enough back that I was able see what lists they had been put on and find more. I then put together all those people on a list and sent it to my new friend – saying “your competitor’s engineers do chatter publicly on Twitter, here’s one link you can click to subscribe to them all in one stream.” Thanks, Twitter.

Several months ago I was attending the Techonomy conference, where wildly innovative people from around the world gathered to discuss using technology to solve big problems. I scraped their names off the conference web site, then paid Amazon’s Mechanical Turk $50 to find each person’s Twitter name, website URL, to determine their gender and whether they lived inside or outside the United States. With that information, I was able to assemble a Twitter List of International Women of Techonomy. It’s a great list to read in Flipboard.

I’ve used the tool Needlebase to discover patterns and benchmarks in social media activity by leading corporate practitioners around the world by scraping Twitter lists, and to scrape and map people who’ve been listed by other users as Journalists who have more than 2,000 followers and live in the South Eastern US. I used Needlebase to scrape the messages and locations of thousands of Tweets from Twitter staff members and find one needle in the haystack that indicated yes, the rumor that Twitter was opening a data center in Utah may have been correct, since an engineer Tweeted that he had begun work that day and geotagged the tweet from the area.

Those are just a few of the things I’ve been able to do thanks to Twitter’s clean URL structure and publicly accessible data. They say that Twitter offers a whole lot of value to those who mostly listen, not just for heavy tweeters. I hope my robots and I will always be welcome to listen in the ways we like to. There are more things we do inside ReadWriteWeb that I can’t discuss publicly and I’ve got some big ideas for things to do with Twitter in my head for the future.

I love Twitter. And I love blogging. I am amazed that Evan Williams has played such a vital role in the creation of both super-disruptive technologies. They are super-disruptive, too! Anyone can now publish at will! Instant global distribution – for free!

Publishing is now interlinked with trackbacks, comments, RSS feeds, @ replies, Twitter lists, profile URLs and so much more structure that lets us analyze it and discover new things! These are world-shaking technologies. Williams has paved the way, and for those of us who wanted to do crazy things with it – he and the company have mostly stayed out of the way.

Thanks for everything, Mr. Williams. Good luck with your next project. Millions of us are very excited to see what it will be. I hope it will be as hackable as the ones you’ve changed the world with so far have been.

Also, it would be cool if you’d follow me on Twitter.

My PR buddy Julie Wohlberg asked me tonight if I knew any good journalists she should invite to a social media conference in Florida called SheCon. ReadWriteWeb’s Sarah Perez in Tampa was apparently inaccessible, so I took a few minutes to explore some possibilities…with web applications! I used the DIY data extraction and normalization service Needlebase, along with Twitter list search engine Tlists and of course Twitter itself to discover a list of journalists in the South of the US who have more than 2000 Twitter followers. (Mapped above) It wasn’t hard to do at all! Here they are in one Twitter list you can follow: Top South East Journalists

What I did was use Tlists to find 3 Twitter lists of 500 journalists (this one, this one and this one), curated by people I was familiar with. Then I used Needlebase to scrape the usernames, locations, number of followers and number of tweets (just for fun) of all the people on those three lists. Then I told Needlebase to exclude anyone with fewer than 2000 followers and show me the remaining ones on a map, grouped by the State they were in. Then I copied their usernames into a text file and sent it to Julie. It was super fun, similar in many ways to this post I put up here last week about the Twitter and LinkedIn habits of Corporate Social Strategists.

This method isn’t complete until you say: hey everyone reading this post, if I missed anyone – let me know and we’ll add you to the list! Machines can work pretty fast, in this case I was slowed down by making a video and talking about it but was able to do all this in about an hour. That’s awesome, but it’s not perfect until there’s a touch of human follow-up too.

How does one do such things at all though? How about I show you a screencast? Hooray! Below are links to 3 videos demonstrating how I did all that. Jing, the free service I used, is limited to 5 minute videos, so there are three and the last one ends a little abruptly. None the less, I hope that you will find them useful and will go out and scrape all kinds of wonderful things for yourself, from all kinds of web pages!

And if you’re able to, check out SheConf in May!

Video 1
Video 2
Video 3

If you watched those videos, I did that 3X with three different Twitter Lists. Yay! Fun times.

Reading up about Twitter and Tunisia today, among other things, I posted the following thought: Whatever else it does, the web’s biggest political impact comes from making it easy for anyone to learn about the world & history, right?

Not everyone agreed with me! I got some great responses, which I’ve curated in the widget below. (Incidentally, I tried using both Curated.by and Storify. Neither worked as well as I wished it would, but I preferred Storify.)

What do you think of this question? I think it makes for a very interesting little discussion! Lots of opportunity for cynicism, but I’m not sure how warranted that is. Do you really believe that learning via the web has been minimal compared to the impact of Farmville? I don’t know if I really believe that.

I don’t want to jinx it, but I’ve been keeping track of citations of my reporting in major media publications (defined as: indexed in Google News and something I am proud of being cited in) and 2010 is going out with a bang. All on ReadWriteWeb of course, not a whole lot going on here on Marshallk.com.

If you’ll indulge me just for a moment: I counted 6 in 2006, 2 in 2007, 12 in 2008, 15 in 2009 and just hit 50 in 2010, including 16 this month. Maybe my counting is just getting better, I don’t know. I am thankful every day for my job and for those of you who read and help me write the things I do. Look out, 2011, here we come.

That from a new poll of just under 100 marketing professionals, performed by the world’s leading marketing analyst firm, Forrester. (“New Media Dominates For Interactive Marketing News & Information“) It seems a little crazy, but the internet is very important to marketers, so publications that focus on it exclusively, with an eye towards the future, make sense for marketers to read.

Mashable serves marketers directly (though it’s been hours since I first wrote about Twitter’s opening the doors for marketers to request ad buys directly and no Mashable coverage yet!) – so it makes sense that they are at the top of this list. TechCrunch has made itself the center of the web universe, so it’s near if not at the top of almost every list.

ReadWriteWeb? Essential reading for interactive marketers? Cool! Thanks, folks!