Of course the best of both worlds are those magical people who are strong in both. People like Nathan Yau, Andrew Vande Moere, Benjamin Wiederkehr, Ben Fry, Jer Thorp and Martin Wattenberg. Then there’s emergent voices online like Cole Nussbaumer, Enrico Bertini and Bryan Connor. See also the new podcast Data Stories. I don’t know all of these people personally, I just discovered them as part of the output of our startup Plexus Engine. Speaking of Plexus, if you’re into data and design – and if you’d like to live in beautiful Portland, Oregon – you should send me an email so we can talk. If you don’t want to move but are interested in rocking the future of the internet with us, you should still get in touch. We’re building something incredible and we’re looking for people who want to build it with us.

• Programatically look at a list of hundreds of webpage URLs and determine what their RSS feed URLs are. All the methods we’ve tried break or miss feeds.
• Send a feed to a feed publishing service like Feedburner and have it cache non-live items in the feed it publishes.
• Build spaghetti-ball messes of ornate processes with lots of RSS feeds without the apps using them timing out.

Anybody know good, scalable solutions to any of these problems?

But Friday afternoon I was at the Fubonn Mall and was surprised to find out that there was no Wikipedia entry for the place! It calls itself the largest Asian mall in Oregon and that certainly seems of encyclopedic significance to me.

Unfortunately, I did not stop and snap a picture of the mall’s giant round front window when I thought about it. I would come to regret that later.

That night I sat in my living room and decided I would create a Wikipedia entry for Fubonn.

It was a lot of fun. I logged in to an account I created years ago but hadn’t used much and I created the new entry. I looked at entries for other businesses and malls around town, clicked the edit button, copied the Wiki code for all the formatting, pasted it into my entry and then changed it to refer to Fubonn. I Googled around, especially in the Google News Archives, and found information to include in the entry. It really enjoyed it. Here it is, a living breathing Wikipedia page about Fubonn.

I saw that other posts had photos of businesses included photos and I wanted to include one of the big front window of Fubonn. There weren’t any I could find in the Creative Commons section of Flickr so I grabbed one from the Fubonn website. Wikipedia seemed to indicate that if a photo was used in an advertisement and I couldn’t reasonably get a photo in another way, then I could use that photo in an entry. Turns out I misread that. My photo got deleted and a long paragraph of confusing explanation was sent my way. The short version: I need to go back to the mall and take my own photo of the front door. Silly, if you ask me – but that’s the rules.

One long-time editor visited my entry, fixed some links, removed a “see also” link to the Fubonn Foursquare page (turns out you’re only allowed to link to other Wikipedia pages in the See Also section – not sure where I could link to what I consider a valuable additional resource off-site) and then placed my entry into the WikiOregon project. That project is dedicated to building out pages about Oregon. The editor classified my entry as of “start” quality (not a stub) and of “low” importance to the project.

I took issue with the low importance designation, especially when I saw some of the other entries that had been classified as of “mid” importance, and so I edited my entry’s Talk page and classified it as of Mid importance to WikiOregon. I am not sure why the page about the 1916 Oregon Ducks football team, which finished in 2nd place, would be more important than the largest Asian mall in the state and the cornerstone of the alleged move of Chinatown from downtown to South East Portland.

There were some parts of the experience that I found confusing and disappointing, but when I woke up in the morning I felt silly for having complained about that the night before on Twitter and Google Plus. This was my first major contribution to the giant sprawling, pseudo-democratic experiment that is Wikipedia. Why am I entitled to just jump in and be praised for everything I do?

New Zealand technologist Nat Torkington said of Sullivan’s experience, “having built a valuable resource, vain nano-Napoleons hide behind hostile UIs.” For what it’s worth, my experience did make me bristle – but I wouldn’t describe what I experienced that way.

The new article rating system says that 9 anonymous people have already rated my article to be of high quality and completeness, that’s cool. And the article history page shows that 4 other people have now made edits to the page. The end result is pretty good so far, I think. I wish my photo was still there, but I think I’m going to try to go back to that mall and take one of my own.

Once I got over the sting of confusing reprimands, my imagination started whirring. I imagined taking my 9 year-old niece back to Fubonn with my friend who’s a professional food photographer. I imagined setting up an interview with the mall’s owner and taking a bunch of high-quality photos to put into the public domain. Wouldn’t that be a great experience for a 9 year old to get to take part in? To collaborate in creating a part of the public record like that?

I think I would like to do something like that with my niece, but I want to think of the best place to do it for. Maybe we could find one place near her home that has an entry we could expand on with some Google searches, then maybe another place that doesn’t have an entry. We could take some photos, take a trip to the library to do some research, and create one ourselves.

I think that would be an incredibly empowering experience for a young person old enough to appreciate it. It was even for me, and I write on the web daily, to accolades from around the world.

There’s something magical about filling out a public resource about a place in the real world. The Geoloqi Wikipedia layer, as well as looking at Wikipedia entries through Google Earth, have created an expectation in my mind that every single place I go should have a Wikipedia entry available about it. I would really like that to be the case.

If Wikipedia can figure out how to welcome more and more new editors onto the site, and I don’t think coddling us is necessary, perhaps that will become reality in the future. It’s an incredibly complicated community management situation though. Danny Sullivan’s experience having his entry about an important woman in technology get deleted is super frustrating and an example of how things can go wrong.

But there’s a whole lot about that’s right about Wikipedia, too. The difference between many Wikipedia entries and old encyclopedia entries on the same topics is so substantial that it deserves to be sung about from mountain tops. (The world before Wikipedia – an honestly horrible vision)

I thought I’d write up these thoughts in hopes it could help some other people jump into the messy world of Wikipedia, too. The world will be a better place for it.

Related: Why I Love the Internet So Much (about editing OpenStreetMap)

After years of writing about startup companies, I’m now building one myself. Specifically, I’m building a company that’s developing a technology based on some of my favorite consulting projects I’ve done for clients over the years: an app and data platform that discovers emerging topical information. It’s a learning-curve busting, “first mover’s advantage” as a service, technology for information workers who want to win. It’s about helping users “skate to where the puck is going to be, not to where it’s been.”

It’s called Plexus Engine, it’s in private beta and you can sign up to be notified when it launches at PlexusEngine.com. A Plexus is a place where nerves branch and rejoin in the body and the Plexus Engine analyzes points of intersection online to detect emerging signals.

What’s it do, specifically? It’s not ready to be talked about much, but I will tell you this:

I’ve built my career as one of the web’s leading technology journalists by making strategic use of lightweight tools for processing data to gain first mover’s advantage.

I’ve also consulted for companies large and small on how to build and use new media technologies, launch products and identify potential hires and industry experts, using tools as well. That’s where Plexus Engine was born.

Now I’m building a technology for everyone to use in order to save time and derive value from the huge sea of data being published online.

Josh Dilworth, founder of Austin’s Jones-Dilworth, who’s done PR for SXSW, Siri, Wolfram Alpha and many more, says – “For years Marshall has had a leg up and now we know why. We are already using Plexus at Jones-Dilworth and it makes us look smarter every day. It’s instant domain knowledge — ideal for getting up to speed in new categories.”

Richard Snee, VP of Marketing at data warehousing company EMC Greenplum, with whom I was consulting when Plexus Engine was born, puts it this way: “For many B2B marketing professionals effective use of social media can be mysterious and frustrating. The work we did with Marshall helped create a blueprint for success in our social media efforts at EMC Greenplum.”

Sam Whitmore, editor of Sam Whitmore’s Media Survey in Oakland, CA says of Plexus: “Mining the info that this technology does, quickly and easily, is money.”

Plexus Engine is going to be especially valuable for people working in marketing and PR, but I think anyone who does business on the web is going to want to use it.

Ok, that’s the end of the short version of the story. You should go to PlexusEngine.com and sign up for beta access. I’ll let you know as soon as more information is available. You can follow @plexusengine on Twitter for updates on the company and you can follow me at @marshallk

***********************

Now, who wants to hear some cool stories about the Internet?

I’ve been learning about how to do this kind of stuff for as long as I’ve been working online. The methods I’ve explored have been complicated, experimental and challenging but now I’m going to productize the lessons I’ve learned in a way that anyone can use them.

Back when I started blogging AOL’s Weblogs Inc. I signed up to get RSS feeds from the key tech companies via SMS alerts. (Using Sameer Patel’s old startup Zaptxt.) No one else was doing that at the time and it helped me report on news before all the other tech blogs. That landed me a job as the first hired writer at TechCrunch.

When I was at TechCrunch, I used a variety of other tools to segment my inbound streams of information and broaden the range of information I could consume. (See Open Sourcing My TechCrunch Work Flow)

At ReadWriteWeb, I’ve used a wide variety of tools to mine signal from a whole lot of noise around the web. Here are a few examples of tips and tricks I’ve employed there so far that I’ve already written about before:

Delicious Data Mining

When social bookmarking service Delicious was being “sunsetted” by Yahoo, I wrote about a system we set up for mining it for streams of valuable signals.

Here’s how it worked: we went through the ReadWriteWeb archives and grabbed URLs of companies and products we’d written about before. Then we took those URLs over to Delicious and we looked up their bookmarking history. We scrolled back to the first 20 user names of people who bookmarked those links, then we copied and pasted them into a spreadsheet. Then we repeated that process 300 times or so. Finally, we sorted the spreadsheet alphabetically and found 15 people who on 5 or more occasions had bookmarked something we had later found of sufficient interest to write about. They had a proven history of finding things early – so we subscribed to an RSS feed of everything those people bookmarked in the future. That worked really well for a long time.

Needlebasing Twitter

One day we caught wind of a local Salt Lake City newspaper that ran a story about a big new data center opening in town with a mystery anchor tenant. The paper believed that the tenant was Twitter, opening its first data center outside of San Francisco – as the company said it would, in a location undisclosed. We used the (now Google-acquired) web app called Needlebase to investigate.

We grabbed the URL of the Twitter List of the staff of Twitter Inc. and we trained Needlebase’s point-and-click screen scraping tool to recognize what a user name, Tweet text and location field (when there was one) looked like on the page of staff Tweets. Then I clicked a button and said “go!”

In just a few minutes, the most recent 1125 Tweets from staff were pulled into Needlebase and we said “show ‘em on a map!” Sure enough, one Twitter network engineer had posted a Tweet with a location attached to it right across the highway from the alleged mystery data center. He’d just left San Francisco, he had Tweeted, and arrived in Salt Lake City ready to get to work.

That Tweet was quickly deleted after we reported on it. Six months later, it was reported that the Salt Lake data center efforts were plagued with all kinds of problems and got called off at great expense. (Here’s a screencast about how to use Needlebase to scrape at least the old Twitter interface, things have changed but it’s an ok intro to Needlebase.)

ReadWriteWeb is where I learned to use Twitter as a journalist and it was only slight hyperbole when I wrote four years ago that Twitter was paying my rent. (It was through my use of Twitter on ReadWriteWeb, by the way, that Mashable learned to make use of Twitter, too.)

Backtyping Your Comments Around the Web

Backtype, a startup that got swallowed up by Twitter, used to offer the coolest feature: an RSS feed for comments posted to blog posts all around the web and signed with a particular URL in the URL field.

We took Robert Scoble’s Most Influential in Tech list of Twitter users, grabbed the home page URLs from all the Twitter bios on that list, then ran those URLs through Backtype and got an RSS feed for any comments posted by the people on the list. For some people we put their feeds in an RSS to Instant Messaging alert system, so whenever Chris Messina posted a comment on any blog around the web and signed it FactoryJoe.com, I’d get an IM within 5 minutes. We got to write several stories before anyone else that way.

Unfortunately, that service doesn’t exist anymore, but it was born from the same kind of thinking as the other examples above: what new fields of data online could I gain programmatic access to, subject to some analysis and then use for strategic advantage?

That’s part of the thinking behind Plexus Engine, too.

I’ve written about lots of other ways to use publicly available data and services to derive value from the web: How to Build a Social Media Cheat Sheet on Almost Any Topic, How to Use Blekko (or any Custom Search Engine) to Rock at Your Job, How to Use Mechanical Turk to Rock at Conference Blogging and even How to Find the Weirdest Stuff on the Internet.

If those kinds of things are exciting to you, I think you’re really going to enjoy Plexus Engine. It’s going to be some internet magic, with a ribbon on top.

I think it’s going to be a must-have technology for anyone who does business on the web. I’m looking forward to showing it to you, as soon as its ready.

I’ll leave his name out of it because I’ve totally copied and pasted things he’s posted online before as the basis for acts of journalism myself!

I do take issue with the idea that the trend of bringing curated social media into other types of media is a bad idea. Here’s why, from my comment on Google+. I edited it to make it more clear.

I respectfully disagree.

1. Had you seen those tweets yourself already? Discovery, curation and contextualization of publicly available information has long been an important part of journalism.

2. If it’s random peoples’ random tweets being shared, that doesn’t sound like a value add, but there certainly is potential there for journalists to integrate multiple types of media to add value. Some Tweets are good to include, some Tweets are not. I find a lot of news on Twitter and sometimes include the tweets themselves in my reporting.

3. I would argue that journalism is expanding and you’re seeing a lot more of new types of journalism: quick hits to catch busy people up on news, curation of reports elsewhere, etc. but there’s still old-fashioned journalism being performed as well. I’m watching the Al Jazeera iPad app right now and it’s great.

I’m also working on a big article about Walmart’s mobile strategy. I’ve been working on it for a week. I’m using lots of online social media, bots, virtual assistants and hope to have 4 or 5 interviews included in my research. In the meantime, though, I’ll probably write 10 other posts for which I didn’t take the time to do interviews. All of that rolled up together = contemporary journalism. Go read some tweets, then go read some longform.org or such things.

I don’t think it’s as dismal as you think.

In fact – I think we’re making a difficult transition into a new golden age of journalism. I hope so, at least.

That said, there is a feeling of pressure to work ever faster. From a previous comment in the same conversation.

It’s hard to scale, but we honestly do try to interview people whenever we can. (I know I totally copied and pasted a comment from you awhile ago though too!) I do probably 5-7 interviews a week by phone or IM for 15 blog posts I write. I wish I could do more, but I have to rely on search and discussion with my own co-workers in most cases. I can’t spend more than 90 minutes on most of those stories and sometimes that precludes being able to connect with someone to interview. Sad but true.

Given all that – online social media is where a lot of conversation is happening and it can be incredibly valuable to news research. Sometimes that’s done well and sometimes that’s done poorly.

Is this a case of the rich getting richer, of the new media ecosystem being concentrated into the arms of a small number of voices, contrary to the interests of consumers? If this was the only way to discover new people to follow, that would be bad. It isn’t and it won’t be though. Like all things, this arrangement is part meritocracy, part democracy, part privilege and some other parts other stuff. It’s complex and there’s more to discuss about it than I can here while I’m riding down the highway on an Amtrak bus and blogging on my phone.

Is this ethically wrong? I don’t think so, but it is sticky that’s for sure. Networks of self-published content are the hot currency of the era and the ecosystem around those networks includes some of us interesting enough, culturally safe enough and commercially viable enough that we make our living publishing on the web, through RSS, to subscribers on Twitter, Facebook and Plus. It’s a beautiful thing, but the challenge will be to not get so cozy with the networks that we both cover and that deliver us this flow that we no longer serve our audiences (or whatever you people reading should be called) with an eye for critique of the network providers themselves.

I’m not on Twitter’s suggested user list but my employer is. I’ll rip into that company at a moment’s notice, publish its secrets when I discover them and just generally maintain a respectful antagonism with them despite their role in the supply chain that turns my thought into bits into (delicious Oregon microbrewed) beer in my belly.

Hopefully Plus didn’t just buy a bunch of unconditionally supportive new friends in the media. Clearly they don’t hold a grudge about my scoop of the details about how the new network would work at SXSW, despite the red-faced shouting at me at the time. I’ve also been very critical of Plus regarding the Real Names policy.

There’s room in my head though to be glad to have been picked for the pickup basketball team while also feeling like the captain of the team sometimes acts like a frat-boy a-hole. It’s a complicated situation and no one is pure and good in it. It’s the future: messy like the present and the past but hopefully a little more just and democratically empowering.

One thing’s for sure: I’ll be disclosing that I’m on the Plus suggested user list in every article I post about the network in the future. Because these days, a free pile of social network connections equal free discourse at scale, free access to answers to many of my questions and other resources that eventually translate to free money and power. And I intend to keep it free because I’m going to work hard to not pay the price of my integrity.

Thanks for your help!

There are probably other examples and as people are telling me on Twitter – it really depends on your definition of Venture Capital.

Why It’s Smart

Regardless, I think that if anyone is going to break a record on funds raised, it’s ok with me that it’s Twitter. I don’t have the time or knowledge to put together a whole post about this on ReadWriteWeb, so instead some notes here.

* Twitter has revolutionized business and public communication in a historically unprecedented way. Never before has it been so easy for anyone to publish quick updates about what they are doing and for that to be read and passed around at scale, in real-time. That’s a really, really big deal.
* Businesses are scrambling to get to Twitter’s advertising products faster than the company can deliver them.
* Twitter comes up with really smart ways to do what it does, like its latest ad product – letting brands pay to have their Tweets show up at the top of the page any time someone who already follows them visits Twitter.com. That’s brilliant.
* This whole thing is just beginning. Twitter’s just beginning, but “social media” is all the more just at its beginning. At least, if you were someone with a huge amount of money made from the old economy, and if you could afford to gamble it on what appears to be a new economy emerging, to make a very serious bet seems like a respectable strategy to me.

Half of that money, reportedly, is going to buy out the hippies that created Twitter, leaving them wealthy enough to go innovate some more, possibly kicking off a PayPal-like wave of new world-changing startups.

The rest of that money is going to go, apparently, towards making Twitter all the more solid, important and ready to be the AT&T to Facebook’s Verizon and Google Plus’s Sprint, or whatever. These could well be the communication platforms of the future though, so I don’t think it’s stupid at all to throw a whole lot of money into them. If you’ve got it. And Digital Sky Technologies, the giant Russian company that’s put comparable sums into Facebook, Zynga and other companies has it. So why not?

This really isn’t my area of expertise, though, venture capital. Maybe there’s good reason to freak out – but I haven’t heard it yet.

Oh does that make me happy. I found a list of the top 25 business web app startups or some such thing today and of course I sent the link to Fancyhands, asking them to go grab the RSS feeds for each of those companies’ blogs or press releases. They did so, like magic, then I plopped those puppies into the form on the page above. It’s a simple function to perform, but it’s a nice time saver. Now I’m ready to subscribe to them all and keep up to date with important developments in those companies. Thanks, Internet!

I don’t know why I hadn’t thought about it this way before. I hope the plan works. One more cool thing about Plus.

I’d post a link to my Plus profile here but I wrote this whole post on my phone, sitting on the sidewalk in front of my house, in the dark. (Cutting sod that’s grown over my walkway.) I’m not hard to find there though and am lots of fun to talk to, I promise.

That’s when I thought “how about I take a list of high-priority individuals, track their comments around the web and use that as a way to sniff for news?” I used Robert Scoble’s Twitter List of Most Influential People in Tech, but it could have been any list of people with home page URLs published in a public, predictable place. If I was a geotechnology beat specialist, I might have used (heck, maybe I still will) a list of geotech industry specialists.

Source of data, available programmatically, with a structured field with which some data can be filtered out from others based on some criteria, criteria data-set available from another source already. Put all of those circumstances together and you’ve got an opportunity.

That’s what I did, as described in this post on BackType’s acquisition today by Twitter. That feature is probably as good as dead, now.

I just thought I’d share that thought-process here. I think about things like this all the time, but especially when I set aside some time for my brain to think about it. You can too. Publicly available, structured data enables all kinds of strategic possibilities.

Below is a picture of my email inbox, with a search for the Fancyhands threads. Below that is a more detailed discussion of each request, but the high level take-aways seem to be: I’m not remembering to use anywhere near my full quota, many of the tasks I request aren’t working out so well but the ones that are working out have been great.

You might think: Marshall, you need to make simpler requests – those are the ones that get the best results. But you’d be surprised at the crazy requests I’ve gotten great results from in months past! Like: send me a spreadsheet of every daily newspaper in the US, its name, its location and its URL. No problem! That was great. I think it depends largely on who happens to answer my request on the other end. I’ve gotten some really sophisticated responses and some really frustrating ones.

1. Most recently I asked the Fancies to find out the name of a browser that Hilary Mason told me about and that my friend Tyler Gillies knew about too, but I couldn’t remember the name of. I said it was a browser that developer types used that let you download all the text on a page in a scrape-type dump. The reply I got back: most developers use Firefox. So…that one didn’t work out so well. I’ve gotten great replies from Fancyhands before for questions like this though.

2. I asked for an OPML file (a bundle of RSS feeds) of the blogs of the top 15 tech incubators in the US, as determined by an analyst firm whose list I sent in my email. This is unusual: it took two weeks to get the results! They worked out great though and I really appreciate it. Good thing I wasn’t in a hurry To be honest the file needed some cleaning up too before I could use it. It looked like someone had Googled for how to create an OPML file, found my own directions from this blog and then done a fairly clumsy job of it. Hey, it was almost free and I didn’t have to do it myself!

3. In this request, I had this crazy idea that someone had built a Greasemonkey script or that there might be some javascript out there that I could tweak that would let me hover over the linked-to Twitter messages now posted on Techmeme and preview them in some sort of pop-up, without having to load a whole new page. This was a fairly freaky request and sure enough, it didn’t end so well. The person who responded at Fancyhands said they sent multiple email requests for help to Twitter Headquarters! LOL. I wrote back, as calmly as I could, and asked “you didn’t mention my name in those emails, did you?” Thankfully, they said they did not. I let that thread end right there.

4. Twitter usernames for the same top 15 tech incubators mentioned above: boom – one day, perfect results. Way to go Fancies.

5. Taking dogs to Europe: what are the rules and restrictions? Quarantine required, etc? Got a real succinct, helpful answer that same evening. Someone Googled and summarized, so I didn’t have to. Worked out great.

6. Customer reviews of retirement communities, and one in particular. This basically doesn’t exist, it appears. Neither I nor the Fancies could find any good sites for this. Next week I will ask Fancyhands to build me just such a site. Kidding!

7. Twitter list of music service employees. This one worked out really well. I used this as a foundation to build one of my favorite Twitter lists and to write this blog post.

8. Please make a Twitter list out of Time Magazine’s Top Twitter Accounts to Follow. This worked out great. Same day, good results.

9. Sprint news summary. I was headed to a telecom conference and asked the Fancies to summarize top news stories over the last 6 months about Sprint. They had done a real good job on a related background research request earlier and had some hopes for this request. Unfortunately, the respondent sent me back 5 crappy links that weren’t good for much and were barely on topic. I said so in an email response and they emailed me back Google News search result URLs for 5 different search terms on the topic. Not so hot.

In summary: It looks like I got 6 solid enough responses over the last 2 months, out of 9 attempts. That means I paid a little over $10 per successful response. I can live with that, for now. I should have thought about more requests to make, as I paid for up to 70 in that time period. I just haven’t taken the time to think of ideas often enough, though. Some months past I’ve sent more requests and gotten some really dazzling responses.

I think it’s a tool I’m still learning to use. That’s been my most recent experience with the service. In case you were wondering, and I know some of you have been.

So now I will just Tweet news when it’s short – and I’ll include a link back to this post explaining why for the first few times I do that. Maybe. What do you think of this idea? It does neglect RSS readers, Facebook users, etc. There is that. If it decreases the amount of click-bait BS around the web though, that could be a net win, no?

Quick photo! Testing testing!! New WordPress iOS app!

I suppose people say it robs publishers of pageviews. Here’s the math I do in my head. If 1 million people read 10 of your articles and 5% of the time they will click through to see your site itself – that’s 500,000 free pageviews you just got. Did you have some other plan in mind that would bring those million people to your content but with a higher conversion rate? Did Flipboard cannibalize your existing plan? I suspect that’s not how it goes down – instead 1m people are exposed to your content that wouldn’t have been otherwise. That sounds win-win to me.

Who am I to say this? I do publish content for a living. Though I don’t own the business I do it for – I have led the growth of that site from a 3 person staff to now 15. Publishing tech news content. That’s where I’m coming from.

I love finding Twitter lists like a good curated collection of Anthropologists or GeoTech pros and subscribing to the links they share on Twitter inside Flipboard.

Maybe there are concerns that I’m missing. Somebody tell me the rest of the story.

* I believe we’ll look back at these days when social media search is primarily for brand monitoring like geocities “under construction”

By this I mean that there are so many more things to search for on the social web than just brand reputation monitoring – people talking about you. That’s clearly what most people are interested in today but I think in the future we’ll see that there’s far more value to glean from discovery of communities of interest, the nature of their interest, patterns and correlations, changing connections over time, weak bonds and weak signals, early hints and wise ruminations. Maybe thinking the world will find wise ruminations of value is a stretch.

But my point is, there are so many more opportunities online beyond covering-your-ass that I think someday the focus on that will seem silly. Give me an introduction to a new voice articulating where the world might be tomorrow and I’ll trade a chance to hear ten utterances by people about me today. Right?

* I believe that someday soon, creation of keyword lists for group discovery will be a hot part of our industry

Finding or ranking influencers is getting hot already, but determining what keywords can help outline a relevant community of interest when found in their discussion remains a big unmet challenge. There’s too much content being produced every moment and producers are so disinterested in structuring their own communication that the creation or discovery of the structure of a community and its communication is most likely to happen from the outside in, I think. At least in large part.

What language do people use that designates them as belonging to a particular community? Lately I’ve been hunting for Twitter Lists of employees of certain companies. Sometimes those lists are called Team, Staff or Work. Sometimes not. In less structured environments, what language gets used casually that can be used to draw a line between a group being watched vs not, between people relevant to a particular query or not.

That Mediasift allows developers to filter the Twitter fire hose for the presence of 10k keywords for the entry level price makes me picture a query like “who are the community of people who use these 10k keywords on Twitter?”. Creation of that list of keywords, to discover a small group inside a group of hundreds of millions of Twitter users just by the language they use, that’s a technology opportunity if you ask me.

* I believe someday NLP (natural language processing) parsing of business from social messages will make both and the stream much more valuable

How much personal communication is appropriate in a business context? That’s what people always want to know in a new medium. Both are valuable and important though. I think someday we’ll have technologies that will be able to tell the difference and give me control over how to view it all. Let’s say I just discover someone who is business relevant to me. I my say “show me a profile of their interests and circumstances on a personal level, but interrupt me during working hours if they post business-relevant messages.”. Or vice-versa.

Such analysis would mke the personal more valuable because we could appreciate it. It would make the business more valuable because the signal to noise ratio would change and allow us to capture more of the must-read content, depending on the nature of our relationship. Both put separately and summarized would mke awesome context.

Plane boarding, just some initial articulation of some thoughts!

Related: See my recent RWW post on How to Track the Future of the Music Industry (Or Almost Anything Else)

I had bigger ambitions based on some robots in the cyborg womb, but it appears they’re not ready to emerge yet. More on that later.

For those unfamiliar with the details, Mr. Williams was a co-founder of both Blogger.com and Twitter. He is a founding father of both blogging and what some people call micro-blogging.

In case you don’t know me, I blog for a living. I get a lot of the story ideas that I blog about from Twitter. I get a lot of my readership from Twitter. That’s just the most direct and personal way that Evan Williams has effected my personal life and the lives of my family members. There are many more.

It wasn’t so long ago that the media available to any of us was very limited. That’s no longer the case. Williams has been vital to the creation of this post-scarcity media world and to a new world of choices, learning, support for previously silenced or marginalized voices and much more. It’s incredible.

I’m very fortunate to be able to make a living thanks to blogging and Twitter – I wrote 3 years ago that Twitter is paying my rent, only to revise that post later to say Twitter was helping pay my first mortgage. From story discovery to access to experts to community collaboration with things like copy editing, Twitter is great for journalism.

If you’ll forgive me for pointing it out, it’s said that there’s no media organization today that gets more value out of Twitter than Mashable and Mashable learned to use Twitter for journalism from me.

I thanked Williams for all that in the only conversation I’ve ever had with him. I was walking into a bathroom while he was walking out, at some tech conference. I stopped him in the doorway, introduced myself and told him I wanted to thank him – that Twitter had been paying my rent for some time. And that it had changed my life in other ways as well. He said something like “cool, nice to meet you” and that’s the last we’ve ever spoken. That’s fine.

Gmail says I’ve had 63 threads with Twitter’s 3 primary communications people. I’ve only had one with Williams. Two years ago I wrote a blog post titled How Twitter’s Staff Uses Twitter (And Why It Could Cause Problems). It was about how Twitter’s leadership didn’t use their own service very much – they didn’t tweet very often, they didn’t follow very many people and they didn’t follow the top developers in the ecosystem. I argued that an abscense of Twitter power users on Twitter’s own staff could lead the company to make decisions that made life less wonderful for power users and developers. Williams told me by email that my concerns were unfounded. Two years later I think those concerns have been validated, though thankfully less than they might have been. We freakish Twitter users have mostly been left alone. (I believe I’ve been right about that, but I’ve certainly been wrong about other things – like the #fixreplies controversy. I was wrong about that.)

At ReadWriteWeb, we write more blog posts before lunch than Williams has written in the last two years. Blogging is decentralized and no decisions Blogger.com could make today would slow us down elsewhere. Twitter, on the other hand, is not decentralized.

But I love Twitter. I love the clean URL structure and public nature of the data. I love the way it lets me find and curate lists of people around a common topic, like people who work in a particular field or at a particular company. I love the way I can subscribe to those lists in interfaces like Tweetdeck or Flipboard.

I’m following almost nine thousand people on Twitter, but it’s just serendipty that leads me to read messages from most of them. A select few hundred I get in a column in Tweetdeck that pops-up high-priority messages whenever that column gets a new Tweet. So I watch news-makers closely, I develop close relationships with tech analysts by replying to their column quickly and I see inside the minds of companies of interest by putting their staff lists in columns.

A few weeks ago I told a man who worked at a big company about watching Foursquare employees chatter in a column on Twitter. He said, “I wish my competitors talked so freely on Twitter, but I’m sure they don’t.” I sent an email to the virtual assistant program FancyHands and asked them to look up engineers at said company on LinkedIn, then search for their names on Twitter and send me those usernames. I got enough back that I was able see what lists they had been put on and find more. I then put together all those people on a list and sent it to my new friend – saying “your competitor’s engineers do chatter publicly on Twitter, here’s one link you can click to subscribe to them all in one stream.” Thanks, Twitter.

Several months ago I was attending the Techonomy conference, where wildly innovative people from around the world gathered to discuss using technology to solve big problems. I scraped their names off the conference web site, then paid Amazon’s Mechanical Turk $50 to find each person’s Twitter name, website URL, to determine their gender and whether they lived inside or outside the United States. With that information, I was able to assemble a Twitter List of International Women of Techonomy. It’s a great list to read in Flipboard.

I’ve used the tool Needlebase to discover patterns and benchmarks in social media activity by leading corporate practitioners around the world by scraping Twitter lists, and to scrape and map people who’ve been listed by other users as Journalists who have more than 2,000 followers and live in the South Eastern US. I used Needlebase to scrape the messages and locations of thousands of Tweets from Twitter staff members and find one needle in the haystack that indicated yes, the rumor that Twitter was opening a data center in Utah may have been correct, since an engineer Tweeted that he had begun work that day and geotagged the tweet from the area.

Those are just a few of the things I’ve been able to do thanks to Twitter’s clean URL structure and publicly accessible data. They say that Twitter offers a whole lot of value to those who mostly listen, not just for heavy tweeters. I hope my robots and I will always be welcome to listen in the ways we like to. There are more things we do inside ReadWriteWeb that I can’t discuss publicly and I’ve got some big ideas for things to do with Twitter in my head for the future.

I love Twitter. And I love blogging. I am amazed that Evan Williams has played such a vital role in the creation of both super-disruptive technologies. They are super-disruptive, too! Anyone can now publish at will! Instant global distribution – for free!

Publishing is now interlinked with trackbacks, comments, RSS feeds, @ replies, Twitter lists, profile URLs and so much more structure that lets us analyze it and discover new things! These are world-shaking technologies. Williams has paved the way, and for those of us who wanted to do crazy things with it – he and the company have mostly stayed out of the way.

Thanks for everything, Mr. Williams. Good luck with your next project. Millions of us are very excited to see what it will be. I hope it will be as hackable as the ones you’ve changed the world with so far have been.

Also, it would be cool if you’d follow me on Twitter.

My PR buddy Julie Wohlberg asked me tonight if I knew any good journalists she should invite to a social media conference in Florida called SheCon. ReadWriteWeb’s Sarah Perez in Tampa was apparently inaccessible, so I took a few minutes to explore some possibilities…with web applications! I used the DIY data extraction and normalization service Needlebase, along with Twitter list search engine Tlists and of course Twitter itself to discover a list of journalists in the South of the US who have more than 2000 Twitter followers. (Mapped above) It wasn’t hard to do at all! Here they are in one Twitter list you can follow: Top South East Journalists

What I did was use Tlists to find 3 Twitter lists of 500 journalists (this one, this one and this one), curated by people I was familiar with. Then I used Needlebase to scrape the usernames, locations, number of followers and number of tweets (just for fun) of all the people on those three lists. Then I told Needlebase to exclude anyone with fewer than 2000 followers and show me the remaining ones on a map, grouped by the State they were in. Then I copied their usernames into a text file and sent it to Julie. It was super fun, similar in many ways to this post I put up here last week about the Twitter and LinkedIn habits of Corporate Social Strategists.

This method isn’t complete until you say: hey everyone reading this post, if I missed anyone – let me know and we’ll add you to the list! Machines can work pretty fast, in this case I was slowed down by making a video and talking about it but was able to do all this in about an hour. That’s awesome, but it’s not perfect until there’s a touch of human follow-up too.

How does one do such things at all though? How about I show you a screencast? Hooray! Below are links to 3 videos demonstrating how I did all that. Jing, the free service I used, is limited to 5 minute videos, so there are three and the last one ends a little abruptly. None the less, I hope that you will find them useful and will go out and scrape all kinds of wonderful things for yourself, from all kinds of web pages!

And if you’re able to, check out SheConf in May!

Video 1
Video 2
Video 3

If you watched those videos, I did that 3X with three different Twitter Lists. Yay! Fun times.

Not everyone agreed with me! I got some great responses, which I’ve curated in the widget below. (Incidentally, I tried using both Curated.by and Storify. Neither worked as well as I wished it would, but I preferred Storify.)

What do you think of this question? I think it makes for a very interesting little discussion! Lots of opportunity for cynicism, but I’m not sure how warranted that is. Do you really believe that learning via the web has been minimal compared to the impact of Farmville? I don’t know if I really believe that.