Various services have different ways for users to separate their “friends” into different groups, viewable by topic, category or type of connection. Facebook is making changes today to make it easier to break your Facebook Newsfeed into groups. That’s going to be very important. The best Twitter applications offer group functionality that the site itself doesn’t. MySpace offers no such feature, yet. The Facebook news prompted us to try to articulate the value of group creation online. By better understanding the value that groups can deliver, we can better strategize our creation of groups.

First we’ll discuss four ways that small groups separated from a full river of news can help you use the social web more effectively. Then, for context, we’ll briefly contrast this with the value of the full stream of information. Using both together is more useful than merely limiting the full stream to a manageably small group of sources on a given topic or of a certain priority.

Forgive me if this is all obvious to you; I know it’s not to everyone. Even if it is, I think there’s value in discussing fundamental qualities of emerging methods of communicating. The assumption in discussing these values is that you’re an ambitious knowledge worker. If that’s not the case then this logic may or may not apply.

The Value of Groups

Prioritization

Pulling high-priority sources out of the full stream and putting them in a special place enables you to catch more of the high-value information those sources publish. Why lose valuable messages in the whole sea of marginally valuable information that we all have access to? High value sources don’t always publish high value information, but the increased likelihood of their doing so warrants putting them in a special place so that the unusually high signal-to-noise ratio they offer is maintained.

Below: I follow thousands of people on FriendFeed but have about 100 people who often discover or make news early in their own group.

Context

Different words and links have different meanings in different contexts. When subscribing to a large set of sources it’s sometimes easy to forget who or what certain sources are when their content comes barreling down a full stream of information. Placing sources into contextual groups helps put messages in context, adding meaning and offering insight into the significance of some content.

Right: I’ve got a list of 300 tech industry analysts on Twitter pulled into a separate group in Tweetdeck. It’s good to know that when I read these messages, they are coming from professional analysts.

Intimacy

We’ve all got far more connections online than it’s realistic to maintain closely. You may be familiar with the concept of the Dunbar number. Researcher Robin Dunbar argues that 100 to 150 is the approximate natural group size in which everyone can really know everyone else.

Serious users of social media often maintain far, far more connections than that, though. What can you do? Strategic creation of groups facilitates social contact disproportionately frequently relative to contact with the entire list of our social connections. That disproportionately frequent contact lends itself well to greater intimacy.

Left: The ReadWriteWeb writers’ chat, an invaluable resource for us in a world swimming with social connections.

Speed

Some sources of information are more important to be up-to-the-minute with than others. Strategic creation of time-sensitive groups allows you to have those groups alone delivered in a way that fits their time-sensitive nature. You don’t want to be interrupted by updates from every source of information you have any interest in - but some sources are worth being interrupted by.

Right: High-priority RSS feeds delivered by IM.

The Value of the Full Stream

Many people are tempted to solve information overload by cutting back on the number of connections and subscriptions they are signed up for online. We argue that this is a mistake; group creation can help capture some of the same benefits of cut-back without incurring the loss of benefits felt by restricting a well-populated stream of information.

Serendipitous Discovery

Do you cancel your cable TV subscription just because you end up not watching most of the shows that are on at a given time? (Maybe with Hulu you do now.) Probably not. Channel surfing is a way to discover new things.

So too with the web; it’s better to have too many options than not enough. Subscribing to a source of information substantially increases the likelihood that you’ll see something good from that source.

Just don’t worry about reading everything. Scan what you can and let fate bring you value from a bulk of undifferentiated information. Problem solved.

Weak Connections

All of us have some social connections that are stronger than others. There’s value in those weak connections, too. Welcoming people into your full stream of social information is how weak connections are built and maintained.

Reciprocity

Some people say that social media makes almost everyone famous, at least to a small group of people. One definition of fame is a circumstance where the number of people who care about what you’re doing is more than you are capable of paying attention to yourself. Online, though, we can all pay a little passing attention to the people who are paying attention to us. People appreciate RSS subscribers; they like friends on Facebook and Twitter. If someone follows you, it only makes sense to follow them back. (I need to follow my own advice better on Twitter.)

If a person isn’t terribly important to you, just don’t include them in a high-priority group. Interact with them when you get the chance. They’ll appreciate the reciprocal connection, though

That’s one take on the strategic value of groups and the full stream of information. What are your thoughts on this topic? Have you come up with any other super-useful ways to build, manage, or find value in groups online? We’d love to hear about it!

Title photo: Your Days - December test Group : 31 Décembre 2006 by Nawal_ CC on Flickr

I spoke for an hour and a half tonight about RSS magic. Lots of jokes, some things I have discussed elsewhere publicly before and a fair amount of philosophical rumination on what it all means. Bill Jackson graciously streamed it live and recorded it, and I’ve embedded it below. Here’s the link to the slide show of links I went through. An hour and a half is a long video, so if you want the “quick and dirty” version - see this post, this post and this one.

The night starts with a round of introductions and I get into the meat of my talk a few minutes in. I hope you enjoy it and find it useful. Thanks to everyone who showed up and helped out.

1. Use other peoples’ lists before building your own.

A few weeks ago I wrote a giant post titled How to Build a Social Media Cheat Sheet on Any Topic. It was a big hit and if you haven’t seen it yet, I’d recommend checking it out. It’s all about how to discover the top online blogs writing about any topic, how to find their most popular posts (according to their readers), how to search against their archives as a reference source and how to connect with the authors elsewhere around the web, on Twitter for example.

Jason Falls wrote a similar post a few days later that offered some more good advice, though. One of the biggest things I took from Jason’s post was that there’s no reason to start building a list from scratch, as I showed readers how to do in my post, when you can find existing lists of resources already created. Jason used Alltop as one of his sources of sources, and though the interface there is not good for regular use, someone has done a whole lot of work finding good blogs on a variety of topics. There’s no need to reproduce that work if you’re interested in one of those topics.

Additionally, Googling for the terms list of blogs about topic X will often lead you to existing lists. I’ve also begun exploring Delicious pages like http://delicious.com/tag/list+blogs+eco, where you’ll find links people have tagged with the words list, blogs and eco. Shazam!

2. You don’t have to search against static lists

Google Custom Search engines are really simple but are one of the most powerful tools I started working with last year. I’ve got probably 30 of them that let me search top blogs on a wide variety of different topics.

Whenever I find or make a list, I always consider turning it into a CSE. Did you know, though, that you can make a CSE that’s automatically populated using an OPML URL? Check out http://www.google.com/cse?cref=http://www.google.com/cse/tools/makecse%3Furl%3Dwww.readwriteweb.com/topeducationblogsgreatposts.opml&loading=1.

You can switch out the OPML URL in that link and search against any OPML file you’ve posted or found online. That way if you update the OPML file, you’ll automatically update the sources included in your Custom Search Engine.

Big thanks is due to Marjolein Hoekstra of Clever Clogs for teaching me that, as she has so many things about RSS.

3. Finding industry people on Twitter is easy.

I’m hesitant to share this one publicly, but what the heck. Have you looked at Twellow? It’s a search engine that searches the user bio fields of peoples’ Twitter profiles. That’s cool, but there’s an advanced search feature that’s even cooler.

Let’s say I want to find people who work at any of the Semantic Web companies listed in RWW’s Top Semantic Web Companies to Watch lists of 2007 or 2008. I just opened my browser to those posts, drug the company names off onto a text editor and then did a little find and replace to create this search query:
http://www.twellow.com/search?q=(Freebase)|(Powerset)|(Twine)|(AdaptiveBlue)|(Hakia)|(Talis)|(TrueKnowledge)|(TripIt)|(Calais)|(Spock)|(collexis)|(Irion)|(Knowledge+Concepts)|(Garlik)|(Zotero)|(Celtx)|(Reveltyix)|(ZoomInfo)|(Kirix)|(Rearden+Commerce)|(OpenQabal)|(ThoughtExpress)|(Lifestrea.ms)|(Ontos)|(BooRah)|(Swotti)|(Inform.com)|(Siri)|(Evri)|(UpTake)|(Imindi)|(Faviki)|(Zemanta)|(HeadUp)|(Daylife)|(SemantiNet)|+(ThoughtExpress)”>http://www.twellow.com/search?q=(Freebase)+|(Powerset)|(Twine)|(AdaptiveBlue)|(Hakia)|(Talis)|(TrueKnowledge)|(TripIt)|(Calais)|(Spock)|(collexis)|(Irion)|(Knowledge+Concepts)|(Garlik)|(Zotero)|(Celtx)|(Reveltyix)|(ZoomInfo)|(Kirix)|(Rearden+Commerce)|(OpenQabal)|(ThoughtExpress)|(Lifestrea.ms)|(Ontos)|(BooRah)|(Swotti)|(Inform.com)|(Siri)|(Evri)|(UpTake)|(Imindi)|(Faviki)|(Zemanta)|(HeadUp)|(Daylife)|(SemantiNet)|(ThoughtExpress”)

I then scanned through those search results for people who appear to work at those companies, opened their Twitter profiles up in new tabs, added them as friends, then captured all the tabs as a slideshow I could share with the rest of the team. Just like that we were all following a bunch of people who worked at top semweb companies. You could do the same thing to jump in quickly to any industry’s community on Twitter.

Those are some of the most useful research tactics I’ve discovered lately - how about you? Feel free to post useful stuff right below in comments!

Lego maze photo CC from Flickr user Anvilon

Check out the 25 comments on the original post as well for some interesting discussion. Some readers were very critical and I’ve tried to offer some critical thoughts as well, but it’s clear that marking on the web is here to stay. Hopefully it will be based on a greater degree of authenticity, usefulness and innovation than marketing generally is known for.

For more personal thoughts on new media marketing, check out two of my old posts here Social Media for Marketing and Thoughts on Product Launch Promotion. Both are a touch out of date but should be a good source of some still-valuable resources and advice.

Speaking of resources, if you’re interested in new media marketing you may appreciate this OPML file of Chris Brogan’s favorite marketing bloggers to watch in 2009. It’s a special file of all their feeds filtered to deliver just their unusually popular posts (filtering performed by Postrank). You can download that file, then import it into your RSS reader and you’ll be kept super smart all year long. I’ll be keeping an eye on those feeds, myself.

If you’d like a short, concentrated injection of smarts along similar lines, check out my consulting services, just like these happy people have.

And now the blog post I promised…

Will 2009 be the big year for corporate transparency, for a global conversation - perhaps for bargain basement online marketing tactics instead of old-school huge commercial campaigns?

Peter Kim, a former Forrester analyst now working on stealth enterprise software company, recently polled 14 of the most high-profile thinkers about social media marketing and asked them what they expected to see 2009 bring. The end product was an attractive 23 page PDF that we’ve embedded below, but we thought we’d pull out some of the thoughts we found most interesting for all you skimmers out there.

Social shopping

“Now with connective technologies like Facebook Connect, Google FriendConnect, and OpenID, consumers will now be able to see reviews, experiences, and critiques from people they actually know and trust.” - Jeremiah Owyang, Forrester Research

Our take: Much as we are concerned about the proprietary nature of Facebook Connect in particular, the ease with which people are able to see feedback left by people they know, with confirmed identities, really could be a game changer.

Access

“Twitter will continue to achieve legitimacy. But more than any push-channel, Twitter will give customers, advocates and critics unprecedented access to corporate personnel and vice versa.” - Scott Monty, Ford Motor Company (Photo, right, by Wendy Piersall)

Our take: This makes sense, and it’s pretty funny to think about. Even the biggest cynics often have a dramatic turn around about Twitter once they start using it, and the intimacy that develops is remarkable. We agree with Monty that this will become increasingly difficult to resist.

Measuring the success of social networks

“Implement listening programs through social media to get real time authentic knowledge that is actionable… Measure with customer service metrics like retention/ satisfaction & social metrics like engagement.” - Rohit Bhargava, Ogilvy (Photo by Shashi Bellamkonda)

“Slowly but surely, we’re going to develop a set of better metrics to help guide, direct and validate ‘commitment’-based marketing and yes, Mr Kim….they will extend beyond the rather short term, blunt metric called ROI”. - Joseph Jaffe, Crayon

Our take: Good luck with that, we’re not optimistic. This is soft stuff and though clear success speaks for itself, all the gradations between success and failure are going to be very hard to quantify.

Quality vs Quantity in Social Media

“I believe we’ll have more focused velvet-rope social networks in 2009 where the tools and the goals match verticals of interest instead of the general commons of Facebook.” - Chris Brogan, New Marketing Labs

“Exclusivity trumps accessibility. Having thousands of friends becomes ’so 2008′ and defriending becomes the hot new trend, driven by overwhelming rivers of newsfeeds.” - Charlene Li, Altimeter Group (Photo by deneyterrio on Flickr)

Our take: Maybe, for some people and in some circumstances this will be the case. We expect most people to find a middle ground between the whole sale slow-down that some seem to expect and an evolutionary adjustment to vastly increased data input.

Making the most of limited budgets

“Dwindling budgets suddenly make low-cost social media look like the pretty girl at the ball. ” - Ann Handley, Marketing Profs

“Companies will struggle with how to control who says what — but will increasingly realize that in an economic downturn, they need all the marketing muscle and leverage they can get and actively encourage.” - Charlene Li

Our take: This makes a lot of sense to us, but we expect that it will be tempered by the fear of totally blowing it. Getting into the social media space and doing it wrong is something that a lot of companies fear getting blown apart for. We expect that to change slowly and only for a limited number of companies.

Here’s the full document embedded below - what do you think of these predictions?

Unfortunately, I haven’t found any way to subscribe to an RSS or email list of those posts - and I’ve looked really hard! Tonight I’m preparing for a presentation I’m giving tomorrow to a medical tech and civil liberites organization and I really wanted to make such a subscription available for them. So I bit the bullet and made it myself. It was not as easy as I’d like and is going to take a few minutes each week for me to maintain - so if any participants are here reading this and would like to take it over, I’ll show you below not just how I created the feed but how you can help too.

Read on for RSS and email subscription options and step by step instructions describing how this was done. I hope the first commenter from the medical blogging community who stops by will break my heart by showing me an existing RSS subscription option that I just haven’t found yet.

Here’s the feed, give it a click: http://feeds.feedburner.com/GrandRoundsFeed

And if you’d like to subscribe by email - here’s a form to do that with.

I tried a lot of things that didn’t work and this is the solution I came up with. It should be reusable in other contexts of course, not just with Grand Rounds.

Step One

Thankfully, there’s already a starting place because Dr. Nicholas Genes, a resident in the Emergency Medicine program at Mount Sinai Hospital in New York City, keeps a Google Calendar of all past and future hosts of the Grand Rounds embedded in a four year old post on his blog.

I clicked the plus button in that embed to subscribe to the calendar in my Google Calendar so I’ll see when and where future editions are being hosted. Unfortunately there’s no permalinks to the particular posts available, it’s just links to the home pages of the host blogs.

Step Two

What I’m going to do is click on each link that comes into my calendar each Tuesday morning, visit the host blog, find the permalink to the Grand Round post and tag it “grandroundsfeed” in my Delicious account.

Step Three

Delicious publishes an RSS feed for everything I as a user tag as “grandroundsfeed.” I took that RSS feed and put it through FeedBurner. Feedburner lets me keep track of how many subscribers there are, add a link to the end of every item that says please visit Marshallk.com and take over maintance of this feed from me please, offer subscription to the links by email, transfer control of the feed to another Feedburner account holder when someone capable volunteers and perhaps most importantly, it lets me switch out the source feed when the time comes to be instead the feed of someone else’s delicious account items tagged “grandroundsfeed” (or any other tag of their choosing).

That’s it! It’s as simple as that. If no one volunteers to tag the new link each week, I’ll just keep doing it mself. It will only take 2 minutes and it’s a great public service for an awesome round up of content. We could create a widget that displays these links each week, we could do all kinds of things with it. I’m fantasizing about a combination of Dapper.net, PostRank and maybe one other tool to create a “best of” Grand Rounds feed that would deliver only the most popular links from the entire collection each week. I probably won’t take the time to try to figure that out, but I think it’s doable.

Subscription options are, again, up above these instructions. Obviously if you’re interested in getting any help with projects like this for your organization, drop me a line, but hopefully these instructions are clear enough that you could do it yourself with one hand tied behind your back.

In the mean time - enjoy the awesome medical blogosphere round ups. If you want to see some examples, recent ones include Dr. Deb’s adorable iTunes playlist version and the wacky Grand Round in the form of medical blog posts re-interpreted as job advice for Barack Obama over at Musings of a Distractable Mind. See also the recent edition at the very nicely produced Nurse Ratched’s Place. Next week is at Canadian Medicine. I just linked to particular recent posts of all of these blogs so they’d get a trackback notification about this post. Is that trackback spam? I don’t think it is.

Happy feed creation in whatever fields you’re in!

I feel conflicted about the decision I have learned that the Expo has made to do a publicity deal with Izea/PayPerPost, who are scumbags. I can understand why the Expo would do so and I can understand why armies of beginning bloggers would work with Izea. However, I do not like the idea of advertisers paying for blog coverage and I do not believe them when they say disclosure is required. See image below, click for full size.

Why did I contribute to the BlogOnExpo? I didn’t know about the Izea partnership until after I already had and I’m not sure how big a deal it is. I do want to be clear though that I am in no way in support of Izea.

That said, I hope you like the content that I and a number of other bloggers contributed to the Expo.

Update: Give it a try, this video may or may not play for you. It was recorded using Jing Project, an application that’s simple to record with but maddening to use once a video is loaded. I get several emails a day saying I’m approaching the bandwidth limits of my free account. I have no idea how many times it’s been viewed or how close I am to that bandwidth requirement though. I can’t resize the file itself to fit into a smaller player, no one has answered my email requesting information despite previous requests from the company for me to try it out again and apparently the only solution to my problems will be to give them a $160 for a year’s subscription. I think I’ll be trying out ScreenFlow instead. But give it a click below, maybe you can watch this video!

Of course this is just one example of a general principle. I hope you can imagine all kinds of other applications that you would like to get limited access to without visiting them, but from inside your RSS reader.

Dapper Will Extract The Data in That Field For You

Select the "create a new Dapp" link. I love Dapper.

Just Enter The URL of the Page You Want to Extract From

Copy and paste in the URL, select RSS feed, then click the "next step" button.

Go Ahead and Log In Through Dapper

Then click on the login button on Facebook, or whatever app you want to enable access to.

Voila, You Are Logged In.

Now click the "add to basket" button and then "next step". You will be warned that Dapper works best with more than one URL in the basket, but you can just say that’s ok and move on with your life.

Now click to define the field you want to extract data from.

1. I clicked on my friends’ names whose birthdays are today. 2. You can see in the preview field below that Dapper properly understood which field I was trying to define and didn’t capture any extra data. 3. Save that field. Update: After a day of using this feed live it looks like I may need to grab a different field on the page as Dapper is bringing me back header info instead of birthdays. I’m going to try repeating this process with the “view all” page instead. Two days later, with two birthday entries instead of one, the feed is working great again. Facebook formatting and wiggling is one thing that needs to be taken into consideration and if this was a more serious matter, a more flexible tool than Dapper might be needed given the circumstances. Double Update - after using this feed for a week or so, it turns out it’s actually pretty good. Not perfect but a heck of a lot better than I thought it was. Trial and error!

Now tell Dapper what that field you selected is and where in an RSS feed you want it to live.

In some cases you might make this the item text, but in this case the title is what we’re looking for. If you wanted to define another field, you will be given that opportunity once you save this one - but in this case we’ll just say "next step."

Preview that baby…

Looks good! Click "next step."

Now you’ve got some fields to fill out…

I probably didn’t need to mark this as private, but I did. Then hit save.

Now tell Dapper what your username and password are so it can login for you.

Hit "fix dates" if there’s no date field already determined (this is a mystery but it works) . Finally, copy and paste that orange RSS link into your favorite feed reader! It may need to be one that supports authentication (log in) and in those cases if it’s high enough priority these days I add it to Netvibes.com

Ta da! That’s all there is to it. Making the feed took me about 3 minutes, much less than making this tutorial did. Again, Facebook is just one use case here - now go knock yourself out coming up with others!

Below is a series of screen shots illustrating the current state of my basic RSS work flow. There are lots of little details, feed discovery and creation techniques and other advanced steps that can be taken - but I’m often asked about the basics. So here they are. I hope you find this useful and feel free to pass it along to a friend. I’ll do my best to answer any questions in comments below. If you’d like a personalized research system like this set up and populated with the most useful feeds for your work, let me know. I’m also working with some other people on a giant post coming soon describing all the things I know how to do with a pile of RSS feeds - I have a consulting project that’s totally open ended so I thought I’d make a list.

Note that I made this post almost entirely with the application ScreenSteps. It was easy and fun, I wanted to try it and it didn’t take too long for me to think of a good topic to try it on.

Netvibes.com

I put a link to Netvibes in my browser toolbar and click it about once an hour. These are high priority feeds that I want to keep an eye on throughout the day. They include things like 1.) Competitors 2.) Manually created feeds for special circumstances and 3.) Groups of feeds spliced together. Netvibes also has a very good mobile version, m.netvibes.com, so I can check these same feeds on my phone.

Google Reader

Once or twice a day I load up my Google Reader app, which I set up as a standalone “single app browser” using FluidApp.com. I try to always read every new item in my “vendors” folder and then I scan what I can from everything else. I oversubscribe, prioritize, read what I can and believe that subscription means there is some chance I’ll see things - not subscribing means there is dramatically less opportunity for me to see the news in a particular feed.

GMail Web Clips

I’ve loaded 5 or 10 feeds into the Gmail "web clips" feature so I can increase the chance that I might see some of the items that come through those feeds.

Snackr.net

For the same reason, I also put the feeds in my Netvibes page into Snackr.net - an Adobe AIR based RSS ticker. It runs at the bottom of my page throughout the day, giving me something to scan while pages load, etc. I regularly find important news stories here.

Zaptxt

The highest priority feeds are run through an RSS to IM/SMS alert system. I use Zaptxt and it gets me lots of stories - before other people get to them. No matter what field you work in, I believe that being among the first to know about key information can provide a huge competitive advantage. There are many services that provide this kind of functionality - you should give some thought to what you would benefit from getting this kind of notification about.

That’s how it goes! There are of course all kinds of ways to drill down - but this is the big picture. It’s a whole lot of fun to build these kinds of work flows for people working in particular niches. Once that’s done there are any number of other advanced things you can do with the information you’re consuming like this. (See this case study, for example, or this one.)

Thanks for stopping by, if you’ve got tips on how you process large amounts of information online I’m sure readers here and I would love to know what they are!

I don’t often blog about particular consulting projects because most of the work I do is with pre-launch companies or for internal use only, but consulting is what I spend one to two thirds of my day doing after I finish blogging at ReadWriteWeb.

The Project

After building out a collection of RSS feeds that attendees could use to track the DEMO conference in January, I was approached by Sun about helping build a blog coverage microsite to track discussion of their giant JavaOne conference that starts next week.

This is an example of one end of the RSS spectrum, most use cases are far simpler - so don’t be scared!

JavaOne is a huge conference where scores of attendees will be blogging about a wide variety of Sun products and announcements. I worked with Sun to create a page called BlogCentral (hopefully to be moved to sun.com/blogcentral by conference time!) that aggregates all the latest and the most popular blog posts about the conference and 15 particular Sun projects and products. It’s like a news dashboard for anyone interested in seeing what’s being written about at JavaOne.

How We Did It

The project ended up being more complicated than I expected but it became relatively simple once I figured out the workflow on my end. I’ll describe it here in detail so that you can do something similar for other events or topics, if you like. Of course you can hire me to make it happen, but here’s a How-to in case you want to do it yourself -you can also just hire me so we can do something else magical together!

Step One: Build the Queries

For the conference in general and for 15 other topics, we used Google Blogsearch to search for blog mentions. For each topic we got a list of keywords that would indicate a post was concerning that topic, then we experimented with different queries strung together using AND, OR and - (not). We ended up with queries like:
“Mobile & Embedded” OR “Mobile and embedded” AND (JavaOne OR CommunityOne)

Did you know you could use parentheses in search queries in Google? I didn’t. Some of the queries were much longer and more complex so at several steps of the way we had to check to make sure the full query made it through the search. Once we saw the basic structure of the search results page URL we could often just make changes there. We used -”Re:” to remove the frequent forum postings.

We used advanced search to get 50 results per page and limit the searches to English language sources (per client request).

Google Blogsearch worked best because it gave the most complete, up to date results. There was a little more spam that had to be filtered out by query than was in Ask.com blogsearch (my usual favorite) but Ask pulled in a lot of press release feeds too.

Unfortunately Google doesn’t deliver the original blog source name in the RSS feed for results. That would have been nice to display on the final page. Blogdigger does deliver the source name but it always publishes “via Blogdigger” as well and the results aren’t nearly as extensive.

Removing the HTML in Titles

Google highlights your search terms in titles and summaries and in at least some cases the feed delivers the bold tags as tags. That looked pretty ugly on the page and was a real pain to remove. I asked friends on Twitter for suggestions and a lot of people tried to help me remove the markup with Yahoo Pipes. Unfortunately, as Yahoo! staff confirmed when I asked them, Pipes has an error with its Regex, the function that should have let me get a new feed with that markup removed. It took awhile to figure that out, I was sure I was doing something wrong! Update: See this Pipe it turns out to remove the bold text. Thanks Yahoo!

Instead, I ended up using my beloved Dapper.net to remove the markup, which was relatively easy using this Dapp. Just swithc out the input URL, identify the title as title and summary as item text and then create a short URL for the service. Note - you should check the “fix date” box for each feed, too, as otherwise Google Blogsearch doesn’t want to give you a publish date for the latest posts and that will mess up future steps. That short URL works as an RSS feed. Then refresh the page and repeat for other queries. Thanks again, Dapper!

Removing Duplicate Items

At this point I took my new RSS URL and went one of two directions. For the full feed of search results for each topic, I wanted to remove duplicate items. For this I went to Feed.Informer (formerly called FeedDigest) and chose to remove duplicates by URL, ignoring anchor links - so that blog comment URLs would get removed once the original post was seen.

This left me with a new RSS URL that I then took to Sun to put on the page.

Filtering for Hotness

From Dapper I also took the blogsearch feed URL over to AideRSS, where we got a new feed of just the 20% most commented on, linked-to and bookmarked blog posts in each search feed. That made up the Popular feed that you’ll see around the JavaOne Blogcentral page.

I put the AideRSS folks through an awful lot but they were very helpful. Hopefully we got some kinks worked out of the system but keep in mind two things: if you can give them a short URL instead of a long one they’ll handle it better and second, almost any really crazy feed will error our the first time you input it. In the background, the processing is happening and if you come back later your ranked feed will be available. Huge thanks to AideRSS for doing something that no one else on the web does - give me a simple feed of the most high-value posts from one source, even a search feed. That is a big part of the value proposition of the final project and I really appreciate them for it.

FeedBurner

Ordinarily I would have run all of the above feeds through Feedburner as a final step. That would have let me change the source feed on the back end without ever having to send the client anything new. Sun didn’t want anything from Feedburner, though, because FB is blocked in China. That’s a real big shame.

Putting Those Puppies on the Page

We used the Planet software to display the most recent items in each feed on the page. It works well enough though I wasn’t terribly involved in this end and can’t say for sure how much granular control there is over feed display. Correction! Actually, that software was a customized version of Roller, I’m now told. Other options include (from most to least complex) MagpieRSS, SimplePie and Feed.Informer’s own output.

Custom Search Engine

While we were at it, we also put a Google Custom Search Engine that indexes just a handful of selected Sun domains on the bottom of the page. Added value with just minutes of extra work!

Conclusion

How’s that for RSS awareness? It’s pretty simple, really and once I got the markup removal figured out it was a lot of fun. My brain hurt at times, but that’s a good thing.

How did Sun feel about the project? Joanne Kisling, the person who brought me in on the project was kind enough to leave the following in comments here:

“I was Marshall’s contact at Sun and he was AWESOME to work with. He really knows his stuff. He dogged every detail and got answers, even when I was thinking we’d hit a dead end. I didn’t see the frustrated Tweets or anything like it; he was nothing but upbeat and pleasant despite some trying circumstances. I really like his determination and willingness to try new things. Great job, Marshall! ”

Thanks Joanne! Hopefully the description of the work above will help other folks do similar kinds of work with less frustration! I’m real happy though with how BlogCentral turned out.

Please feel free to ask any questions you have about the process, I’d love to give a quick bit of help if you’re trying to do something like this yourself. Of course if you’d like me to do something like this with you, availability is limited but drop me a line.

Happy RSS Awareness Day, everyone!

Logo from http://enterpriserssdayofaction.wikispaces.com/