I feel conflicted about the decision I have learned that the Expo has made to do a publicity deal with Izea/PayPerPost, who are scumbags. I can understand why the Expo would do so and I can understand why armies of beginning bloggers would work with Izea. However, I do not like the idea of advertisers paying for blog coverage and I do not believe them when they say disclosure is required. See image below, click for full size.

Why did I contribute to the BlogOnExpo? I didn't know about the Izea partnership until after I already had and I'm not sure how big a deal it is. I do want to be clear though that I am in no way in support of Izea.

That said, I hope you like the content that I and a number of other bloggers contributed to the Expo.

Update: Give it a try, this video may or may not play for you. It was recorded using Jing Project, an application that's simple to record with but maddening to use once a video is loaded. I get several emails a day saying I'm approaching the bandwidth limits of my free account. I have no idea how many times it's been viewed or how close I am to that bandwidth requirement though. I can't resize the file itself to fit into a smaller player, no one has answered my email requesting information despite previous requests from the company for me to try it out again and apparently the only solution to my problems will be to give them a $160 for a year's subscription. I think I'll be trying out ScreenFlow instead. But give it a click below, maybe you can watch this video!

Of course this is just one example of a general principle. I hope you can imagine all kinds of other applications that you would like to get limited access to without visiting them, but from inside your RSS reader.

Dapper Will Extract The Data in That Field For You

Select the "create a new Dapp" link. I love Dapper.

Just Enter The URL of the Page You Want to Extract From

Copy and paste in the URL, select RSS feed, then click the "next step" button.

Go Ahead and Log In Through Dapper

Then click on the login button on Facebook, or whatever app you want to enable access to.

Voila, You Are Logged In.

Now click the "add to basket" button and then "next step". You will be warned that Dapper works best with more than one URL in the basket, but you can just say that's ok and move on with your life.

Now click to define the field you want to extract data from.

1. I clicked on my friends' names whose birthdays are today. 2. You can see in the preview field below that Dapper properly understood which field I was trying to define and didn't capture any extra data. 3. Save that field. Update: After a day of using this feed live it looks like I may need to grab a different field on the page as Dapper is bringing me back header info instead of birthdays. I'm going to try repeating this process with the "view all" page instead. Two days later, with two birthday entries instead of one, the feed is working great again. Facebook formatting and wiggling is one thing that needs to be taken into consideration and if this was a more serious matter, a more flexible tool than Dapper might be needed given the circumstances. Double Update - after using this feed for a week or so, it turns out it's actually pretty good. Not perfect but a heck of a lot better than I thought it was. Trial and error!

Now tell Dapper what that field you selected is and where in an RSS feed you want it to live.

In some cases you might make this the item text, but in this case the title is what we're looking for. If you wanted to define another field, you will be given that opportunity once you save this one - but in this case we'll just say "next step."

Preview that baby...

Looks good! Click "next step."

Now you've got some fields to fill out...

I probably didn't need to mark this as private, but I did. Then hit save.

Now tell Dapper what your username and password are so it can login for you.

Hit "fix dates" if there's no date field already determined (this is a mystery but it works) . Finally, copy and paste that orange RSS link into your favorite feed reader! It may need to be one that supports authentication (log in) and in those cases if it's high enough priority these days I add it to Netvibes.com

Ta da! That's all there is to it. Making the feed took me about 3 minutes, much less than making this tutorial did. Again, Facebook is just one use case here - now go knock yourself out coming up with others!

Below is a series of screen shots illustrating the current state of my basic RSS work flow. There are lots of little details, feed discovery and creation techniques and other advanced steps that can be taken - but I'm often asked about the basics. So here they are. I hope you find this useful and feel free to pass it along to a friend. I'll do my best to answer any questions in comments below. If you'd like a personalized research system like this set up and populated with the most useful feeds for your work, let me know. I'm also working with some other people on a giant post coming soon describing all the things I know how to do with a pile of RSS feeds - I have a consulting project that's totally open ended so I thought I'd make a list.

Note that I made this post almost entirely with the application ScreenSteps. It was easy and fun, I wanted to try it and it didn't take too long for me to think of a good topic to try it on.

Netvibes.com

I put a link to Netvibes in my browser toolbar and click it about once an hour. These are high priority feeds that I want to keep an eye on throughout the day. They include things like 1.) Competitors 2.) Manually created feeds for special circumstances and 3.) Groups of feeds spliced together. Netvibes also has a very good mobile version, m.netvibes.com, so I can check these same feeds on my phone.

Google Reader

Once or twice a day I load up my Google Reader app, which I set up as a standalone "single app browser" using FluidApp.com. I try to always read every new item in my "vendors" folder and then I scan what I can from everything else. I oversubscribe, prioritize, read what I can and believe that subscription means there is some chance I'll see things - not subscribing means there is dramatically less opportunity for me to see the news in a particular feed.

GMail Web Clips

I've loaded 5 or 10 feeds into the Gmail "web clips" feature so I can increase the chance that I might see some of the items that come through those feeds.

Snackr.net

For the same reason, I also put the feeds in my Netvibes page into Snackr.net - an Adobe AIR based RSS ticker. It runs at the bottom of my page throughout the day, giving me something to scan while pages load, etc. I regularly find important news stories here.

Zaptxt

The highest priority feeds are run through an RSS to IM/SMS alert system. I use Zaptxt and it gets me lots of stories - before other people get to them. No matter what field you work in, I believe that being among the first to know about key information can provide a huge competitive advantage. There are many services that provide this kind of functionality - you should give some thought to what you would benefit from getting this kind of notification about.

That's how it goes! There are of course all kinds of ways to drill down - but this is the big picture. It's a whole lot of fun to build these kinds of work flows for people working in particular niches. Once that's done there are any number of other advanced things you can do with the information you're consuming like this. (See this case study, for example, or this one.)

Thanks for stopping by, if you've got tips on how you process large amounts of information online I'm sure readers here and I would love to know what they are!

I don't often blog about particular consulting projects because most of the work I do is with pre-launch companies or for internal use only, but consulting is what I spend one to two thirds of my day doing after I finish blogging at ReadWriteWeb.

The Project

After building out a collection of RSS feeds that attendees could use to track the DEMO conference in January, I was approached by Sun about helping build a blog coverage microsite to track discussion of their giant JavaOne conference that starts next week.

This is an example of one end of the RSS spectrum, most use cases are far simpler - so don't be scared!

JavaOne is a huge conference where scores of attendees will be blogging about a wide variety of Sun products and announcements. I worked with Sun to create a page called BlogCentral (hopefully to be moved to sun.com/blogcentral by conference time!) that aggregates all the latest and the most popular blog posts about the conference and 15 particular Sun projects and products. It's like a news dashboard for anyone interested in seeing what's being written about at JavaOne.

How We Did It

The project ended up being more complicated than I expected but it became relatively simple once I figured out the workflow on my end. I'll describe it here in detail so that you can do something similar for other events or topics, if you like. Of course you can hire me to make it happen, but here's a How-to in case you want to do it yourself -you can also just hire me so we can do something else magical together!

Step One: Build the Queries

For the conference in general and for 15 other topics, we used Google Blogsearch to search for blog mentions. For each topic we got a list of keywords that would indicate a post was concerning that topic, then we experimented with different queries strung together using AND, OR and - (not). We ended up with queries like:
"Mobile & Embedded" OR "Mobile and embedded" AND (JavaOne OR CommunityOne)

Did you know you could use parentheses in search queries in Google? I didn't. Some of the queries were much longer and more complex so at several steps of the way we had to check to make sure the full query made it through the search. Once we saw the basic structure of the search results page URL we could often just make changes there. We used -"Re:" to remove the frequent forum postings.

We used advanced search to get 50 results per page and limit the searches to English language sources (per client request).

Google Blogsearch worked best because it gave the most complete, up to date results. There was a little more spam that had to be filtered out by query than was in Ask.com blogsearch (my usual favorite) but Ask pulled in a lot of press release feeds too.

Unfortunately Google doesn't deliver the original blog source name in the RSS feed for results. That would have been nice to display on the final page. Blogdigger does deliver the source name but it always publishes "via Blogdigger" as well and the results aren't nearly as extensive.

Removing the HTML in Titles

Google highlights your search terms in titles and summaries and in at least some cases the feed delivers the bold tags as tags. That looked pretty ugly on the page and was a real pain to remove. I asked friends on Twitter for suggestions and a lot of people tried to help me remove the markup with Yahoo Pipes. Unfortunately, as Yahoo! staff confirmed when I asked them, Pipes has an error with its Regex, the function that should have let me get a new feed with that markup removed. It took awhile to figure that out, I was sure I was doing something wrong! Update: See this Pipe it turns out to remove the bold text. Thanks Yahoo!

Instead, I ended up using my beloved Dapper.net to remove the markup, which was relatively easy using this Dapp. Just swithc out the input URL, identify the title as title and summary as item text and then create a short URL for the service. Note - you should check the "fix date" box for each feed, too, as otherwise Google Blogsearch doesn't want to give you a publish date for the latest posts and that will mess up future steps. That short URL works as an RSS feed. Then refresh the page and repeat for other queries. Thanks again, Dapper!

Removing Duplicate Items

At this point I took my new RSS URL and went one of two directions. For the full feed of search results for each topic, I wanted to remove duplicate items. For this I went to Feed.Informer (formerly called FeedDigest) and chose to remove duplicates by URL, ignoring anchor links - so that blog comment URLs would get removed once the original post was seen.

This left me with a new RSS URL that I then took to Sun to put on the page.

Filtering for Hotness

From Dapper I also took the blogsearch feed URL over to AideRSS, where we got a new feed of just the 20% most commented on, linked-to and bookmarked blog posts in each search feed. That made up the Popular feed that you'll see around the JavaOne Blogcentral page.

I put the AideRSS folks through an awful lot but they were very helpful. Hopefully we got some kinks worked out of the system but keep in mind two things: if you can give them a short URL instead of a long one they'll handle it better and second, almost any really crazy feed will error our the first time you input it. In the background, the processing is happening and if you come back later your ranked feed will be available. Huge thanks to AideRSS for doing something that no one else on the web does - give me a simple feed of the most high-value posts from one source, even a search feed. That is a big part of the value proposition of the final project and I really appreciate them for it.

FeedBurner

Ordinarily I would have run all of the above feeds through Feedburner as a final step. That would have let me change the source feed on the back end without ever having to send the client anything new. Sun didn't want anything from Feedburner, though, because FB is blocked in China. That's a real big shame.

Putting Those Puppies on the Page

We used the Planet software to display the most recent items in each feed on the page. It works well enough though I wasn't terribly involved in this end and can't say for sure how much granular control there is over feed display. Correction! Actually, that software was a customized version of Roller, I'm now told. Other options include (from most to least complex) MagpieRSS, SimplePie and Feed.Informer's own output.

Custom Search Engine

While we were at it, we also put a Google Custom Search Engine that indexes just a handful of selected Sun domains on the bottom of the page. Added value with just minutes of extra work!

Conclusion

How's that for RSS awareness? It's pretty simple, really and once I got the markup removal figured out it was a lot of fun. My brain hurt at times, but that's a good thing.

How did Sun feel about the project? Joanne Kisling, the person who brought me in on the project was kind enough to leave the following in comments here:

"I was Marshall's contact at Sun and he was AWESOME to work with. He really knows his stuff. He dogged every detail and got answers, even when I was thinking we'd hit a dead end. I didn't see the frustrated Tweets or anything like it; he was nothing but upbeat and pleasant despite some trying circumstances. I really like his determination and willingness to try new things. Great job, Marshall! "

Thanks Joanne! Hopefully the description of the work above will help other folks do similar kinds of work with less frustration! I'm real happy though with how BlogCentral turned out.

Please feel free to ask any questions you have about the process, I'd love to give a quick bit of help if you're trying to do something like this yourself. Of course if you'd like me to do something like this with you, availability is limited but drop me a line.

Happy RSS Awareness Day, everyone!

Logo from http://enterpriserssdayofaction.wikispaces.com/

Read on for a Flash audio player and links that we discussed.


Check out the play button below where you can listen to the interview using the Yahoo Mediaplayer, which was super easy to include here! It did change the color of the text on this page, though. Anyway, here's the interview.

Dave Winer interview of yours truly.

Links we discussed include:
*The ReadWriteTalk interview with the founders of FriendFeed - where by the way, I've since learned that there is a lot more than just RSS going on over there. Check out that interview, there's a transcript available too. The differentiation starts at about 8 mins in.
*The RWW Primer on Lifestreaming
*Our list of 35 lifestreaming services
*The popularity filtering service we talked about was AideRSS and the feed scraping service is Dapper.net

Here's part 1, the action starts a little bit in and there's more advanced stuff in parts two and three. Brief notes for the whole talk are below.

Coincidentally, I'm also working on a book proposal for a publisher interested in a work concerning RSS. The original title was "RSS for Power Users" but the number 1 bit of feedback I've gotten is that the title has to change! Otherwise, feedback so far has been on balance good. Below is that proposal, just pasted in as I'm on the run, but I'd love your feedback too in comments. Thanks for stopping by!

Notes from presentation

RSS for Business
Notes for presentation by Marshall Kirkpatrick, tech blogger and consultant. For more information see marshallk.com and my daily posts at readwriteweb.com

Thanks for attending this session!

basic idea: bring the world to you by RSS, push out your contribution of things back out by RSS and make enough points of contact with the web at large that you can start to gain traction.

What does a feed reader look like?

-Netvibes
- NNW (NetNewsWire - see also FeedDemon for Windows desktop feed reader.)
-Google Reader
- zaptxt
- sidebar of marshallk, revenuerecognition.com
- mobile, m.netvibes

Why a biz wants to publish a feed:

- don't ask people to come back to your site to check for new information - they won't.
- RSS turns visitors into stakeholders into informed spokespeople
- Email is good but RSS is where an increasing number of tech savvy people want their web updates.
- Info available in RSS is easy to mashup and work magic with - see the rest of this presentation for examples.

Why a biz wants to read feeds -

- Know the market landscape without visiting sites
- Search automatically
- Identify new people, resources and opportunities
- First movers' advantage

Information oveload and feeds

- It's not like email - you don't have to read it all
- If you don't subscribe there's much lower chance you'll see it
- You can filter or delegate but the ultimate filter is your own eyes and mind.

Building yourself a reading list

- top blogs in your niche
how to identifying them - see post at http://tinyurl.com/35svf9

(aside: build a Google custom search engine with these top reference sites/blogs while you're at it)

- vendor blogs, press releases

- search feeds
* ask.com for blogs, yahoo and/or topix for news, live.com for web if you like, terraminds for twitter
* try searching for your org name, link to your site, top exec names, competitors names, key terms?

- tag feeds? del.icio.us/popular/concept del.icio.us/tag/concept1+concept2

Advanced stuff

-aiderss.com filter for popularity
- pipes.yahoo for combining, filtering

-email newsletters to feeds
https://mail.google.com/mail/feed/atom/label

- scraped feeds: dapper.net, feedfire, feedyes
- zaptxt.com rss to im/sms
- feeddigest, magpie RSS for posting
------

Download the original attachment

Book Proposal from Marshall Kirkpatrick

1. About the author(s)

Edited out here for modesty, but check this blog's About page if you're unfamiliar with my work.

2. Proposal summary

"RSS for Power Users" would be a book about using RSS and related web applications to gain competitive advantage as a knowledge worker in any field that intersects with media and the internet.

3. Book description

The book will teach readers how to gain first-mover's advantage in an information-based workplace. It will illustrate how to use RSS for rapid and automatic identification of market opportunities both pro-active and reactive. Feed readers, feed scraping/splicing and filtering and various tools to increase your information capacity with less time invested. Best practices for feed publishing will also be included. Readers need this information because all markets are growing more dependent on information, faster paced and more competitive.

4. Competing or comparable books

There are technical books available for programming with RSS and related syndication protocols but this book will be for the power end-user. Most power-users today have to learn through their own experience alone or through tutorial blog posts like those I write and that are very well received.

5. Readers/Market

RSS is a pretty fundamental technology and is thus of wide, general interest. Marcom practitioners, lawyers, journalists, consultants and academics are particularly interested in taking the leap from elementary to advanced use of RSS. Advanced use of RSS is an accessible way that anyone interested in leadership, personal advancement or self-preservation in an information economy can change their daily routine to get a better handle on the torrent of info available online.

6. Book size and illustrations

Please estimate:

a. The number of published pages to within a 50-page range

I'm guessing 250 pages.

b. The number of figures (line drawings + screen shots)

I'm going to guess 75.

7. Tentative table of contents with annotations

Background: To get an idea of the details included in the following TOC see these blog posts.

-Open Sourcing my TechCrunch Workflow

My post about how I use a variety of feed readers to break industry news.
http://marshallk.com/open-sourcing-my-techcrunch-work-flow (shortcut via http://snurl.com/mtc )

-How to find the weirdest stuff on the internet

My post about systematically discovering the top sources in any niche and filtering those sources for the most popular items in them.
http://www.readwriteweb.com/archives/the_weirdest_stuff_on_the_internet.php (http://snurl.com/rwwweird )

- The Glory, Bliss and How-to of Screen Scraping for RSS

A post about creating feeds from pages that don't offer them.
http://snurl.com/scraping

- Social Media for Marketing: What We’ve Done at SplashCast So Far

A post about using RSS reading and publishing for marketing purposes.
http://snurl.com/socnetmarcom

- Teaching RSS: A Discussion

A post about how I taught people about RSS two years ago, much of it still useful today.
http://snurl.com/teachrss

Proposed TOC

I. A Brief History of RSS
[High level history and explanation of the basics just so that power users have it as a foundation.]

II. Power RSS - the Big Picture
[How to deal with information overload the Power RSS way, what first mover's advantage looks like and a discussion of why RSS is so useful in so many different ways.]
a. The paradigm of power RSS (it aint like email, etc.)
b. The need for speed - first mover's advantage in the context of RSS
c. The pliability of RSS

III. Discovering the Best Feeds
[Half of the story is building your reading list. There are clear technical steps that can make this far more effective than accidental discovery of sources.]
a. The diversity of sources
b. Systematized source discovery
c. Scraping, filtering and splicing feeds
d. OPML and feed sharing

IV. Making the Most of the Feeds You Find
[This is the day-to-day of integrating feed reading into your work.]
a. Picking a feed reading suite
1. Start pages
2. Full readers
3. Mobile
4. Real time alerts
b. Acting on the items delivered
1. New media/online actions
2. Leveraging RSS in the context of traditional media
c. Staying a step ahead
1. Maintenance
2. Forecasting

IV. Advanced Feed Publishing
[Whether you're an individual or an organization, there are some important steps to take to optimize the impact of the feeds you publish.]
a. Full feed publishing
1. Best practices
2. Analytics
3. Advertising
b. Tag feeds
c. Other feeds

V. The Future of RSS
[Forecasting what possibilities could come next based on the bleeding edge today.]
a. Semantic web
b. Machines and feeds
c. Invisible feeds - RSS "under the covers"

Thoughts?

The highlights of the post are the link to this Wired article about the economy of screen scraping (great article) and this screencast about how to use Dapper that I finally recorded.

These types of tools are things I use often in my consulting work. They are really exciting and worth a close examination by anyone who reads this blog.

An OPML file is an outline. In this case, it's a bundle of RSS feeds that can be moved into and out of any RSS reader as a group. No matter what RSS reader you use, it can import and export OPML files. It's real handy. If PR people, for example, would send me one OPML file of all their clients' blogs and a news search feed for each of those clients' company names - I would throw it into my reader and have a long term connection with all their news. It would build name recognition if nothing else, but I'd likely find something in there someday to write about too. There's a billion other reasons to use OPML - just ask yourself in what circumstances you can imagine sending someone else one link or file that contains a collection of dynamic sources on any topic. I know these are the sorts of questions that keep me up at night.

Here's how you do it...

There's lots of ways, but most of them take too long or don't work as well as they ought to. Start a Netvibes account. Any RSS reader will export an OMPL file, but Netvibes makes it easy to make a little one that contains just a few select feeds. Create a new tab. Click on the "add content" link in the top left of the page. Add an RSS feed. Then add another RSS feed, and another and another until you're done. Then click on "settings" in the top right. Then say "backup" and export in OPML format. UPDATE: If you've updated to Ginger in Netvibes, you can now find your OMPL file via Add Content- Add a Feed - Export

A file will land on your desktop. Here's one to check out for an example: econews.opml Save Link As that baby and it will land on your computer.

Open it with a text editor like textpad or whatever is on your computer - not Word! Go in and look at the file. Change the title of it. You can see what the structure of an individual feed looks like in this format - outline type="rss" title="VibeAgent Blog" text="VibeAgent Blog" xmlUrl="http://www.vibeagent.com/blog/?feed=rss2" htmlUrl="http://www.vibeagent.com/blog" / See that? DON"T BE SCARED!

Change the title so it's not "Netvibes OPML" - give it a good title. Now delete any extra items that got in there by mistake. Don't break anything but go ahead and do it. Now, do a Save As and call it mytitleofmyfilethingy.OPML. Give it the .OPML suffix.

Next, send it to me, or to whoever you want to send it to. You can send it as an email attachment or you can put it on your server and just send a link, that's extra nice.

SOME PEOPLE will laugh at me for suggesting that you do it this way. I don't care, though, no one is using OPML as is so there's no time for purists. The above method will work just fine. Get comfy and you can do new and exciting things with OPML.

Small sustainability-minded businesses now have a greater opportunity than ever before to experience "information power parity" with larger and more traditionally oriented competitors. There are a new class of tools, specifically blogs and RSS syndication, that enable businesses to:
1. speak directly to consumers without being dependent on anyone else for media coverage
2. to turn marketing into issue leadership and website visitors into subscribed stakeholders
3. to gain first-mover's advantage on industry news and the moves of competitors by automating your online research.

I'd like to write a how-to overview of those possibilities in the sustainable industries context. It would be based on the experiences I described in this article: social media for marketing...

I think that's pretty good but I'm interested in your opinion of the ideas and articulation. I've been a big believer since I got into this business in the power of advanced RSS to help anyone smart become a leader in their corner of the information economy - whatever that might be. It's worked very well for me, I'd like to spend some time with green businesses helping them do the same. It's always a challenge explaining it, though, particularly before getting to do a visual demonstration. At that point everything becomes clear, but how do you open the conversation with ambitious people unfamiliar with the concepts?