Tutorial – Desktop News Feeds

Wouldn't it be great if you could automatically gather updates from all your favorite online sources and make them subtly accessible in your favorite browser, to peruse when you feel like reading headlines, without wasting time or screen space and without even touching them unless you want to read a full article? Would you like to do this in the same way, whatever your preferred Linux environment? In this tutorial, I show you how by integrating two distinct sets of open standards.

First, I explain what RSS feeds are and how to download as many of them as you want with scripts that can run periodically as automated cron jobs. Second, you will see how to use the output of those scripts to generate desktop icons, or root menus, that open the news you want to read straight into your browser. You can do this in ways that work with little or no tweaking in the great majority of Linux window managers or desktop environments available today.

Only your imagination will limit what you can do when you know how to download news automatically from the web or regenerate window manager menus. I mention some ways to reuse what you learn here in the Conclusions of the tutorial.

RSS

Depending on who you ask, RSS stands for Rich Site Summary or Really Simple Syndication. From the RSS Primer [1], RSS:

• … is a format for delivering regularly changing web content. Many websites syndicate their content as an "RSS Feed" to whoever wants it.
• … allows you easily to stay informed by retrieving the latest content directly from your sites of interest.

To see what an RSS feed looks like, try opening some of the URLs used in this tutorial in your browser. The openness of RSS has led to the creation of many interfaces for reading news efficiently and processing them in countless ways.

Getting RSS news and other notifications from all your sources – everything from personal blogs to comments in forums to media corporations – through one interface of your choice is not only a huge time saver, it can eliminate many of the annoyances and much of the surveillance built into social networks and services like Google or Facebook. It is quite hard to underestimate how much good RSS can do on these fronts, if users and publishers together just started to exploit its full potential.

RSS Aggregators

RSS aggregators are graphical or console programs that periodically download your RSS feeds of choice and display all the corresponding headlines in one window. I assembled the tricks describe here, because, as much as I like RSS, over time I grew more and more dissatisfied with the very concept behind all standard RSS aggregators. On one hand, I wanted something that would not distract me: no desktop notifications, scrolling tickers, or anything like that, thanks. I wanted to access the news when I felt like it. On the other hand, an RSS feed is basically little more than a list of web pages. Why should I browse headlines in a specialized program, instead of the browser I'm already using, heavily configured just to browse web pages? Last but not least, I wanted something as automatic and WM-independent as possible.

Retrieving RSS Feeds

To embed headlines in your desktop, you first need a system that downloads, parses, and reorders your RSS feeds. The very simple way I am going to show you is easy to extend or reimplement in other languages. The tools you will need are:

• A short Python script (Listing 1) that downloads one feed and converts its most important data to a simple format.

• A Bash script that uses the Python script to download all the feeds and merge the result into a list of headlines.

Lines 3 to 6 of get_rss.py (Listing 1) load the Python libraries needed to read command-line arguments, connect to websites, and format dates. Lines 8 and 9 set up a socket (i.e., a communication channel) and its related timeout. The arguments passed to the script are saved in the feed_name and feed_url variables.

The real magic happens in line 13, where, using the previously established socket, the feedparser module retrieves and dumps the whole RSS feed into the d array. Without getting into detail, each element of that array contains the complete set of data (publication date, title, URL, and more) for a single news item.

The loop in lines 15 to 24 simply extracts and prints to standard output – prepending feed_name – the only fields of each item I need to proceed: date, title, and URL (see s.link in line 24).

Lines 16 to 19 check to see whether the date of publication of an item exists (line 16) and, if so, saves it to the date variable in a format that is both readable and suitable for further processing.

All the other scripts that use get_rss.py output need one headline per line with fields separated by pipe symbols; therefore, lines 20 to 23 retrieve the title of the current item, replace newlines with spaces, and replace any pipes that might already be in the title with double dashes.

If you run get_rss.py at the prompt with LinMag as feed name and the Linux Magazine feed URL,

marco~> get_rss.py LinMag http://www.linux-magazine.com/rss/feed/lmi_news

your result will look like Listing 2 (note that for readability in all listings like this, I replace all real URLs with <URL> and shorten all titles).

As already mentioned, the next step to efficient RSS usage is to run get_rss.py from another script that I call rssmixer (Listing 3). This script reads the RSS feeds it should download from a plain text file (FEED_LIST, line 4) in a simple format. Each line contains the URL of one RSS feed, preceded by a label of your choice and separated by the pipe character; for example:

CNNtop|http://rss.cnn.com/rss/edition_us.rss
LinMag|http://www.linux-magazine.com/rss/feed/lmi_news

The loop in lines 8 to 13 reads $FEED_LIST one line at a time, splitting it with the cut command to save the $FEED_NAME and $FEED_URL of the current feed in dedicated variables. In line 12, get_rss.py uses those variables to dump titles, URLs, and dates of the current feed into the file $FEED_DIR/$FEED_NAME.feed.

Line 15 first extracts from all the files in $FEED_DIR the lines that contain URLs (grep http), removing the file names (with cut). The resulting lines are then sorted on their second field (-k 2), in reverse order (-r); only the first five (head -5) are saved to the $RSSMIX file, resulting in a file like that in Listing 4.

Of all the lines in this tutorial, line 15 in Listing 3 is the one you should customize the most to suit your own needs and preferences. As is, it returns only the five most recent headlines from all feeds combined – from newest to oldest. However, this is not the only possibility. The headlines you see with these scripts depend on how you tweak that line and how often you run rssmixer as a cron job.

I'll return to this issue in the Conclusions, but now I'll show you how to use the rssmixer results to build icons and menus for your news on your Linux desktop.