The Newsbeuter RSS Feedreader

"Newsbeuter" is a pun on the German word "Wildbeuter", which means "hunter-gatherer". During the stone age, people hunted and gathered their food, and these days, they hunt and gather news and information. Credits for this idea goes to Clifford Wolf, who submitted it to a little competiton that was started when I got aware that the original name would violate French and European registered trademarks.

Newsbeuter is available as source package. Simply go to http://synflood.at/newsbeuter.html and download the latest source package, which is usually in the .tar.gz file format. Alternatively, you can check out the latest development source tree from the newsbeuter SVN repository by running the following command on the commandline:

svn co http://bereshit.synflood.at/svn/noos/trunk

Newsbeuter depends on a number of libraries to function correctly. This table lists these dependencies. Please be aware that the list libraries may themselves depend on other libraries. These dependencies are not listed here. Please also be aware that you need a recent C++ compiler. Currently, newsbeuter has only been tested with GCC.

If you intend to modify the filter language parser, you will also need Coco/R for C++, which you can download from http://www.ssw.uni-linz.ac.at/coco/. The Coco/R binary must be installed as "coco-cpp" in your PATH. Debian users only need to install the package "coco-cpp".

After you've downloaded and installed the dependencies mentioned above, you can start compiling and installing newsbeuter. To compile newsbeuter, simply run "make" in the source tree. After a short time, this should complete successfully, and you can go on with installation by running "make install". By default, this will install the "newsbeuter" binary to the /usr/local/bin directory. You can provide an alternative installation path using the prefix parameter, e.g. running "make install prefix=/opt/newsbeuter" will install the binary to the directory /opt/newsbeuter/bin.

# a comment
max-items        100 # such comments are possible, too
browser          links
show-read-feeds  no

unbind-key       R
bind-key         ^R    reload-all

It is possible to configure custom color settings in newsbeuter. The basic configuration syntax is:

color <element> <foreground color> <background color> [<attribute> ...]

This means that if you configure colors for a certain element, you need to provide a foreground color and a background color as a minimum. The following colors are supported:

The "default" color means that the terminal's default color will be used.

Optionally, you can also add one or more attributes. The following attributes are supported:

Currently, the following elements are supported:

The default color configuration of newsbeuter looks like this:

background   white   black
listnormal   white   black
listfocus    yellow  blue   bold
info         yellow  blue   bold
article      white   black

It is very likely that you have used other RSS feed readers before. In this case, it is practical to migrate the previous configuration to newsbeuter. The vast amount of RSS feed readers allows the export of subscriptions via OPML files. OPML is an XML file format that was designed to save outlines, and has found its primary use in the import and export of feed subscriptions between different RSS feed readers.

The best thing to start with is to export your subscriptions from the old reader. Usually, RSS feed readers have appropriate menu itmes available to do so.

Snownews provides a script to convert your current subscription file into an OPML file:

snow2opml > ~/blogroll.opml

This command creates from your Snownews configuration a file blogroll.opml in your home directory. To export the subscription list from raggle, the following command is necessary:

raggle --export-opml ~/blogroll.opml

When you have exported the subscriptions from your old RSS feed reader, you can import them into newsbeuter:

newsbeuter -i ~/blogroll.opml

Don't worry, newsbeuter won't destroy your existing configuration, or add subscriptions more than once: every URL that is added to the subscription list is checked before whether it is already in the list, and is only added if not. This makes it possible to merge several OPML files into your subscription list.

If your old RSS feed reader was able to structure your subscriptions in hierarchies, and reflected this structure in the exported OPML file, newsbeuter doesn't throw away this information (although it doesn't support hierarchies), but generates tags from it. Tags are newsbeuter's way of organizing subscriptions in a non-hierarchical way. More information on the use of tags can be found below.

Imagine the following folder hierarchy:

|- News
| |- Europe
| `- International
|- IT
| |- Linux
| |- Windows
| `- Programming
|   |- C++
|   |- Ruby
|   `- Erlang
`- Private

Subscriptions found in the folder "Private" will be tagged with "Private", subscriptions in the folder "International" will be tagged with "News" and "News/International", subscriptions in the folder "Erlang" will be tagged ith "IT", "IT/Programming" and "IT/Programming/Erlang", and so on. This means that when you select the tag "Programming" in newsbeuter, you will see all subscriptions that were in the "Programming" folder or one of its subfolders before. This means that you will lose virtually nothing of your previously configured structure.

Newsbeuter comes with the possibility to categorize or "tag", as well call it, RSS feeds. Every RSS feed can be assigned 0 or more tags. Within newsbeuter, you can then select to only show RSS feeds that match a certain tag. That makes it easy to categorize your feeds in a flexible and powerful way.

Usually, the _{/.newsbeuter/urls file contains one RSS feed URL per line. To assign a tag to an RSS feed, simply attach it as a single word, separated by blanks such as space or tab. If the tag needs to contain spaces, you must use quotes (") around the tag (see example below). An example} /.newsbeuter/urls file may look like this:

http://blog.fefe.de/rss.xml?html interesting conspiracy news "cool stuff"
http://rss.orf.at/news.xml news orf
http://www.heise.de/newsticker/heise.rdf news interesting

When you now start newsbeuter with this configuration, you can press "t" to select a tag. When you select the tag "news", you will see all three RSS feeds. Pressing "t" again and e.g. selecting the "conspiracy" tag, you will only see the http://blog.fefe.de/rss.xml?html RSS feed. Pressing "^T" clears the current tag, and again shows all RSS feeds, regardless of their assigned tags.

A special type of tag are tags that start with the tilde character ("_{"). When such a tag is found, the feed title is set to the tag name (excluding the} character). With this feature, you can give feeds any title you want in your feed list:

http://rss.orf.at/news.xml "~ORF News"

From version 0.4 on, newsbeuter contains support for Snownews extensions. The RSS feed readers Snownews and Liferea share a common way of extending the readers with custom scripts. Two mechanisms, namely "execurl" and "filter" type scripts, are available and supported by newsbeuter.

An "execurl" script can be any program that gets executed and whose output is interpreted as RSS feed, while "filter" scripts are fed with the content of a configured URL and whose output is interpreted as RSS feed.

The configuration is simple and straight-forward. Just add to your ~/.newsbeuter/urls file configuration lines like the following ones:

exec:~/bin/execurl-script
filter:~/bin/filter-script:http://some.test/url

The first line shows how to add an execurl script to your configuration: start the line with "exec:" and then immediately append the path of the script that shall be executed. If this script requires additional parameters, simply use quotes:

"exec:~/bin/execurl-script param1 param2"

The second line shows how to add a filter script to your configuration: start the line with "filter:", then immediately append the path of the script, then append a colon (":"), and then append the URL of the file that shall be fed to the script. Again, if the script requires any parameters, simply quote:

"filter:~/bin/filter-script param1 param2:http://url/foobar"

In both cases, the tagging feature as described above is still available:

exec:~/bin/execurl-script tag1 tag2 "quoted tag"
filter:~/bin/filter-script:http://some.test/url tag3 tag4 tag5

A collection of such extension scripts can be found on this website: http://kiza.kcore.de/software/snownews/snowscripts/extensions

If you want to write your own extensions, refer to this website for further instructions: http://kiza.kcore.de/software/snownews/snowscripts/writing

Since version 0.7, newsbeuter contains a plugin-based bookmarking system. When a user bookmarks a link (possible in the article list, in the article view, and in the URL view), he is asked for the URL to bookmark (already preset with the URL of the current selection), the bookmark title (in most cases preset with the title of the current selection) and the bookmark description. After the question for the description, an external program, configured via the configuration command "bookmark-cmd", is executed with 3 commandline parameters. The plugin itself implements the actual bookmark saving (e.g. writing the bookmark to an external file, or storing it to a del.icio.us account). When everything went OK, the plugin simply exits. In case something goes wrong while saving the bookmark, it writes out an error message as a single line. This error message is then presented to the user from within newsbeuter.

Newsbeuter comes with an example plugin, which implements a simple tab-separated bookmark file. This example can be found in the "doc" subdirectory.

Like other text-oriented software, newsbeuter contains an internal commandline to modify configuration variables ad hoc and to run own commands. It provides a flexible access to the functionality of newsbeuter which is especially useful for advanced users.

To start the commandline, type ":". You will see a ":" prompt at the bottom of the screen, similar to tools like vi(m) or mutt. You can now enter commands. Pressing the return key executes the command (possibly giving feedback to the user) and closes the commandline. You can cancel entering commands by pressing the ESC key. Currently, the following commands are available:

Newsbeuter provides a powerful filter language that enables the user to filter the content of many dialogs, such as the feed list or the article list. The basic concept is that every feed and every article has a number of attributes which can then be compared with user-supplied values, and these comparisons and be logically AND'ed, OR'ed and grouped.

Examples for simple filter expressions are:

unread_count > 0
rssurl =~ "^https:"

Logically connecting and grouping such expressions looks like in the following examples:

( unread_count > 0 and unread_count < 10 ) or total_count > 100
( author =~ "Frank" or author =~ "John" ) and ( title =~ "Linux" or title =~ "FreeBSD" )

The possibilities for combining such queries is endless, sky (actually: the available memory) is the limit.

To filter your feeds, press "F" in the feed list, enter your filter expression, and press enter. To clear the filter, press Ctrl-F. To filter the articles in the article list, press "F", enter your expression, and press enter. Clearing the filter works the same as before. Be aware that only certain attributes work in both dialogs. The table below lists all available attributes and their context, i.e. an attribute that belongs to a feed can only be matched in the feed list, while an attribute that belongs to an article can only be matched in the article list.

Note that it's also possible to filter for feed attributes when you query for article attributes. This is because every article is internally linked to the feed from which it was downloaded.

Sometimes, a user is confronted with certain content he doesn't want to read, e.g. on topics the user is not interested in or articles from certain people he doesn't want to read. In Usenet, such functionality within software is traditionally called a "killfile", i.e. based on the content of this "killfile", articles that match certain conditions do not get displayed and are not presented to the user at all.

In newsbeuter, such a "killfile" can be implemented on a per-article basis via the configuration file. The most important configuration command for this is "ignore-article":

ignore-article "*" "title =~ \"Gentoo\""
ignore-article "http://synflood.at/blog/index.php?/feeds/index.rss2" "title =~ \"newsbeuter\""

The basic format is that the user specifies an RSS feed for which the ignore shall be applied ("*" matches all RSS feeds), and then a filter expression (see previous section). If newsbeuter hits an article in the specified RSS feed that matches the specified filter expression, then this article is ignored and never presented to the user. The configuration itself can contain as many ignore-article commands as desired.

Query feeds are a mechanism of newsbeuter to define custom "meta feeds" by using newsbeuter's built-in filter language. A query feed is a feed that is aggregated from all currently downloaded articles of all feeds. To narrow down the set of articles, the user has to specify a filter. Only articles that match this filter are added to the query feed. A query feed is updated whenever it is entered in the feed list. When you change the unread flag of an article, this is reflected in the feed where the article was originally fetched.

To define a query feed, the user has to add a line to the file ~/.newsbeuter/urls in the following format:

query:<name of feed>:<filter expression> [<tag> ...]

The "query:" in the beginning tells newsbeuter that it's a query feed, "<name of feed>" specifies the name under which the query feed shall be displayed in the feed list, and "<filter expression>" is the filter expression that shall be used. Like every other feed, a query feed can be tagged to organize it like a regular feed.

A good example for the user of this feature is a query feed that contains all unread articles:

"query:Unread Articles:unread = \"yes\""

Note the quotes that are necessary around the complete query "URL" and the backslashes that are necessary the escape the quotes in the filter expression.

If you want to combine several feeds to one single feed, a good solution is to tag the feeds that you want to combine with one certain tag, and then create a query feed that only displays articles from feeds with that certain tag:

http://domain1.tld/feed.xml fun news tag1
http://domain2.tld/?feed.rss private jokes tag1
http://domain3.tld/feeds.rss news
"query:tag1 Articles:tags # \"tag1\""

In this example, the feeds http://domain1.tld/feed.xml and http://domain2.tld/?feed.rss are aggregated into the query feed named "tag1 Articles", but the feed http://domain3.tld/feeds.rss is not.

Basically, the possibility of what can be realized with query feeds is only limited by what can be queried from articles and feeds with the filter language and by your creativity.

Bloglines is a web-based RSS feed reader. Newsbeuter provides a synchronization feature that makes it possible to download unread articles from Bloglines (which can then be optionally be marked as read). This is useful in cases where you want to use the web frontend e.g. at work, and use newsbeuter e.g. when you're at home. It is also possible to use newsbeuter as an offline reader for Bloglines.

To enable Bloglines support, you first need to set the following configuration variable:

urls-source "bloglines"

By default, this is set to "local", which means that the list of configured URLs is retrieved from ~/.newsbeuter/urls. By setting it to "bloglines", newsbeuter will retrieve the list of retrieved URLs directly from Bloglines. This means that you can only either use the locally configured URLs or the URLs from Bloglines. Combination of both is not possible.

After setting the URLs source, you also need to set the Bloglines authentication configuration:

bloglines-auth "username@emailaddress.com:yourpassword"

In the configuration variable "bloglines-auth", you provide your Bloglines login (i.e. your email address) and your password, separated by a colon (":"). Optionally, you can configure that all messages that are downloaded are marked as read in the Bloglines web frontend:

bloglines-mark-read "yes"

After you've configured newsbeuter for Bloglines, you can start it up. Newsbeuter will then download the RSS feed URLs subscribed in Bloglines, which you can then reload.

In this configuration, you need the possibility to connect to Bloglines. In case you don't have access to Bloglines (e.g. a temporary network outage, or you sit in a location without connectivity, e.g. a train), you can start newsbeuter in offline mode:

newsbeuter -o

With this commandline switch, you can read the articles that you have already downloaded without any network connectivity. This functionality makes newsbeuter a true offline client for Bloglines.

The OPML online subscription mode works similar to the Bloglines synchronization mode, except that no information about read articles is synchronized back. When enabled, all feeds that are listed in the feed list will be taken from an OPML that is downloaded from a freely configurable URL.

To enable this mode, the following configuration needs to be done:

urls-source "opml"
opml-url "<opml url>"

"opml" must be specified as source for the feed URLs, and the URL of the OPML file needs to be specified, too. As with Bloglines synchronization mode, the offline mode via "newsbeuter -o" also works with OPML online subscription mode.

To support custom categorization of articles by the user, it is possible to flag an article. A valid flag is any character from A to Z and from a to z. Every article can be flagged with up to 52 different flags, i.e. every letter from the Roman alphabet in upper and lower case. Flagging is easy: just select an article in the article list, or enter the article view, and press ^E. This will start the flag editor. By pressing enter, the new flags are saved. You can cancel by pressing the ESC key.

The flags of an article can be used in every filter expression. The flags of an article are always ordered, and when new flags are added, ordering is immediately restored. This behaviour can also be relied upon when querying articles via the filter language.

If an article contains one or more flags, it is marked with an "!" in the article list. In the article view, all flags (if available) are listed.

In newsbeuter, it's possible to define macros to execute more than one command at once. A macro is configured using the "macro" configuration command. The first parameter to "macro" is the key, all parameters afterwards are operations (as listed in the "Available Operations" table above), optionally with parameters on their own, separated by the ";" character. Here's a simple example:

macro k open ; reload ; quit

When the user presses the macro prefix ("," by default) and then the "k" key, the three operations "open", "reload" and "quit" will be executed subsequently.

Newsbeuter contains a powerful format string system to make it possible for the user to configure the format of various aspects of the application, such as the format of entries in the feed list or in the article list.

Format strings are similar to those that are found in the "printf" function in the C programming language. A format sequence begins with the % character, followed by optional alignment indication: positive numbers indicate that the text that is inserted for the sequence shall be padded right to a total width that is specified by the number, while negative number specify left padding. Followed by the padding indication comes the actual sequence identifier, which is usually a single letter.

In addition, newsbeuter provides other, more powerful sequences, such as "%>[char]", which indicates that the text right to the sequence will be aligned right on the screen, and characters between the text on the left and the text on the right will be filled by "[char]". Another powerful is the conditional sequence, "%?[char]?[format 1]&[format 2]?": if the text of the sequence identifier "[char]" is non-empty, then "[format 1]" will be evaluted and inserted, otherwise "[format 2]" will be evaluted and inserted. The "&" and "[format 2]" are optional, i.e. if the identifier's text is empty, then an empty string will be inserted.

The following tables show what sequence identifiers are available for which format:

A podcast is a media file distributed over the internet using syndication feeds such as RSS, for later playback on portable players or computers. Newsbeuter contains support for downloading and saving podcasts. This support differs a bit from other podcast aggregators or "podcatchers" in how it is done.

Podcast content is transported in RSS feeds via special tags called "enclosures". Newsbeuter recognizes these enclosures and stores the relevant information for every podcast item it finds in an RSS feed. What the user then can do is to add the podcast download URL to a download queue. Alternatively, newsbeuter can be configured to automatically do that. This queue is stored in the file $HOME/.newsbeuter/queue.

The user can then use the download manager "podbeuter" to download these files to a directory on the local filesystem. Podbeuter comes with the newsbeuter package, and features a look and feel very close to the one of newsbeuter. It also shares the same configuration file.

A usual "use case" is to configure newsbeuter to automatically enqueue newly found podcast download URLs. Then, the user reloads the podcast RSS feeds in newsbeuter, and after that, he/she uses podbeuter to view the current queue, and either selectively download certain files or automatically download them all together by pressing "a" within podbeuter.

This section was kindly provided by Elrond.

SQLite, the db used by newsbeuter, supports triggers. These are small snippets of SQL that get executed inside the database by the database engine. They're stored inside the db and the normal user (including newsbeuter itself) doesn't see them. Just the db seems to do some magic: Like changing some values when you change another value.

So what is this good for when looking at newsbeuter? Well first of it's a hack. The real answer should be to use application logic (do it inside newsbeuter, not in the db). So: Don't use this, unless you know, what you're doing, and unless you have some sort of backup.