The Newsbeuter RSS Feedreader

Andreas Krennmair


Table of Contents

Introduction
Why "Newsbeuter"?
Installation
Downloading Newsbeuter
Dependencies
Compiling and Installing
First Steps
Example Configuration
Configuring Colors
Migrating from other RSS Feed Readers
Advanced Features
Tagging
Scripts and Filters (Snownews Extensions)
Bookmarking
Command Line
Filter Language
Killfiles
Query Feeds
Bloglines Synchronization
OPML Online Subscription Mode
Flagging Articles
Macro Support
Format Strings
Using SQLite Triggers with newsbeuter
Feedback
License

Introduction

Newsbeuter is an RSS feedreader. RSS is a number of widely-used XML formats to transmit, publish and syndicate articles, for example news or blog articles. Newsbeuter is designed to be used on text terminals on Unix or Unix-like systems such as Linux, FreeBSD or Mac OS X. NetBSD is currently not supported, due to technical limitations in the iconv() implementation.

Why "Newsbeuter"?

"Newsbeuter" is a pun on the German word "Wildbeuter", which means "hunter-gatherer". During the stone age, people hunted and gathered their food, and these days, they hunt and gather news and information. Credits for this idea goes to Clifford Wolf, who submitted it to a little competiton that was started when I got aware that the original name would violate French and European registered trademarks.

Installation

This chapter describes how to compile and install newsbeuter from source.

Downloading Newsbeuter

Newsbeuter is available as source package. Simply go to http://synflood.at/newsbeuter.html and download the latest source package, which is usually in the .tar.gz file format. Alternatively, you can check out the latest development source tree from the newsbeuter SVN repository by running the following command on the commandline:

svn co http://bereshit.synflood.at/svn/noos/trunk

Dependencies

Newsbeuter depends on a number of libraries to function correctly. This table lists these dependencies. Please be aware that the list libraries may themselves depend on other libraries. These dependencies are not listed here. Please also be aware that you need a recent C++ compiler. Currently, newsbeuter has only been tested with GCC.

If you intend to modify the filter language parser, you will also need Coco/R for C++, which you can download from http://www.ssw.uni-linz.ac.at/coco/. The Coco/R binary must be installed as "coco-cpp" in your PATH. Debian users only need to install the package "coco-cpp".

Compiling and Installing

After you've downloaded and installed the dependencies mentioned above, you can start compiling and installing newsbeuter. To compile newsbeuter, simply run "make" in the source tree. After a short time, this should complete successfully, and you can go on with installation by running "make install". By default, this will install the "newsbeuter" binary to the /usr/local/bin directory. You can provide an alternative installation path using the prefix parameter, e.g. running "make install prefix=/opt/newsbeuter" will install the binary to the directory /opt/newsbeuter/bin.

First Steps

After you've installed newsbeuter, you can run it for the first time by typing "newsbeuter" on your command prompt. This will bring you the following message:

Error: no URLs configured. Please fill the file /Users/ak/.newsbeuter/urls with RSS feed URLs or import an OPML file.
newsbeuter 0.8
usage: ./newsbeuter [-i <file>|-e] [-u <urlfile>] [-c <cachefile>] [-h]
        -r              refresh feeds on start
        -e              export OPML feed to stdout
        -i <file>       import OPML file
        -u <urlfile>    read RSS feed URLs from <urlfile>
        -c <cachefile>  use <cachefile> as cache file
        -C <configfile> read configuration from <configfile>
        -v              clean up cache thoroughly
        -h              this help

This means that newsbeuter can't start without any configured feeds. To add feeds to newsbeuter, you can either add URLs to the configuration file $HOME/.newsbeuter/urls or you can import an OPML file by running "newsbeuter -i blogroll.opml". To manually add URLs, open the file with your favorite text editor and add the URLs, one per line:

http://rss.cnn.com/rss/cnn_topstories.rss
http://newsrss.bbc.co.uk/rss/newsonline_world_edition/front_page/rss.xml

If you need to add URLs that have restricted access via username/password, simply provide the username/password in the following way:

http://username:password@hostname.domain.tld/feed.rss

In order to protect username and password, make sure that $HOME/.newsbeuter/urls has the appropriate permissions.

Now you can run newsbeuter again, and it will present you with a controllable list of the URLs that you configured previously. You can now start downloading the feeds, either by pressing "R" to download all feeds, or by pressing "r" to download the currently selected feed. You can then select a feed you want to read, and by pressing "Enter", you can go to the article list for this feed. This works even while the downloading is still in progress. You can now see the list of available articles by their title. A "N" on the left indicates that an article wasn't read yet. Pressing Enter brings you to the content of the article. You can scroll through this text, and also run a browser (default: lynx) to view the complete article if the content is empty or just an abstract or a short description. Pressing "q" brings you back to the article list, and pressing "q" again brings you back to the feed list. Pressing "q" a third time then closes newsbeuter.

Newsbeuter caches the article that it downloads. This means that when you start newsbeuter again and reload a feed, the old articles can still be read even if they aren't in the current RSS feeds anymore. Optionally you can configure how many articles shall be preserved by feed so that the article backlog doesn't grow endlessly (see below).

Several aspects of newsbeuter's behaviour can be configured via a configuration file, by default $HOME/.newsbeuter/config. This configuration files contains lines in the form "<config-command> <arg1> …". The configuration file can also contain comments, which start with the # character and go as far as the end of line. If you need to enter a configuration argument that contains spaces, use quotes (") around the whole argument.

Searching for articles is possible in newsbeuter, too. Just press the "/" key, enter your search phrase, and the title and content of all articles are searched for it. When you do a search from the list of feeds, all articles of all feeds will be searched. When you do a search from the article list of a feed, only the articles of the currently viewed feed are searched.

Table 1. Configuration Commands

Configuration Command Argument(s) Default Description Example
always-display-description [true/false] false If true, then the description will always displayed even if e.g. a content:encoded tag has been found. always-display-description true
article-sort-order <sortfield>[-<direction>] date The sortfield specifies which article property shall be used for sorting (currently available: date, title, flags, author, link, guid). The optional direction specifies the sort direction ("asc" specifies ascending sorting, "desc" specifies descending sorting. for date, "desc" is default, for all others, "asc" is default). article-sort-order author-desc
articlelist-format <format> "%4i %f %D %?T?;%-17T; ?%t" This variable defines the format of entries in the article list. See below for more information on format strings (note that the semicolon should actually be a vertical bar; this is a limitation in AsciiDoc). articlelist-format "%4i %f %D %?T?;%-17T; ?%t"
auto-reload [yes/no] no If enabled, all feeds will be automatically reloaded at start up and then continuously after a certain time has passed (see reload-time). auto-reload yes
bind-key <key> <operation> n/a Bind key <key> to <operation>. This means that whenever <key> is pressed, then <operation> is executed (if applicable in the current context). A list of available operations can be found below. bind-key ^R reload-all
bookmark-cmd <command> "" If set, the configured command will be used as bookmark plugin. Please refer to the documentation for further information on bookmark plugins. bookmark-cmd "~/bin/my-bookmark-plugin.sh"
bloglines-auth <login>:<password> "" Set the Bloglines username and password. This is necessary for the Bloglines synchronization mode. bloglines-auth "username@emailaddress.com:mypassword"
bloglines-mark-read [yes/no] no If set to yes, it will mark all articles that are downloaded from Bloglines as read. bloglines-mark-read yes
bookmark-cmd <bookmark-command> "" If set, then <bookmark-command> will be used as bookmarking plugin. See the documentation on bookmarking for further information. bookmark-cmd "~/bin/delicious-bookmark.sh"
browser <browser-command> lynx Set the browser command to use when opening an article in the browser. browser w3m
cache-file <path> "~/.newsbeuter/cache.db" This configuration option sets the cache file. This is especially useful if the filesystem of your home directory doesn't support proper locking (e.g. NFS). cache-file "/tmp/testcache.db"
cleanup-on-quit [yes/no] yes If yes, then the cache gets locked and superfluous feeds and items are removed, such as feeds that can't be found in the urls configuration file anymore. cleanup-on-quit no
color <element> <fgcolor> <bgcolor> [<attr> …] n/a Set the foreground color, background color and optional attributes for a certain element color background white black
confirm-exit [yes/no] no If set to yes, then newsbeuter will ask for confirmation whether the user really wants to quit newsbeuter. confirm-exit yes
datetime-format <date/time format> %b %d This format specifies the date/time format in the article list. For a detailed documentation on the allowed formats, consult the manpage of strftime(3). datetime-format "%D, %R"
define-filter <name> <filter> n/a With this command, you can predefine filters, which can you later select from a list, and which are then applied after selection. This is especially useful for filters that you need often and you don't want to enter them every time you need them. define-filter "all feeds with fun tag" "tags # \"fun\""
error-log <path> "" If set, then user errors (e.g. errors regarding defunct RSS feeds) will be logged to this file. error-log "~/.newsbeuter/error.log"
feedlist-format <format> "%4i %n %11u %t" This variable defines the format of entries in the feed list. See below for more information on format strings. feedlist-format " %n %4i - %11u -%> %t"
html-renderer <path> internal If set to "internal", then the internal HTML renderer will be used. Otherwise, the specified command will be executed, the HTML to be rendered will be written to the command's stdin, and the program's output will be displayed. This makes it possible to use other, external programs, such as w3m, links or lynx, to render HTML. html-renderer "w3m -dump -T text/html"
ignore-article <feed> <filterexpr> n/a If a downloaded article from <feed> matches <filterexpr>, then it is ignored and not presented to the user. This command is further explained in the "kill file" section below. ignore-article "*" "title =~ \"Windows\""
include <path> n/a With this command, you can include other files to be interpreted as configuration files. This is especially useful to separate your configuration into several files, e.g. key configuration, color configuration, … include "~/.newsbeuter/colors"
macro <macro key> <command list> n/a With this command, you can define a macro key and specify a list of commands that shall be executed when the macro prefix and the macro key are pressed. macro k open ; reload ; quit
max-items <number> 0 Set the number of articles to maximally keep per feed. If the number is set to 0, then all articles are kept. max-items 100
notify-format <string> "newsbeuter: finished reload, %f unread feeds (%n unread articles total)" Format string that is used for formatting notifications. See the chapter on format strings for more information. notify-format "%d new articles (%n unread articles, %f unread feeds)"
notify-program <path> "" If set, then the configured program will be executed if new articles arrived (through a reload). The first parameter of the called program contains the notification message. notify-program "~/bin/my-notifier"
notify-screen [yes/no] no If yes, then a "privacy message" will be sent to the terminal, containing a notification message about new articles. This is especially useful if you use terminal emulations such as GNU screen which implement privacy messages. notify-screen yes
notify-xterm [yes/no] no If yes, then the xterm window title will be set to a notification message about new articles. notify-xterm yes
podcast-auto-enqueue [yes/no] no If yes, then all podcast URLs that are found in articles are added to the podcast download queue. See below for more information on podcast support in newsbeuter. podcast-auto-enqueue yes
opml-url <url> "" If the OPML online subscription mode is enabled, then the list of feeds will be taken from the OPML file found on this location. opml-url "http://host.domain.tld/blogroll.opml"
proxy <server:port> n/a Set the proxy to use for downloading RSS feeds. proxy localhost:3128
proxy-auth <auth> n/a Set the proxy authentication string. proxy-auth user:password
refresh-on-startup [yes/no] no If yes, then all feeds will be reloaded when newsbeuter starts up. This is equivalent to the -r commandline option. refresh-on-startup yes
reload-only-visible-feeds [yes/no] no If yes, then manually reloading all feeds will only reload the currently visible feeds, e.g. if a filter or a tag is set. reload-only-visible-feeds yes
reload-time <number> 60 The number of minutes between automatic reloads. reload-time 120
save-path <path> ~/ The default path where articles shall be saved to. If an invalid path is specified, the current directory is used. save-path "~/Saved Articles"
show-read-feeds [yes/no] yes If yes, then all feeds, including those without unread articles, are listed. If no, then only feeds with one or more unread articles are list. show-read-feeds no
suppress-first-reload [yes/no] no If yes, then the first automatic reload will be suppressed if auto-reload is set to yes. suppress-first-reload yes
text-width <number> 0 If set to a number greater than 0, then all HTML will be rendered to this maximum line length. If set to 0, the terminal width will be used. text-width 72
unbind-key <key> n/a Unbind key <key>. This means that no operation is called when <key> is pressed. unbind-key R
urls-source <source> "local" This configuration command sets the source where URLs shall be retrieved from. By default, this is ~/.newsbeuter/urls. Alternatively, you can set it to "bloglines", which enables newsbeuter's Bloglines synchronization mode, or to "opml", which enables newsbeuter's OPML online subscription mode. urls-source "bloglines"
use-proxy [yes/no] no If yes, then the configured proxy will be used for downloading the RSS feeds. use-proxy yes
user-agent <user agent string> "" If set to a non-zero-length string, this value will be used as HTTP User-Agent header for all HTTP requests. user-agent "Lynx/2.8.5rel.1 libwww-FM/2.14"

Table 2. Available Operations

Operation:Default key:Description
open ENTER Open the currently selected feed or article.
quit q Quit the program or return to the previous dialog (depending on the context).
reload r Reload the currently selected feed.
reload-all R Reload all feeds.
mark-feed-read A Mark all articles in the currently selected feed read.
mark-all-feeds-read C Mark articles in all feeds read.
save s Save the currently selected article to a file.
next-unread n Jump to the next unread article.
prev-unread p Jump to the previous unread article.
open-in-browser o Opens the URL associated with the current article.
help ? Runs the help screen.
toggle-source-view ^U Toggles between the HTML view and the source view in the article view.
toggle-article-read N Toggle the read flag for the currently selected article.
toggle-show-read-feeds l Toggle whether read feeds should be shown in the feed list.
show-urls u Show all URLs in the article in a list (similar to urlview).
clear-tag ^T Clear current tag.
select-tag t Select tag.
open-search / Opens the search dialog. When a search is done in the article list, then the search operation only applies to the articles of the current feed, otherwise to all articles.
enqueue e Add the podcast download URL of the current article (if any is found) to the podcast download queue (see below for more information on podcast support).
reload-urls ^R Reload the URLs configuration file.
redraw ^L Redraw the screen.
cmdline : Open the command line.
set-filter F Set a filter.
select-filter f Select a predefined filter.
clear-filter ^F Clear currently set filter.
bookmark ^B Bookmark currently selected article or URL.
edit-flags ^E Edit the flags of the currently selected article.
next-unread-feed ^N Go to the next feed with unread articles. This only works from the article list.
prev-unread-feed ^P Go to the previous feed with unread articles. This only works from the article list.
delete-article D Delete the currently selected article.
purge-deleted $ Purge all article that are marked as deleted from the article list.
up UP Goes up one item in the list.
down DOWN Goes down one item in the list.
pageup PPAGE Goes up one page in the list.
pagedown NPAGE Goes down one page in the list.

Keys, as used in the bind-key configuration command, use a special syntax. Lowercase keys, uppercase keys and special characters are written literally. The Enter key is written as "ENTER", while the Esc key is written as "ESC". The function keys F1 to F12 are written as "F1" to "F12". The Space key is written as "SPACE". Key combinations with the Ctrl key, such as Ctrl-R, are written as ^R. Please be aware that all Ctrl-related key combinations need to be written in uppercase. The following identifiers for keys are supported:

  • ENTER (Enter key)
  • BACKSPACE (backspace key)
  • LEFT (left cursor)
  • RIGHT (right cursor)
  • UP (up cursor)
  • PPAGE (page up cursor)
  • NPAGE (page down cursor)
  • DOWN (down cursor)
  • ESC (Esc key)

Example Configuration

# a comment
max-items        100 # such comments are possible, too
browser          links
show-read-feeds  no
unbind-key       R
bind-key         ^R    reload-all

Configuring Colors

It is possible to configure custom color settings in newsbeuter. The basic configuration syntax is:

color <element> <foreground color> <background color> [<attribute> ...]

This means that if you configure colors for a certain element, you need to provide a foreground color and a background color as a minimum. The following colors are supported:

  • black
  • red
  • green
  • yellow
  • blue
  • magenta
  • cyan
  • white
  • default

The "default" color means that the terminal's default color will be used.

Optionally, you can also add one or more attributes. The following attributes are supported:

  • standout
  • underline
  • reverse
  • blink
  • dim
  • bold
  • protect
  • invis

Currently, the following elements are supported:

  • listnormal: a normal list item
  • listfocus: the currently selected list item
  • info: the info bars on top and bottom
  • background: the application background
  • article: the article text

The default color configuration of newsbeuter looks like this:

background   white   black
listnormal   white   black
listfocus    yellow  blue   bold
info         yellow  blue   bold
article      white   black

Migrating from other RSS Feed Readers

It is very likely that you have used other RSS feed readers before. In this case, it is practical to migrate the previous configuration to newsbeuter. The vast amount of RSS feed readers allows the export of subscriptions via OPML files. OPML is an XML file format that was designed to save outlines, and has found its primary use in the import and export of feed subscriptions between different RSS feed readers.

The best thing to start with is to export your subscriptions from the old reader. Usually, RSS feed readers have appropriate menu itmes available to do so.

Snownews provides a script to convert your current subscription file into an OPML file:

snow2opml > ~/blogroll.opml

This command creates from your Snownews configuration a file blogroll.opml in your home directory. To export the subscription list from raggle, the following command is necessary:

raggle --export-opml ~/blogroll.opml

When you have exported the subscriptions from your old RSS feed reader, you can import them into newsbeuter:

newsbeuter -i ~/blogroll.opml

Don't worry, newsbeuter won't destroy your existing configuration, or add subscriptions more than once: every URL that is added to the subscription list is checked before whether it is already in the list, and is only added if not. This makes it possible to merge several OPML files into your subscription list.

If your old RSS feed reader was able to structure your subscriptions in hierarchies, and reflected this structure in the exported OPML file, newsbeuter doesn't throw away this information (although it doesn't support hierarchies), but generates tags from it. Tags are newsbeuter's way of organizing subscriptions in a non-hierarchical way. More information on the use of tags can be found below.

Imagine the following folder hierarchy:

|- News
| |- Europe
| `- International
|- IT
| |- Linux
| |- Windows
| `- Programming
|   |- C++
|   |- Ruby
|   `- Erlang
`- Private

Subscriptions found in the folder "Private" will be tagged with "Private", subscriptions in the folder "International" will be tagged with "News" and "News/International", subscriptions in the folder "Erlang" will be tagged ith "IT", "IT/Programming" and "IT/Programming/Erlang", and so on. This means that when you select the tag "Programming" in newsbeuter, you will see all subscriptions that were in the "Programming" folder or one of its subfolders before. This means that you will lose virtually nothing of your previously configured structure.

Advanced Features

Tagging

Newsbeuter comes with the possibility to categorize or "tag", as well call it, RSS feeds. Every RSS feed can be assigned 0 or more tags. Within newsbeuter, you can then select to only show RSS feeds that match a certain tag. That makes it easy to categorize your feeds in a flexible and powerful way.

Usually, the /.newsbeuter/urls file contains one RSS feed URL per line. To assign a tag to an RSS feed, simply attach it as a single word, separated by blanks such as space or tab. If the tag needs to contain spaces, you must use quotes (") around the tag (see example below). An example /.newsbeuter/urls file may look like this:

http://blog.fefe.de/rss.xml?html interesting conspiracy news "cool stuff"
http://rss.orf.at/news.xml news orf
http://www.heise.de/newsticker/heise.rdf news interesting

When you now start newsbeuter with this configuration, you can press "t" to select a tag. When you select the tag "news", you will see all three RSS feeds. Pressing "t" again and e.g. selecting the "conspiracy" tag, you will only see the http://blog.fefe.de/rss.xml?html RSS feed. Pressing "^T" clears the current tag, and again shows all RSS feeds, regardless of their assigned tags.

A special type of tag are tags that start with the tilde character (""). When such a tag is found, the feed title is set to the tag name (excluding the character). With this feature, you can give feeds any title you want in your feed list:

http://rss.orf.at/news.xml "~ORF News"

Scripts and Filters (Snownews Extensions)

From version 0.4 on, newsbeuter contains support for Snownews extensions. The RSS feed readers Snownews and Liferea share a common way of extending the readers with custom scripts. Two mechanisms, namely "execurl" and "filter" type scripts, are available and supported by newsbeuter.

An "execurl" script can be any program that gets executed and whose output is interpreted as RSS feed, while "filter" scripts are fed with the content of a configured URL and whose output is interpreted as RSS feed.

The configuration is simple and straight-forward. Just add to your ~/.newsbeuter/urls file configuration lines like the following ones:

exec:~/bin/execurl-script
filter:~/bin/filter-script:http://some.test/url

The first line shows how to add an execurl script to your configuration: start the line with "exec:" and then immediately append the path of the script that shall be executed. If this script requires additional parameters, simply use quotes:

"exec:~/bin/execurl-script param1 param2"

The second line shows how to add a filter script to your configuration: start the line with "filter:", then immediately append the path of the script, then append a colon (":"), and then append the URL of the file that shall be fed to the script. Again, if the script requires any parameters, simply quote:

"filter:~/bin/filter-script param1 param2:http://url/foobar"

In both cases, the tagging feature as described above is still available:

exec:~/bin/execurl-script tag1 tag2 "quoted tag"
filter:~/bin/filter-script:http://some.test/url tag3 tag4 tag5

A collection of such extension scripts can be found on this website: http://kiza.kcore.de/software/snownews/snowscripts/extensions

If you want to write your own extensions, refer to this website for further instructions: http://kiza.kcore.de/software/snownews/snowscripts/writing

Bookmarking

Since version 0.7, newsbeuter contains a plugin-based bookmarking system. When a user bookmarks a link (possible in the article list, in the article view, and in the URL view), he is asked for the URL to bookmark (already preset with the URL of the current selection), the bookmark title (in most cases preset with the title of the current selection) and the bookmark description. After the question for the description, an external program, configured via the configuration command "bookmark-cmd", is executed with 3 commandline parameters. The plugin itself implements the actual bookmark saving (e.g. writing the bookmark to an external file, or storing it to a del.icio.us account). When everything went OK, the plugin simply exits. In case something goes wrong while saving the bookmark, it writes out an error message as a single line. This error message is then presented to the user from within newsbeuter.

Newsbeuter comes with an example plugin, which implements a simple tab-separated bookmark file. This example can be found in the "doc" subdirectory.

Command Line

Like other text-oriented software, newsbeuter contains an internal commandline to modify configuration variables ad hoc and to run own commands. It provides a flexible access to the functionality of newsbeuter which is especially useful for advanced users.

To start the commandline, type ":". You will see a ":" prompt at the bottom of the screen, similar to tools like vi(m) or mutt. You can now enter commands. Pressing the return key executes the command (possibly giving feedback to the user) and closes the commandline. You can cancel entering commands by pressing the ESC key. Currently, the following commands are available:

Table 3. Available Commandline Commands

Command Syntax Description Example
quit quit Quit newsbeuter. quit
save save <filename> Save the currently select article to disk. This works in the article list and in the article view. save ~/important.txt
set set <variable>[=<value>] Set configuration variable <variable> to <value>. If no value is specified, the current value is printed out set reload-time=15
tag tag <tagname> Only display feeds with the tag <tagname>. tag news
goto goto <case-insensitive substring> Go to the next feed whose name contains the case-insensitive substring. goto foo
n/a <number> Jump to the entry with the index <number> (usually seen at the left side of the list). This currently works for the feed list and the article list. 30

Filter Language

Newsbeuter provides a powerful filter language that enables the user to filter the content of many dialogs, such as the feed list or the article list. The basic concept is that every feed and every article has a number of attributes which can then be compared with user-supplied values, and these comparisons and be logically AND'ed, OR'ed and grouped.

Examples for simple filter expressions are:

unread_count > 0
rssurl =~ "^https:"

Logically connecting and grouping such expressions looks like in the following examples:

( unread_count > 0 and unread_count < 10 ) or total_count > 100
( author =~ "Frank" or author =~ "John" ) and ( title =~ "Linux" or title =~ "FreeBSD" )

The possibilities for combining such queries is endless, sky (actually: the available memory) is the limit.

To filter your feeds, press "F" in the feed list, enter your filter expression, and press enter. To clear the filter, press Ctrl-F. To filter the articles in the article list, press "F", enter your expression, and press enter. Clearing the filter works the same as before. Be aware that only certain attributes work in both dialogs. The table below lists all available attributes and their context, i.e. an attribute that belongs to a feed can only be matched in the feed list, while an attribute that belongs to an article can only be matched in the article list.

Table 4. Available Comparison Operators

Operator Meaning
= test for equality ("==" works, too)
!= test for inequality; logical negation of = operator
=~ test whether regular expression matches
!~ logical negation of the =~ operator
< less than
> greater than
<= less than or equal
>= greater than or equal
# contains; this operator matches if a word is contained in a list of space-separated words (useful for matching tags, see below)
!# contains not; the negation of the # operator

Table 5. Available Attributes

Attribute Context Meaning
title article article title
link article article link
author article article author
content article article body
date article publication date of the article
guid article a unique identifier of the article
unread article indicates whether the article has been read
enclosure_url article the URL of a possible enclosure (e.g. podcast file)
enclosure_type article the MIME type of the enclosure URL
flags article The set of flags of the article
feedtitle feed, article title of the feed
description feed, article feed description
feedlink feed, article link to the feed
feeddate feed, article publication date of the feed
rssurl feed, article RSS URL of the feed
unread_count feed, article number of unread articles in the feed
total_count feed, article total number of articles in the feed
tags feed, article all tags that are associated with the feed

Note that it's also possible to filter for feed attributes when you query for article attributes. This is because every article is internally linked to the feed from which it was downloaded.

Killfiles

Sometimes, a user is confronted with certain content he doesn't want to read, e.g. on topics the user is not interested in or articles from certain people he doesn't want to read. In Usenet, such functionality within software is traditionally called a "killfile", i.e. based on the content of this "killfile", articles that match certain conditions do not get displayed and are not presented to the user at all.

In newsbeuter, such a "killfile" can be implemented on a per-article basis via the configuration file. The most important configuration command for this is "ignore-article":

ignore-article "*" "title =~ \"Gentoo\""
ignore-article "http://synflood.at/blog/index.php?/feeds/index.rss2" "title =~ \"newsbeuter\""

The basic format is that the user specifies an RSS feed for which the ignore shall be applied ("*" matches all RSS feeds), and then a filter expression (see previous section). If newsbeuter hits an article in the specified RSS feed that matches the specified filter expression, then this article is ignored and never presented to the user. The configuration itself can contain as many ignore-article commands as desired.

Query Feeds

Query feeds are a mechanism of newsbeuter to define custom "meta feeds" by using newsbeuter's built-in filter language. A query feed is a feed that is aggregated from all currently downloaded articles of all feeds. To narrow down the set of articles, the user has to specify a filter. Only articles that match this filter are added to the query feed. A query feed is updated whenever it is entered in the feed list. When you change the unread flag of an article, this is reflected in the feed where the article was originally fetched.

To define a query feed, the user has to add a line to the file ~/.newsbeuter/urls in the following format:

query:<name of feed>:<filter expression> [<tag> ...]

The "query:" in the beginning tells newsbeuter that it's a query feed, "<name of feed>" specifies the name under which the query feed shall be displayed in the feed list, and "<filter expression>" is the filter expression that shall be used. Like every other feed, a query feed can be tagged to organize it like a regular feed.

A good example for the user of this feature is a query feed that contains all unread articles:

"query:Unread Articles:unread = \"yes\""

Note the quotes that are necessary around the complete query "URL" and the backslashes that are necessary the escape the quotes in the filter expression.

If you want to combine several feeds to one single feed, a good solution is to tag the feeds that you want to combine with one certain tag, and then create a query feed that only displays articles from feeds with that certain tag:

http://domain1.tld/feed.xml fun news tag1
http://domain2.tld/?feed.rss private jokes tag1
http://domain3.tld/feeds.rss news
"query:tag1 Articles:tags # \"tag1\""

In this example, the feeds http://domain1.tld/feed.xml and http://domain2.tld/?feed.rss are aggregated into the query feed named "tag1 Articles", but the feed http://domain3.tld/feeds.rss is not.

Basically, the possibility of what can be realized with query feeds is only limited by what can be queried from articles and feeds with the filter language and by your creativity.

Bloglines Synchronization

Bloglines is a web-based RSS feed reader. Newsbeuter provides a synchronization feature that makes it possible to download unread articles from Bloglines (which can then be optionally be marked as read). This is useful in cases where you want to use the web frontend e.g. at work, and use newsbeuter e.g. when you're at home. It is also possible to use newsbeuter as an offline reader for Bloglines.

To enable Bloglines support, you first need to set the following configuration variable:

urls-source "bloglines"

By default, this is set to "local", which means that the list of configured URLs is retrieved from ~/.newsbeuter/urls. By setting it to "bloglines", newsbeuter will retrieve the list of retrieved URLs directly from Bloglines. This means that you can only either use the locally configured URLs or the URLs from Bloglines. Combination of both is not possible.

After setting the URLs source, you also need to set the Bloglines authentication configuration:

bloglines-auth "username@emailaddress.com:yourpassword"

In the configuration variable "bloglines-auth", you provide your Bloglines login (i.e. your email address) and your password, separated by a colon (":"). Optionally, you can configure that all messages that are downloaded are marked as read in the Bloglines web frontend:

bloglines-mark-read "yes"

After you've configured newsbeuter for Bloglines, you can start it up. Newsbeuter will then download the RSS feed URLs subscribed in Bloglines, which you can then reload.

In this configuration, you need the possibility to connect to Bloglines. In case you don't have access to Bloglines (e.g. a temporary network outage, or you sit in a location without connectivity, e.g. a train), you can start newsbeuter in offline mode:

newsbeuter -o

With this commandline switch, you can read the articles that you have already downloaded without any network connectivity. This functionality makes newsbeuter a true offline client for Bloglines.

OPML Online Subscription Mode

The OPML online subscription mode works similar to the Bloglines synchronization mode, except that no information about read articles is synchronized back. When enabled, all feeds that are listed in the feed list will be taken from an OPML that is downloaded from a freely configurable URL.

To enable this mode, the following configuration needs to be done:

urls-source "opml"
opml-url "<opml url>"

"opml" must be specified as source for the feed URLs, and the URL of the OPML file needs to be specified, too. As with Bloglines synchronization mode, the offline mode via "newsbeuter -o" also works with OPML online subscription mode.

Flagging Articles

To support custom categorization of articles by the user, it is possible to flag an article. A valid flag is any character from A to Z and from a to z. Every article can be flagged with up to 52 different flags, i.e. every letter from the Roman alphabet in upper and lower case. Flagging is easy: just select an article in the article list, or enter the article view, and press ^E. This will start the flag editor. By pressing enter, the new flags are saved. You can cancel by pressing the ESC key.

The flags of an article can be used in every filter expression. The flags of an article are always ordered, and when new flags are added, ordering is immediately restored. This behaviour can also be relied upon when querying articles via the filter language.

If an article contains one or more flags, it is marked with an "!" in the article list. In the article view, all flags (if available) are listed.

Macro Support

In newsbeuter, it's possible to define macros to execute more than one command at once. A macro is configured using the "macro" configuration command. The first parameter to "macro" is the key, all parameters afterwards are operations (as listed in the "Available Operations" table above), optionally with parameters on their own, separated by the ";" character. Here's a simple example:

macro k open ; reload ; quit

When the user presses the macro prefix ("," by default) and then the "k" key, the three operations "open", "reload" and "quit" will be executed subsequently.

Format Strings

Newsbeuter contains a powerful format string system to make it possible for the user to configure the format of various aspects of the application, such as the format of entries in the feed list or in the article list.

Format strings are similar to those that are found in the "printf" function in the C programming language. A format sequence begins with the % character, followed by optional alignment indication: positive numbers indicate that the text that is inserted for the sequence shall be padded right to a total width that is specified by the number, while negative number specify left padding. Followed by the padding indication comes the actual sequence identifier, which is usually a single letter.

In addition, newsbeuter provides other, more powerful sequences, such as "%>[char]", which indicates that the text right to the sequence will be aligned right on the screen, and characters between the text on the left and the text on the right will be filled by "[char]". Another powerful is the conditional sequence, "%?[char]?[format 1]&[format 2]?": if the text of the sequence identifier "[char]" is non-empty, then "[format 1]" will be evaluted and inserted, otherwise "[format 2]" will be evaluted and inserted. The "&" and "[format 2]" are optional, i.e. if the identifier's text is empty, then an empty string will be inserted.

The following tables show what sequence identifiers are available for which format:

Table 6. Available Identifiers for feedlist-format

Identifier Meaning
d Feed description
i Feed index
l Feed link
L Feed RSS URL
n "unread" flag field
t Feed title
u "unread/total" field

Table 7. Available Identifiers for articlelist-format

Identifier Meaning
a Article author
D Publication date
f Article flags
i Article index
t Article title
T If the article list displays articles from different feeds, then this identifier contains the title of the feed to which the article belongs.

Table 8. Available Identifiers for notify-format

Identifier Meaning
n Number of unread articles
f Number of unread feeds
d Number of new unread articles (i.e. that were added through the last reload)
D Number of new unread feeds (i.e. that were added through the last reload)
Examples
feedlist-format "%4i %n %11u %t"
articlelist-format "%4i %f %D %?T?|%-17T| ?%t"
notify-format "%d new articles (%n unread articles, %f unread feeds)"
Podcast Support

A podcast is a media file distributed over the internet using syndication feeds such as RSS, for later playback on portable players or computers. Newsbeuter contains support for downloading and saving podcasts. This support differs a bit from other podcast aggregators or "podcatchers" in how it is done.

Podcast content is transported in RSS feeds via special tags called "enclosures". Newsbeuter recognizes these enclosures and stores the relevant information for every podcast item it finds in an RSS feed. What the user then can do is to add the podcast download URL to a download queue. Alternatively, newsbeuter can be configured to automatically do that. This queue is stored in the file $HOME/.newsbeuter/queue.

The user can then use the download manager "podbeuter" to download these files to a directory on the local filesystem. Podbeuter comes with the newsbeuter package, and features a look and feel very close to the one of newsbeuter. It also shares the same configuration file.

Table 9. Podbeuter Configuration Commands

Configuration Command Argument(s) Default Description Example
download-path <path> ~/ Specifies the directory where podbeuter shall download the files to. download-path "~/Downloads"
max-downloads <number> 1 Specifies the maximum number of parallel downloads when automatic download is enabled. max-downloads 3
player <player command> "" Specifies the player that shall be used for playback of downloaded files. player "mp3blaster"

Table 10. Available Operations in Podbeuter

Operation Default key Description
quit q Quit the program.
pb-download d Download the currently selected URL.
pb-cancel c Cancel the currently selected download.
pb-delete D Delete the currently selected URL from the queue.
pb-purge P Remove all finished and deleted downloads from the queue and load URLs that were newly added to the queue.
pb-toggle-download-all a Toggle the "automatic download" feature where all queued URLs are downloaded one after the other. The "max-downloads" configuration option controls how many downloads are done in parallel.
pb-increase-max-dls + Increase the "max-downloads" option by 1.
pb-decrease-max-dls - Decrease the "max-downloads" option by 1. If the option is already 1, no further decrease is possible.

A usual "use case" is to configure newsbeuter to automatically enqueue newly found podcast download URLs. Then, the user reloads the podcast RSS feeds in newsbeuter, and after that, he/she uses podbeuter to view the current queue, and either selectively download certain files or automatically download them all together by pressing "a" within podbeuter.

Using SQLite Triggers with newsbeuter

This section was kindly provided by Elrond.

SQLite, the db used by newsbeuter, supports triggers. These are small snippets of SQL that get executed inside the database by the database engine. They're stored inside the db and the normal user (including newsbeuter itself) doesn't see them. Just the db seems to do some magic: Like changing some values when you change another value.

So what is this good for when looking at newsbeuter? Well first of it's a hack. The real answer should be to use application logic (do it inside newsbeuter, not in the db). So: Don't use this, unless you know, what you're doing, and unless you have some sort of backup.

Example

So after the "don't use it" you still want to know, what one can do? So here's an example.

Suppose you have a strange feed where the articles become "new" by just changing their subject, and nothing else changes. The body is just empty, and the URL keeps the same. This feed really exists. It's the "updated software rss feed" of some major company and the title just contains the name of the driver and version number. And the URL points to the download page. newsbeuter considers articles only as new, when they have a new UniqueID (this is good). So those articles are never marked as new (unread) ever again.

So what can we do? We do some magic: We let the db test if newsbeuter changes the subject and then let itself mark the article again as unread.

  1. You need the sqlite3 command line tool (available via apt-get install sqlite3 on Debian) or some other tool to do direct sql on the sqlite database.
  2. Start sqlite3 with the newsbeuter db:

    Rivendell:~/.newsbeuter% sqlite3 cache.db
    SQLite version 3.4.2
    Enter ".help" for instructions
    sqlite>
  3. Create the trigger:

    sqlite> create trigger update_item_title update of title on rss_item
              > for each row when old.title != new.title
              > begin
              >   update rss_item set unread = 1 where rowid == new.rowid;
              > end;
  4. Leave sqlite3 with <Ctrl-D> or .quit.

That's it. newsbeuter (well, its db) now marks articles as unread when their title changes. And nicely enough this works all inside newsbeuter, no need to restart it so that it rereads the cache, that magically modifies itself. It just works.

Feedback

If you want to tell us something related to newsbeuter, don't hesitate to send an email: ak-newsbeuter@synflood.at

Alternatively, you can reach the newsbeuter developers on IRC: channel #newsbeuter on irc.freenode.net.

If you want to report newsbeuter bugs, please use this issue tracker: http://code.google.com/p/newsbeuter/issues/list

License

MIT/X Consortium License

©opyright 2006-2008 Andreas Krennmair <ak@synflood.at>

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.