Hacks & Code

Und wieder einmal ein Eintrag der Sorte "schöne Ideen, die es wert sind, ausprobiert zu werden, aber wohl nie implementiert werden, außer es findet sich jemand, der dem Andreas $VIELGELD zuwirft":

mika und ich haben vor ein paar Tagen über das Problem der "database wedges" bei Subversion mit BDB-Backend und allgemein darüber, dass BDB scheisse ist, und immer wieder Datenverlust verursacht, diskutiert. Deswegen muss ein neues Backend her, das nichts mit der berüchtigten Berkeley DB zu tun hat.

Ich bin bei Recherchen dazu auf ein paar interessante Dinge gestoßen: einerseits gibt es libjio, mit dem man transaktionsorientiert auf Dateien schreiben kann, was sich ziemlich nett anhört. Weiters bin ich auf tdb gekommen, eine "Datenbank" ähnlich GDBM und BDB, jedoch ziemlich klein und mit Locking etc (naja, BDB kann das auch, aber GDBM nicht). Der Autor der libjio bietet auch einen Patch für tdb an, allerdings ist der ziemlich rudimentär, da er nicht die coolen libjio-Features nützt, sondern nur die Unix-I/O-Wrapper.

Meine Idee dazu ist, das ldbm-Backend von OpenLDAP dahingehend anzupassen, dass tdb als Alternative zu GDBM, NDBM, etc. verwendet werden kann, und tdb wiederum so zu erweitern, dass die libjio wesentlich besser integriert wird. Wenn das getan ist, und funktioniert, ist es natürlich interessant, sich die Performance dieses neuen Backend anzusehen, und mit dem BDB-Backend zu vergleichen, und dann die Robustheit bei großen Datenmengen und hoher Last über lange Zeit zu überprüfen. Bei mehr als 1000000 (in Worten: eine Million) Einträgen und hoher Last über mehrere Stunden ist mir bei Performance-Tests das BDB-Backend nämlich schon zweimal abgeraucht. Das ist übrigens auch meine negative Erfahrung mit BDB, neben den üblichen "database wedges" bei Subversion.

Auf jeden Fall: wer bereit ist, Geld für so eine Untersuchung auszugeben (wahrscheinlich eh keiner), möge sich bei mir melden.

Today, Nico and me released tpp 1.2, the ingenious text presentation program. That new release is completely refactored, in fact it shares probably five percent of the pre-1.2 version of tpp. We implemented all but one feature from the previous version -- we kicked out LaTeX support. But we added LinuxDoc support instead, but it's currently broken, so better don't use (or fix it, we're happy about every patch that we get). Other additional features are support for line wrapping, the possibility to set a footer and a header, a reload function where you can immediately reload the file that you've currently opened instead of quitting and restarting tpp, and support for transparent terminals.

One thing that I always regretted about mutt that hardly any new features were added. A few years ago, I wanted to have a certain feature included in mutt, but when I posted this on the mutt mailing list, the only answer I got was that the authors don't want to add any new features anymore. I forgot about this again, well, until a few days ago.

And that was when Sven Guckes was ranting about the mutt development process in at.linux, closing with the sentence "mutt is dead" (he got famous in Austria's Linux scene for his legendary demo session at Linuxdays 2001 in St. Pölten, with quotes like "elm is dead"). At around the same time, another annoyance of mutt hit me that I wanted to have fixed once and for all.

So I started searching around and found lots and lots of mutt patches that added very interesting and useful features or fixed that little annoyance that always itched but that was not annoying enough to have it fixed. So, I thought, why not use the creative potential of all these people out there, take the mutt source, and enhance it with a number of patches out there. And so I (finally) had the idea of forking mutt, making it a more feature-rich email client than mutt ever was. Because open source is about feature competition after all, isn't it? Another advanced email client would definitely heat up the email client "market" (if you can say so) in the open source scene. And so I forked, added a number of interesting patches (and I still keep adding), fixed a few bugs by myself, and put it online under the name "mutt next generation", or short, mutt-ng. Of course, development will be very active in the next few weeks/months, depending on how much I need to do to get a really nice email client.

I think a fork was clearly necessary, as mutt development got totally stuck after the last release, and forks at the right time in the past did either produce very viable alternatives or helped the original project. Emacs/XEmacs and gcc/egcs come to my mind. So, my hope is that the word gets spread, that people use and test mutt-ng, and don't hesitate to submit their own patches, because unlike the mutt developers, I'm very happy to integrate feature patches and bugfixes.

Today I stumbled across The Heirloom Toolchest, a collection of standard Unix utilities, taken from all the ancient Unix source code published by Caldera a few years ago. They're packaged ready-to-compile, and quite small, yet providing all the functionality you usually need. This package is definitely a good thing, as it provides a good base for building a small system based on diet libc.

Just in case somebody has to struggle with zlib's low-level inflate/deflate routines like me, you can find some example code here.

Today I uploaded klumpat, a simple logging framework for the C programming language, and especially for dietlibc applications.

The idea is that you have a set of specialized loggers, e.g. for the console, files, syslog or logging to other loggers, which you create with a specialized function, and then log to them with a unified logging function. Every logger has a loglevel, which specifies the threshold from which loglevel on messages shall be logged. One special feature is the so-called meta logger: it's a logger to dispatch a log message to 0 to n other loggers, e.g. a debug logfile (with loglevel DEBUG), an error logfile (with loglevel ERROR) and a console logger (with loglevel NOTICE). The API itself is trivial, as the following example shows:

klumpat flogger = kl_new_file("file",DEBUG,"testlog.txt");
klumpat clogger = kl_new_console("console",ERROR);
klumpat * logger = kl_new_meta("meta",DEBUG);
kl_add_logger(logger,flogger);
kl_add_logger(logger,clogger);
kl_log(logger,DEBUG,"this is a debug message");
kl_log(logger,NOTICE,"this is a notice");
kl_log(logger,ERROR,"this is an error message");
kl_delete(logger);

Another nice thing is that messages in different loglevels are getting colored differently when being printed to the console. That makes the visual perception of e.g. error messages much easier.

Why the name "klumpat"? Well, "klumpat" means (roughly translated) "crap". It was just a random name that came to my mind when I started implementing the whole thing.

Just for the records: I noticed a salt generation weakness in htpasswd when using it in MD5 mode and reported it to the Apache developers.

Update: nobody replied to my email, so I went to the Apache website where found out that you now need to submit patches via their BTS. gna Well, I did that too.

Removing memory leaks on Symbian OS is really a hairy thing. Just in case you have to do programming on Symbian OS, if you get a panic with the reason code "ALLOC 0x<32bit-hex>", use either the technique described on this page. If you don't have MS Visual Studio (like me, I'm stuck with Metroworks CodeWarrior), use the following code snippet:

TBuf<255> buf;

buf.Format(_L("CFileDescription address: %x"),(TUint)this);

LOG_DEBUG(buf);

Replace the LOG_DEBUG with your own logging facility, and put the code snippet into every object's constructor. If you encounter a panic as described above, find the hex value in your logfile, and you know what type of object it is that is not properly freed and thus causes the panic.

One of my programs, tpp (1, 2, 3), has finally entered Debian unstable: http://packages.debian.org/unstable/graphics/tpp. Many thanks go out to Nico Golde, who is both co-author and maintainer of the Debian package.

Today I announced on the diet libc mailing list that I managed to get the PostgreSQL [[Object-relational_database|ORDBMS]] running together with diet libc. This is pretty cool, IMHO, and the first widely-used DBMS that runs with diet libc.

In the last few days, I wrote yet another POP3 daemon (I already wrote one before), based on dietlibc and libowfat and using Maildir as storage backend. Expect a release soon, at least after I did more interoperability testing. BTW: currently, the stripped binary has a size of about 23k, statically linked! What would also be nice are performance tests. While I use fork() to handle several sessions simultaneously, I also used mmap(2) to map the mail files into memory instead of read(2)ing it into memory, just to make this process faster.

Last friday, tpp 0.2 has been released. tpp now does colors, sliding in text from left, right, the bottom and even the top. A good number of these features has been developed by Nico Golde, who joined the tpp development team, and also already created tpp Debian packages.



Another feature that will be fully usable by the next release is the tpp-to-LaTeX converter. As a small example, have a look at tpp-ac-am.tpp, an example presentation that can also be found in the source distribution, and at tpp-ac-am.pdf. The PDF has been generated from tpp-ac-am.tex which has been generated from the .tpp file by the latest development version of tpp. This looks quite promising already, doesn't it?

According to this page, the following 19 bytes of not quite valid "HTML" make IE crash (yes, I just tried it, and it works):

<style></style>@;/*

As requested in one comments, here are some tpp screenshots:

• a normal tpp slide


• tpp's slide overview


• tpp's formatting capabilities demonstrated


• that's what source code looks like with tpp

This weekend, I wrote tpp ("text presentation program"), an ncurses-based presentation tool. The idea is really simple: you write your presentation in a simple description language (similar to MagicPoint) using your favorite editor, and then display the slides on any text terminal that is supported ncurses, be it an old VT100, the Linux console w/ or w/o framebuffer or a simple xterm (or any other terminal emulator for X11).



I had the idea for tpp on my way home from Anif to Linz last friday, and as soon as I came home, I went to my computer, and started hacking it using Ruby. In the late evening, a first usable version was ready. And on saturday I made some cleanups and added a few more features. Actually, developing tpp went a lot quicker than I thought, but this is yet another example that Ruby is extremely good for rapid prototyping, and in this case, the prototype became the actual program. Or maybe not...? Rewrites of tpp in C & ncurses are gladly accepted.