This is yet another release entry of digup. This time, however, it is a major release with lots of new improvements and some old fixes:

• Added new flags --modify-window=number to allow larger deltas in the modification time of files. This is a fixup for checking backups copied onto FAT filesystems, because these store the time stamp with a precision of only 2 seconds. Since this is a common operation, the shortcut -w/--windows will set the allowed modification window to 1.
• Also added a new flag --exclude-marker=file, which makes the recursive file system walk skip all directories (and their sub-directories) containing a special marker file usually called .nobackup in other programs. This option is persistent and stored in the created digest file, so it must only be specified once and then in all subsequent runs the same directory marker is used to selectively skip over special directories.
• Added a --restrict=pattern temporary exclusion flag. If specified, all paths containing the pattern as a substring are temporarily skipped. This allows one to quickly check a specific folder or file structure for modifications. Especially useful in combination with -c to perform a full digest check on only a subset of the files.
• One more smaller fixup: using 64-bit file offsets on Window, which allows files larger than 2GB to work correctly.

For more information and the new version see the digup web page.

Fixed another severe bug in the digup tool: on the amd64 architecture the tool crashed when writing the digest file, thanks goes to Daniel D. for reporting and fixing this bug.

The bug was caused by the variable arguments lists va_list used twice in the fprintfcrc() function. Apparently, on the amd64 platform va_start() and va_end() must be called twice even when passed the list to vsprintf().

For more information and the new version see the digup web page.

Fixed a two bugs in the digup tool: added large file support when compiling the program and fixed a string allocation bug.

This new version enables large file support by using long long variables for size. Furthermore, a string allocation bug was fixed which occured when using -t and -f command line parameters.

For more information and the new version see the digup web page.

Fixed a small bug in the stx-execpipe library: add large file support when compiling the library.

This bug switches on the large file support functions. Without this fix a pipeline reading or writing files >2GB will not function properly. The fix is to #define _FILE_OFFSET_BITS 64 when compiling the library's code.

For more information and the source code see the stx-execpipe web page.

The STX C++ library series has been extended today with a new installation: the STX Execution Pipe library, in short STX ExecPipe. It is the solution to an issue that I encountered in writing a small backup application. This backup tool collects some file to backup and then calls tar and xz followed by ncftpput.

However, I could not find any useful C++ library that allows convenient chaining of external programs like used in everyday shell piping. This pipe line functionality is based on the system calls fork(), exec() and pipe(), which are not easy to use correctly. After writing some ad-hoc functions to call one or two external programs, I decided to tackle this basic problem once and for all. The result is the stx-execpipe library.

Using the library a C++ program can build a sequence of external programs with command line parameters. These programs are connected by the library just like a shell pipe: stdout of the preceding stage goes into stdin of the next one. The input and output of the whole pipeline can be redirected to a plain fd, a file or saved in a std::string.

One very interesting feature of the library is to insert intermediate processing functions into a pipe line. The data can be intercepted and passed back to the parent process for manipulation or just inspection. This was necessary to calculate the SHA1 digest of a backup tarball simultaneously to uploading it.

For more information and the source code see the stx-execpipe web page.

The following small code snippet exemplifies the flexibility of the stx-execpipe solution:

stx::ExecPipe ep;

// first stage calls tar
std::vector<std::string> tarargs;
tarargs.push_back("tar");
tarargs.push_back("--create");
tarargs.push_back("--verbose");
tarargs.push_back("--no-recursion");
tarargs.push_back("/path/to/some/files");
tarargs.push_back("/path/to/more/files");
ep.add_execp(&tarargs);

// second stage compresses the tarball
ep.add_execp("xz", "-9");

// third stage intercepts data for a SHA1 digest
Sha1Function sha1tar;
ep.add_function(&sha1tar);

// fourth stage sends the tarball via FTP
std::vector<std::string> ftpargs;
ftpargs.push_back("ncftpput");
ftpargs.push_back("-c");
ftpargs.push_back("ftp.upload-to-host.net");
ftpargs.push_back("/path/to/ftpfile.tar.gz");
ep.add_execp(&ftpargs);

if (ep.run().all_return_codes_zero()) {
    std::cout << "Backup upload complete." << std::endl
}
else {
    // error processing...
}

Published yet another C++ template library using a B-tree. This time the solution is a disk-based read-only key-value mapping using a "packed, sequential" B-tree as index structure.

All applications mapping a large number of constant, integral keys to string or data blobs can benefit from this library. The database structure is highly compact and contains self-verification checksums over both key and value areas.

stx-cbtreedb is a direct contender with cdb and tinycdb, which however are based on hash tables and do not retain key proximity. Compared to other full-fledged B-tree implementations like BerkeleyDB or TokyoCabinet, the stx-cbtreedb is very small, faster and the database files have much less overhead due to read-only optimizations.

For more information and the source code see the stx-cbtreedb web page.

My second LibriVox recording is finished! The basic idea behind LibriVox is to read public domain texts and to put the recordings back into the public domain. More about that on librivox.org.

As with the last recording, my personal motivation is to practice my rusty English pronunciation. By reading and rereading the texts I believe my English will become more fluent and in the end also achieve better articulation.

The second recording is "Double Take" by Richard Wilson, a science fiction short story available from Project Gutenberg. I have also made a quick LaTeX typesetted version of the Gutenberg etext for more comfortable reading and with slight corrections: doubletake_wilson_text.pdf (369 KB).

Story Summary / Teaser

Pacing through a high-speed spy adventure, young Paul Asher finds himself going around in circles -- very peculiar circles indeed!

Audio Recording - Runtime: 26:29

MP3 encoded with standard lame VBR preset (obviously better/smaller than CBR 128 kbps)

doubletake_wilson_tb_vbr.mp3 (19.1 MB)

The recording is now also available on archive.org in the LibriVox Short Science Fiction Collection 033.

Published a small, but very useful console tool to update md5sum or shasum digest files. It will read existing md5sum.txt files and add new files to it without rereading the whole directory tree.

This makes digup very useful to update and verify incremental archives like chronological data storages or music collections, which are nowadays commonly stored and backuped on hard disks. Using a full file digest scan even slowly creeping bad blocks on old hard disks can be detected. By using a crontab entry, this check can be performed unattended and routinely.

For more information, the source code and binaries for various platforms see the digup web page.

After two weeks of hard work, my first LibriVox recording is finally finished! The basic idea behind LibriVox is to read public domain texts and to put the recordings back into the public domain. More about that on librivox.org.

My personal motivation behind recording, besides a lasting inclination towards fantasy and science fictions texts, is to practice my rusty English pronunciation. By reading and rereading the texts I believe my English will get more fluent and in the end also achieve better articulation.

The first recording is "The Last Evolution" by John W. Campbell, Jr. a science fiction short story available from Project Gutenberg. I have also made a quick LaTeX typesetted version of the Gutenberg etext for more comfortable reading and with slight corrections: lastevolution_campbell_text.pdf (153 KB).

Story Summary / Teaser

A future Earth population is unexpectedly attacked by powerful extraterrestrial Outsiders. However, the highly developed civilization of humans and machines quickly builds up forces and defends itself. This desperate, life threatening struggle against the invading aliens leads to The Last Evolution.

Audio Recording - Runtime: 58:42

MP3 encoded with standard lame VBR preset (obviously better than CBR)

lastevolution_campbell_tb_vbr.mp3 (42.7 MB)

If someone drops me a line with a good reason for uploading 250 MB, then I can also provide a lossless 24-bit FLAC file. Just burning a CD is not a good enough reason.

The recording is now also available on archive.org in the LibriVox Short Science Fiction Collection 026.

Released a minor updated source code version for Flex Bison C++ Example. The example source code is released into the public domain or, at your option, under the Do What The Fuck You Want To Public License (WTFPL).

This minor bugfix release fixes up two simple compilation issues with the newest bison version 2.4.1.

For more information and the download package see the Flex Bison C++ Example web page.