Just in time before my short holiday I've released rdup 0.5.7. See the project page for more info.

This release removes syslog support from rdup-snap, as it didn't work correctly on Mac OS X and you can do the same with logger for instance.

rdup-hist

rdup-hist is a small tool that will display files from your backup (inspired by plan9's yesterday):

% rdup-hist -b /raid/backup gtodo
200803/20 1000 1000 1082 /home/miekg/gtodo
200803/19 1000 1000 1082 /home/miekg/gtodo
200803/18 1000 1000 1082 /home/miekg/gtodo
200803/17 1000 1000 1082 /home/miekg/gtodo

And you can also diff between the current file and the one in the backup:

%  rdup-hist -b /raid/backup -d gtodo
--- /raid/backup/elektron/200803/20/home/miekg/gtodo 
+++ gtodo       2008-03-20 08:46:17.292675581 +0100
@@ -1,5 +1,7 @@
 Feature for metacity

+o Fix rdup
+
 o for gnome-terminal
     show sub/tabs in window-manager when focussing with your mouse

You can also go back in history:

% rdup-hist -b /raid/backup  -7w todo
200801/31 1000 1000 128 /home/miekg/todo
200801/30 1000 1000 159 /home/miekg/todo

So this file already existed seven weeks (-7w) ago. See rdup-hist(1).

The default VIM7 install is equipped with the English dictionary, but as I'm Dutch I wanted to use the Dutch spelling, unfortunately these spell files are not included in Ubuntu. The following article is a short howto on how to add this wonderful feature to your VIM.

Dutch spelling

Download your language from the debian experimental archive.

I needed Dutch so I downloaded:

/main/v/vim-spellfiles/vim-spellfiles-nl20060604-1all.deb

Install this with:

dpkg -i vim-spellfiles-nl_20060604-1_all.deb

This will put some files in /usr/share/vim/addons. By default VIM will not look in that directory. To fix this I'm creating some symlinks from my local .vim directory:

cd ~/.vim/spell
ln -s /usr/share/vim/addons/spell/* .

Now VIM should be able to find your spell files.

.vimrc

Add the following to your .vimrc file:

setlocal spell spelllang=en
set spellfile=~/.vim/spellfile.{encoding}.add

With this you enable spell checking for the English language.

key mapping

I've added some key mappings to enable Dutch or English checking or to disable it entirely. I'm using control-E for English, control-N for Dutch (nederlands) and control-O for off.

map     <C-E>    :setlocal spell spelllang=en_us<CR>
imap    <C-E>    <ESC>:setlocal spell spelllang=en_us<CR>i
map     <C-O>    :setlocal spell spelllang=<CR>
imap    <C-O>    <ESC>:setlocal spell spelllang=<CR>i
map     <C-N>    :setlocal spell spelllang=nl<CR>
imap    <C-N>    <ESC>:setlocal spell spelllang=nl<CR>i

autocmd

For some filetypes want to use a different language, so I also have these:

autocmd Filetype c setlocal spell spelllang=en
autocmd Filetype perl setlocal spell spelllang=en
autocmd Filetype mail setlocal spell spelllang=nl

So my default settings when I'm emailing is to use the Dutch language.

Highlighting

The default highlighting seems to suck, but this guy had a fix. Add the following to your color file are to you .vimrc:

:highlight clear SpellBad
:highlight SpellBad term=standout ctermfg=1 term=underline cterm=underline
:highlight clear SpellCap
:highlight SpellCap term=underline cterm=underline
:highlight clear SpellRare
:highlight SpellRare term=underline cterm=underline
:highlight clear SpellLocal
:highlight SpellLocal term=underline cterm=underline

This gives you some nice underlining, not to fancy, but just enough to make you notice :).

Using it

The following key are important when using VIM spelling:

• z= when a word is spelled incorrectly get a listing of alternatives
• zG add the word to the spelling file, it is a Good word.
• zW mark the word as bad in the spelling file.

Happy VIM-ing

Vim tip 21 allows you to copy text between VIM session running in different terminals! No more using your mouse to copy, just yank in one terminal and paste in another!

Put this in your .vimrc:

set clipboard=unnamed
set go+=a

rdup 0.5.6 is released, it fixes some bugs of which the most important is that the pathname argument is normalized. As such

../../.././././////.././../../

is reworked to

/

See the rdup project page for more information.

Quick download link.

Although hdup isn't really maintained anymore, a lot of people keep on using it (which is fine of course). Also it's all opensource, so if somebody wants to fix what needs to be fixed this can also be done.

The reason I consider rdup a better tool is that I can rewrite it with rdup. Incidentally the oneliners I'm describing here are a shorthand for the actual algorithm implemented in hdup.

Core of the algorithm

1. Let rdup generate the files
2. tar the file into a tarfile
3. mcrypt the archive - or whatever
4. scp it somewhere else

This boils down to:

rdup | tar && mcrypt <tar-ball> && scp <tar-ball> user@remotehost:

Note: I know that is not all hdup can do, but its the core of hdup's functionality.

Note2: we only need the filenames from rdup, hence the -F'%n\n' flag.

Note3: all temp. config files are stored in the directory ~/.hdup.

Note4: I'm assuming bash as the shell here, important for the $HOSTNAME variable for instance (I don't have that one in zsh...)

Full Dump, ie monthly

We want to dump /home and /var for starters:

We use the normal way to do this with rdup:

rm -f ~/.hdup/filelist
rdup -F'%n\n' ~/.hdup/filelist /home /var | \
tar -T - --create --no-recursion --gzip --file \
/tmp/$HOSTNAME.$(date +'%Y-%m-%d').monthly.tar.gz

I'm using some shell foo to create the correct archive name.

Incrementals, weekly and daily

Because of how rdup works in contrast to hdup you can create unlimited incremental dump against the filelist used by the previous rdup run, in this case ~/.hdup/filelist. I'm cutting a corner here as I'm only implementing daily-backups. The script is almost identical, 'cept for the naming and keeping the filelist:

rdup -F'%n\n' ~/.hdup/filelist /home /var | \
tar -T - --create --no-recursion --gzip --file \
/tmp/$HOSTNAME.$(date +'%Y-%m-%d').daily.tar.gz

Encryption

Just as with hdup encryption is a second step, first the archive is created then it is encrypted. First create a file with your secret key:

echo "my secret key" > ~/.hdup/key

Then encrypt our archive:

cat /tmp/$HOSTNAME.$(date +'%Y-%m-%d').daily.tar.gz | \
mcrypt -f ~/.hdup/key -F - > \
/tmp/$HOSTNAME.$(date +'%Y-%m-%d').daily.tar.gz.mc
rm -f /tmp/$HOSTNAME.$(date +'%Y-%m-%d').daily.tar.gz

Remote storage

This was the whole point of hdup, putting your encrypted tar-file on somebody else's computer. In hdup I'm using scp, so why not here?

scp /tmp/$HOSTNAME.$(date +'%Y-%m-%d').daily.tar.gz.mc \ 
user@remotehost

Putting it all together

This is an untested shell script fragment, just to convey the idea! It should be trivial to write your own script, based on the above information. (I you do, please let me know).

case $TYPE in
monthly)
    ARCH_NAME=/tmp/$HOSTNAME.$(date +'%Y-%m-%d').monthly.tar.gz
    rm -f ~/.hdup/filelist
    ;;
daily)
    ARCH_NAME=/tmp/$HOSTNAME.$(date +'%Y-%m-%d').daily.tar.gz
    ;;
esac
# $DIRS holds all the dirs to be backed up
rdup -F'%n\n' ~/.hdup/filelist $DIRS | \
tar -T - --create --no-recursion --gzip $ARCH_NAME

if $encryption; then
cat $ARCH_NAME | mcrypt -f ~/.hdup/key -F - > $ARCH_NAME.mc && \
rm -f $ARCH_NAME
ARCH_NAME=$ARCH_NAME.mc
fi

if $remote; then
scp $ARCH_NAME user@remote:/tmp
fi
echo $ARCH_NAME

So with a few lines and rdup I can make hdup, agreed not all feature are there, but this can be easily helped by extending the script.

Usually people use SSH as a replacement for rsh, which is of course a good thing. SSH uses encryption to transport your password to the remote server for authentication.

But SSH can do more, you can use a public/private key pair and set it up in such a way (google around for howto's), that SSH will only transport a public key over the Internet. This way no passwords are transported, so even if someone breaks the encryption, no harm is done. Well... at least your private key is still safe (for now).

naive solution

In /etc/ssh/sshd_config there are 2 keywords that must be configured to turn this behavior on: PasswordAuthentication and UsePAM.

The incorrect way to configure this is to use the following config snippet:

PasswordAuthentication no
UsePAM no

This will give you the desired result, but also has a side effect: it disables PAM. PAM is used for more than password checking, it also sets up your account and can configure other stuff. So disabling it is not a wise thing to do.

right solution

The following snippet is the correct config for sshd_config:

PasswordAuthentication no
UsePAM yes

Now, we only need to configure PAM (This is under Debian/Linux). In /etc/pam.d/ssh it says

# Standard Un*x authentication.
@include common-auth

This piece of code will ask for you password, which is now handled in ssh itself by means of the key-exchange. So it can be disabled, but a better way is the following.

a new pam.d/ssh

Create a common-deny, with the following content:

#
# /etc/pam.d/common-deny - always deny
#
auth    required        pam_deny.so

And in /etc/pam.d/ssh change

# Standard Un*x authentication.
@include common-auth

to

# Standard Un*x authentication.
# always deny
@include common-deny

conclusion

Now sshd uses a secure key exchange for authentication, it still uses PAM and PAM is configured in such a way that password authentication for sshd always fails.

Goal

The name of the game is getting a minimum maintenance RAID server up and running. The OS should be bootable from an USB stick and the RAID disks will be SATA disks.

Installation and setup

This section handles the initial installation and setup procedure. By keeping the OS small (and on USB) it will (hopefully) make backups and restores more easy,

USB stick

First install Debian/Ubuntu on the USB stick. I created two partitions on the 512 MB sized stick: /boot and /.

Where /bootr is 100 MB, which is probably a bit large. Use ext2 (not a journaling filesystem! As this ages the flash stick too much). Next use debootstrap to install to OS. I've kept the install down to about 300 MB, by only installing the essentials and removing unneeded cruft.

I've followed this site to install Ubuntu on the stick.

Booting

Luckily my system allows me to boot from USB so this all worked nicely; and I can login into my new (soon to be) fileserver system.

The only problem was that any SATA disks were detected before the USB flash disk. Depending on the number of disks, the USB disk would be /dev/sda (with 0 disks) or /dev/sdd (with 3 disks)

This would make it impossible to set the root device in grub and /etc/fstab. I solved this by using the device names from /dev/disk/by-id/, in this case:

/dev/disk/by-id/usb-_USB_FLASH_DRIVE_19661A000079-part2

This name can also be used in /etc/fstab. Linux will automaticly convert this into the correct /dev/sdX. This kind of stuff makes me happy.

SATA hardware

I've bought two promise TX 2 SATA controllers, with 2 SATA ports. This gives me a total of 4 ports. I need 3 ports to start with. These work fine with Ubuntu.

The harddisks are mounted in a Chieftec SATA disk bracket. This allows me to easily remove faulty disks from the RAID array. And it has nice lights on the front. And a fan to keep the disks cool.

Remote logging

To safe the flash drive it is advised to forward all logging to another host. This is really simple to setup, google for it.

Setup RAID

For setting up the RAID array I've looked at the following sites:

• software-RAID-HOWTO-2.html
• software-raid-in-ubuntu

They sort-of helped, but looking at mdadm's manpage gave me the info I needed. It boiled down to the following:

Fdisk the disks:

fdisk /dev/sd?

set the disk type to 0xfd. This is for autodetection (I'm not using this currently).

Then create the RAID-5 array:

mdadm --verbose --create /dev/md0  -c128 -lraid5 -pls -n3 /dev/sda1 /dev/sdb1 /dev/sdc1

Create a filesystem:

mkfs.xfs /dev/md0

And that is it -- only an emergency procedure remains.

NFS, setup and mountpoint

The raid array is mounted under /raid. This directory is suid with the group name 'shared'.

This directory is exported using the kernel NFS server.

Tweaks

I've choosen not to install cron as this is not needed for such a simple machine. I did install:

• munin, get statistics about the server
• monit, monitor important processes, mdadm for instance.
• ntp, time synchronisation
• smartmontool, extra disk monitoring
• postfix, email allerts, etc.

Also, with lots of effort I got munin to monitor my SATA disks with smartctl.

I've been running a development release of metacity for some time know and I really like it. Especially with the new composer that you can enable.

I've made some screenshots of my setup:

drop shadows

Every menu and window has a drop shadow now, this shows the workspace switcher. Too bad this hasn't got a fixup (yet?).

alt-tabbing

The new alt-tab screen. This looks quite a bit like the one from Compiz, but its very nice to have this in metacity itself.

I want more!

This already looks very good. I'm hoping that the workspace switching will also be beefed up.

Further more I really want something like Expose!

older article that I've revived.

Problem

The problem: you have a laptop and you're not always connected to the Internet. Still you want to sent mail even when you're offline. You cannot use just any mail server out there, 'cause a lot of them don't relay. So you must use your own mail server.

You'll need:

1. postfix, only used for queuing and forwarding the mail
2. openSSH, for setting up a tunnel

OpenSSH config

You will need to create a ssh tunnel to your mail server. This is the command I use:

ssh -2 -N -f -L 10025:elektron.atoom.net:25 miekg@elektron.atoom.net 2>/dev/null

Of course you also want to setup ssh so that you can login without typing a password.

postfix config

Next you must tell postfix to use you're tunnel. It is also important to keep postfix from doing MX lookups. In the main.cf of postfix add the following:

relayhost = [127.0.0.1]:10025   # use the tunnel, no MX lookups
defer_transports = smtp               # only send when online
mydestination = localhost.localdomain   # not sure if this is needed

system config

Put the startup commands in /etc/dhclient-exit-hooks. This way every time you get a IP number from a dhcp-server the tunnel is re-established and the mail is flushed, /etc/dhclient-exit-hooks:

# start an ssh tunnel to elektron
ssh -2 -N -f -L 10025:elektron.atoom.net:25 miekg@elektron.atoom.net 2>/dev/null

# run the mailqueue
/usr/sbin/sendmail -q

Also make a cronjob that runs every now and then to flush the queue:

# run queue every 5 minutes
*/5 * * * *     root  test -x /usr/sbin/sendmail && \ 
nice -n10 /usr/sbin/sendmail -q

That's it. Mail should now be queued until it can be delivered via you're own mail server.

I wanted to count all the books we own, and furthermore classify them in a few categories. Ofcourse this would mean a lot of typing, so the idea is to use a barcode reader. With a little help from books.google you can get the author, genre, title from a book back.

With that you can easily make a small library.

Hard- and software setup

I've bought I scanner somehwere online, from 50 to 100 euros you can buy a decent barcode scanner.

Next I wrote a Perl script that would use the ISBN number from the book and would fetch the information from google.

Archiving the books

Database

I'm using SQLite3 as a backend. The "database" was initialized with the following tables:

CREATE TABLE books (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    isbn INTEGER,
    title VARCHAR,
    author VARCHAR,
    genre VARCHAR,
    year_published INTEGER,
    epoch_added INTEGER,
    loc_code VARCHAR
);

The loc_code columns may be used in the future as I may implemented a location code for each book.

CREATE TABLE lending (
    id INTEGER,
    isbn INTEGER,
    date INTEGER,
    who VARCHAR
);

The lending table isn't used yet.

Perl script

I'm using a small Perl script to retrieve the information from google and to parse the author, title and stuff. This is a simple regexp based parsing - no fancy HTML parsing or whatever. The database backend used is SQLite.

The gist of the script is:

Read in the barcode from STDIN. Fire off links to get the HTML.

while(<>) {
    chomp;
    if (! /^\d+$/) {
    print "** Invalid\n";
    next;
    }
    my @html = `links -width 200 -dump "http://books.google.com/books?q=+$_&btnG=Search+Books"`;
    # title,author,genre,year of first publication
    my @det = details(@html);
    if (scalar @det == 0) {
    print "** Onbekend\n";
    } else {
    my $epoch = time();

    # genre
    if ($det[2] =~ /pages/) {
        # problably couldn't fetch it
        $det[2] = "none";
    }
    # first published, if empty, make it up
    if ($det[3] eq "") {
        $det[3] = "0";
    }

    if (retrieve $_) {
        insert $_, $det[0], $det[1], $det[2], $det[3], $epoch;
    } else {
        print "** Boek already there: $_: $det[0]\n";
    }
    }
}

Parse the HTML This function reads in the HTML and returns a list with the author and title details. This in done in details.

Normalize the genre The genre isn't standardized, so I needed to map them to genre I want. This is done in normalize_genre.

Check if the book is already there, the function retrieve does a simple database lookup:

sub retrieve {
    my $isbn = shift;
    my $sth = $dbh->prepare('SELECT * FROM books WHERE isbn = ?')
    or die "Couldn't prepare statement: " . $dbh->errstr;
    $sth->execute($isbn);
    while (my @data = $sth->fetchrow_array()) {
    #print "@data" . "\n";
    }
    # ... We have to do this after the while loop that fetches 
    # whatever rows were available, because with some
    # databases you don't know how many rows there were until after 
    # you've gotten them all.
    my $row = $sth->rows;
    $sth->finish;
    if ($dryrun == 0) {
    return ($row == 0);
    } else {
    return 1;
    }
}

If we don't have the book yet - do a simple insert:

sub insert {
    my ($isbn, $title, $author, $genre, $year, $epoch) = @_;

    $genre = lc normalize_genre $genre;
    print join "|", $title, $author, $genre, $year, $epoch . "\n";
    if ($dryrun == 0) {
    my $sth = $dbh->prepare('INSERT INTO books VALUES (?, ?, ?, ?, ?, ?, ?, ?)')
        or die "Couldn't prepare statement: " . $dbh->errstr;
    $sth->execute(undef, $isbn, "$title", "$author", "$genre", $year, $epoch, undef) 
        or die "Couldn't insert book: " . $dbh->errstr;
    $sth->finish;
    } else {
    print "** No commit\n";
    }
}

Extra tools

With SQLite you can write shell oneliners to get the info from your database:

sqlite3 biblio.db "SELECT isbn,title,genre,author FROM books ORDER BY genre,title"

Or to get it ordered on genre, this has one argument, the genre (stored in $1):

sqlite3 biblio.db "SELECT title, author FROM books WHERE genre LIKE '%$1' ORDER BY title";

From this last command I generate a LaTeX file to create a list of books per genre. Note I'm using marginnote, each new letter will be printed in the margin and with the default options of LaTeX you can only have like 13 of these margin paragraphs and I needed more. For the rest it's simple LaTeX and Perl:

#!/usr/bin/perl -w
use strict;

my $genre = shift @ARGV;
print <<EOF;
\\documentclass[a4paper]{article}
\\setlength{\\parindent}{0mm}
\\usepackage{fullpage}
\\usepackage{marginnote}
\\renewcommand{\\familydefault}{\\sfdefault}

\\begin{document}
\\pagestyle{empty}
EOF
print "\\emph{\\textbf{\\Large $genre}}\\\\\n";

my $lastletter = "";
while (<>) {
chomp;
s/&/\\&/g;
my ($t, $a) = split /\|/;
my $c = substr $t, 0, 1;
print "\\marginnote{$c}" if ($c ne $lastletter);
print "\\textbf{$t}" . "," . "{\\small $a}" . "\\\\\n";
$lastletter = $c;
}
print <<EOF;
\\end{document}
EOF

Download

For the entire Perl script, click here.

I've had the "pleasure" to use Mac OS X a few times and compaired to, for instance GNOME it just sucks. A few points:

• the dock: I close windows, they stay active in the dock? Why!?
• apple+tab: only switches between different window classes. Ever had two Iterms open? Good luck. (Yes, I know, you should then use a different key-combination. Why!?)
• resizing: the mouse pointers doesn't change. I really like the little hand I get in gnome when I move windows.
• finder: new window -> Applications -> Firefox. What's wrong with a simple (start)menu?
• placement of the cntlr key: this is soo annoying.

Another point in case: valhenson.livejournal.com/11711.html You cannot enable follow-focus-mouse.

The only cool thing is that resume/suspend just works and that the wireless is flawlessly detected. This is also the only reason I don't have Linux running on it...

My point being: this is a Mac with supposedly the best GUI ever. Well not in my book...

I've reworked my older DNS filesystem Perl script to make something nicer. It's still not perfect, but works much better.

You can now cat files for instance. The filesizes are something that should also be fixed, if set them to 512 bytes.

From a DNS standpoint some other things can be improved. Right now everything is shown from a zone (everything that can be discovered that is), however this also includes glue...

To get this running you'll need FUSE (should be installed if you run Linux) and Perl with the Perl Fuse module installed. I'm running Ubuntu Gutsy and everything could be installed very easily.

Furthermore you'll need two files, a Fdns.pm module and a fdns.pl Perl script. Place both in a directory and type:

mkdir theworld
./fdns theworld

Now the world of DNS is mounted on theworld. Now you can do the following:

$ cd theworld
$ ls
# this might take a WHILE!!!!
...
eu/                  ns2.undp.org             utama.bolnet.bo
eur1.nipr.mil        ns2.univie.ac.at         uucp-gw-1.pa.dec.com
eur2.nipr.mil        ns2.uz                   uucp-gw-2.pa.dec.com
euro-ns1.cw.net      ns2.yemen.net.ye         uy/
euro-ns2.cw.net      ns2.zamnet.zm            uz/
euro-ns3.cw.net      ns3.aalnet.net           va/
euterpe.william.org  ns3.arnes.eu             vc/
f5.nstld.com         ns3-auth.sprintlink.net  ve/
...

This is part of the root zone, you see the eu delegation, so that is a directory. There is also a nl directory in there. Let go there:

$ cd nl
$ ls
@

So nl doesn't do axfr and all that could be discovered is contained in the file @. Lets check that one out:

$ cat @
nl.  5988  IN SOA ns.domain-registry.nl. postmaster.domain-registry.nl. (
                  2008030205      ; Serial
                  7200    ; Refresh
                  900     ; Retry
                  2419200 ; Expire
                  900 )   ; Minimum TTL
nl.     85188   IN      NS      ns3.nic.nl.
nl.     85188   IN      NS      ns4.nic.nl.
nl.     85188   IN      NS      ns-ext.isc.org.
nl.     85188   IN      NS      ns-nl.nic.fr.
nl.     85188   IN      NS      ns.domain-registry.nl.
nl.     85188   IN      NS      ns2.nic.nl.
nl.     85188   IN      NS      nl1.dnsnode.net.

This is the SOA record for nl. Lets see if there any funny delegations:

$ cd fghjfhfjdhdj
cd: no such file or directory: fghjfhfjdhdj

Nope, that one does not exists, maybe there are funny names?

$ cat fjhfdjfd  
cat: fjhfdjfd: No such file or directory

Nope. Okay, why not descend into some directory I know should be there.

$ cd miek
$ ls
@  a  localhost  www
$ cat a
a.miek.nl.      3600    IN      A       192.168.1.2
$ cat www
www.miek.nl.    3600    IN      CNAME   a.miek.nl.

So every file has the name of a label, and the contents are RRs who have that label. Esp. funny are zones with wildcards *, as you can cat any name you wish.

color output of ls

It's wise to disable that when you are roaming around in this filesystem. As every discovered name is statted, which means another DNS lookup.

Feedback

Comments and patches are welcome.