For some general usage and remarks on Amazon's S3 service see this link

In this post I will document how rdup-s3 works (as it is finished now). If you want to access the code now you have to checkout trunk from http://www.miek.nl/projects/rdup/svn/trunk/ using subversion.

Please note there is currently a 5 Gigabyte limit on the file size, rdup-s3 will not check for this and happily will start the upload - you have been warned!

Listing my buckets

$ rdup-s3 -i key-id -k secret-id ls
rdup-test1

Ah, so I have one bucket named rdup-test1.

Listing files inside that bucket

$  rdup-s3 -i key-id -k secret-id \ 
   -b rdup-test1 ls 
           9787 rdup.c

So, 1 file with a size 9787 bytes.

Deleting files from a bucket

$ rdup-s3 -i key-id -k secret-id \
  -b rdup-test1 rm rdup.c

No news is good news.

Deleting a bucket

$ rdup-s3 -i key-id -k secret-id \
  -b rdup-test1 rm

No news is good news - so its gone.

Upload a file named and create bucket

Now I'm uploading (put) rdup.c to the bucket rdup-backup. The remote name is also rdup.c (the last argument to rdup-s3).

$ cat rdup.c | rdup-s3 -i key-id -k secret-id \
  -b rdup-backup put rdup.c

Did it work:

$ rdup-s3 -i key-id -k secret-id ls
rdup-backup

Yes!, the bucket was created on the fly.

$ rdup-s3 -i key-id -k secret-id \
  -b rdup-backup ls  
           9787 rdup.c

And it is back again.

Time to upload my compressed and encrypted backup to Amazon :)

Steps:

1. Read: How to use nice fonts in LaTeX

2. Install the software:

   sudo apt-get install texlive-xetex

   sudo apt-get install lcdf-typetools

   The latter one is needed for otfinfo to determine the font family name.

3. Read /usr/share/doc/texlive-doc/xetex/XeTeX-reference.pdf

4. Use xelatex to create the pdf of your LaTeX documents

If you get the following error install the package lmodern

 LaTeX Error: File `lmodern.sty' not found.

Finally I can use truetype fonts in my LaTeX documentation. Sweet!

Just came by this, an article I wrote for Cisco (Internet Protocol Journal) back in 2004.

The article

Post 0.6.1:

• rethink the whole rdup-utils stuff. There are better ways possible to do the actual backup. Like sending the list of filenames to a remote server and then starting the backup. This way the remote side can check how far along the backup is; makes it possible to resume a backup.
• split of the utils in a seperate package.

Maybe in 0.6.1:

• move to Gnode in stead of GTree. This gives me more flexibility. I want to print the list from rdup depth-first; first the files then the directories. All utility must then make directories on demand. When the finally do see the directory they can correctly set the permissions. This is needed for directories which may have a 600 permission, which you can read as root, but you can not remotely dump them (with ssh and as a normal user), because the rdup cannot enter the directory after creating and setting this permission on it.

Definitely in 0.6.1:

• remove the i18n from the Perl utils - the hack I was using when you don't want i18n breaks with Perl 5.10.
• removed the -a flag, which utilizes extended attributes for storing the original owner and group information (again needed for remote dumps). This only worked on Linux and Solaris and was slow. I never liked this hack anyway, so now I'm removing it.
• Amazon S3 support in the form of rdup-s3, see this blog entry

DSA 1617-1 a security update from Debian says they forgot to update the SE Linux policy that would allow BIND9 to randomize its source ports...

I always found SE Linux to be too complex for my needs, but now it turns out you can actually be more vulnerable when you run SE Linux.

I know enough.

.nl employs a system for registering a domain name that works as follows:

1. you setup a zone
2. you register the zone name
3. .nl checks the zone from your nameservers
4. if the zone is found to be OK, your name is registered
5. if it is not ok, you have to fix your zone or the name is already claimed (you can check that before hand)

This conflicts with how most ISPs work. Where one just updates the provisioning database and it (ideally) goes right to the registry to claim the name. Then after some time you "dump" the provisioning database on the network and automatically configure the zone for the customer.

Wouldn't it be nice when you could do the same for the .nl registry? The thing you need is some kind of fake nameserver that will tell the registry just what it wants to hear: For all .nl zones it will tell the bot, 'Yes, I'm authoritative for that name and these are the records.' And it gives out the standard SOA, MX and others. See the technical requirements for what exactly is needed.

With such a fake nameserver in place the registry process becomes:

1. register the name with .nl
2. .nl will query your fake nameserver
3. if OK, provision the zone for the client
4. (some time later) send in a nameserver change request to give the zone the correct nameservers.

Note: the big difference here is that you don't have to go restarting services on your network right away. You can do it when you are ready for it. And that is a big plus for most ISPs.

fake nameserver

The following Perl scripts starts a nameserver on two IP's. You need to configure a host with two IP's as this is one of SIDN's requirements. When it runs it will return exactly what .nl wants to hear for all .nl queries. If gives back a FORMERR for all none .nl names.

Download the script

usage

Starting the server:

   ./dnsserver.pl
   Creating TCP socket 127.0.0.1#15353 - done.
   Creating UDP socket 127.0.0.1#15353 - done.

A sample query:

dig -p 15353 @127.0.0.1 mx miek.nl

; (1 server found)
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 36820
;; flags: qr aa rd; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;miek.nl.           IN  MX

;; ANSWER SECTION:
miek.nl.        3600    IN  MX  100 mx1.miek.nl.
miek.nl.        3600    IN  MX  100 mx2.miek.nl.

;; Query time: 18 msec
;; SERVER: 127.0.0.1#15353(127.0.0.1)
;; WHEN: Fri Jul 25 11:44:51 2008
;; MSG SIZE  rcvd: 65

I'm trying to move my noisy server to the cellar I'm left with the question how do I cater for my desktop computer needs. I would be nice to have a thin client, but then you have the problem that local devices do not work: because you are actually working on the server, it needs to "know" somehow that you have locally inserted an USB disk. This sucks.

So how about a thick client, a client with enough power to run a full blown Linux distro. This works but has the following disadvantages:

• more hard disks that can break
• more maintenance of the machine
• FANS, which make NOISE

What I need is a machine that is something of a cross between a thin and a thick client. Let's call it a lazy client. With a lazy client, you will get the low maintenance of a thin client, with the power of a thick client.

Lazy client

My lazy client is a Linux machine, which is not too powerful, but fast enough do most of the jobs. The requirements for such a machine are:

• no hard disk
• no fans
• no local configuration
• enough memory
• fast video card

In short I should be able to play video/music, develop my programs, next to the normal desktop chores.

Implementation

I'm still pondering this, but I'm thinking of the following:

• mount /home via NFS from the server
• boot from USB or cdrom (with a r/w live like system)
• any configuration would be done via puppet. Also see this blog but preferable no configuration what so ever is needed on the client
• user management via LDAP or just rsync /etc/passswd and friends to the client. I prefer a LDAP solution.

Small script which display "teletekst" pages from the Dutch teletext (or the Belgium, but I've never used that).

I'm not the original author, but feel free to grab your copy.

#!/usr/bin/perl
#
# Dit scriptje laat de NOS teletext zien
#
# (c) 2001, 2002, 2003 by Bas Zoetekouw <bas@debian.org>
# All rights reserved.
# 
# Naar een idee van Wouter Bergmann-Tiest
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
# 1. Redistributions of source code must retain the above copyright
#    notice and this list of conditions.
# 2. Redistributions in binary form must reproduce the above copyright
#    notice and this list of conditions in the
#    documentation and/or other materials provided with the distribution.
#
#
## 2003-01-06: niew teletekst format
## 2003-01-06: kleurtjes
## 2003-01-06: meer html entities toegevoegd (dank Richard)
## 2003-01-22: VRT support
## 2003-02-07: kleur support aangepast

#use strict;

#my $SOURCE = "VRT"; 
my $SOURCE = "NOS"; 
my $LYNX;

foreach my $l (qw|/usr/bin/links /usr/bin/lynx lynx|) {
if (-x $l) {
    $LYNX=$l; last;
}
}

my $color = 1;
if ((scalar @ARGV > 0) && ($ARGV[0] eq "-m")) {
$color = 0;
shift @ARGV;
}

my %COLORS = ( "000000" => "\e[30m",  #black
       "black"  => "\e[30m",  #black
       "ff0000" => "\e[31m",  #red
       "red"    => "\e[31m",  #red
       "00ff00" => "\e[32m",  #green
       "lime"   => "\e[32m",  #green
       "0000ff" => "\e[34m",  #blue
       "blue"   => "\e[34m",  #blue
       "ffff00" => "\e[33m",  #yellow
       "yellow" => "\e[33m",  #yellow
       "ff00ff" => "\e[35m",  #magenta
       "fuchsia"=> "\e[35m",  #magenta
       "00ffff" => "\e[36m",  #cyan
       "aqua"   => "\e[36m",  #cyan
       "ffffff" => "\e[37m",  #white
       "white"  => "\e[37m",  #white
       "reset"  => "\e[0m");  #reset color
my %HTML   = ( "&quot;" => '"',
    "&shy;"  => "",
    "&lt;"   => '<',
    "&amp;"  => '&',
    "&gt;"   => '>');


sub display(\@) {
my @Output=@{(shift)};
my ($print, $last) = (0, 0);
foreach my $line (@Output) {
#   print "--> $line";
    ($print = 1) if ($line =~ s/^.*<pre>//i);
    ($last  = 1) if ($line =~ s/<\/pre>.*$//i);

    next unless ($print==1);

    if ($color==1) { 
    foreach (keys %COLORS) {
        $line =~ s/COLOR=$_[^>]*>/>$COLORS{"$_"}/ig;
    } 
    }
    foreach (keys %HTML) {
    $line =~ s/$_/$HTML{"$_"}/ig;
    } 
    $line =~ s,<[^>]+>,,g;

    last if ($last == 1);

    print $line;

} 
print $COLORS{"reset"};
}

my ($Pagina, $SubPagina) = @ARGV;

$Pagina    = "100"  if  (scalar @ARGV == 0);
$SubPagina = "1"    if  (scalar @ARGV <= 1);  

my @Output = ("");
if ($SOURCE eq "NOS") {
$SubPagina = sprintf("%02i", $SubPagina);
@Output = `$LYNX -source 'http://teletekst.nos.nl/tekst/${Pagina}-${SubPagina}.html'`;
} elsif ($SOURCE eq "VRT") {
@Output = `$LYNX -source 'http://193.121.55.225/tt/tt.php?p=${Pagina}/${SubPagina}&g=0&s=0&r=0&x=1'`
}

display @Output;

This is an older article that I've brought back to live This was original written at the time of Linux 2.6.0

So be ware!

update

As it turns out, the script below never did work correctly (see the bottom of this page for an updated version). The reason it did work for me was that sshn INET_2 never blocked outgoing port 25...until October the 8th. This was at the same time I was experimenting with linux2.6 (go figure). When sshn started to block port 25 outgoing, the script broke. For telnet session to port 25 it worked correctly, but actual sending of email was already done from a high port. This was always (incorrectly) routed through INET_2.

See below ("stopped working") for an updated (working) script.

network layout

                                                       ________    
                                           +------------+        /
                                           | INET_1     |       |
                             +-------------+ Chello     +-------
         __                  |             | 10Mb/cable |     /
     ___/  \_         +------+-------+     +------------+    |
   _/        \__      |    eth0      |                      /
  /             \ eth3|              |                      |
 | Local network -----+ Linux router |                      |     Internet   
  \_ 192.168/16__/    |              |                      |
    \__     __/       |    eth2      |                      \
       \___/     eth1 +------+-------+     +------------+    |
      ___             |      |             | INET_2     |     \
   __/   \___         |      +-------------+ SSHN       +-------
  /          \        |                    | 100Mb/fiber|       |
 |   WaveLan  \       |                    +------------+        \________   
  \_172.16/16  -------+
    \__      _/
       \____/

 Based upon: http://lartc.org/howto/lartc.rpdb.multiple-links.html

My server is located in the LAN (192.168/16) and does mail (port 25), cvs (port 2401) and www (port 80). These ports are redirected from interface eth2 to the local address of the server: 192.168.1.2.

The firewall runs the external DNS and the ntp time protocol.

The main problem here is that the on INET_2 I cannot use port 25 'cause it is filtered by the provider. Thus all my email must be routed via INET_1. It is forwarded from INET_1:25 to 192.168.1.2:25.

first try

Is was very helpful to look at: Routing for multiple uplinks/providers but this howto stops where is gets interesting: using DNAT and SNAT and multiple routes to the internet.

The biggest problem for me was that I couldn't get email to be routed back correctly. I've had the following lines in my firewall scripts:

# mark mail packets
iptables -A PREROUTING -t mangle -p tcp --dport 25 -s 192.168.1.2 -j MARK --set-mark 2
iptables -A FORWARD -t mangle -p tcp --dport 25 -s 192.168.1.2 -j MARK --set-mark 2
iptables -A INPUT -t mangle -p tcp --dport 25 -s 192.168.1.2  -j MARK --set-mark 2
# REDIRECT mail seperate
iptables -A PREROUTING -t nat -p tcp -i ${INET} --dport 25 -j DNAT --to 192.168.1.2:25

And then used, ip rule add fwmark 2 pref 10 table mail.out to get the mail out on the correct interface.

BUT THIS DIDN'T WORK!

I saw packets coming to my server but they were not routed back correctly.

solution

My mistake was that I was marking packets with destination port 25 (the --dport 25 part in the rules above). But due to the fact that packets ogoing to port 25 where already DNATed, the destination port was altered. In stead of marking packets going to port 25 I should have marked packets coming from port 25 (--sport 25). So the rules now look like this:

# mark mail packets
iptables -A PREROUTING -t mangle -p tcp --sport 25 -s 192.168.1.2 -j MARK --set-mark 2
iptables -A FORWARD -t mangle -p tcp --sport 25 -s 192.168.1.2 -j MARK --set-mark 2
iptables -A INPUT -t mangle -p tcp --sport 25 -s 192.168.1.2  -j MARK --set-mark 2
# REDIRECT mail separate
iptables -A PREROUTING -t nat -p tcp -i ${INET} --dport 25 -j DNAT --to 192.168.1.2:25

Again routing on the mark with ip rule add fwmark 2 pref 10 table mail.out gives the desired result. My email is routed over the correct interface. Of course the MX records for my domains only list INET_1.

stopped working

Wow for some reason it all stopped working. Just when I upgraded everything to linux 2.6.0, although I don't think that has something to with it. My sendmail daemon DOES NOT send mail from port 25 any more, it now uses a high port. This totally garbles up my scripts as that was the assumption. Stay tuned as I post my updated script.

As stated in the beginning this has nothing to do with linux 2.6.0, but everything with my provider blocking port 25 outgoing was well. Sigh

new solution

I had to change the firewall script to look at the destination port. If it was 25 and the packet was coming from my mail server (192.168.1.2) it should be routed over the cable connection. However as I'm also relaying for people who have the same fiber provider (and thus also a blocked port 25) the script is a bit more complicated. I share a internal network with those guys. Within this network we can send mail to port 25. So for these guys the email should be routed over the fiber connection instead over the cable connection.

So the script needs to do this:

• dnat incoming mail requests to my server
• route all outgoing email to the cable wire
• route all outgoing other traffic to the fiber connection
• make an exception for internally relayed email (3 hosts at the moment)

the script

It now looks little this: (I've garbled the IP addresses a bit, comments in the code)

# let these guys jump to the ACCEPT rule, thereby NOT marking the packets
# and get them routed via the fiber connection

RELAY="IP1 IP2 IP3"
for i in $RELAY; do

${iptab} -A PREROUTING -t mangle -p tcp --dport 25 -s 192.168.1.2 -d ${i} -j ACCEPT
${iptab} -A FORWARD -t mangle -p tcp --dport 25 -s 192.168.1.2 -d ${i} -j ACCEPT
${iptab} -A INPUT -t mangle -p tcp --dport 25 -s 192.168.1.2 -d ${i} -j ACCEPT
done

## END RELAY ##

# this is the stuff at the looks the dest. port and gives it a mark

# still alive, rule wise?
${iptab} -A PREROUTING -t mangle -p tcp --dport 25 -s 192.168.1.2 -j MARK --set-mark 2
${iptab} -A FORWARD -t mangle -p tcp --dport 25 -s 192.168.1.2 -j MARK --set-mark 2
${iptab} -A INPUT -t mangle -p tcp --dport 25 -s 192.168.1.2  -j MARK --set-mark 2
# mark secure pop3 packets
${iptab} -A PREROUTING -t mangle -p tcp --dport 995 -s 192.168.1.2 -j MARK --set-mark 2
${iptab} -A FORWARD -t mangle -p tcp --dport 995 -s 192.168.1.2 -j MARK --set-mark 2
${iptab} -A INPUT -t mangle -p tcp --dport 995 -s 192.168.1.2  -j MARK --set-mark 2

# also look at the source port, other wise no server can connect to port 25
# to deliver email

# still alive, rule wise?
${iptab} -A PREROUTING -t mangle -p tcp --sport 25 -s 192.168.1.2 -j MARK --set-mark 2
${iptab} -A FORWARD -t mangle -p tcp --sport 25 -s 192.168.1.2 -j MARK --set-mark 2
${iptab} -A INPUT -t mangle -p tcp --sport 25 -s 192.168.1.2  -j MARK --set-mark 2
# mark secure pop3 packets
${iptab} -A PREROUTING -t mangle -p tcp --sport 995 -s 192.168.1.2 -j MARK --set-mark 2
${iptab} -A FORWARD -t mangle -p tcp --sport 995 -s 192.168.1.2 -j MARK --set-mark 2
${iptab} -A INPUT -t mangle -p tcp --sport 995 -s 192.168.1.2  -j MARK --set-mark 2

# DNAT to the mail server
# REDIRECT mail seperate
${iptab} -A PREROUTING -t nat -p tcp -i ${INET} --dport 25 -j DNAT --to 192.168.1.2:25
# pop3ssl
${iptab} -A PREROUTING -t nat -p tcp -i ${INET} --dport 995 -j DNAT --to 192.168.1.2:995

I have this script running right now and it performs wonderful.

This info can also be found on the Internet, but I thought I compile it here too.

Problem

You want to save something (say passwords) in a file and protect the file with a master password (say with your PGP/GPG key). This can be cumbersome unless you have vim.

The following examples will handle *.gpg files differently, vim will decrypt the file when opening it and encrypt the file when writing it. Also the use of a swap file is disabled.

vim config

The following config will make vim behave like described above manner:

augroup gpg
au!
au BufReadPre,FileReadPre *.gpg set viminfo=
au BufReadPre,FileReadPre *.gpg set noswapfile
au BufReadPost *.gpg :%!gpg -q -d
au BufReadPost *.gpg | redraw!
au BufWritePre *.gpg :%!gpg --default-recipient-self -q -e -a
au BufWritePost *.gpg u
au VimLeave *.gpg :!clear
augroup END

Just add it to your ~/.vimrc and your are set.

I've found this config somewhere on the Internet (lost the reference), so it's not my idea.

You all, of course, know about the fc command. From bash's help system:

fc: fc [-e ename] [-nlr] [first] [last] or fc -s [pat=rep] [cmd]

fc is used to list or edit and re-execute commands from the history list. FIRST and LAST can be numbers specifying the range, or FIRST can be a string, which means the most recent command beginning with that string.

Now I had the following problem: you have a file with shell commands in it. Next you want to select a few lines from this file to be executed in your running shell. A way to do this is:

• edit the file with vi
• save this edited file under a different name, say $file
• execute it: . $file

But this is cumbersome and I wanted the executed lines to be added to my shell's history. So that is why I created the function: f():

• edit a file, exec what's left
• you are finished editing
• add what is executed to the history (fc -R)

It needs to be a shell function, because as an external executable you cannot alter your current shell history.

Synopsis: f filename

This is the body of the function:

f() {
if [[ ! -f $1 ]]; then return 1; fi

copy=$(mktemp ${TMPPREFIX:-/tmp/shell}.XXXXXXX)
if cp $1 $copy; then
    if ${EDTIOR:-vi} $copy; then
    $SHELL $copy
    # add to hist
    fc -R ${copy}
    rm -f $copy
    fi
   else
    return 1
fi
}

How does it work?

• Check if we have an argument
• If so, copy the file to a temporary file
• Edit this temporary file
• Execute the contents when the editor did a normal exit
• Add the results to the current history (fc -R)
• Remove the temporary file

Well, after giving Xmonad a chance I've reverted back to XFCE. This time with a 1-pixel wide window decoration. Short story: I really like the tiling of Xmonad (or any other tiling window manager). But you do miss some flexibility if you really want to move a window. Also I needed a panel and system tray, which are available as third-party tools, but aren't the real deal.

So back to XFCE.

But cntrl-enter starts a terminal, and I'm looking for a tool that will automatically "tile" all my X-terms in a single workspace; call it a lame-ass expose.

zsh prompt

I've further tweaked my prompt to remove any consonants from the hostname, thereby shortening it a bit more:

echo ${1//[aeuio]/}

is all the magic that is needed.