With monit services are restarted, ever since I've installed zabbix I wanted the same functionallity. Turns out this is possible, but it takes some configuration.

Also see the zabbix manual, where it gets interesting from page 160 onwards.

In zabbix go to Configuration->Actions.

Add a new 'Action Operation' in which you want to run a remote command.

1. Operation type: remote command
2. Remote command: host:script in my (test) case elektron:/home/miekg/bin/zabbix_service {TRIGGER.NAME}: {STATUS}

And zabbix_service is now a shell script which will echo its arguments to a file in /tmp.

Also be sure to set EnableRemoteCommands=1 in zabbix_agentd.conf and restart zabbix.

When enabled I do see something in /tmp:

% ls /tmp/zabbix*
-rw-rw-r-- 1 zabbix zabbix 144 2009-08-20 11:04 /tmp/zabbix_test
% cat /tmp/zabbix*
SSH server is down on elektron: ON
Sshd is not running on elektron: ON

So this is starting to work nicely, however there a a few issues with it. The script:

• runs only on the host specified (here: elektron);
• runs under the user zabbix;
• needs to parse its arguments.

host groups

Reading from the manual you can use the syntax:

hostgroup#command

instead of

host:command

So (in my case) using atoom# should fix creating actions for all my hosts.

Running privileged command

At page 162 it say:

One may be interested in using sudo to give access to privileged commands.

So it must be done with sudo.

Parsing the argument

Are there any other macros (page 87 in the manual) which can be of use? Looking at some:

{TRIGGER.ID}    Numeric trigger ID which triggered this action.
{TRIGGER.KEY}   Key of first item of the trigger which caused a notification.

I've added these to my little test script, let's see what comes out of it.

SSH server is down on elektron: ON 13009 net.tcp.service[ssh]
Sshd is not running on elektron: ON 13014 proc.num[sshd]

Indeed a number is added (13009) and a net.tcp.service[ssh] string. That is somewhat more easy to parse, but still...

From the looks of it, the {TRIGGER.ID}s differs per host, so you cannot use them to check the failure of (say) the SSH daemon for all hosts. The {TRIGGER.KEY} looks much more portable and parseable in that respect.

In case you are interested the trigger ids can be found by going to the trigger screen of zabbix and clicking on a trigger. In the URL it has a triggerid=xxxxx string.

I finally went with the following macros:

atoom#/home/miekg/bin/zabbix_service "{TRIGGER.KEY}:{STATUS}:{HOSTNAME}"

Which gives the following output:

proc.num[sshd]:OFF:elektron

...and we have a string I can parse! :)

Todo

I've left to following items on my todo list

• Configure sudo to give zabbix more powers - so that it is allowed to restart services;
• Write a proper script, which can restart a service;
• Flap detection;
• Failure detection, stop restarting after n tries.

Had a great ATB trip today together with Ans. We only biked for about 35 km, but it felt like 50 because of the warm (27° celsius) weather.

The shell (in this case bash) is packed with features. So much so that you will probably never ever learn them all. Brace expansion is one of those things.

What is it? In the bash manual it says:

Brace Expansion

Brace expansion is a mechanism by which arbitrary strings may be generated. This mechanism is similar to pathname expansion, but the filenames generated need not exist

So what is it? You can make your shell generate strings, like so:

$ echo {one,two}
one two

Or, somewhat more useful

$ echo prefix_{one,two}
prefix_one prefix_two

Or

$ echo prefix{,two}
prefix prefixtwo

Or

$ echo {1..4}
1 2 3 4

Think of it as sort of a pathname expansion, but the filenames do not have to exist.

More down to the earth examples include: moving a file to the same name, but then with .bak added.

Cumbersome way:

$ mv mylongfilename mylongfilename.bak

Or

$ mv mylongfilename{,.bak}

Which first generates nothing, and leaves mylongfilename and a mylongfilename with .bak appended.

With this you can create neat little scripts, like the following one, which I use for archiving my E-mail. I'm especially fond of the

rmdir ~/Mailback/$BCK/{old,sent}/{new,tmp}

1 int main(int c) {
2     printf("This is nice");
3     exit(1);
4 }

Just bought anothor EeePC, this time the 900HD variant. Installing Ubuntu as we speak and giving Windows the boot. The 4 minute encounter with Windows XP (which felt like 4 hours) gave me the impression of a fast machine. This was just a quick peek to see if the hardware was working.

Going with Ubuntu 9.04 - then removing Pulseadio and (maybe) upgrading the kernel to 2.6.30.4.

The trick is to disable the hardisk in the BIOS and to disable all quick boot stuff in there too. So that my SD card with the Ubuntu install will boot.

Wireless does not work during the install, not a biggy, but I still plugged in a cable, just to have networking during the install. Gave the little fellow a name flo, partitioned the disks and of we go.

Brainless Ubuntu install follows

And we can reboot. This is now a worthy laptop for my wife.

Welcome Flo!

I've just released rdup 1.0.2, this adds nothing exiting; it just fixes some bugs. Libnettle has become mandatory and rdup-simple and rdup-ln are installed in /usr/bin now.

Download and project page.

rdup 1.0.1 never got released, but it is tagged in git

Excellent idea Ton!

How to make a custom filter in git to expand the string $Hash$ to something more usefull, ala the $Id$ (which git already supports), but then with more info (committer, date, etc.).

Which also helped in this case was the Pro Git book which is, as of now, a must buy.

But as always is the case between Ton and me, I find his scripts too long :-) So I miekified his solution.

First we need two scripts, one to expand $Hash$ and another one to collapse it again:

git.expand:

#!/bin/sh
id=$(git show -s --pretty=format:%h\ %ci\ ${SUDO_USER:-${LOGNAME}}%n)
sed -e 's!\([[:space:]]*\$[H]ash\)\$!\1: '"${id} "'\$!'

and git.collapse:

#!/bin/sh
sed -e 's!\([[:space:]]*\$[H]ash\):.*\$!\1\$!'

And a final check to see if they are working:

 % echo '$Hash$' | git.expand
 $Hash: 8c84acf 2009-08-11 19:28:20 +0200 miekg $

Looking nice.

% echo '$Hash$' | git.expand | git.collapse
$Hash$

Wonderful - the scripts work. Now we need to set up the filtering in git.

Filtering

As Ton also explained you need to add the filtering commands to your global ~/.gitconfig and then edit a .gitattributes to enable this on a per repository basis.

In ~/.gitconfig add:

[filter "hash"]
smudge = git.expand
clean = git.collapse

And in your .gitattributes add:

*   filter=hash

See gitattributes(5) for more information on this.

Test

Now run a little test:

% echo '$Hash$' > h
% git add h
% git commit -m'testadd' -- h
Created commit b5189fd: testadd
 1 files changed, 1 insertions(+), 0 deletions(-)
 create mode 100644 h

And, and... and...:

% cat h
$Hash$

Huh, unexpanded? Well yes, you need to remove and checkout the file for it all to work:

% rm h && git checkout -- h && cat h
$Hash: b5189fd 2009-08-11 19:50:19 +0200 miekg $

Putting it all together

So now I only need a little wrapper around vi and git to do all this automatically. So here is vit2:

1 int main(int c) {
2     printf("This is nice");
3     exit(1);
4 }

That's all folks!

And thanks Ton.

In Perl you have this:

% perl -e 'print "a" x 5, "\n"'
aaaaa

With that you can easily create a separator string consisting out of 60 spaces.

I always missed this in my shell - until now.

In Zsh have the following expansion:

l:expr::string1::string2:

Pad the resulting words on the left. Each word will be truncated if
required and placed in a field expr characters wide.

See zsh.dotsrc.org. There is also a r: variant which operates in the same way.

And lo and behold:

% echo ${(r:40::-:)A}
----------------------------------------

No more:

A="------------------------------------"

While rechecking my rdup test-suite one of the tests failed. On closer inspection it was due to the following line:

DAY_BEFORE=$(( $(date +%d) - 1 ))

When $(date +%d) is 10 this yields:

$ DAY_BEFORE=$(( 10 - 1 ))
$ echo $DAY_BEFORE
9

Also with octal values (those starting with a leading zero), it also works:

$ DAY_BEFORE=$(( 06 - 1 )) 
$ echo $DAY_BEFORE
5

So when does this go wrong? When the day is 08 or 09 (as it is today):

$ DAY_BEFORE=$(( $(date +%d) - 1 ))
bash: 09: value too great for base (error token is "09")

My fix was not to let date generate the leading zero in the first place:

$ DAY_BEFORE=$(( $(date +%_d) - 1 ))
$ echo $DAY_BEFORE
8

Moral of the story:

Be careful with leading zeros in Bash, as they might get interpreted as octal values.

A customer wanted some performance figures for a ESX cluster we built. We used our own atop and the trusted workhorse of plotting Gnuplot to make some performance plots.

Creating the data

As said, we used atop for this, it has a nice data gathering mode which prints out lines like these:

13:40:07  cpu   %usr  %nice   %sys  %irq %softirq   %steal  %wait  %idle
13:40:08  all      0      0      0     0        1        0      4     95
13:40:09  all      0      0      1     0        0        0      0     99

We were interested in the values of %usr, %sys, %irq + %softirq, %wait and %idle.

Normalizing the data

To matter slightly easier we decided to normalize the data a bit;

1. convert the time stamp to Unix epoch
2. kill the first 5 lines of output as this is a header

In short, the following tiny Perl program was used for the conversion.

#!/usr/bin/perl -w
use Time::Local;
use strict;
<>;<>;<>;<>;<>;<>; # kill first 5 lines
my @l;
while(<>) {
chomp;
if (/^#/) { print; next }
@l = split;
my ($h, $min, $sec) = split /:/, $l[0];
shift @l;
$h =~ s/^0//; # don't make it look octal
$min =~ s/^0//;
$sec =~ s/^0//;
my $time = timelocal($sec, $min, $h, (localtime)[3,4,5]);
print "$time @l", "\n"; # show the normalized line
}

Generating the plot

This was actually the most difficult step as Gnuplot is such a complicated program - you can do a lot of plotting with it. So it was a matter of getting it to plot exactly as we needed it to plot.

We wanted to plot the data as a histogram were each bar in the plot would show the percentage of time spend in the different statuses (usr, irq, sys and idle). Gnuplot calls this a rowstacked histogram. With this knowledge we came to the following Gnuplot script (script.gnuplot):

set yrange [0:100]
set xrange [0:]
set ylabel "% of total"
set xlabel "seconds"
set style data histograms
set style histogram rowstacked
set style fill solid
set key outside
set terminal jpeg
# output to /tmp/a.jpg
set output "/tmp/a.jpg"
# read the data from /tmp/a
plot '/tmp/a' u 3 t '%usr' lt rgb 'red', \
'' u 5 t '%sys' lt rgb 'black', \
'' u 6 t '%irq' lt rgb 'gray', '' u 7 t '%softirq' lt rgb 'gray', \
'' u 9 t '%wait' lt rgb 'yellow' , '' u 10 t '%idle' lt rgb 'blue'

Gnuplot would read its data from /tmp/a and generate a jpg named /tmp/a.jpg.

The full Monty

After gathering the performance data, making the plots was as easy as:

./normalize < performance.data > /tmp/a && \
gnuplot gnuplot.script && \
rm /tmp/a

Then /tmp/a.jpg could be renamed to something more suitable.

So you think you know git... today I found out something "funny". In a git repository:

$ git log
$ fatal: object 1fcc8de9361c56e538ff35d8cc4b07a9c95b7bf3 is corrupted

Okay, WTF? Lets look in the .git directory:

$ cd .git/objects/1f
$ ls -l
total 4
-r--r--r-- 1 root root 1057 2009-06-23 19:03 03db070bcb47bff3f8106f2ec7028b3496aaa8
-r--r--r-- 1 root root    0 2009-08-03 21:41 cc8de9361c56e538ff35d8cc4b07a9c95b7bf3

Ah, 0 bytes, that explains it (probably due to the weird reboot of last night and ext4).

Okay, lets fix this:

$ mv cc8de9361c56e538ff35d8cc4b07a9c95b7bf3 /tmp
$ git status
# On branch master
fatal: bad object HEAD

Hmmm, not good, let look in .git/refs/heads

$ cd .git/refs/heads
$ ls -l
total 0

Hmm, nothing, peeking in another git repo says me that you need a file master here which holds and object ref (a SHA1 string). Lets create one. First find the latest hash:

$ find . -mtime -1 -print
./objects/c4/eb5cdc3347dfc12e3fc9d5c47b8e20217d38c7

Okay, lets put that in the master file:

$ cat > master
c4eb5cdc3347dfc12e3fc9d5c47b8e20217d38c7
^D

And git status came back to live - so something is working again :) Now try to add something:

$ git add <somefile>
fatal: bad tree-ish HEAD

Oh oh... Let "fix" the archive some more:

$ git gc

That did the trick, my git repository is now fubar...

$ git co
fatal: You are on a branch yet to be born

Nice.

$ git branch master
fatal: A branch named 'master' already exists.

Hmmm, really nice.

$ git branch
error: branch 'master' does not point at a commit

$ git co master
fatal: reference is not a tree: master

The only way to fix this for now was to zapp the entire .git directory:

rm -rf .git
git init

And create a brand new repository.

Finally made the switch to jackd which works so much better than Pulseaudio. Ubuntu did not make this easy, but with some perseverance it works. One of the things I really hate about Pulseaudio is that when I click on a new song in audacious it would take about 1 second before the audio stabilized. Also with flash audio would stutter for the first few seconds.

First (if you don't care about gnome-desktop):

apt-get remove pulseaudio pulseaudio-utils

I'M pulseaudio FREE!!

Then followed this.

Install jackd and friends, read this, and try:

jackd --verbose -d alsa -r 44100 -d hw:1

(You might need hw:0 instead of hw:1 - I have two sound cards)

Now the hard task of making alsa work with jack. You are missing libasound_module_pcm_jack.so in Ubuntu...

This does not work on Ubuntu - you need to custom build your own libasound2-plugins package. You can get my 32 bit (Intel) versions of them here lib64asound2-plugins1.0.18-1ubuntu4i386.deb and libasound2-plugins1.0.18-1ubuntu4i386.deb

Custom libasound2 package

Prerequisites

apt-get install libjack-dev
apt-get build-dep libasound2-plugins
apt-get source libasound2-plugins

Now, the plugins aren't build with jack support - go figure. So you'll need to edit debian/rules in alsa-plugins-1.0.18/. Uncomment all the jack stuff you see, except the following:

# install $(INSTALL_UAG) jack/.libs/libasound_module_pcm_jack.so \
#   debian/libasound2-plugins/usr/lib/alsa-lib/libasound_module_pcm_jack.so.2.0.0
# ln -s libasound_module_pcm_jack.so.2.0.0 \
#   debian/libasound2-plugins/usr/lib/alsa-lib/libasound_module_pcm_jack.so

After this you can build the package with:

sudo dpkg-buildpackage -b -us -uc

Now you only have to take care of starting a jack-daemon on your desktop start up and you are back in audio heaven.

Yes, what a nice post on ssh escape keys.

So you need to press <enter> and then ~ for the escape character to work!