April 2009 Archives
Resize RAID1 array without knowing which disk is which
I want to grow my RAID1 array from 2x250GB to 2x500GB. The following resource has helped my a lot.
Update
It all worked, allthough the kernel decided to crash during the offline ext4 resize. But I finally have my extra disk space
% df -h | grep md6before:
/dev/md6 77G 67G 6.7G 91% /volafter:
/dev/md6 306G 67G 224G 23% /vol
Now to my problem/solution:
On Linux you often have the following problem: You are
looking or touching some piece of hardware and
you are asking yourself: "is this card eth0 or eth1?"
or "is this disk /dev/sda or /dev/sdb?".
In my case I didn't know which disk had what device name. Solution?
Rip out a random disk and replace it with an empty new disk (no partition) and hope for the best.
Some scary moments come to pass
Well after rebooting, I got the question if I wanted to boot the degraded array. After saying 'yes' I was back into my system.
Turns out I had ripped out /dev/sda. Time to mark this on the
case:
Now to restore the partition table of the old disk to the new one:
sfdisk /dev/sda < sda.sfdisk
Where sda.sfdisk is the partition info saved with sfdisk -d
/dev/sda.
Now my last partition on the disk was /dev/sda4 - luckily not in a logical
partition, as I don't know how to extend that. So I could easily delete this
partition and recreate it so that it fills the entire size of the new
disk. Some simple fdisk-foo will help you here.
Now re-add all the partitions back to the RAID1 array:
mdadm --add /dev/md0 /dev/sda1
And this for all 8 partitions.
And then... we wait for the RAID1 rebuild:
watch cat /proc/mdstat
Just to be sure I re-installed grub with grub-install /dev/sda.
Rinse and repeat for the other disk and after that has finished
rebuilding I can resize the file system on /dev/sda4. Which is
% grep sda4 /proc/mdstat
md6 : active raid1 sda4[2] sdb4[1]
So
e2fsck -f /dev/md6
resize2fs /dev/md6
Note this was all done with kernel 2.6.29.1 and the ext4 file system.
See
this post
or some info on that. And only 1 kernel crash during the offline resize!
Rebuilding the RAID1 array
I'm rebuilding my RAID1 array, so everything is a bit slow at the moment.
Massaging visitors for nice(r) html
For my web statistics I use visitors,
to generate static html. Now the default html of visitors is nice,
but I wanted to integrate it with my default website theme.
Unfortunately visitors does not have the option to generate bare
html. So Perl and shell to the rescue!
Stripping the style sheet
The html as generated by visitors include the following style
sheet:
<style>
td {
font-size: 10px;
padding: 0;
}
.barfill {
background-color: #96ef94;
border-left: 1px;
border-right: 1px;
... /* and more */
</style>
No I wanted to strip this using Perl and its marvelous flipflop (..)
operator.
1 int main(int c) { 2 printf("This is nice"); 3 exit(1); 4 }
Line 5. shows the flip-flop operator in action. The ..-operator
becomes true when <style> is seen (and stays true!) and becomes false
when </style> is seen. Sort of like a state machine in one line of
code. So if I find that I'm between the two style-tags in the file I
don't print anything except for the first time (line 6). Then I insert
my own modified style sheet.
header and footer
I also need to insert my own header and footer, this can be easily done
with some here documents in the shell. And some annoying tags need
to be stripped: sed to the rescue:
visitors -T -A -P http://www.miek.nl \
/var/log/apache2/miek.nl-access.log | \
sed -e 's/^<html>//' -e 's/^<hea d>//' \
-e 's/^<body>//' -e 's|^</head>||' \
-e 's|</body>||' -e 's|</html>||' -e 's/\[//'
Note that to keep everything consistent I use m4 to generate
templates, these templates get converted to html and those
are then used by nanoblogger.
Complete script
So the complete script for use in my site becomes:
#!/bin/zsh
# make a nice visitors page for use on miek.nl
( cat <<HEADER
...insert my header here...
HEADER
# create the content
visitors -T -A -P http://www.miek.nl \
/var/log/apache2/miek.nl-access.log | \
sed -e 's/^<html>//' -e 's/^<hea d>//' \
-e 's/^<body>//' -e 's|^</head>||' \
-e 's|</body>||' -e 's|</html>||' -e 's/\[//'
cat <<FOOTER
...insert my footer here...
FOOTER
) | perl -we '
# now fix the css style in the m5 file
# remove it completely and replace it with my own
my $printed = 0;
while (<>) {
if (m|<style>| .. m|</style>|) {
system("cat /home/miekg/bin/visit.style") if !
$printed;
$printed = 1;
} else {
print
}
}' > /home/miekg/miek.nl/server/visit.m5
# build the html, finally!
( cd /home/miekg/miek.nl; make -f m5.Makefile server/visit.html )
Recursively build playlists
My music is stored and sorted in /shared/music. To make it more
accessible for my Dreambox I want have the following playlists.
/shared/music/
<complete list of everything below>
/shared/music/P
<complete list of everything below P>
/shared/music/P/Prodigy
<complete list of everything below P/Prodigy>
First I was playing around with some Perl and came up with the following solution:
% cat m3u
#!/usr/bin/perl
while(<>) {
chomp;
chdir $_;
system("fapg -r . > playlist.m3u");
}
Which would then become
find /shared/music -type d | m3u
Only 2 things wrong with this solution
- Perl isn't needed
playlist.m3uis a bad filename sorting wise
For some odd reason I could not write it strait away in shell, but after some coffee it became
find /shared/music -type d | ( while read dir; do
cd "$dir" && fapg -r . > 00-PLAYLIST.m3u
done )
Note however that the usual find caveats apply: directories with a trailing space or directories with newlines in the name will not work.
My First Linux
I've always kept my first Linux CDs as a reminder of that great time when I was first exploring this unexplored territory.
These are two (bad) shots of the actual CD, maybe I should put the content of them online again? Kernel version 1.2.8, gcc version old, non working X on the default install, that kind of stuff... I was hooked forever to this stuff.
1995!
Transcoding video with mencoder
I've bought a dreambox 500, which is one of the smallest model they make, but I only want to use it to stream content from my pc to the tv.
When you start streaming there is almost always not right with the
video stream... its is not encoded in mpeg2 which means the dm500
can't show it, or if it is in the right format there is something
wrong with the aspect ratio.
So enter mencoder, which can be used for
all kinds of transcoding. I wanted to convert avi files to mpeg2 (or
DVD format) for showing on the dm. I'm now using the following shell
script:
1 int main(int c) { 2 printf("This is nice"); 3 exit(1); 4 }
Where you give it basename of the file you want to convert.
I also needed to a presize filter to make the video appear in the correct size
on the tv screen, so the scale=presize=pal filter is needed in this case.
Oracle and OpenSolaris
I've never like (Open)Solaris. Maybe the kernel rules, but the userland
completely sucks, something like apt should have been implemented
in (Open)Solaris years ago.
Now that Oracle owns SUN and thus Solaris they should rebrand it. Maybe OpenSoracle or even OpenSorry.
Ubuntu, LUKS disk encryption
Found this very nice howto, which I almost followed to the letter, except I did it in Ubuntu and currently more stuff is working out of the box, so you don't need to do all the steps.
I'm working on an USB stick which is /dev/sdb1 in my system.
Steps I did take:
apt-get install cryptsetup hashalotcryptsetup --verbose --verify-passphrase luksFormat /dev/sdb1cryptsetup luksOpen /dev/sdb1 funnydisknamemkfs.ext4 -O extents,uninit_bg,dir_index /dev/mapper/funnydisknamecryptsetup luksClose /dev/mapper/funnydiskname
Now pull out your disk and re-insert. If everything went well, your GNOME, XFCE, KDE env should now popup an 'This disk is encrypted, please enter the passphrase' question.
I don't know why yet, but I've just created a Twitter
account. I'm still figuring out
what utilities I should use for it: gwibber (GUI) and twidge
(command line) look nice.
In other news I also made a public Linkedin.com profile.
The source of this blog item consists out of 398 characters (excluding the title).
Hmmm...
Greylisting
Well, I finally caved in and implemented greylisting for miek.nl.
I must say, it really helps (for now). I've also made the rdup mailing
list subscriber only, because I was fed up with deleting email from
the mailman's non-subscriber queue. I.e. non-subscriber email is
now discarded.
notify of new comments
In my quest to build a comment system for my nanoblogger/shell/m4/php blogging setup I have now implemented a comment notification system. When giving a comment you can click 'Notify me of followup comments by email'. Each approved comment will then be mailed to the email address given. This will of course only happen until the commenting is closed on this entry.
Doing so will get you these kind of email messages:
Subject: Comment notify miek.nl `Windows 7?'
To: miek@miek.nl
Date: Tue, 21 Apr 2009 11:05:27 +0200 (CEST)
Hello,
This is an email to inform you of a new comment
at http://miek.nl.
It is posted under the blog entry titled `Windows 7?'
<a href="http://miek.nl">Miek Gieben</a>
Hallo
--
Comment notify sent by http://miek.nl - Tue Apr 21 11:05:27 CEST 2009
Notifies will be sent until commenting is closed
The only thing not implemented is a direct link to the blog entry with the new comment. It is doable to recreate the original URL, but for now this is sufficient.
Stuff on the todo: check if the email address is valid (syntax wise) and make it easy to remove someone from the notification list.
Windows 7?
So Windows Vista development was started in 2002 and released it as "stable" at the end of 2006. And then it still didn't give us what Microsoft had promised, like
- WinFS
- Avalon, trimmed down to Aero?
Soon (maybe after a year) they started with the development of Windows 7. Now I read on slashdot that a RC1 release is scheduled RSN.
So developing a entire new OS from scratch(?) takes about 4 years in Redmond and now they want you to believe Windows 7 is a completely new OS and will be better than Vista -- even though development took only a year?
If you believe everything Microsoft says you can always use the following quote, which I have been hearing from people use Microsoft software (even in the DOS era) for years now.
Yes, I know this version sucks, but the next version that comes out will be much better!
Coding Erlang
For the past year or so I've been trying to learn a new language called Erlang.
I've found this nice document, which can be used when learning the language. So here I am on Saturday evening doing some Erlang exercises :-)
Anyhow, I was doing exercise 3.4:
- Erlang provides a function lists:append which joins two lists together, implement you own function append that performs the append operation. (Do NOT peek; the answer is given in figure 3.11).
And the answer given is:
1 int main(int c) { 2 printf("This is nice"); 3 exit(1); 4 }
However, when I test this I get:
% erl
1> c(append).
{ok,append}
2> append:append([4,5,6], [1,2,3]).
[4,[5,[6,[1,2,3]]]]
Which made me suspicious. Checking with the official BIF:
3> lists:append([4,5,6], [1,2,3]).
[4,5,6,1,2,3]
Hmm. With a little help from "Programming Erlang" I found the ++
operator for adding lists together. So my final solution became:
1 int main(int c) { 2 printf("This is nice"); 3 exit(1); 4 }
Checking:
4> c(append).
{ok,append}
5> append:append([4,5,6], [1,2,3]).
[4,5,6,1,2,3]
The only ugly thing IMHO is the creation of the one element list with
[H] at the last line.
rdup 0.9.1 released
rdup version 0.9.1 has just been released. I'm hoping this is the last release before 1.0.0. A bug in rdup's -c output has been fixed.
See the project page for
more information or just download the
damn thing
right away. It's sha1 sum is 2f32cc18c1c58666443d0c3456b97d5c61376511
in case you care.
git release script
This is more a blog to document my own practices, than anything else.
While converting to git I discovered git archive which can create
tar archives directly from your git repository. So basically the
script is a wrapper around git archive.
1 int main(int c) { 2 printf("This is nice"); 3 exit(1); 4 }
release process
My release process now becomes
git pull, check for new commitsgit tag 0.9.1, tag the next releasegit push, push the tag to the bare git repogit log 0.9.0..0.9.1, check the log between this and the previous release../git-release 0.9.1, make the tar.bz2 for this release- update my site and send a new release email to the rdup list
rdup -c output bug
Piping rdup -c output right into rdup-up
rdup -c /dev/null ~/bin | ./rdup-up -t /vol/backup
fails with the following error:
rdup-up: First character should '-' or '+', `01BLOCK00000' at line: 8
This was reported to me by Jared Henley. It is fixed with this commit
You can work around it with, by putting rdup-tr in the pipeline, like
so:
rdup /dev/null ~/bin | rdup-tr | ./rdup-up -t /vol/backup
Note: rdup is called without the -c flag!
I wanted to release a new version of rdup anyhow, so expect a 0.9.1 release soonish.
/proc/sys/kernel/pid_max
In kernel version 2.6.29.1:
# /bin/echo -n 40000 > /proc/sys/kernel/pid_max
/bin/echo: write error: Invalid argument
In one older kernel I tried it still worked, so I've submitted a bug.
OpenLDAP 2.4 cn=config
OpenLDAP uses a cn=config DIT to configure the server since version
2.4. I'm always into new stuff, but I must admit that I rather liked
editing /etc/ldap/slapd.conf to configure the server. Anyhow being
able to store ACLs in the tree is a big plus, but for configuring minor
stuff (like indexes) it makes live more difficult.
The following site was an excellent tool in helping me configure OpenLDAP. For a list of current attributes names, see for instance here
Configuring an index
In OpenLDAP you can configure a index by using the following in
slapd.conf
index cn,uid,uidNumber eq
And then reload your ldap server. So how to translate this to the new style of configuring openldap?
Lets first see what the current indexed attributes are
# ldapsearch -xLLL -b cn=config -D cn=admin,cn=config -W olcDatabase={1}hdb olcDbIndex
Enter LDAP Password:
dn: olcDatabase={1}hdb,cn=config
olcDbIndex: objectClass eq
We look in the cn=config tree as the admin user. All OpenLDAP items
are prefixed with olc (Open Ldap Configuration?). In our first defined
database there is only an index on the objectClass.
We can now use ldapmodify to add indexes (we add three in this case):
# ldapmodify -x -D cn=admin,cn=config -W
Enter LDAP Password:
dn: olcDatabase={1}hdb,cn=config
add: olcDbIndex
olcDbIndex: cn eq
olcDbIndex: uid eq
olcDbIndex: uidNumber eq
modifying entry "olcDatabase={1}hdb,cn=config"
^D
Recheck what we've got
# ldapsearch -xLLL -b cn=config -D cn=admin,cn=config -W olcDatabase={1}hdb olcDbIndex
Enter LDAP Password:
dn: olcDatabase={1}hdb,cn=config
olcDbIndex: objectClass eq
olcDbIndex: cn eq
olcDbIndex: uid eq
olcDbIndex: uidNumber eq
Looking good. Notice that you don't have to restart your ldap server as this change is being picked up at once.
Site move
If you can read this, it means the server move was successful. I'm once again hosting everything from my basement. On a quad core Intel machine running Ubuntu with a shiny new 2.6.29.1 kernel.
Whisky
Ever after we have been to Japan, I got a taste for whisky. After seeing a documentary on National Geographic on the history of whisky I decided to try some Irish whisky, like Jameson.
Now I have made the mistake of buying some Scottish crap called "Glen Talloch", which bills itself as being a "very old" whiskey. Too bad it is disgusting.
So I'm back to Jameson right now.
anti-virus scanners are useless
So there you are with your windows XP or Vista. You know your way around technical stuff so you run (several?) anti-virus scanners.
Now I was always intrigued by computer viruses, so I bought a book about it from Dr. Fred B. Cohen, called "A Short Course on Computer Viruses" (2nd edition, ISBN: 0-471-00768-4). See this link for some texts out of the book.
Fred Cohen has invented the concept of a computer virus. At the time (around 1983) nobody believed that such software could be written and that it would work... I think this was even demonstrated on UNIX systems, oh the irony :-)
Anyhow in this book, there is the following text, paragraph 3.2 Technical Defenses with Major Flaws (page 64)
.. That means you cannot write a program D that correctly determines whether or not other programs are viruses, unless that program: runs forever without a result - or has an infinite number of false positives - or has an infinite number of false negatives - or has combinations of these three problems
So basically you can not write a program that will always correctly identify a virus.
Still people tried, so now we have virus scanners. In paragraph 3.2.6 "Virus Scanner", page 72,73, Cohen says the following (wording slightly changed)
A first problem is that they [virus scanners] are only good against known viruses in personal computers.
A second problem is that they tend to take a noticeable amount of time to scan a system or network for these patterns.
A third problem is that in order to remain effective, a scanner must be update often.
With the Internet the last problems seems to be solved, but infections also spread faster, so basically the update problem remains as bad as it was in 1983.
So the inventor of computer viruses writes in 1983 that anti-virus scanners are useless.
And there you are in 2009, running two (or even more) anti-virus scanners to make you feel safe. Who are you fooling?
There are so many alternatives.
LVM: good, LVM snapshots: bad
Well, today I was looking into using LVM snapshots to allow a client OCN use Linux as a Netapp replacement.... Boy was I in for a disappointment.
LVM an sich is working great, but the moment you turn on snapshots the (in this case) write performance goes to hell. Using LVM is easy enough. The system I was on has 32 GB ram and 2 disk arrays with hardware RAID.
Setting up:
pvcreate /dev/sdb1
pvcreate /dev/sdc1
vgcreate volume1 /dev/sdb1 /dev/sdc1
vgchange -a y volume1
lvcreate -n lv-vol1 -L1.00T volume1
mke2fs -j /dev/volume1/lv-vol1
mount /dev/volume1/lv-vol1 /vol
and you are ready. Creating a snapshot can be done with
lvcreate -s -n snap-lv-vol1 -L50G volume1
Sadly enough such a snapshots isn't enlarged automatically when it turns
out to be too small (Netapp has this feature). With a lvresize this
can be easily remedied.
Making a snapshot with the normal chunk size
of 8.0 KB kill performance once in a while you will see a kcopyd
process which is copying blocks from your live file system to the
snapshot(s). I've witnessed a hanging file system, which took minutes to
spring back to live -- totally not acceptable. Even when LVM works
okay, turning on snapshots kills performance.
If you search for "LVM linux snapshot performance" you will get a lot of complaints, but no solutions. In short: LVM is often billed as the solution for everything and those very handy snapshots are great to have. Too bad it turns out these snapshots are not production ready. I'm totally not comfortable to deploy LVM and snapshots in any production environment.
To fake the snapshots we are going to deploy rsnapshot which uses
rsync to fake snapshots.
We were also really tempted to deploy OpenSolaris with ZFS, the only thing holding us back is that this would introduce another platform that must be administered. Other than that:
I want something like ZFS on Linux!
(Yes, I know of btrfs)
Linux networking bonding
See this?
bond0: Warning: failed to get speed and duplix from eth*
Do this:
insmod bonding miimon=100
why?
When using Linux network bonding the kernel may be to quick to enslave the interfaces. When an interface is too slow to report it capabilities it will be set to 100Mbit and Full Duplex. Which is a bit sad when you have 6 Gigabit network cards...
vim text objects
For a few weeks months now I'm learnings how to use
VIM text-objects. There is
an extended help wth help text-objects in VIM. I'm trying to condense the
VIM help in a smaller blog entry here.
What are a text-objects in vim? Text-objects are things like a
'paragraph' or the text between braces or something like a
word. Text-objects can be used with the normal vim commands y, d and
c. To make you really understand it, I can recommend using control-V
to actually visually select your text object.
In vim's text-objects documentation they talk about an important property. A text-object selection is either inner or not...
Some text-object selectors:
aw- a word, works on wordsiw- inner word, works on wordsab- brace, works with text between ( and )ib- inner brace, works with text between ( and )...- there are many more
Note: the i is used to denote inner selection, the a for
normal ones.
Consider the following text:
(passed init=/sbin/bootchartd)
(this is some stuff from Ubuntu's bootchart implementation).
Now an inner brace selection will be
passed init=/sbin/bootchartd
without the braces. An inner text-object selector will always select less than the normal block selection. The normal brace selection will select
(passed init=/sbin/bootchartd)
with the braces. You can check this for your self with the
combo: control-V+a+b (normal selection) or control-V+i+b (inner
selection). Note the selection works from anywhere inside the
braces: it does not matter where the cursor is positioned.
With dib you delete the inner brace text-object.
The word text-object works by letting you select only the word (= inner selection) or the word + whitespace (= normal selection). Thus, with the same sentence as above and the cursor positioned on the p of passed.
(passed init=/sbin/bootchartd)
control-V+i+w, selects
(passed init=/sbin/bootchartd)
In contrast with, control-V+a+w, which selects
(passed init=/sbin/bootchartd)
With the space after 'passed' also.
See vim's online help for more info, but I hope this little text was helpful to you (It certainly helped me).
Server upgrade with ldap
As I'm on an upgrade roll I decided to upgrade my new server too. It is configured with kerberos and ldap... and this is were the trouble.
ldap
When a service is upgraded in Ubuntu/Debian it is first stopped and than
later restarted when the new files are there. When all your user
information is kept in ldap, the following sucks:
Preparing to replace slapd 2.4.11-0ubuntu6.1 (using
.../slapd_2.4.15-1ubuntu3_i386.deb) ...
Stopping OpenLDAP: slapd.
Dumping to /var/backups/slapd-2.4.11-0ubuntu6.1:
- directory dc=atoom,dc=net... done.
Unpacking replacement slapd ...
And there goes the neighborhood... everything slows to a crawl, because
every getpwnam call needs to timeout before /etc/passwd and friends
are consulted.
So a control-C (at the wrong time of course, so this gave also
problems) the upgrade process to fix /etc/nsswitch.conf and disable
ldap-lookups for now:
sudo vi /etc/nsswitch.conf
vi: error while loading shared libraries: libgailutil.so.18: cannot
open shared object file: No such file or directory
oh... oh... Luckily there is also nano, but as a vim addict I find
nano very confusing. After figuring out that control-X will save the
file I can finally make the change.
