It is slow, but I found out the reason… However in order to avoid this slowness, I will must not use an important feature of the operating system.

Since the upgrade to 9.04, I noticed that the hard disk led was blinking long minutes. Updating software took long minutes, however downloading was quit fast. Tried to investigate with atop, tried to kill processes that appeared to access the hard disk etc. No real improvement. Often the system seemed to be freezing…

But after investigating few hours, the unexpected result came out. It was difficult to find the cause because the bad was not always happening. And finally I remarked that the difference was between rebooting and restarting from suspend! And the hdparm test hast confirmed it:

sudo hdparm -tT /dev/sda
/dev/sda:
Timing cached reads:   1978 MB in  2.00 seconds = 988.92 MB/sec
Timing buffered disk reads:  162 MB in  3.08 seconds =  52.64 MB/sec
misi@kaktusz:~$ sudo hdparm -tT /dev/sda

/dev/sda:
Timing cached reads:   1072 MB in  2.00 seconds = 536.09 MB/sec
Timing buffered disk reads:    4 MB in  5.50 seconds = 744.96 kB/sec

The funny thing is that the rest of the hard disks were behaving normally after suspend, so only the root filesystems hard disk was the evil…

So… is this a bug in kernel (2.6.28-11-generic)?

Anyway, for me the solution is to avoid suspend, shut down, and boot when needed, but wait 50000 seconds for boot

Due to the complex dependencies in the linux boot process the safest way to boot is to run the different initialization scripts sequentially. The result is a long boot process, subsystems are initialized earlier than needed etc.

There are different approaches like upstart, initng, runit, einit that try to improve the boot process on linux, each approach concentrating on several aspects, like running initialization scripts in parallel, handling device generated events in a more flexible way etc. Ubuntu for example comes now with upstart as default init, but it failes to use even at minimum the strength of the ideas behind upstart.

With the intent to speed up the linux process both for desktop and for embedded devices like ubuntu, I started to write some scripts that generates native upstart scripts based on the information from the /etc/init.d/* scripts. As first stage it would be enough to create the native upstart script as wrapper scripts around the /etc/init.d/* scripts.

One could speed up the boot process if initialization scripts could be run in parallel. As we know certain scripts need to have other scripts already started before. To simplify my description I will use the terms target as synonim for the script and dependencies for scripts that need to be started. Having the dependency relation ship we identify two categories of scripts: dependants and dependees. Of course the dependee can be dependant and so on… It is a general requirement that there should be no cyclic dependencies in sysVinit scripts.

The most trivial approach is to create upstart scripts for the dependees containing following

start on starting …. #list those targets that require the current one

pre-start exec /etc/init.d/$MYSCRIPT start

pre-stop exec /etc/init.d/$MYSCRIPT stop

the dependants (targets) would contain sthg. like:

stop on stopping … #list those dependencies that if stopped, the current one should be stopped

pre-start exec /etc/init.d/$MYSCRIPT start

pre-stop exec /etc/init.d/$MYSCRIPT stop

the two above would be merged like:

start on starting …. #list those targets that require the current one

stop on stopping … #list those dependencies that if stopped, the current one should be stopped

pre-start exec /etc/init.d/$MYSCRIPT start

pre-stop exec /etc/init.d/$MYSCRIPT stop

the first version of a python script that would generate such wrappers would be sthg. like… (GPL!!!!)

def createfile(file, startingon):
f = open(”/etc/event.d/upgen/” + file, ‘w’)
f.write(’#file automatically generated by upgen \n\n’)
for dep in startingon:
f.write(’start on starting ‘ + dep + ‘\n\n’)
f.write(’\n\n’)
f.write(’pre-start exec /etc/init.d/’ + file + ‘ start \n\n’)
f.write(’pre-stop exec /etc/init.d/’ + file + ‘ stop\n\n’)
f.close()

def createfiles(startingonmap):
for file in startingonmap:
createfile(file, startingonmap[file])

depfile=’/etc/init.d/.depend.boot’

startingonmap={}

for line in open(depfile, ‘r’).readlines():
#skip lines with
if (line.find(’TARGETS =’) != -1) or (line.find(’INTERACTIVE =’) != -1):
pass
else:
t = line.split(’:')
dependant = t[0]
dependies = t[1].split()
for dependy in dependies:
inversemap = {}
try:
inversemap = startingonmap[dependy]
except:
pass
inversemap[dependant] = 1
startingonmap[dependy]=inversemap

createfiles(startingonmap)

here: http://www.mobiphil.com/?p=33 I was writing about my bad experience with nfs4. Switched back to 3, and things work just NICE…

After months of fight and frustration with nfs4 on Ubuntu, I switched to nfs3. And finally I can see the  light at the end of the tunnel! As I was describing here, I made some measurement about copy speed of small files towards nfs server and back. Copying to the server was almost 10 times slower with nfs4 than copying from the server (same harddisk type both on server and client).

Now with nfs3 the speeds are the same in both directions!!!

By having more and more devices around at home, it makes sense to keep all data in one place. I have a server for longer time, that runs a samba server, nfs file server and is acting as host for VirtualBox and Xen virtual machines. I had finally time to complete another phase of this “move everything to server”, by using a central home directory for all the clients.

The experiment was so far not completely successful as the writing speed towards the server is horribly low. I made the following experiments

1. Measured the ip speed with iperf in both directions between the client and server.

on the server:

xxx@yyyyyyyyy:~$ iperf -s
————————————————————
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
————————————————————
[ 4] local 192.168.12.5 port 5001 connected with 192.168.12.11 port 56444
[ 4] 0.0-10.0 sec 753 MBytes 630 Mbits/sec

on the client

uuuuu@qqq:~$ iperf -c 192.168.12.5
————————————————————
Client connecting to 192.168.12.5, TCP port 5001
TCP window size: 16.0 KByte (default)
————————————————————
[ 3] local 192.168.12.11 port 56444 connected with 192.168.12.5 port 5001
[ 3] 0.0-10.0 sec 753 MBytes 631 Mbits/sec

so the transfer rate is quite good (630Mbits/sec) for a Gbit card on both side as the server has a slower PCI.

2. Copied a 100MByte file from the server to the client, it took 1.6 sec

3. copied a the same 100MByte file from the client and it took 60sec!!! so 10 times slower!!! Cannot beleive….

Here is the line from /etc/fstab for the /home folder

192.168.12.5:/XXXXXX/home /home nfs4 auto,rw,nodev,sync,_netdev,noatime,proto=tcp,retry=10,rsize=32768,wsize=32768,hard,intr 0

4. By changing the sync parameter to async there was some significant improvement, but still far from perfect… copying from the server remained 1.6 sec, and copy to the server dropped to 3.6 sec. The same copy on the server, locally takes 1.7 sec.

This means that copying through nfs is twice slower than a local copy without having network bottleneck.

4. Copying 1000 files of 100kbyte from server to client took ~3sec, but back to the server 45secs!!!!! This is really painful. This made it clear, why programs like firefox slowed down since the nfs home. Firefox writes several small files quite often, so …

5. And now some comparison about (half) alternatives. Copying with scp the same 1000 files to the server took 12secs, so almost a quarter of the time spent with nfs.

Important to mention that there is no router between the computers, they are connected with a cat6 cable (gigabit ethernet).

Any suggestion if this copy time could be decreased?