Tuning an HP ProLiant MicroServer

I recently became the owner of an HP ProLiant MicroServer N54L, a small box with a bunch of hard drive bays and a low-power AMD Turion dual-core processor. I’ve also been tweaking it quite heavily, resulting in a dramatic improvement in network performance and a significant drop in power draw – and this is how I did it. If you’re not running Linux, mind, everything past Step 1 is likely of no use to you.

Warning: The advice here worked for me, but some of it – in particular turning off journalling and using an aggressive spin-down profile on the drives – is hardly best-practice. It can even shorten the lifespan of certain components. Where something can have a deleterious effect, I’ll highlight it – but go into this warned that not everything I’ve done here is for everyone.

Step 1: Unlock the BIOS
The MicroServer is HP’s entry-level server, and as a result is missing several features of higher-end models – in particular hot-swap drive bays. This isn’t a physical restriction, but a fake restriction put in place by HP – and it can be removed by installing a modified BIOS. Doing so also allows all the SATA ports to run at 3Gb/s – two are limited to 1.5Gb/s by default – and fixes a flaw in the original BIOS that prevents the NIC from operating under certain operating systems.

HP has recently taken the decision to lock its BIOS updates behind a warranty wall: if you don’t have a warranty, you can’t get BIOS updates. That’s not very nice, so it’s a good job that someone has uploaded a pre-modified BIOS. The naughty person. Just download the file, write the image to a USB stick with dd (or ImageWriter if you’re on Windows) and power the server on with the stick in a port. Wait for the DOS prompt to appear and power off – you’re done.

Step 2: Network Tuning
Out of the box, I found the MicroServer’s network performance to be poor indeed under Ubuntu 13.10. Some of the blame has to go to the cheap Broadcom NIC, but I figured I could do better.

First, edit /etc/rc.local and stick the following lines into it, just above ‘exit 0’:

ethtool -G em1 rx 511
ifconfig em1 txqueuelen 1000
defaultroute=`ip route | grep "^default" | head -1`
ip route change $defaultroute initcwnd 10

This forces the NIC to use its entire ring buffer for RX packets – by default only 200 bytes of the 511 bytes available get used – and increases the transmit queue length to a figure better suited to gigabit network traffic. The default route is also tweaked.

Next, edit /etc/sysctl.conf and add the following:

fs.file-max = 100000
net.core.netdev_max_backlog = 50000
net.core.optmem_max = 40960
net.core.rmem_default = 16777216
net.core.rmem_max = 16777216
net.core.wmem_default = 16777216
net.core.wmem_max = 16777216
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.all.log_martians = 1
net.ipv4.conf.all.send_redirects = 0
net.ipv4.ip_local_port_range = 10000 65000
net.ipv4.tcp_fin_timeout = 10
net.ipv4.tcp_max_syn_backlog = 30000
net.ipv4.tcp_max_tw_buckets = 2000000
net.ipv4.tcp_rfc1337 = 1
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_sack=0
net.ipv4.tcp_slow_start_after_idle = 0
net.ipv4.tcp_timestamps=0
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_wmem = 4096 65536 16777216
net.ipv4.udp_rmem_min = 8192
net.ipv4.udp_wmem_min = 8192
vm.swappiness = 10

A reboot will bring the box up with your new settings, or you can force a reload with sysctl -p (or sudo sysctl -p if you’re not root.) Combined, these two tweaks resulted in the throughout of the system going from about 40MB/s to 120MB/s, so it’s definitely worth doing.

Step 3: Power Tuning
The idle power of the MicroServer wasn’t great when I was finished tuning the network, pulling around 40W from the wall. A little careful tuning of the system, though, and that dropped to 21W – a significant saving.

First, we’ll discuss the fixes that won’t cause you any problems. Open /etc/rc.local again and add the following lines:

echo 'min_power' > '/sys/class/scsi_host/host2/link_power_management_policy'
echo 'min_power' > '/sys/class/scsi_host/host3/link_power_management_policy'
echo 'min_power' > '/sys/class/scsi_host/host4/link_power_management_policy'
echo 'min_power' > '/sys/class/scsi_host/host5/link_power_management_policy'
echo '0' > '/proc/sys/kernel/nmi_watchdog'
echo '1500' > '/proc/sys/vm/dirty_writeback_centisecs'
echo 'auto' > '/sys/bus/pci/devices/0000:00:18.3/power/control'
echo 'auto' > '/sys/bus/pci/devices/0000:00:00.0/power/control'
echo 'auto' > '/sys/bus/pci/devices/0000:00:01.0/power/control'
echo 'auto' > '/sys/bus/pci/devices/0000:00:06.0/power/control'
echo 'auto' > '/sys/bus/pci/devices/0000:01:05.0/power/control'
echo 'auto' > '/sys/bus/pci/devices/0000:00:11.0/power/control'
echo 'auto' > '/sys/bus/pci/devices/0000:00:12.0/power/control'
echo 'auto' > '/sys/bus/pci/devices/0000:00:13.0/power/control'
echo 'auto' > '/sys/bus/pci/devices/0000:00:13.2/power/control'
echo 'auto' > '/sys/bus/pci/devices/0000:00:18.4/power/control'
echo 'auto' > '/sys/bus/pci/devices/0000:00:12.2/power/control'
echo 'auto' > '/sys/bus/pci/devices/0000:00:18.2/power/control'
echo 'auto' > '/sys/bus/pci/devices/0000:00:14.0/power/control'
echo 'auto' > '/sys/bus/pci/devices/0000:00:14.1/power/control'
echo 'auto' > '/sys/bus/pci/devices/0000:00:14.3/power/control'
echo 'auto' > '/sys/bus/pci/devices/0000:00:14.4/power/control'
echo 'auto' > '/sys/bus/pci/devices/0000:00:16.0/power/control'
echo 'auto' > '/sys/bus/pci/devices/0000:00:16.2/power/control'
echo 'auto' > '/sys/bus/pci/devices/0000:00:18.0/power/control'
echo 'auto' > '/sys/bus/pci/devices/0000:00:18.1/power/control'
echo 'auto' > '/sys/bus/pci/devices/0000:02:00.0/power/control'

That’ll have a small but noticeable impact on power draw at idle, enabling a bunch of typically-off power management features. If you’ve got additional USB or PCIe devices in your MicroServer, you may find that this doesn’t catch all of ’em; install PowerTOP and go to the ‘Tunables’ tab to see if there’s anything labelled ‘Bad,’ then toggle it to ‘Good.’

That’s a start, but for real savings we need to spin the drives down. This may or may not be possible in your usage scenario: mechanical drives have a limited number of load/unload cycles before they break, and the last thing you want to be doing is having it constantly spinning down and spinning back up again. Because I’m not always accessing the MicroServer – I back up to it once a day, stream media from it during the evening, and otherwise mostly leave it alone – I can get away with a reasonably aggressive power-saving plan; you may well need to adjust the numbers below to suit your own needs.

If you want to continue, start with adding the following to /etc/rc.local:

hdparm -S 60 /dev/sda /dev/sdb /dev/sdc /dev/sdd
hdparm -B 1 /dev/sdb /dev/sdc /dev/sdd
hdparm -M 128 /dev/sdb /dev/sdc /dev/sdd

That tells the system to spin all four hard drives down after 300 seconds of inactivity. If you ain’t got four hard drives, edit the line accordingly. Some drives may not sleep on their own; if that’s the case, have a look at the Powernap tool and use it in conjunction with ‘hdparm -y’ to force a drive to spin down. I use it on my system because one drive of a mirror sleeps while the other stays awake; Powernap watches for this, and when the first drive sleeps it forces the second into the low-power state.

The second command puts the drives in the most aggressive power management level possible, while the third puts it into the quietest possible acoustic management mode. Note that I haven’t included the boot drive sda there: the drive that came with my MicroServer doesn’t support either of those settings. Try running the command yourself, and if your drives do support it then add them to the above list.

One drive that won’t spin down is your boot drive – in my case, the 250GB drive the MicroServer came with. If that’s a problem, consider switching to a solid-state drive (SSD) instead. If you’re truly desperate to save as much power as possible but have no cash left, however, there are options. First, move /var – the bit of the disk that sees the most activity, in particular due to the log files held in /var/log, to a spare USB flash drive you’ve got lying around. You’ll need at least 1GB, preferably 2GB or more to allow for sudden growth.

The procedure for moving /var isn’t straightforward, and is rife with danger – it’s perfectly possible to end up with a non-booting system. If you’re still reading, here’s how I did it (as root) with the USB stick inserted as /dev/sde. First, edit /etc/fstab and insert a line to mount the USB stick as /var, and while you’re at it edit the root mount point to disable access time writing.

/dev/sde1       /var    ext4    defaults,noatime,nodiratime     0       1
/dev/sda1       /       ext4    noatime,nodiratime,errors=remount-ro 0       1

I’d recommend using the drive’s UUID (visible with the blkid command) instead of the device node, ‘cos you don’t want the system to get confused and mount the wrong drive. Next, populate the drive:

mkfs.ext4 /dev/sde1
mkdir /tempvar
mount -t ext4 /dev/sde1 /tempvar
cp -rfp /var -T /tempvar
umount /tempvar
mount /dev/sde1 /var
reboot

When the server comes up – assuming nothing went wrong – /var is now running from the USB drive, saving writes to the mechanical drive. However, you may find the mechanical drive still doesn’t spin down. In my case, this was ‘cos I was running a journalling file system – ext4 – which was always writing data to the disk. Simple fix: turn journalling off.

Another warning: turning journalling off is a really bad idea. Journalling is there to ensure that the file system doesn’t get corrupted. Turn it off, and there’s a non-zero chance that a sudden reboot – a power-cut, say – will result in a corrupt root file system and a non-booting server. I’ve got a UPS and a backup of the file system, so it’s a chance I’m willing to take – but consider the implications long and hard before continuing.

If you’re still convinced you want to do this, you can turn ext4’s journalling option off by remounting the root filesystem read-only and issuing the following command:

tune2fs -O "^has_journal" /dev/sda1

With that, the root drive is now free to spin down – resulting in the aforementioned 21W idle power draw. But seriously, don’t do this. Just buy a cheap SSD instead.