HIDI: How I Did It/ posts/ Online Resizing

Recently, the forkit.org server started running out of space on /home and /var. This isn't really a problem, for we use LVM, the Logical Volume Manager on marvin. LVM allows us to dynamically change the size of partitions. We currently have 2 147GB SCSI Ultra320 disks in a RAID1 configuration (mirrored). 15GB of that 147GB is unallocated. I wanted to add 10GB to /home and 2GB to /var. Given our historical usage patterns, this should allow us to live for another year or so before we need to look into more disk space, and leaves us 3GB for "emergency" expansions.

So, marvin is a production server. Obviously, since it is a hobby-machine, there are now SLAs or such, but a fair number of people depend on it for webhosting and email and the like. We try to minimize downtime as much as possible. I initially thought that I would be able to do the partition resizing without any downtime. The ext3 filesystem (which is what we use) supports online resizing, which should mean that I can resize the partition (using LVM), and the filesystem (using resize2fs) without needing to unmount on reboot the server. A Good Thing!

The typical mechanism to add 2GB to the /var partition would be:

# lvresize -L +2G /dev/mapper/vg-lv_var
# resize2fs /dev/mapper/vg-lv_var

I was able to do the lvresize portion - growing the partition. However, when I attempted to resize the filesystem, resize2fs complained:

# resize2fs /dev/mapper/vg-lv_var
resize2fs 1.40-WIP (14-Nov-2006)
Filesystem at /dev/mapper/vg-lv_var is mounted on /var; on-line resizing required
resize2fs: Filesystem does not support online resizing

Well, after a bit of Googling, it appears that our ext3 partition was created long enough ago that the default mkfs configuration did not include support for online resizing. Specifically, the resize_inode option was not set. Well, bummer. There is apparently a script out there that will set that option, but requires that the filesystem be offline when running. That doesn't really help us here, as the point is to avoid taking /var offline, which is a somewhat difficult thing to do on a running system, and even more difficult to do on a system that is remotely hosted.

So what is the solution? Well, as I just alluded to, we'll need to unmount /home and /var. /home can be done while the system is up, we just need to be careful to stop services like mail delivery and cron. /var is much more difficult to unmount. Lots of things need /var to run. You are much better served by booting into Single User Mode for doing work on the /var partition. Single User Mode isn't really a great option for us, as marvin is hosted in Louisville, KY, and I'm in Schenectady, NY. Luckily, Jeff (another marvin owner/operator) lives in Louisville, and can go in to get physical access if needed. This is our fallback.

The actual solution is to reboot marvin, force a fsck and insert the resizing commands into the boot sequence before non-root partitions are mounted.

To force the fsck is simple. First, create a file '/forcefsck'. This tells the boot system (on Debian systems at least) that a filesystem check should be run during the next boot. By default, this check won't fix errors automatically. This is fine for systems where you can sit there during boot, but colocated systems without network consoles like marvin should probably be set to automatically fix the errors. Do this by setting FSCKFIX=yes in /etc/rcS.d/S30checkfs.sh.

The next part, resizing the filesystems, requires writing a script to do the resize, and scheduling the script to run after the check of the filesystems, and before the mounting. This can be done by making a script called /etc/rcS.d/S32resize-disks. The scripts in /etc/rcS.d/ are run before the runlevel-dependent scripts, and basically initialize the system. The filesystems will be checked at any number after '30', and they are mounted by script S35mountall.sh. So using the number '32' insures the resize will happen after the check, and before the mounting.

> cat S32resize-disks
#!/bin/sh
PATH=/sbin:/bin
LOGFILE=/var/log/fsck/resize
logsave -as $LOGFILE resize2fs /dev/mapper/vg0-lv_var
logsave -as $LOGFILE resize2fs /dev/mapper/vg0-lv_home

The logsave prefix to the resize2fs command causes the output to be saved to the specified logfile, and stored in memory until /var is mounted.

Now, just reboot and hope! It took marvin approximately 30 minutes to do the reboot, fsck and resize. After boot, just remove the S32resize-disks script (don't need it to run on subsequent reboots). The /forcefsck file should have been removed automatically.

Enjoy your new space!