So today I found myself staring at the Red Hat LVM documentation wherein I began to silently cuss. You see, I needed to double the LUN size for MySQL’s usage on one of my servers. These servers are our first that are directly attached into the SAN and running Linux, so we’re working a bit off the map here. To make matters more difficult, we’re using multipathd to manage the SAN connections between the system and SAN, so it wasn’t very clear on what exactly I should do in this case. The LVM docs are somewhat … lacking.
Thanks to the wonder of Google and sheer dumb luck, I ran across this post on one of the RHEL5 mailing lists. In that case, the author wasn’t sure if it was the right method. But, I had a secondary system and was willing to run with scissors for a moment since we’re not fully in production yet.
In short, the basic flow of operations is:
- Figure out the multipath I/O device name.
- Figure out the underlying device IDs (or device names)
- Issue the resize of the LUN in your SAN.
- Tell the kernel to rescan the underlying device IDs so it sees the new LUN size.
- Tell multipathd that a resize has occurred.
- Issue a pvresize so LVM knows it has more extents to work with now.
- Issue an lvresize to increase the logical volume size.
- Run resize2fs and do an online resize of the filesystem.
- Make popcorn.
- Watch a movie.
The first time I attempted the process (in a similar, but not quite fashion), I caused the system hang all LVM commands. I turned off multipathd thinking that it would need to be off while I did the resize. This appears to not be a healthy way to do it because I ended up having to warm cycle the system. This is the point where I stopped reading the LVM documentation and found the mailing list post.   Tried it out and it worked.
So, without further ado …
# multipath -ll mpath0
mpath0 (360060160dac711004c6fa9d07c7cde11) dm-2 DGC,RAID 10
[size=50G][features=1 queue_if_no_path][hwhandler=1 emc][rw]
\_ round-robin 0 [prio=2][active]
\_ 1:0:1:0 sdc 8:32 [active][ready]
\_ 2:0:1:0 sde 8:64 [active][ready]
\_ round-robin 0 [prio=0][enabled]
\_ 1:0:0:0 sdb 8:16 [active][ready]
\_ 2:0:0:0 sdd 8:48 [active][ready]
# for i in `multipath -ll mpath0 | grep sd | awk '{print $2}'`; do
> echo $i ; done
1:0:1:0
2:0:1:0
1:0:0:0
2:0:0:0
# for i in `multipath -ll mpath0 | grep sd | awk '{print $3}'`; do
> blockdev --rereadpt /dev/$i ; done
BLKRRPART: Input/output error
BLKRRPART: Input/output error
# multipathd -k"resize multipath mpath0"
ok
# multipath -ll mpath0
mpath0 (360060160dac711004c6fa9d07c7cde11) dm-2 DGC,RAID 10
[size=100G][features=1 queue_if_no_path][hwhandler=1 emc][rw]
\_ round-robin 0 [prio=2][enabled]
\_ 1:0:1:0 sdc 8:32 [active][ready]
\_ 2:0:1:0 sde 8:64 [active][ready]
\_ round-robin 0 [prio=0][enabled]
\_ 1:0:0:0 sdb 8:16 [active][ready]
\_ 2:0:0:0 sdd 8:48 [active][ready]
# pvresize /dev/mapper/mpath0
Physical volume "/dev/mpath/mpath0" changed
1 physical volume(s) resized / 0 physical volume(s) not resized
# lvresize -L 50G /dev/VolGroupMySQL/mysql-san
Extending logical volume mysql-san to 50.00 GB
Logical volume mysql-san successfully resized
# resize2fs /dev/VolGroupMySQL/mysql-san
resize2fs 1.39 (29-May-2006)
Filesystem at /dev/VolGroupMySQL/mysql-san is mounted on /var/lib/mysql; on-line resizing required
Performing an on-line resize of /dev/VolGroupMySQL/mysql-san to 13107200 (4k) blocks.
The filesystem on /dev/VolGroupMySQL/mysql-san is now 13107200 blocks long.
# df
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
9.7G 3.3G 6.0G 36% /
/dev/sda1 122M 13M 103M 11% /boot
tmpfs 3.9G 0 3.9G 0% /dev/shm
/dev/mapper/VolGroupMySQL-mysql--san
50G 795M 46G 2% /var/lib/mysql
One thing to note is the BLKRRPART errors from blockdev. This appears to be “normal” as far as I can tell. The kernel through some log messages (included below), but they appear harmless as far as I can discern. The SCSI notices occurred when I issued the blockdev command. The device-mapper multipath warning came from the multipathd resize command.
SCSI device sdc: 209715200 512-byte hdwr sectors (107374 MB)
sdc: Write Protect is off
sdc: Mode Sense: 87 00 00 08
SCSI device sdc: drive cache: write through
sdc: detected capacity change from 53687091200 to 107374182400
sdc: unknown partition table
SCSI device sde: 209715200 512-byte hdwr sectors (107374 MB)
sde: Write Protect is off
sde: Mode Sense: 87 00 00 08
SCSI device sde: drive cache: write through
sde: detected capacity change from 53687091200 to 107374182400
sde: unknown partition table
SCSI device sdb: 209715200 512-byte hdwr sectors (107374 MB)
sdb: test WP failed, assume Write Enabled
sdb: asking for cache data failed
sdb: assuming drive cache: write through
sdb: detected capacity change from 53687091200 to 107374182400
sdb:<6>sd 1:0:0:0: Device not ready: <6>: Current: sense key: Not Ready
Add. Sense: Logical unit not ready, manual intervention required
end_request: I/O error, dev sdb, sector 0
printk: 68 messages suppressed.
Buffer I/O error on device sdb, logical block 0
unable to read partition table
SCSI device sdd: 209715200 512-byte hdwr sectors (107374 MB)
sdd: test WP failed, assume Write Enabled
sdd: asking for cache data failed
sdd: assuming drive cache: write through
sdd: detected capacity change from 53687091200 to 107374182400
sdd:<6>sd 2:0:0:0: Device not ready: <6>: Current: sense key: Not Ready
Add. Sense: Logical unit not ready, manual intervention required
end_request: I/O error, dev sdd, sector 0
unable to read partition table
device-mapper: multipath emc: long trespass command will be send
device-mapper: multipath emc: honor reservation bit will not be set (default)
device-mapper: multipath: Using dm hw handler module emc for failover/failback and device management.
device-mapper: multipath emc: emc_pg_init: sending switch-over command
