[linux-lvm] Dead device and cleaning up LVM

[Ray Van Dolson]

I have a volgroup that was created on top of an iSCSI mounted block
device (target was stgt on RHEL 5.3 and initiator is the default iSCSI
initiator in RHEL 4.7).

Through a goof the target went offline and I tried shutting down the
initiator software before disabling the volume group on the initiator
with vgchange -an. Now I'm left with a bit of a mess including iSCSI
initiator modules that won't completely unload. I'm trying to avoid a
reboot and we have been theorizing that we somehow need to tell LVM to
"let go" of the remaining ghosts of the iSCSI device before things can
go back to normal.

The /dev/sdXX device is gone, but how can I convice LVM to quit trying
to use /dev/dm-2? Can I edit the /etc/lvm/cache/.cache file and remove
entries there?

I'm seeing the following stuff in dmesg:

scsi7 (0:1): rejecting I/O to dead device

And all the various pvscan/vgscan/lvscan tools show something similar
to the following:

/dev/dm-2: read failed after 0 of 4096 at 4398038056960: Input/output error
/dev/dm-2: read failed after 0 of 4096 at 4398038114304: Input/output error
/dev/dm-2: read failed after 0 of 4096 at 0: Input/output error
/dev/dm-2: read failed after 0 of 4096 at 4096: Input/output error
/dev/dm-2: read failed after 0 of 4096 at 0: Input/output error

Anyone know how I can remove this device both from the system and from
LVM safely? I can reboot, but saving that as a last resort if
possible.

Also, any best practices or suggestions for using iSCSI backed block
devices as volgroup members would be appreciated. I'm worried about
LVM not recognizing the volume group of physical volumes correctly on
reboots...

Thanks,
Ray

[Bryn M. Reeves]

Post by Ray Van Dolson
Anyone know how I can remove this device both from the system and from
LVM safely? I can reboot, but saving that as a last resort if
possible.

You don't mention the name of your VG/LV but you should be able to
remove the active lvm devices with "dmsetup remove", e.g.:

dmsetup remove $VG-$LV

For each LV in each VG that was using the iSCSI devices.

Post by Ray Van Dolson
Also, any best practices or suggestions for using iSCSI backed block
devices as volgroup members would be appreciated. I'm worried about
LVM not recognizing the volume group of physical volumes correctly on
reboots...

You might need to label any file system mounts in /etc/fstab with
"_netdev" and enable the netfs service to have the VGs automatically
activated/mounted at boot time.

Regards,
Bryn.

[Ray Van Dolson]

Post by Bryn M. Reeves

Post by Ray Van Dolson
Anyone know how I can remove this device both from the system and from
LVM safely? I can reboot, but saving that as a last resort if
possible.

You don't mention the name of your VG/LV but you should be able to
dmsetup remove $VG-$LV
For each LV in each VG that was using the iSCSI devices.

Beautiful. This did the trick! I can now also unload the iSCSI kernel
modules correctly on the initiator.

Thanks,
Ray

[Ray Van Dolson]

Post by Ray Van Dolson

Post by Bryn M. Reeves

Post by Ray Van Dolson
Anyone know how I can remove this device both from the system and from
LVM safely? I can reboot, but saving that as a last resort if
possible.

You don't mention the name of your VG/LV but you should be able to
dmsetup remove $VG-$LV
For each LV in each VG that was using the iSCSI devices.

Beautiful. This did the trick! I can now also unload the iSCSI kernel
modules correctly on the initiator.

Also, just an FYI in case anyone else runs across this thread... to get
my filesystem back (pvscan didn't detect the LVM at all):

# pvcreate --uuid 3HpAVg-fRz0-HWXp-EFxK-khVO-IvN6-J4a6w1 /dev/sdb
(Found the old UUID by examining the /etc/lvm/backup directory)
# vgcfgrestore FTP1
# vgchange -ay FTP1
# fsck.ext3 /dev/FTP1/LogVol00
# mount -a

Back in action!

Ray

[Bryn M. Reeves]

Post by Ray Van Dolson

Post by Ray Van Dolson

Post by Bryn M. Reeves

Post by Ray Van Dolson
Anyone know how I can remove this device both from the system and from
LVM safely? I can reboot, but saving that as a last resort if
possible.

You don't mention the name of your VG/LV but you should be able to
dmsetup remove $VG-$LV
For each LV in each VG that was using the iSCSI devices.

Beautiful. This did the trick! I can now also unload the iSCSI kernel
modules correctly on the initiator.

Also, just an FYI in case anyone else runs across this thread... to get
# pvcreate --uuid 3HpAVg-fRz0-HWXp-EFxK-khVO-IvN6-J4a6w1 /dev/sdb

If you also specify "--restorefile=" and the path to the current
metadata backup in /etc/lvm, pvcreate will ensure that the new PV can
accommodate the VG you want to restore to it. It'll almost certainly
always work without this if you always use the default options but by
specifying --restorefile you can be completely sure.

Post by Ray Van Dolson
(Found the old UUID by examining the /etc/lvm/backup directory)
# vgcfgrestore FTP1
# vgchange -ay FTP1
# fsck.ext3 /dev/FTP1/LogVol00
# mount -a
Back in action!

Glad to hear it!

Regards,
Bryn.

[Stefan Monnier]

You don't mention the name of your VG/LV but you should be able to remove
dmsetup remove $VG-$LV

IIUC you can also use

lvchange -an /dev/<VG>/<LV>

at leas that's what I do when an LVM volume becomes unavailable
(e.g., because the corresponding USB drive was disconnected).
I guess "vgchange -an" might work as well.


Stefan