[linux-lvm] No device found for PV but is actually fine

[Michal Svoboda]

Hi all,

I have a weird issue:

# pvs
No device found for PV JszGc1-S16s-bsFQ-weAl-aj8W-d6y4-BeVqcN.
PV VG Fmt Attr PSize PFree
/dev/sdc data2 lvm2 a-- 298.09g 0
/dev/sdd1 data lvm2 a-- 298.09g 0

# pvscan
No device found for PV JszGc1-S16s-bsFQ-weAl-aj8W-d6y4-BeVqcN.
No device found for PV JszGc1-S16s-bsFQ-weAl-aj8W-d6y4-BeVqcN.
PV /dev/sdc VG data2 lvm2 [298.09 GiB / 0 free]
PV /dev/sdd1 VG data lvm2 [298.09 GiB / 0 free]
Total: 2 [596.18 GiB] / in use: 2 [596.18 GiB] / in no VG: 0 [0 ]

# pvscan --cache
Found duplicate PV JszGc1S16sbsFQweAlaj8Wd6y4BeVqcN: using /dev/sde2 not /dev/md0
Found duplicate PV JszGc1S16sbsFQweAlaj8Wd6y4BeVqcN: using /dev/sdf2 not /dev/sde2

The VG on the "missing" PV is called "system", but:

# vgdisplay /dev/system
No device found for PV JszGc1-S16s-bsFQ-weAl-aj8W-d6y4-BeVqcN.
--- Volume group ---
VG Name system
System ID
Format lvm2
Metadata Areas 0
Metadata Sequence No 28
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 8
Open LV 8
Max PV 0
Cur PV 1
Act PV 0
VG Size 232.88 GiB
PE Size 4.00 MiB
Total PE 59616
Alloc PE / Size 33792 / 132.00 GiB
Free PE / Size 25824 / 100.88 GiB
VG UUID fQoJ8p-BI3D-BcEH-BiQX-JJqf-p6TT-6KIoRt

Actually all LVs on the "system" VG are present and accounted for. The
LVs contain the root file system, and some virtual machine drives, all
of which are running *fine* (and the system in fact boots up from the
"missing" PV :). But I can't resize:

# lvextend -L +16G /dev/system/vm-monitor
No device found for PV JszGc1-S16s-bsFQ-weAl-aj8W-d6y4-BeVqcN.
Cannot change VG system while PVs are missing.
Consider vgreduce --removemissing.

The "system" VG is situated on a md raid1. I think it all happened after
I replaced all the drives of that md with mdadm --replace with bigger
disks. But I can't be sure, because the system in fact works and except
the inability to resize there are no problems.

I have googled around a bit, but to no avail. Can someone help me
diagnose and repair this?

Version info:

# pvs --version
LVM version: 2.02.105(2) (2014-01-20)
Library version: 1.02.84 (2014-01-20)
Driver version: 4.27.0

# cat /proc/version
Linux version 3.14.0-4-ARCH (***@var-lib-archbuild-extra-x86_64-thomas) (gcc version 4.8.2 20140206 (prerelease) (GCC) ) #1 SMP PREEMPT Wed Apr 9 21:11:25 CEST 2014


Thanks,
Michal Svoboda

[Jack Waterworth]

Hi Michael,

Check your pvs output to see if any of your pvs have the 'm'
attribute. This means the device has been marked as missing in the metadata.

You can remove this attribute by running the vgextend command.

# vgextend --restoremissing /dev/device

I usually see this happen on virtual machines. Is this setup running on
a virt?

Jack Waterworth
Senior Technical Support Engineer
Red Hat Global Support Services North America
919.754.4625

Post by Michal Svoboda
Hi all,
# pvs
No device found for PV JszGc1-S16s-bsFQ-weAl-aj8W-d6y4-BeVqcN.
PV VG Fmt Attr PSize PFree
/dev/sdc data2 lvm2 a-- 298.09g 0
/dev/sdd1 data lvm2 a-- 298.09g 0
# pvscan
No device found for PV JszGc1-S16s-bsFQ-weAl-aj8W-d6y4-BeVqcN.
No device found for PV JszGc1-S16s-bsFQ-weAl-aj8W-d6y4-BeVqcN.
PV /dev/sdc VG data2 lvm2 [298.09 GiB / 0 free]
PV /dev/sdd1 VG data lvm2 [298.09 GiB / 0 free]
Total: 2 [596.18 GiB] / in use: 2 [596.18 GiB] / in no VG: 0 [0 ]
# pvscan --cache
Found duplicate PV JszGc1S16sbsFQweAlaj8Wd6y4BeVqcN: using /dev/sde2 not /dev/md0
Found duplicate PV JszGc1S16sbsFQweAlaj8Wd6y4BeVqcN: using /dev/sdf2 not /dev/sde2
# vgdisplay /dev/system
No device found for PV JszGc1-S16s-bsFQ-weAl-aj8W-d6y4-BeVqcN.
--- Volume group ---
VG Name system
System ID
Format lvm2
Metadata Areas 0
Metadata Sequence No 28
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 8
Open LV 8
Max PV 0
Cur PV 1
Act PV 0
VG Size 232.88 GiB
PE Size 4.00 MiB
Total PE 59616
Alloc PE / Size 33792 / 132.00 GiB
Free PE / Size 25824 / 100.88 GiB
VG UUID fQoJ8p-BI3D-BcEH-BiQX-JJqf-p6TT-6KIoRt
Actually all LVs on the "system" VG are present and accounted for. The
LVs contain the root file system, and some virtual machine drives, all
of which are running *fine* (and the system in fact boots up from the
# lvextend -L +16G /dev/system/vm-monitor
No device found for PV JszGc1-S16s-bsFQ-weAl-aj8W-d6y4-BeVqcN.
Cannot change VG system while PVs are missing.
Consider vgreduce --removemissing.
The "system" VG is situated on a md raid1. I think it all happened after
I replaced all the drives of that md with mdadm --replace with bigger
disks. But I can't be sure, because the system in fact works and except
the inability to resize there are no problems.
I have googled around a bit, but to no avail. Can someone help me
diagnose and repair this?
# pvs --version
LVM version: 2.02.105(2) (2014-01-20)
Library version: 1.02.84 (2014-01-20)
Driver version: 4.27.0
# cat /proc/version
Thanks,
Michal Svoboda
_______________________________________________
linux-lvm mailing list
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

[Marian Csontos]

Post by Michal Svoboda
Hi all,
# pvs
No device found for PV JszGc1-S16s-bsFQ-weAl-aj8W-d6y4-BeVqcN.
PV VG Fmt Attr PSize PFree
/dev/sdc data2 lvm2 a-- 298.09g 0
/dev/sdd1 data lvm2 a-- 298.09g 0
# pvscan
No device found for PV JszGc1-S16s-bsFQ-weAl-aj8W-d6y4-BeVqcN.
No device found for PV JszGc1-S16s-bsFQ-weAl-aj8W-d6y4-BeVqcN.
PV /dev/sdc VG data2 lvm2 [298.09 GiB / 0 free]
PV /dev/sdd1 VG data lvm2 [298.09 GiB / 0 free]
Total: 2 [596.18 GiB] / in use: 2 [596.18 GiB] / in no VG: 0 [0 ]
# pvscan --cache
Found duplicate PV JszGc1S16sbsFQweAlaj8Wd6y4BeVqcN: using /dev/sde2 not /dev/md0
Found duplicate PV JszGc1S16sbsFQweAlaj8Wd6y4BeVqcN: using /dev/sdf2 not /dev/sde2

Peter, is this an error in LVM filtering or is manual filter
modification required?

Michal(-e), I suggest modifying global_filter to accept only the devices
you want in the VG.

-- Martian

Post by Michal Svoboda
# vgdisplay /dev/system
No device found for PV JszGc1-S16s-bsFQ-weAl-aj8W-d6y4-BeVqcN.
--- Volume group ---
VG Name system
System ID
Format lvm2
Metadata Areas 0
Metadata Sequence No 28
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 8
Open LV 8
Max PV 0
Cur PV 1
Act PV 0
VG Size 232.88 GiB
PE Size 4.00 MiB
Total PE 59616
Alloc PE / Size 33792 / 132.00 GiB
Free PE / Size 25824 / 100.88 GiB
VG UUID fQoJ8p-BI3D-BcEH-BiQX-JJqf-p6TT-6KIoRt
Actually all LVs on the "system" VG are present and accounted for. The
LVs contain the root file system, and some virtual machine drives, all
of which are running *fine* (and the system in fact boots up from the
# lvextend -L +16G /dev/system/vm-monitor
No device found for PV JszGc1-S16s-bsFQ-weAl-aj8W-d6y4-BeVqcN.
Cannot change VG system while PVs are missing.
Consider vgreduce --removemissing.
The "system" VG is situated on a md raid1. I think it all happened after
I replaced all the drives of that md with mdadm --replace with bigger
disks. But I can't be sure, because the system in fact works and except
the inability to resize there are no problems.
I have googled around a bit, but to no avail. Can someone help me
diagnose and repair this?
# pvs --version
LVM version: 2.02.105(2) (2014-01-20)
Library version: 1.02.84 (2014-01-20)
Driver version: 4.27.0
# cat /proc/version
Thanks,
Michal Svoboda
_______________________________________________
linux-lvm mailing list
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

[Michal Svoboda]

Post by Marian Csontos
Michal(-e), I suggest modifying global_filter to accept only the
devices you want in the VG.

Hi,

I've tried playing with filter and global_filter in lvm.conf, but it did
not help - as if the conf file was ignored. I tried blacklisting and
whitelisting. Does something need to be restarted/rebooted for these
settings to take effect?

I then did some more experiments and eventually solved the issue.

First I tried pvscan --cache /dev/md0, at which point md0 was "accepted"
as existing PV, and the "system" VG would show up in vgdisplay. I was
even able to resize one of the LVs. However, after running pvscan --cache
with no extra agruments, I would receive the duplicate PV errors as in
previous e-mail and the "system" VG would disappear from vgdisplay. :-o

Finally I solved the issue by changing initcpio hook from "mdadm_udev"
to "mdadm". This is arch-linux specific thing that changes stuff that is
exported to the initial ramdisk used for boot. User-wise, that change
would stop using udev auto-magic for assembling md arrays, but use
/etc/mdadm.conf directly instead. I don't know the implementation details.

After that I receive *no more errors* about either missing or duplicate
PV and all VGs/LVs are displayed by the respective display commands. So
for me issue is solved but if anyone wants to get to the bottom I am
open to do some more experiments.


Michal Svoboda

[Marian Csontos]

Post by Michal Svoboda

Post by Marian Csontos
Michal(-e), I suggest modifying global_filter to accept only the
devices you want in the VG.

Hi,
I've tried playing with filter and global_filter in lvm.conf, but it did
not help - as if the conf file was ignored. I tried blacklisting and
whitelisting. Does something need to be restarted/rebooted for these
settings to take effect?

Oh yes, you would need to restart lvmetad (on Fedora/RHEL/CentOS
systemctl restart lvm2-lvmetad.service, not sure it is the same on Arch.)

You may want to scrub the RAID as there may be data not written to both
legs.

IMO it would be the best to take over udev rules and systemd units from
Fedora as Peter spends lot of time to keep them up to date and working.

I have seen some threads on Arch forum yesterday related to boot
problems with LVM on mdadm after upgrading from lvm2-2.02.105 to .106 -
I think people there will appreciate if you add a comment...

-- Martian

Post by Michal Svoboda
I then did some more experiments and eventually solved the issue.
First I tried pvscan --cache /dev/md0, at which point md0 was "accepted"
as existing PV, and the "system" VG would show up in vgdisplay. I was
even able to resize one of the LVs. However, after running pvscan --cache
with no extra agruments, I would receive the duplicate PV errors as in
previous e-mail and the "system" VG would disappear from vgdisplay. :-o
Finally I solved the issue by changing initcpio hook from "mdadm_udev"
to "mdadm". This is arch-linux specific thing that changes stuff that is
exported to the initial ramdisk used for boot. User-wise, that change
would stop using udev auto-magic for assembling md arrays, but use
/etc/mdadm.conf directly instead. I don't know the implementation details.
After that I receive *no more errors* about either missing or duplicate
PV and all VGs/LVs are displayed by the respective display commands. So
for me issue is solved but if anyone wants to get to the bottom I am
open to do some more experiments.
Michal Svoboda
_______________________________________________
linux-lvm mailing list
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

[Michal Svoboda]

Post by Marian Csontos
Oh yes, you would need to restart lvmetad (on Fedora/RHEL/CentOS
systemctl restart lvm2-lvmetad.service, not sure it is the same on Arch.)

Thanks - yeah it's the same.

Post by Marian Csontos
You may want to scrub the RAID as there may be data not written to
both legs.

Thanks for this suggestion too. I did that, found some small amount of
mismatch blocks. Don't know if they are linked to this problem but
better safe than sorry.

Michal Svoboda