[linux-lvm] what creates the symlinks in /dev/<volgroup> ?

Command vgmknodes

On Mon, Jun 20, 2016 at 10:52 PM, Chris Friesen

Post by Chris Friesen
Hi,
Can someone tell me what creates the /dev/<volgroup>/<volume symlinks? Is
this LVM or udev (and if udev, do you know which rule)?
I'm seeing some interesting behaviour where if I create a thin pool it
creates a symlink for the pool, but once I create a thin volume within the
pool then the pool symlink disappears.
Thanks,
Chris
_______________________________________________
linux-lvm mailing list
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Chris Friesen

2016-06-20 23:13:21 UTC

It appears that "vgscan --mknodes" also creates this symlink, and creates it
pointing to /dev/mapper/<name> instead of to /dev/dm-X like udev does.

controller-1:/dev# vgscan -v --mknodes
Wiping cache of LVM-capable devices
Wiping internal VG cache
Reading all physical volumes. This may take a while...
Using volume group(s) on command line.
Found volume group "nova-local" using metadata type lvm2
Found volume group "cgts-vg" using metadata type lvm2
Found volume group "cinder-volumes" using metadata type lvm2
Using logical volume(s) on command line.
The link /dev/cinder-volumes/cinder-volumes-pool should have been created by
udev but it was not found. Falling back to direct link creation.

controller-1:/dev# ls -l /dev/cinder-volumes/
total 0
lrwxrwxrwx 1 root root 7 Jun 20 22:43 anchor-lv -> ../dm-9
lrwxrwxrwx 1 root root 49 Jun 20 23:09 cinder-volumes-pool ->
/dev/mapper/cinder--volumes-cinder--volumes--pool

This is a bad thing, because running the above or "vgmknodes" and then running
"vgchange -an cinder-volumes" will leave the /dev/cinder-volumes directory with
a dangling symlink in it.

This in turn breaks /usr/lib/ocf/resource.d/heartbeat/LVM, which (perhaps
erroneously) uses the existence of a non-empty /dev/<volume_group> directory as
a test to see if the volume group is active or not.

Chris

Found it. /usr/lib/udev/rules.d/11-dm-lvm.rules is what makes the
/dev/<VG>/<LV> symlink for normal devices, but for a thin pool with a thin
volume in it it will exit without making a symlink because
DM_UDEV_DISABLE_SUBSYSTEM_RULES_FLAG=1 is set.
Given that both vgmknodes and the udev rules come from the lvm2 package, it'd be
nice if they agreed on whether or not a symlink should be created and if so
where it should link to.
Chris

controller-1:/dev# vgmknodes -v
Using logical volume(s) on command line.
Found same device /dev/sda6 with same pvid Fw6C1IZABbnspIT23RbOGf2DXuB4zMhS
Found same device /dev/sda5 with same pvid 1s1dPDodojAS0kqToIRy4hiXFjCp2t2o
Found same device /dev/sda6 with same pvid Fw6C1IZABbnspIT23RbOGf2DXuB4zMhS
Found same device /dev/drbd4 with same pvid 6yk0HVTSbU9Amo6mwPKt3nZPnBIXBlhR
Found same device /dev/sda5 with same pvid 1s1dPDodojAS0kqToIRy4hiXFjCp2t2o
The link /dev/cinder-volumes/cinder-volumes-pool should have been created by
udev but it was not found. Falling back to direct link creation.
So it thinks that udev should have already made it. Also, the symlink it
generates looks different than the symlinks already in /dev/cinder-volumes, in
that it points to /dev/mapper/<name> rather than pointing to the "real"
/dev/dm-X device like the others.
controller-1:/dev# ls -l /dev/cinder-volumes/
total 0
lrwxrwxrwx 1 root root 7 Jun 20 17:57 anchor-lv -> ../dm-9
lrwxrwxrwx 1 root root 49 Jun 20 21:48 cinder-volumes-pool ->
/dev/mapper/cinder--volumes-cinder--volumes--pool
lrwxrwxrwx 1 root root 8 Jun 20 17:57
volume-0bc1df18-45d0-4477-9c57-36876d3f82d4 -> ../dm-19
lrwxrwxrwx 1 root root 8 Jun 20 17:57
volume-2fff261f-8860-4b86-8b2e-49bddcf47e9b -> ../dm-17
lrwxrwxrwx 1 root root 8 Jun 20 17:57
volume-48744604-6b02-4f11-ba02-3f692d109953 -> ../dm-15
lrwxrwxrwx 1 root root 8 Jun 20 17:57
volume-8dabc793-e46d-4849-a2fb-dd3d4bc2c988 -> ../dm-20
lrwxrwxrwx 1 root root 8 Jun 20 17:57
volume-be3c9ddb-a6eb-43ca-ac37-5554756a4c13 -> ../dm-16
lrwxrwxrwx 1 root root 8 Jun 20 17:57
volume-eef89318-fa8e-4ca2-a8a7-fe8e143d8792 -> ../dm-14
lrwxrwxrwx 1 root root 8 Jun 20 17:57
volume-f11a7d88-1ad3-4e89-a594-e824019725bb -> ../dm-18
Given the above, I think that something other than vgmknodes must be involved.
Chris

Post by Ilya Boka
Command vgmknodes
On Mon, Jun 20, 2016 at 10:52 PM, Chris Friesen

_______________________________________________
linux-lvm mailing list
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Zdenek Kabelac

2016-06-21 09:07:19 UTC

Post by Chris Friesen
It appears that "vgscan --mknodes" also creates this symlink, and creates it
pointing to /dev/mapper/<name> instead of to /dev/dm-X like udev does.
controller-1:/dev# vgscan -v --mknodes
Wiping cache of LVM-capable devices
Wiping internal VG cache
Reading all physical volumes. This may take a while...
Using volume group(s) on command line.
Found volume group "nova-local" using metadata type lvm2
Found volume group "cgts-vg" using metadata type lvm2
Found volume group "cinder-volumes" using metadata type lvm2
Using logical volume(s) on command line.
The link /dev/cinder-volumes/cinder-volumes-pool should have been created by
udev but it was not found. Falling back to direct link creation.
controller-1:/dev# ls -l /dev/cinder-volumes/
total 0
lrwxrwxrwx 1 root root 7 Jun 20 22:43 anchor-lv -> ../dm-9
lrwxrwxrwx 1 root root 49 Jun 20 23:09 cinder-volumes-pool ->
/dev/mapper/cinder--volumes-cinder--volumes--pool
This is a bad thing, because running the above or "vgmknodes" and then running
"vgchange -an cinder-volumes" will leave the /dev/cinder-volumes directory
with a dangling symlink in it.
This in turn breaks /usr/lib/ocf/resource.d/heartbeat/LVM, which (perhaps
erroneously) uses the existence of a non-empty /dev/<volume_group> directory
as a test to see if the volume group is active or not.

This whole thread is presentint one continues sets of bugs and misunderstandings.

On modern Linux system - it should be ONLY udev to create any links in /dev
dir. So lvm2 does not create ANY links unless forced to do.

So the clear answer here is 'it is udev rule that creates links'

Now the second part - you system is likely misconfigured.
It's admin responsibility to filter out devices in a way lvm2 does not get any
duplicates. If the same device (same UUID) is seen multiple times,
lvm2 has no way to know which of them is the right one to use.

Since you have not even shown the version of lvm2 in use and whather lvmetad
is in use - there is no way to give you any more hints...

Regards

Zdenek

Chris Friesen

2016-06-21 15:22:44 UTC

I'm using the stock CentOS7 version, I think.

LVM version: 2.02.130(2)-RHEL7 (2015-12-01)
Library version: 1.02.107-RHEL7 (2015-12-01)
Driver version: 4.33.0

So are you saying that nobody should run "vgscan --mknodes" on a system where
udev is managing the symlinks?

Personally, I think that running that command should use the same logic as the
udev rules to decide whether or not to create a symlink, and should create the
symlink pointing to the same place as the udev rules.

I'm not sure what you're talking about as far as duplicates, I'm not seeing any
duplicate devices. The problem I see is that "vgscan --mknodes" or "vgmknodes"
will both create an additional symlink when compared to the udev rules, and the
additional symlink is not deleted when I deactivate the volume group.

For what it's worth, in my lvm.conf I have "use_lvmetad = 0" and

global_filter = [ "a|/dev/sda|", "a|/dev/drbd4|", "a|/dev/sda6|", "r|.*|" ]

Chris

This whole thread is presentint one continues sets of bugs and misunderstandings.
On modern Linux system - it should be ONLY udev to create any links in /dev dir.
So lvm2 does not create ANY links unless forced to do.
So the clear answer here is 'it is udev rule that creates links'
Now the second part - you system is likely misconfigured.
It's admin responsibility to filter out devices in a way lvm2 does not get any
duplicates. If the same device (same UUID) is seen multiple times,
lvm2 has no way to know which of them is the right one to use.
Since you have not even shown the version of lvm2 in use and whather lvmetad is
in use - there is no way to give you any more hints...
Regards
Zdenek
_______________________________________________
linux-lvm mailing list
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Zdenek Kabelac

2016-06-22 09:23:07 UTC

Post by Chris Friesen
I'm using the stock CentOS7 version, I think.
LVM version: 2.02.130(2)-RHEL7 (2015-12-01)
Library version: 1.02.107-RHEL7 (2015-12-01)
Driver version: 4.33.0
So are you saying that nobody should run "vgscan --mknodes" on a system where
udev is managing the symlinks?

Yes - on such system this command should be used only in case of 'emergency',
udev doesn't work properly and you need links.

The links however will not be known to udev and likely whole system is
going to be crashing soon or is misconfigured in major way.

Post by Chris Friesen
I'm not sure what you're talking about as far as duplicates, I'm not seeing
any duplicate devices. The problem I see is that "vgscan --mknodes" or
"vgmknodes" will both create an additional symlink when compared to the udev
rules, and the additional symlink is not deleted when I deactivate the volume
group.
For what it's worth, in my lvm.conf I have "use_lvmetad = 0" and
global_filter = [ "a|/dev/sda|", "a|/dev/drbd4|", "a|/dev/sda6|", "r|.*|" ]

And now we are getting to the point on your problem:

"a|/dev/sda|" will also match "a|/dev/sda6|"
(and /dev/sda5...)

If you would like to get only 'exact' '/dev/sda' you would need to use

^/dev/sda$

otherwise '/dev/sda' may appear anywhere as substring of your device path.

Regards

Zdenek

Chris Friesen

2016-06-22 14:52:04 UTC

Yes - on such system this command should be used only in case of 'emergency',
udev doesn't work properly and you need links.
The links however will not be known to udev and likely whole system is
going to be crashing soon or is misconfigured in major way.

Okay, I'll see if I can get the call to vgscan removed. But even so wouldn't it
make sense to have vgscan use the same logic as udev in terms of what symlinks
to make and where to make them to?

"a|/dev/sda|" will also match "a|/dev/sda6|"
(and /dev/sda5...)
If you would like to get only 'exact' '/dev/sda' you would need to use
^/dev/sda$
otherwise '/dev/sda' may appear anywhere as substring of your device path.

Yes, the "/dev/sda" is intentional since we have multiple /dev/sda* devices that
form part of various separate volume groups. Really it's the /dev/sda6 that is
extra here since it's already matched.

The problematic issue I'm seeing with udev/vgscan is actually with the volume
group on /dev/drbd4.

I really don't think I'm having any problems with duplicates...just with vgscan
creating extra symlinks compared to udev, that udev doesn't clean up.

Chris

Zdenek Kabelac

2016-06-23 08:34:05 UTC

Yes - on such system this command should be used only in case of 'emergency',
udev doesn't work properly and you need links.
The links however will not be known to udev and likely whole system is
going to be crashing soon or is misconfigured in major way.

Okay, I'll see if I can get the call to vgscan removed. But even so wouldn't
it make sense to have vgscan use the same logic as udev in terms of what
symlinks to make and where to make them to?

It *IS* using same logic.

If the link is not there - the bug is in your udev rules.

When udev is properly configured, vgscan should not show missing link.

It's worth to note - 'dm' subsystem is in general 'ignored' by udev and 'dm'
user has to provide proper rules. If your drbd device is missing,
then something wrong is with rules...

Likely 'drbd' devices are not properly scanned by udev (and I assume they
would not even work on system with use_lvmetad=1).

Yes, the "/dev/sda" is intentional since we have multiple /dev/sda* devices
that form part of various separate volume groups. Really it's the /dev/sda6
that is extra here since it's already matched.
The problematic issue I'm seeing with udev/vgscan is actually with the volume
group on /dev/drbd4.
I really don't think I'm having any problems with duplicates...just with
vgscan creating extra symlinks compared to udev, that udev doesn't clean up.

Ahh - ok then - I thought you wanted to see only sda6 on your machine.

Regards

Zdenek

Chris Friesen

2016-06-23 16:35:59 UTC

Yes - on such system this command should be used only in case of 'emergency',
udev doesn't work properly and you need links.
The links however will not be known to udev and likely whole system is
going to be crashing soon or is misconfigured in major way.

Okay, I'll see if I can get the call to vgscan removed. But even so wouldn't
it make sense to have vgscan use the same logic as udev in terms of what
symlinks to make and where to make them to?

It *IS* using same logic.
If the link is not there - the bug is in your udev rules.
When udev is properly configured, vgscan should not show missing link.

It doesn't seem to work this way in practice on a stock CentOS system. Here's
the sequence:

1) create a volume group:
"vgcreate chris-volumes /dev/loop2"
At this point there is no /dev/chris-volumes directory.

2) Create a thin pool in the volume group:
"lvcreate -L 1.8GB -T chris-volumes/chris-volumes-pool"

Now udev creates a /dev/chris-volumes directory with a link for the thin pool:
[***@centos7 centos]# ls -l /dev/chris-volumes
total 0
lrwxrwxrwx. 1 root root 7 Jun 23 12:22 chris-volumes-pool -> ../dm-9

3) Create a thin volume in the thin pool:
"lvcreate -V1G -T chris-volumes/chris-volumes-pool -n thinvolume"

Now the link for the thin pool itself has disappeared:
[***@centos7 centos]# ls -l /dev/chris-volumes
total 0
lrwxrwxrwx. 1 root root 8 Jun 23 12:23 thinvolume -> ../dm-11

(At this point /dev/mapper/chris--volumes-chris--volumes--pool-tpool points to
dm-9 and /dev/mapper/chris--volumes-chris--volumes--pool points to dm-10.)

4) If I run "vgscan --mknodes", it re-creates the thin pool link, but pointing
to the /dev/mapper name instead of directly to the /dev/dm-*. Also, it's
indirectly pointing to /dev/dm-10 where before it was pointing to /dev/dm-9:

[***@centos7 centos]# vgscan --mknodes
Configuration setting "snapshot_autoextend_percent" invalid. It's not part of
any section.
Configuration setting "snapshot_autoextend_threshold" invalid. It's not part
of any section.
Reading all physical volumes. This may take a while...
Found volume group "chris-volumes" using metadata type lvm2
Found volume group "centos" using metadata type lvm2
Found volume group "cinder-volumes" using metadata type lvm2
The link /dev/chris-volumes/chris-volumes-pool should have been created by
udev but it was not found. Falling back to direct link creation.
[***@centos7 centos]# ls -l /dev/chris-volumes
total 0
lrwxrwxrwx. 1 root root 47 Jun 23 12:25 chris-volumes-pool ->
/dev/mapper/chris--volumes-chris--volumes--pool
lrwxrwxrwx. 1 root root 8 Jun 23 12:23 thinvolume -> ../dm-11

5) If I run "vgchange -an chris-volumes", I'm left with a /dev/chris-volumes
directory containing a broken symlink (broken because
/dev/mapper/chris--volumes-chris--volumes--pool doesn't exist anymore):

[***@centos7 centos]# ls -l /dev/chris-volumes
total 0
lrwxrwxrwx. 1 root root 47 Jun 23 12:25 chris-volumes-pool ->
/dev/mapper/chris--volumes-chris--volumes--pool

It looks like /usr/lib/udev/rules.d/11-dm-lvm.rules is what makes the
/dev/<VG>/<LV> symlink for normal devices, but for a thin pool with a thin
volume in it it will exit without making a symlink because
DM_UDEV_DISABLE_SUBSYSTEM_RULES_FLAG=1 is set.

[***@centos7 centos]# udevadm info /dev/chris-volumes/chris-volumes-pool
P: /devices/virtual/block/dm-10
N: dm-10
E: DM_UDEV_DISABLE_SUBSYSTEM_RULES_FLAG=1

Chris

Zdenek Kabelac

2016-06-23 17:21:44 UTC

Yes - on such system this command should be used only in case of 'emergency',
udev doesn't work properly and you need links.
The links however will not be known to udev and likely whole system is
going to be crashing soon or is misconfigured in major way.

Okay, I'll see if I can get the call to vgscan removed. But even so wouldn't
it make sense to have vgscan use the same logic as udev in terms of what
symlinks to make and where to make them to?

It *IS* using same logic.
If the link is not there - the bug is in your udev rules.
When udev is properly configured, vgscan should not show missing link.

It doesn't seem to work this way in practice on a stock CentOS system. Here's
"vgcreate chris-volumes /dev/loop2"
At this point there is no /dev/chris-volumes directory.

ONLY active volume do have links

Never ever there is supposed to be directory entry for VG without any active LV.

There is not a term 'active VG' as such - it always is in connection with
active LV - thus directory without any active LV inside is pure bug - if you
see it - report it as regular BZ..

Post by Chris Friesen
"lvcreate -L 1.8GB -T chris-volumes/chris-volumes-pool"
total 0
lrwxrwxrwx. 1 root root 7 Jun 23 12:22 chris-volumes-pool -> ../dm-9
"lvcreate -V1G -T chris-volumes/chris-volumes-pool -n thinvolume"
total 0
lrwxrwxrwx. 1 root root 8 Jun 23 12:23 thinvolume -> ../dm-11
(At this point /dev/mapper/chris--volumes-chris--volumes--pool-tpool points to
dm-9 and /dev/mapper/chris--volumes-chris--volumes--pool points to dm-10.)

correct

Post by Chris Friesen
4) If I run "vgscan --mknodes", it re-creates the thin pool link, but pointing
to the /dev/mapper name instead of directly to the /dev/dm-*. Also, it's

Only 'final' resolving device does matter.

if link is /dev/vg/lv -> /dev/mapper/vg-lv -> /dev/dm-XXX
or /dev/vg/lv -> /dev/dm-XXX - it does not matter.

There are some 'tricks' related to thin-pool maintainance.

Unused thin-pool is 'public' LV - has /dev/vg/lv link.
Used thin-pool by lvm2 is 'private' LV - doesn't have /dev/vg/lv link.

All device should always have entry in /dev/mapper/ - either links
to real devices or direct nodes (on older systems)

lvm2 users are always supposed to use ONLY /dev/vg/lv devices for access.

Post by Chris Friesen
Configuration setting "snapshot_autoextend_percent" invalid. It's not part
of any section.
Configuration setting "snapshot_autoextend_threshold" invalid. It's not part
of any section.

fix your lvm.conf (uncomment sections)

Post by Chris Friesen
Reading all physical volumes. This may take a while...
Found volume group "chris-volumes" using metadata type lvm2
Found volume group "centos" using metadata type lvm2
Found volume group "cinder-volumes" using metadata type lvm2
The link /dev/chris-volumes/chris-volumes-pool should have been created by

Ok - there seems to be internal bug in lvm2 - which incorrectly hints
link creation for this case.

There should not have been /dev/vg/pool link - this is correctly marked
for udev - but incorrectly for udev validation.

However the bug is actually not so much important - it just links
to 'wrapper' device - and eventually we will resolve the problem even without
this extra device in table.

Regards

Zdenek

Chris Friesen

2016-06-23 18:02:47 UTC

fix your lvm.conf (uncomment sections)

Ok - there seems to be internal bug in lvm2 - which incorrectly hints
link creation for this case.
There should not have been /dev/vg/pool link - this is correctly marked
for udev - but incorrectly for udev validation.
However the bug is actually not so much important - it just links
to 'wrapper' device - and eventually we will resolve the problem even without
this extra device in table.

The problem that it causes for me is that when I run "vgchange -an
chris-volumes" it leaves the /dev/chris-volumes with a broken symlink in it
because udev doesn't remove the symlink added by vgscan.

This causes the LVM OCF script in the "resource-agents" package to break,
because it is using the existance of the /dev/vg directory as a proxy for
whether the volume group is active (or really as you said earlier, whether there
are active volumes within the volume group).

I reported this as a bug to the "resource-agents" package developers, and they
said that they can't actually call lvm commands in their "status" routines
because there have been cases where clustered LVM hung when querying status,
causing the OCF script to hang and monitoring to fail.

Ultimately I'll see if I can work around it by not calling "vgscan --mknodes".
Originally it was added in to fix some problems, but that was a while back so
things may behave properly now.

Thanks for your help,
Chris

Zdenek Kabelac

2016-06-24 11:00:03 UTC

fix your lvm.conf (uncomment sections)

Ok - there seems to be internal bug in lvm2 - which incorrectly hints
link creation for this case.
There should not have been /dev/vg/pool link - this is correctly marked
for udev - but incorrectly for udev validation.
However the bug is actually not so much important - it just links
to 'wrapper' device - and eventually we will resolve the problem even without
this extra device in table.

The problem that it causes for me is that when I run "vgchange -an
chris-volumes" it leaves the /dev/chris-volumes with a broken symlink in it
because udev doesn't remove the symlink added by vgscan.

Yep - as said - in normal circumstance use should NOT run 'vgmknodes'
as the created links will not be known/visible to udev - so this behavior is
correct.

Post by Chris Friesen
This causes the LVM OCF script in the "resource-agents" package to break,
because it is using the existance of the /dev/vg directory as a proxy for
whether the volume group is active (or really as you said earlier, whether
there are active volumes within the volume group).
I reported this as a bug to the "resource-agents" package developers, and they
said that they can't actually call lvm commands in their "status" routines
because there have been cases where clustered LVM hung when querying status,
causing the OCF script to hang and monitoring to fail.
Ultimately I'll see if I can work around it by not calling "vgscan --mknodes".

yes please, start with this one...

vgmknodes are really supposed to be used on some unique urgent problem - not
executed in script every hour...

But yes - lvm2 will need to fix link creation of pool-in-use...

Post by Chris Friesen
Originally it was added in to fix some problems, but that was a while back so
things may behave properly now.

Regards

Zdenek

Zdenek Kabelac

2016-07-01 06:51:33 UTC