Discussion:
[linux-lvm] vgcreate failed when using iscsi + dm-multipath + lvm
Neutron Sharc
2017-03-14 01:12:06 UTC
Permalink
Hi all,

I'm testing a storage prototype that combines iscsi, multipath, and
lvm. The model allows iscsi client to attach/detach multiple iscsi
LUNs exposed by storage server. Here are the steps for client to
attach to server storage:

1. iscsi initiator creates multiple LUNs (local at /dev/sde, /dev/sdf,
etc), for example:
$ iscsiadm -m node -p a.b.c.d:3260 -T xxxx -l

2.dm-multipath to create a multipath device for each of the iscsi LUN
(/dev/mapper/xyz, etc), for example:
$ multipath -r /dev/sde

In current test every multipath device has only one iscsi lun such as:

36000000076c10ff42e1fa3dd909bb3cf dm-47 HCD,VIRTUAL-DISK
size=140G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
`- 138:0:0:1 sdae 65:224 active ready running

3. create a volume group that combines several multipath devices together.
$ pvcreate -f -y /dev/mapper/xxx
$ vgcreate vol1 /dev/mapper/xxx /dev/mapper/yyy /dev/mapper/zzz

After repeating step 1/2/3 to create several volume groups, I hit
couple of error scenarios as listed below. These errors seem to happen
at random order:

error 1:
Physical volume '/dev/mapper/360000000557b690283509c3709c4fc0b' is
already in volume group 'vol8',
Unable to add physical volume
'/dev/mapper/360000000557b690283509c3709c4fc0b' to volume group
'vol14'.

error 2:
Found duplicate PV W9rMQ0GuoNw9rnzK8x47OVEY044A2mgT: using
/dev/mapper/360000000a1d91e849f66dd53a61d8fa9
not /dev/mapper/360000000e2ddd2a9380e9159902d9f73, Using duplicate
PV /dev/mapper/360000000a1d91e849f66dd53a61d8fa9 without holders,
replacing /dev/mapper/360000000e2ddd2a9380e9159902d9f73,

error 3:
Can't open /dev/mapper/360000000e2ddd2a9380e9159902d9f73 exclusively.
Mounted filesystem?
Unable to add physical volume
'/dev/mapper/360000000e2ddd2a9380e9159902d9f73' to volume group
'vol2'.



I'm very sure all PVs are freshly created and there is no duplicated PVs at all.

===========
/etc/multipath.conf:
defaults {
path_grouping_policy failover
polling_interval 3
path_selector "round-robin 0"
features "1 queue_if_no_path"
}

============
lvm.conf: I use config coming from lvm2 package. The only change is:
auto_activation_volume_list = []


===============
Here is the software pkg version:
Ubuntu 16.04
lvm2 2.02.133-1ubuntu10
multipath-tools: 0.5.0+git1.656f8865-5ubuntu2.4
open-iscsi: 2.02.133-1ubuntu10
David Teigland
2017-03-14 14:04:33 UTC
Permalink
Post by Neutron Sharc
Physical volume '/dev/mapper/360000000557b690283509c3709c4fc0b' is
already in volume group 'vol8',
Unable to add physical volume
'/dev/mapper/360000000557b690283509c3709c4fc0b' to volume group
'vol14'.
Found duplicate PV W9rMQ0GuoNw9rnzK8x47OVEY044A2mgT: using
/dev/mapper/360000000a1d91e849f66dd53a61d8fa9
not /dev/mapper/360000000e2ddd2a9380e9159902d9f73, Using duplicate
PV /dev/mapper/360000000a1d91e849f66dd53a61d8fa9 without holders,
replacing /dev/mapper/360000000e2ddd2a9380e9159902d9f73,
Can't open /dev/mapper/360000000e2ddd2a9380e9159902d9f73 exclusively.
Mounted filesystem?
Unable to add physical volume
'/dev/mapper/360000000e2ddd2a9380e9159902d9f73' to volume group
'vol2'.
Fairly common problem, lvm needs to see only the multipath device and not
the components. Make sure lvm.conf has multipath_component_detection
enabled. There was also a recent related fix:
lvmetad: use udev to ignore multipath components during scan
Neutron Sharc
2017-03-14 22:40:56 UTC
Permalink
Thank you David!

I grabbed lvm2 src (git://sourceware.org/git/lvm2.git) and build from
tag v2_02_168. It seems the basic "./configure" doesn't include a lot
of components such as lvmetad. What are the config options you would
recommend to use? Thanks.



-Shawn
Post by David Teigland
Post by Neutron Sharc
Physical volume '/dev/mapper/360000000557b690283509c3709c4fc0b' is
already in volume group 'vol8',
Unable to add physical volume
'/dev/mapper/360000000557b690283509c3709c4fc0b' to volume group
'vol14'.
Found duplicate PV W9rMQ0GuoNw9rnzK8x47OVEY044A2mgT: using
/dev/mapper/360000000a1d91e849f66dd53a61d8fa9
not /dev/mapper/360000000e2ddd2a9380e9159902d9f73, Using duplicate
PV /dev/mapper/360000000a1d91e849f66dd53a61d8fa9 without holders,
replacing /dev/mapper/360000000e2ddd2a9380e9159902d9f73,
Can't open /dev/mapper/360000000e2ddd2a9380e9159902d9f73 exclusively.
Mounted filesystem?
Unable to add physical volume
'/dev/mapper/360000000e2ddd2a9380e9159902d9f73' to volume group
'vol2'.
Fairly common problem, lvm needs to see only the multipath device and not
the components. Make sure lvm.conf has multipath_component_detection
lvmetad: use udev to ignore multipath components during scan
David Teigland
2017-03-15 14:29:23 UTC
Permalink
Post by Neutron Sharc
Thank you David!
I grabbed lvm2 src (git://sourceware.org/git/lvm2.git) and build from
tag v2_02_168. It seems the basic "./configure" doesn't include a lot
of components such as lvmetad. What are the config options you would
recommend to use? Thanks.
Finding configure options is a hassle. One way to go is to look at the
lvm package build logs for your distribution, and start with the configure
used there (perhaps removing some things you know you don't need). The
configure used by the latest Fedora 26 build is

from
https://kojipkgs.fedoraproject.org//packages/lvm2/2.02.168/4.fc26/data/logs/x86_64/build.log

./configure --build=x86_64-redhat-linux-gnu --host=x86_64-redhat-linux-gnu
--program-prefix= --disable-dependency-tracking --prefix=/usr
--exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc
--datadir=/usr/share --includedir=/usr/include --libdir=/usr/lib64
--libexecdir=/usr/libexec --localstatedir=/var --sharedstatedir=/var/lib
--mandir=/usr/share/man --infodir=/usr/share/info
--with-default-dm-run-dir=/run --with-default-run-dir=/run/lvm
--with-default-pid-dir=/run --with-default-locking-dir=/run/lock/lvm
--with-usrlibdir=/usr/lib64 --enable-lvm1_fallback --enable-fsadm
--with-pool=internal --enable-write_install --with-user= --with-group=
--with-device-uid=0 --with-device-gid=6 --with-device-mode=0660
--enable-pkgconfig --enable-applib --enable-cmdlib --enable-dmeventd
--enable-blkid_wiping --enable-python2-bindings --enable-python3-bindings
--with-cluster=internal --with-clvmd=corosync --enable-cmirrord
--with-udevdir=/usr/lib/udev/rules.d --enable-udev_sync
--with-thin=internal --enable-lvmetad --with-cache=internal
--enable-lvmpolld --enable-lockd-dlm --enable-lockd-sanlock
--enable-dbus-service --enable-notify-dbus
Neutron Sharc
2017-03-15 20:04:30 UTC
Permalink
Thank you David. I found build log at:
https://launchpadlibrarian.net/254074158/buildlog_ubuntu-xenial-amd64.lvm2_2.02.133-1ubuntu10_BUILDING.txt.gz

After builting lvm 2.02_168 and installed, it seems multipath-tools
has some conflict with lvm2 I may have to rebuild mulitpath-tools
from src again.

David: do you usually rebuild lvm2 and multipath-tools at the same time?
Post by David Teigland
Post by Neutron Sharc
Thank you David!
I grabbed lvm2 src (git://sourceware.org/git/lvm2.git) and build from
tag v2_02_168. It seems the basic "./configure" doesn't include a lot
of components such as lvmetad. What are the config options you would
recommend to use? Thanks.
Finding configure options is a hassle. One way to go is to look at the
lvm package build logs for your distribution, and start with the configure
used there (perhaps removing some things you know you don't need). The
configure used by the latest Fedora 26 build is
from
https://kojipkgs.fedoraproject.org//packages/lvm2/2.02.168/4.fc26/data/logs/x86_64/build.log
./configure --build=x86_64-redhat-linux-gnu --host=x86_64-redhat-linux-gnu
--program-prefix= --disable-dependency-tracking --prefix=/usr
--exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc
--datadir=/usr/share --includedir=/usr/include --libdir=/usr/lib64
--libexecdir=/usr/libexec --localstatedir=/var --sharedstatedir=/var/lib
--mandir=/usr/share/man --infodir=/usr/share/info
--with-default-dm-run-dir=/run --with-default-run-dir=/run/lvm
--with-default-pid-dir=/run --with-default-locking-dir=/run/lock/lvm
--with-usrlibdir=/usr/lib64 --enable-lvm1_fallback --enable-fsadm
--with-pool=internal --enable-write_install --with-user= --with-group=
--with-device-uid=0 --with-device-gid=6 --with-device-mode=0660
--enable-pkgconfig --enable-applib --enable-cmdlib --enable-dmeventd
--enable-blkid_wiping --enable-python2-bindings --enable-python3-bindings
--with-cluster=internal --with-clvmd=corosync --enable-cmirrord
--with-udevdir=/usr/lib/udev/rules.d --enable-udev_sync
--with-thin=internal --enable-lvmetad --with-cache=internal
--enable-lvmpolld --enable-lockd-dlm --enable-lockd-sanlock
--enable-dbus-service --enable-notify-dbus
p***@yahoo.com
2017-03-15 01:08:19 UTC
Permalink
How about posting mdadm status and pvs output? Are you doing this with a script in rapid fire? What happens ‎if you slow it down with a sleep? Why the -f flag to pvcreate?
Loading...