Neutron Sharc
2017-03-14 01:12:06 UTC
Hi all,
I'm testing a storage prototype that combines iscsi, multipath, and
lvm. The model allows iscsi client to attach/detach multiple iscsi
LUNs exposed by storage server. Here are the steps for client to
attach to server storage:
1. iscsi initiator creates multiple LUNs (local at /dev/sde, /dev/sdf,
etc), for example:
$ iscsiadm -m node -p a.b.c.d:3260 -T xxxx -l
2.dm-multipath to create a multipath device for each of the iscsi LUN
(/dev/mapper/xyz, etc), for example:
$ multipath -r /dev/sde
In current test every multipath device has only one iscsi lun such as:
36000000076c10ff42e1fa3dd909bb3cf dm-47 HCD,VIRTUAL-DISK
size=140G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
`- 138:0:0:1 sdae 65:224 active ready running
3. create a volume group that combines several multipath devices together.
$ pvcreate -f -y /dev/mapper/xxx
$ vgcreate vol1 /dev/mapper/xxx /dev/mapper/yyy /dev/mapper/zzz
After repeating step 1/2/3 to create several volume groups, I hit
couple of error scenarios as listed below. These errors seem to happen
at random order:
error 1:
Physical volume '/dev/mapper/360000000557b690283509c3709c4fc0b' is
already in volume group 'vol8',
Unable to add physical volume
'/dev/mapper/360000000557b690283509c3709c4fc0b' to volume group
'vol14'.
error 2:
Found duplicate PV W9rMQ0GuoNw9rnzK8x47OVEY044A2mgT: using
/dev/mapper/360000000a1d91e849f66dd53a61d8fa9
not /dev/mapper/360000000e2ddd2a9380e9159902d9f73, Using duplicate
PV /dev/mapper/360000000a1d91e849f66dd53a61d8fa9 without holders,
replacing /dev/mapper/360000000e2ddd2a9380e9159902d9f73,
error 3:
Can't open /dev/mapper/360000000e2ddd2a9380e9159902d9f73 exclusively.
Mounted filesystem?
Unable to add physical volume
'/dev/mapper/360000000e2ddd2a9380e9159902d9f73' to volume group
'vol2'.
I'm very sure all PVs are freshly created and there is no duplicated PVs at all.
===========
/etc/multipath.conf:
defaults {
path_grouping_policy failover
polling_interval 3
path_selector "round-robin 0"
features "1 queue_if_no_path"
}
============
lvm.conf: I use config coming from lvm2 package. The only change is:
auto_activation_volume_list = []
===============
Here is the software pkg version:
Ubuntu 16.04
lvm2 2.02.133-1ubuntu10
multipath-tools: 0.5.0+git1.656f8865-5ubuntu2.4
open-iscsi: 2.02.133-1ubuntu10
I'm testing a storage prototype that combines iscsi, multipath, and
lvm. The model allows iscsi client to attach/detach multiple iscsi
LUNs exposed by storage server. Here are the steps for client to
attach to server storage:
1. iscsi initiator creates multiple LUNs (local at /dev/sde, /dev/sdf,
etc), for example:
$ iscsiadm -m node -p a.b.c.d:3260 -T xxxx -l
2.dm-multipath to create a multipath device for each of the iscsi LUN
(/dev/mapper/xyz, etc), for example:
$ multipath -r /dev/sde
In current test every multipath device has only one iscsi lun such as:
36000000076c10ff42e1fa3dd909bb3cf dm-47 HCD,VIRTUAL-DISK
size=140G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
`- 138:0:0:1 sdae 65:224 active ready running
3. create a volume group that combines several multipath devices together.
$ pvcreate -f -y /dev/mapper/xxx
$ vgcreate vol1 /dev/mapper/xxx /dev/mapper/yyy /dev/mapper/zzz
After repeating step 1/2/3 to create several volume groups, I hit
couple of error scenarios as listed below. These errors seem to happen
at random order:
error 1:
Physical volume '/dev/mapper/360000000557b690283509c3709c4fc0b' is
already in volume group 'vol8',
Unable to add physical volume
'/dev/mapper/360000000557b690283509c3709c4fc0b' to volume group
'vol14'.
error 2:
Found duplicate PV W9rMQ0GuoNw9rnzK8x47OVEY044A2mgT: using
/dev/mapper/360000000a1d91e849f66dd53a61d8fa9
not /dev/mapper/360000000e2ddd2a9380e9159902d9f73, Using duplicate
PV /dev/mapper/360000000a1d91e849f66dd53a61d8fa9 without holders,
replacing /dev/mapper/360000000e2ddd2a9380e9159902d9f73,
error 3:
Can't open /dev/mapper/360000000e2ddd2a9380e9159902d9f73 exclusively.
Mounted filesystem?
Unable to add physical volume
'/dev/mapper/360000000e2ddd2a9380e9159902d9f73' to volume group
'vol2'.
I'm very sure all PVs are freshly created and there is no duplicated PVs at all.
===========
/etc/multipath.conf:
defaults {
path_grouping_policy failover
polling_interval 3
path_selector "round-robin 0"
features "1 queue_if_no_path"
}
============
lvm.conf: I use config coming from lvm2 package. The only change is:
auto_activation_volume_list = []
===============
Here is the software pkg version:
Ubuntu 16.04
lvm2 2.02.133-1ubuntu10
multipath-tools: 0.5.0+git1.656f8865-5ubuntu2.4
open-iscsi: 2.02.133-1ubuntu10