[linux-lvm] activation mode=degraded doesn't work with thin pool

[Alexander 'Leo' Bergolth]

Hi!

I just tested lv activation with a degraded raid1 thin pool.
Unfortunately it looks like activation mode=degraded only works for
plain raid1 lvs. If you add a thin pool, lvm won't activate it in
degraded mode. (Unless you specify --activationmode partial, which is
IMHO rather dangerous.)

Is this the intended behavior?

-------------------- 8< --------------------
# lvcreate --type raid1 -m 1 -n thinmeta -L100m vg1 /dev/sda2 /dev/sdb2
Logical volume "thinmeta" created.
# lvcreate --type raid1 -m 1 -n Thin -L2g vg1 /dev/sda2 /dev/sdb2
Logical volume "Thin" created.
# lvconvert -y --type thin-pool --poolmetadata vg1/thinmeta vg1/Thin
Thin pool volume with chunk size 64.00 KiB can address at most 15.81 TiB of data.
WARNING: Converting logical volume vg1/Thin and vg1/thinmeta to thin pool's data and metadata volumes with metadata wiping.
THIS WILL DESTROY CONTENT OF LOGICAL VOLUME (filesystem etc.)
Converted vg1/Thin_tdata to thin pool.

# vgchange -a n vg1
0 logical volume(s) in volume group "vg1" now active

### filter one leg of the raid: global_filter = [ "r|^/dev/sdb2|" ]
# systemctl restart lvm2-lvmetad.service
# pvscan --cache

# vgchange -a y
WARNING: Device for PV WzDZS1-Hj9B-RZCa-uhOc-pY9L-1rLN-GB5Zs1 not found or rejected by a filter.
Refusing activation of partial LV vg1/Thin. Use '--activationmode partial' to override.
0 logical volume(s) in volume group "vg1" now active
-------------------- 8< --------------------

(--activationmode partial works)

I am using lvm2-2.02.171-8.el7.x86_64.

Cheers,
--leo

[Alexander 'Leo' Bergolth]

Post by Alexander 'Leo' Bergolth
I just tested lv activation with a degraded raid1 thin pool.
Unfortunately it looks like activation mode=degraded only works for
plain raid1 lvs. If you add a thin pool, lvm won't activate it in
degraded mode. (Unless you specify --activationmode partial, which is
IMHO rather dangerous.)

Unfortunately I cannot even replace a faulty PV if a thin pool is present.

lvm thinks that the thin pool is a partial (not just degraded) LV, so
removing the missing PV would also remove the thin pool! (see below)

Is there any way to work around this problem?

Cheers,
--leo

P.S.: I filed a bugreport for this:
https://bugzilla.redhat.com/show_bug.cgi?id=1504044

-------------------- 8< --------------------
# pvcreate /dev/vdb
Physical volume "/dev/vdb" successfully created.
# pvcreate /dev/vdc
Physical volume "/dev/vdc" successfully created.
# vgcreate vg_test /dev/vdb /dev/vdc
Volume group "vg_test" successfully created
# lvcreate --type raid1 -m 1 -n thinmeta -L100m vg_test /dev/vdb /dev/vdc
Logical volume "thinmeta" created.
# lvcreate --type raid1 -m 1 -n Thin -L2g vg_test /dev/vdb /dev/vdc
Logical volume "Thin" created.
# lvconvert -y --type thin-pool --poolmetadata vg_test/thinmeta vg_test/Thin
Thin pool volume with chunk size 64.00 KiB can address at most 15.81
TiB of data.
WARNING: Converting logical volume vg_test/Thin and vg_test/thinmeta
to thin pool's data and metadata volumes with metadata wiping.
THIS WILL DESTROY CONTENT OF LOGICAL VOLUME (filesystem etc.)
Converted vg_test/Thin_tdata to thin pool.

# vgchange -a n vg_test
0 logical volume(s) in volume group "vg_test" now active

### add global_filter = [ "r|^/dev/vdc$|" ] to lvm.conf
# systemctl restart lvm2-lvmetad.service
# pvscan --cache

# pvs
WARNING: Device for PV na0HS1-ZcQt-bnt0-tFfB-Zl20-irIg-FffGxH not
found or rejected by a filter.
PV VG Fmt Attr PSize PFree
/dev/vda2 vg_sys lvm2 a-- <12.00g <2.00g
/dev/vdb vg_test lvm2 a-- <12.00g 9.79g
[unknown] vg_test lvm2 a-m <12.00g 9.89g

# vgchange -a y vg_test
WARNING: Device for PV na0HS1-ZcQt-bnt0-tFfB-Zl20-irIg-FffGxH not
found or rejected by a filter.
Refusing activation of partial LV vg_test/Thin. Use '--activationmode
partial' to override.
0 logical volume(s) in volume group "vg_test" now active

# vgchange -a y --activationmode=partial vg_test
PARTIAL MODE. Incomplete logical volumes will be processed.
WARNING: Device for PV na0HS1-ZcQt-bnt0-tFfB-Zl20-irIg-FffGxH not
found or rejected by a filter.
1 logical volume(s) in volume group "vg_test" now active

### removing the missing PV would also remove my thin pool!
# vgreduce --removemissing vg_test
WARNING: Device for PV na0HS1-ZcQt-bnt0-tFfB-Zl20-irIg-FffGxH not
found or rejected by a filter.
WARNING: Partial LV Thin needs to be repaired or removed.
WARNING: Partial LV Thin_tmeta needs to be repaired or removed.
WARNING: Partial LV Thin_tdata needs to be repaired or removed.
WARNING: Partial LV Thin_tmeta_rimage_1 needs to be repaired or removed.
WARNING: Partial LV Thin_tmeta_rmeta_1 needs to be repaired or removed.
WARNING: Partial LV Thin_tdata_rimage_1 needs to be repaired or removed.
WARNING: Partial LV Thin_tdata_rmeta_1 needs to be repaired or removed.
There are still partial LVs in VG vg_test.
To remove them unconditionally use: vgreduce --removemissing --force.
WARNING: Proceeding to remove empty missing PVs.

-------------------- 8< --------------------

Post by Alexander 'Leo' Bergolth
Is this the intended behavior?
-------------------- 8< --------------------
# lvcreate --type raid1 -m 1 -n thinmeta -L100m vg1 /dev/sda2 /dev/sdb2
Logical volume "thinmeta" created.
# lvcreate --type raid1 -m 1 -n Thin -L2g vg1 /dev/sda2 /dev/sdb2
Logical volume "Thin" created.
# lvconvert -y --type thin-pool --poolmetadata vg1/thinmeta vg1/Thin
Thin pool volume with chunk size 64.00 KiB can address at most 15.81 TiB of data.
WARNING: Converting logical volume vg1/Thin and vg1/thinmeta to thin pool's data and metadata volumes with metadata wiping.
THIS WILL DESTROY CONTENT OF LOGICAL VOLUME (filesystem etc.)
Converted vg1/Thin_tdata to thin pool.
# vgchange -a n vg1
0 logical volume(s) in volume group "vg1" now active
### filter one leg of the raid: global_filter = [ "r|^/dev/sdb2|" ]
# systemctl restart lvm2-lvmetad.service
# pvscan --cache
# vgchange -a y
WARNING: Device for PV WzDZS1-Hj9B-RZCa-uhOc-pY9L-1rLN-GB5Zs1 not found or rejected by a filter.
Refusing activation of partial LV vg1/Thin. Use '--activationmode partial' to override.
0 logical volume(s) in volume group "vg1" now active
-------------------- 8< --------------------
(--activationmode partial works)
I am using lvm2-2.02.171-8.el7.x86_64.
Cheers,
--leo

[Gionatan Danti]

Post by Alexander 'Leo' Bergolth

Post by Alexander 'Leo' Bergolth
I just tested lv activation with a degraded raid1 thin pool.
Unfortunately it looks like activation mode=degraded only works for
plain raid1 lvs. If you add a thin pool, lvm won't activate it in
degraded mode. (Unless you specify --activationmode partial, which is
IMHO rather dangerous.)

Unfortunately I cannot even replace a faulty PV if a thin pool is present.

From you command history, I can see you are trying to remove the failed
device (to re-add a good device later).
Instead, can you try with lvconvert --repair or lvconvert --replace? Do
they works?

Thanks.

[Alexander 'Leo' Bergolth]

Post by Gionatan Danti

Post by Alexander 'Leo' Bergolth

Post by Alexander 'Leo' Bergolth
I just tested lv activation with a degraded raid1 thin pool.
Unfortunately it looks like activation mode=degraded only works for
plain raid1 lvs. If you add a thin pool, lvm won't activate it in
degraded mode. (Unless you specify --activationmode partial, which is
IMHO rather dangerous.)

Unfortunately I cannot even replace a faulty PV if a thin pool is present.

From you command history, I can see you are trying to remove the failed
device (to re-add a good device later).
Instead, can you try with lvconvert --repair or lvconvert --replace? Do
they works?

lvconvert --repair doesn't work because there is also a linear LV
("boot") on this VG, which already has the error target assigned instead
of the missing PV:

-------------------- 8< --------------------
# lvconvert --repair /dev/vg_test/boot /dev/vdd
WARNING: Disabling lvmetad cache for repair command.
WARNING: Not using lvmetad because of repair.
Couldn't find device with uuid 5ORr1v-kn3T-ROJW-quEg-0Lp5-3kdX-M3jr8S.
Attempt to replace failed RAID images (requires full device resync)?
[y/n]: y
vg_test/boot has components with error targets that must be removed
first: vg_test/boot_rimage_1.
Try removing the PV list and rerun. the command.
Failed to remove the specified images from vg_test/boot.
Failed to replace faulty devices in vg_test/boot.
-------------------- 8< --------------------

(But removing the missing PV doesn't work because of the Thin volume
which is considered as partial as described previously.)


lvconvert --replace also refuses to process bcause of the missing PV:
-------------------- 8< --------------------
# lvconvert --replace 5ORr1v-kn3T-ROJW-quEg-0Lp5-3kdX-M3jr8S
vg_test/Thin_tdata /dev/vdd
Couldn't find device with uuid 5ORr1v-kn3T-ROJW-quEg-0Lp5-3kdX-M3jr8S.
Cannot change VG vg_test while PVs are missing.
Consider vgreduce --removemissing.
Cannot process volume group vg_test
-------------------- 8< --------------------

--leo

[Alexander 'Leo' Bergolth]

Post by Alexander 'Leo' Bergolth

Post by Alexander 'Leo' Bergolth
I just tested lv activation with a degraded raid1 thin pool.
Unfortunately it looks like activation mode=degraded only works for
plain raid1 lvs. If you add a thin pool, lvm won't activate it in
degraded mode. (Unless you specify --activationmode partial, which is
IMHO rather dangerous.)

Unfortunately I cannot even replace a faulty PV if a thin pool is present.

FYI: I managed to remove the missing PV using a rather dirty workaround:
I replaced the missing devices with the error target using dmsetup and edited and restored the VG config:

-------------------- 8< --------------------
### starting point:

### one PV is missing (one leg of the boot LV and of the Thin-subvolumes)
# lvs -a -olv_name,lv_attr,segtype,devices vg_test
Couldn't find device with uuid cUHx7d-Gl8O-TX2l-mi5B-QCx6-so00-cS0M4z.
LV Attr Type Devices
Thin twi-aotzp- thin-pool Thin_tdata(0)
[Thin_tdata] rwi-aor-p- raid1 Thin_tdata_rimage_0(0),Thin_tdata_rimage_1(0)
[Thin_tdata_rimage_0] iwi-aor--- linear /dev/vdb(53)
[Thin_tdata_rimage_1] Iwi-aor-p- linear [unknown](53)
[Thin_tdata_rmeta_0] ewi-aor--- linear /dev/vdb(52)
[Thin_tdata_rmeta_1] ewi-aor-p- linear [unknown](52)
[Thin_tmeta] ewi-aor-p- raid1 Thin_tmeta_rimage_0(0),Thin_tmeta_rimage_1(0)
[Thin_tmeta_rimage_0] iwi-aor--- linear /dev/vdb(27)
[Thin_tmeta_rimage_1] Iwi-aor-p- linear [unknown](27)
[Thin_tmeta_rmeta_0] ewi-aor--- linear /dev/vdb(26)
[Thin_tmeta_rmeta_1] ewi-aor-p- linear [unknown](26)
boot rwi-a-r-p- raid1 boot_rimage_0(0),boot_rimage_1(0)
[boot_rimage_0] iwi-aor--- linear /dev/vdb(1)
[boot_rimage_1] Iwi-aor-p- linear [unknown](1)
[boot_rmeta_0] ewi-aor--- linear /dev/vdb(0)
[boot_rmeta_1] ewi-aor-p- linear [unknown](0)
[lvol0_pmspare] ewi-----p- linear [unknown](2613)
thintest Vwi-aotzp- thin

# vgreduce --removemissing vg_test --force
Couldn't find device with uuid cUHx7d-Gl8O-TX2l-mi5B-QCx6-so00-cS0M4z.
WARNING: Removing partial LV vg_test/Thin.
Logical volume vg_test/thintest in use.

### the Thin subvolumes still have the missing PV attached:
# lvs -a -olv_name,lv_attr,segtype,devices vg_test
Couldn't find device with uuid cUHx7d-Gl8O-TX2l-mi5B-QCx6-so00-cS0M4z.
LV Attr Type Devices
Thin twi-aotzp- thin-pool Thin_tdata(0)
[Thin_tdata] rwi-aor-p- raid1 Thin_tdata_rimage_0(0),Thin_tdata_rimage_1(0)
[Thin_tdata_rimage_0] iwi-aor--- linear /dev/vdb(53)
[Thin_tdata_rimage_1] Iwi-aor-p- linear [unknown](53)
[Thin_tdata_rmeta_0] ewi-aor--- linear /dev/vdb(52)
[Thin_tdata_rmeta_1] ewi-aor-p- linear [unknown](52)
[Thin_tmeta] ewi-aor-p- raid1 Thin_tmeta_rimage_0(0),Thin_tmeta_rimage_1(0)
[Thin_tmeta_rimage_0] iwi-aor--- linear /dev/vdb(27)
[Thin_tmeta_rimage_1] Iwi-aor-p- linear [unknown](27)
[Thin_tmeta_rmeta_0] ewi-aor--- linear /dev/vdb(26)
[Thin_tmeta_rmeta_1] ewi-aor-p- linear [unknown](26)
boot rwi-a-r-r- raid1 boot_rimage_0(0),boot_rimage_1(0)
[boot_rimage_0] iwi-aor--- linear /dev/vdb(1)
[boot_rimage_1] vwi-aor-r- error
[boot_rmeta_0] ewi-aor--- linear /dev/vdb(0)
[boot_rmeta_1] ewi-aor-r- error
[lvol0_pmspare] ewi-----p- linear [unknown](2613)
thintest Vwi-aotzp- thin

### create and execute the dmsetup commands used to replace it:
vg="vg_test"
devs="Thin_tdata_rimage_1 Thin_tdata_rmeta_1 Thin_tmeta_rimage_1 Thin_tmeta_rmeta_1"
for d in $devs; do
d2="$vg-$d"
t1="$(dmsetup table "$d2")"
t2="${t1/ linear */ error}"
echo dmsetup reload "$d2" --table \""$t2"\"
echo dmsetup suspend --noflush "$d2"
echo dmsetup resume "$d2"
done

dmsetup reload vg_test-Thin_tdata_rimage_1 --table "0 20971520 error"
dmsetup suspend --noflush vg_test-Thin_tdata_rimage_1
dmsetup resume vg_test-Thin_tdata_rimage_1
dmsetup reload vg_test-Thin_tdata_rmeta_1 --table "0 8192 error"
dmsetup suspend --noflush vg_test-Thin_tdata_rmeta_1
dmsetup resume vg_test-Thin_tdata_rmeta_1
dmsetup reload vg_test-Thin_tmeta_rimage_1 --table "0 204800 error"
dmsetup suspend --noflush vg_test-Thin_tmeta_rimage_1
dmsetup resume vg_test-Thin_tmeta_rimage_1
dmsetup reload vg_test-Thin_tmeta_rmeta_1 --table "0 8192 error"
dmsetup suspend --noflush vg_test-Thin_tmeta_rmeta_1
dmsetup resume vg_test-Thin_tmeta_rmeta_1

\cp /etc/lvm/backup/vg_test /etc/lvm/vg_test-before-error-target.cfg
\cp /etc/lvm/backup/vg_test /etc/lvm/vg_test-error-target.cfg
### edit vg_test-error-target.cfg:
### remove the missing PV
### edit Thin_tdata_rimage_1 Thin_tdata_rmeta_1 Thin_tmeta_rimage_1 Thin_tmeta_rmeta_1
### you may have to remove the lvol0_pmspare volume if it resided on the missing PV

# diff -u /etc/lvm/vg_test-before-error-target.cfg /etc/lvm/vg_test-error-target.cfg
--- /etc/lvm/vg_test-before-error-target.cfg 2017-10-27 11:54:31.546944978 +0200
+++ /etc/lvm/vg_test-error-target.cfg 2017-10-27 12:01:15.156352344 +0200
@@ -21,17 +21,6 @@

physical_volumes {

- pv0 {
- id = "cUHx7d-Gl8O-TX2l-mi5B-QCx6-so00-cS0M4z"
- device = "[unknown]" # Hint only
-
- status = ["ALLOCATABLE"]
- flags = ["MISSING"]
- dev_size = 25165824 # 12 Gigabytes
- pe_start = 2048
- pe_count = 3071 # 11.9961 Gigabytes
- }
-
pv1 {
id = "FlCr4G-d80M-g9zW-5A2r-qvim-Ayi1-Ce1Rsa"
device = "/dev/vdb" # Hint only
@@ -295,12 +284,7 @@
start_extent = 0
extent_count = 25 # 100 Megabytes

- type = "striped"
- stripe_count = 1 # linear
-
- stripes = [
- "pv0", 27
- ]
+ type = "error"
}
}

@@ -316,12 +300,7 @@
start_extent = 0
extent_count = 1 # 4 Megabytes

- type = "striped"
- stripe_count = 1 # linear
-
- stripes = [
- "pv0", 26
- ]
+ type = "error"
}
}

@@ -379,12 +358,7 @@
start_extent = 0
extent_count = 2560 # 10 Gigabytes

- type = "striped"
- stripe_count = 1 # linear
-
- stripes = [
- "pv0", 53
- ]
+ type = "error"
}
}

@@ -400,33 +374,7 @@
start_extent = 0
extent_count = 1 # 4 Megabytes

- type = "striped"
- stripe_count = 1 # linear
-
- stripes = [
- "pv0", 52
- ]
- }
- }
-
- lvol0_pmspare {
- id = "1tNgGe-mRW5-fKXs-1eVG-t91E-40JP-YdGqZi"
- status = ["READ", "WRITE"]
- flags = []
- creation_time = 1509097937 # 2017-10-27 11:52:17 +0200
- creation_host = "bastel-c7-lvm.wu.ac.at"
- segment_count = 1
-
- segment1 {
- start_extent = 0
- extent_count = 25 # 100 Megabytes
-
- type = "striped"
- stripe_count = 1 # linear
-
- stripes = [
- "pv0", 2613
- ]
+ type = "error"
}
}
}


### write this config to all configured PVs
# vgcfgrestore --force -f /etc/lvm/vg_test-error-target.cfg vg_test
WARNING: Forced restore of Volume Group vg_test with thin volumes.
Restored volume group vg_test

### now the missing PV is completely removed from the VG:
# lvs -a -olv_name,lv_attr,segtype,devices vg_test
LV Attr Type Devices
Thin twi-aotz-- thin-pool Thin_tdata(0)
[Thin_tdata] rwi-aor-r- raid1 Thin_tdata_rimage_0(0),Thin_tdata_rimage_1(0)
[Thin_tdata_rimage_0] iwi-aor--- linear /dev/vdb(53)
[Thin_tdata_rimage_1] vwi-aor-r- error
[Thin_tdata_rmeta_0] ewi-aor--- linear /dev/vdb(52)
[Thin_tdata_rmeta_1] ewi-aor-r- error
[Thin_tmeta] ewi-aor-r- raid1 Thin_tmeta_rimage_0(0),Thin_tmeta_rimage_1(0)
[Thin_tmeta_rimage_0] iwi-aor--- linear /dev/vdb(27)
[Thin_tmeta_rimage_1] vwi-aor-r- error
[Thin_tmeta_rmeta_0] ewi-aor--- linear /dev/vdb(26)
[Thin_tmeta_rmeta_1] ewi-aor-r- error
boot rwi-a-r-r- raid1 boot_rimage_0(0),boot_rimage_1(0)
[boot_rimage_0] iwi-aor--- linear /dev/vdb(1)
[boot_rimage_1] vwi-aor-r- error
[boot_rmeta_0] ewi-aor--- linear /dev/vdb(0)
[boot_rmeta_1] ewi-aor-r- error
thintest Vwi-aotz-- thin

### lvconvert --repair will work now:
# lvconvert -y --repair /dev/vg_test/boot
WARNING: Disabling lvmetad cache for repair command.
WARNING: Not using lvmetad because of repair.
Faulty devices in vg_test/boot successfully replaced.
# lvconvert -y --repair /dev/vg_test/Thin_tmeta
WARNING: Disabling lvmetad cache for repair command.
WARNING: Not using lvmetad because of repair.
Faulty devices in vg_test/Thin_tmeta successfully replaced.
# lvconvert -y --repair /dev/vg_test/Thin_tdata
WARNING: Disabling lvmetad cache for repair command.
WARNING: Not using lvmetad because of repair.
Faulty devices in vg_test/Thin_tdata successfully replaced.

# lvs -a -olv_name,lv_attr,sync_percent,segtype,devices vg_test
LV Attr Cpy%Sync Type Devices
Thin twi-aotz-- thin-pool Thin_tdata(0)
[Thin_tdata] rwi-aor--- 100.00 raid1 Thin_tdata_rimage_0(0),Thin_tdata_rimage_1(0)
[Thin_tdata_rimage_0] iwi-aor--- linear /dev/vdb(53)
[Thin_tdata_rimage_1] iwi-aor--- linear /dev/vdd(53)
[Thin_tdata_rmeta_0] ewi-aor--- linear /dev/vdb(52)
[Thin_tdata_rmeta_1] ewi-aor--- linear /dev/vdd(52)
[Thin_tmeta] ewi-aor--- 100.00 raid1 Thin_tmeta_rimage_0(0),Thin_tmeta_rimage_1(0)
[Thin_tmeta_rimage_0] iwi-aor--- linear /dev/vdb(27)
[Thin_tmeta_rimage_1] iwi-aor--- linear /dev/vdd(27)
[Thin_tmeta_rmeta_0] ewi-aor--- linear /dev/vdb(26)
[Thin_tmeta_rmeta_1] ewi-aor--- linear /dev/vdd(26)
boot rwi-a-r--- 100.00 raid1 boot_rimage_0(0),boot_rimage_1(0)
[boot_rimage_0] iwi-aor--- linear /dev/vdb(1)
[boot_rimage_1] iwi-aor--- linear /dev/vdd(1)
[boot_rmeta_0] ewi-aor--- linear /dev/vdb(0)
[boot_rmeta_1] ewi-aor--- linear /dev/vdd(0)
thintest Vwi-aotz-- thin
-------------------- 8< --------------------


For the wishlist: :-)
An lvm command to (maybe forcibly) manually set a raid sub-lv to faulty
(i.e. replace the PV with the error target) would be nice.


Cheers,
--leo