[linux-lvm] unable to remove snapshot of raid1 leg

Discussion:

Alexander 'Leo' Bergolth

2017-10-23 14:19:35 UTC

Hi!

As the raid1 check of my two-drive LV recently reported a mismatch count

0, I did some forensics.

One experiment was to create a snapshot of one leg of the mirror and try
to mount the snapshot. This worked fine but unfortunately I cannot
remove the snapshot now!

lvm wants to remove the origin too:
-------------------- 8< --------------------
# lvremove vg_test/snap_leg_0
Do you really want to remove active origin logical volume vg_test/testlv_rimage_0 with 1 snapshot(s)? [y/n]: n
Logical volume vg_test/testlv_rimage_0 not removed.
-------------------- 8< --------------------
(I have added the full history of a test setup below.)

Could anyone provide a hint how to get rid of the snapshot?
(Maybe manually using dmsetup commands and metadata backup editing?)

Thanks,
--leo

-------------------- snip! --------------------
# lvcreate --type raid1 -m 1 -n testlv -L100m vg_test /dev/vdb /dev/vdc
Logical volume "testlv" created.
# lvcreate -s -L50m -n snap_leg_0 /dev/mapper/vg_test-testlv_rimage_0
Using default stripesize 64.00 KiB.
Rounding up size to full physical extent 52.00 MiB
Logical volume "snap_leg_0" created.

# lvremove vg_test/snap_leg_0
Do you really want to remove active origin logical volume vg_test/testlv_rimage_0 with 1 snapshot(s)? [y/n]: n
Logical volume vg_test/testlv_rimage_0 not removed.

# lvs -a -o+devices,lv_uuid
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert Devices LV UUID
root vg_sys -wi-ao---- 5.00g /dev/vda2(0) fqp4Np-mLyL-y0OS-nufS-0hsT-7ijc-lGD9Za
swap vg_sys -wi-ao---- 2.00g /dev/vda2(2048) uNG1ht-eOKQ-y8Ji-yWSU-ZkZS-ZTIe-1M82mW
var vg_sys -wi-ao---- 3.00g /dev/vda2(1280) nDwCsP-vn38-6AUb-Dqer-VkLv-QHs3-CfVNCR
snap_leg_0 vg_test swi-a-s--- 52.00m [testlv_rimage_0] 0.00 /dev/vdb(26) f5AXqK-lDNe-hNrO-AW7G-yFvl-ctxU-fivKQf
testlv vg_test rwi-a-r--- 100.00m 100.00 testlv_rimage_0(0),testlv_rimage_1(0) Av5Y8Z-NBTH-tfN2-hCbW-Sur0-ebGU-SHBlsk
[testlv_rimage_0] vg_test owi-aor--- 100.00m /dev/vdb(1) OJyEep-Lgfm-B20o-4rfl-d38O-QByz-8oBa5z
[testlv_rimage_1] vg_test iwi-aor--- 100.00m /dev/vdc(1) J4AP4p-Mhfo-4oPn-SOFP-CGAi-riyL-CFr1fk
[testlv_rmeta_0] vg_test ewi-aor--- 4.00m /dev/vdb(0) Hgd58x-5Z2C-phkD-cIvl-KbPB-UWPz-THyZ3h
[testlv_rmeta_1] vg_test ewi-aor--- 4.00m /dev/vdc(0) mvehrT-Ofci-nWpm-A43m-iBpl-YW1U-Z1Z9x4

# lvremove --select lv_uuid=f5AXqK-lDNe-hNrO-AW7G-yFvl-ctxU-fivKQf
Do you really want to remove active origin logical volume vg_test/testlv_rimage_0 with 1 snapshot(s)? [y/n]: n
Logical volume vg_test/testlv_rimage_0 not removed.
-------------------- snip! --------------------

--
e-mail ::: Leo.Bergolth (at) wu.ac.at
fax ::: +43-1-31336-906050
location ::: IT-Services | Vienna University of Economics | Austria

Alexander 'Leo' Bergolth

2017-10-23 14:29:12 UTC

Permalink

Post by Alexander 'Leo' Bergolth
One experiment was to create a snapshot of one leg of the mirror and try
to mount the snapshot. This worked fine but unfortunately I cannot
remove the snapshot now!
-------------------- 8< --------------------
# lvremove vg_test/snap_leg_0
Do you really want to remove active origin logical volume vg_test/testlv_rimage_0 with 1 snapshot(s)? [y/n]: n
Logical volume vg_test/testlv_rimage_0 not removed.
-------------------- 8< --------------------
(I have added the full history of a test setup below.)

The output of lvremove -vvv is available at:

http://leo.kloburg.at/tmp/lvm-remove-raid1-leg-snapshot/lvremove-vvv.txt

I am using lvm2-2.02.171-8.el7.x86_64.

--leo

--
e-mail ::: Leo.Bergolth (at) wu.ac.at
fax ::: +43-1-31336-906050
location ::: IT-Services | Vienna University of Economics | Austria

Heinz Mauelshagen

2017-10-23 14:44:17 UTC

Permalink

Hi Alexander,

LVM snapshots are meant to be used on the user visible raid1 LVs.
You found a bug allowing it to be used on its hidden legs.

Removing such per leg snapshot should be possible after the raid1 LV got
deactivated.

Heinz

Post by Alexander 'Leo' Bergolth
Hi!
As the raid1 check of my two-drive LV recently reported a mismatch count

0, I did some forensics.

One experiment was to create a snapshot of one leg of the mirror and try
to mount the snapshot. This worked fine but unfortunately I cannot
remove the snapshot now!
-------------------- 8< --------------------
# lvremove vg_test/snap_leg_0
Do you really want to remove active origin logical volume vg_test/testlv_rimage_0 with 1 snapshot(s)? [y/n]: n
Logical volume vg_test/testlv_rimage_0 not removed.
-------------------- 8< --------------------
(I have added the full history of a test setup below.)
Could anyone provide a hint how to get rid of the snapshot?
(Maybe manually using dmsetup commands and metadata backup editing?)
Thanks,
--leo
-------------------- snip! --------------------
# lvcreate --type raid1 -m 1 -n testlv -L100m vg_test /dev/vdb /dev/vdc
Logical volume "testlv" created.
# lvcreate -s -L50m -n snap_leg_0 /dev/mapper/vg_test-testlv_rimage_0
Using default stripesize 64.00 KiB.
Rounding up size to full physical extent 52.00 MiB
Logical volume "snap_leg_0" created.
# lvremove vg_test/snap_leg_0
Do you really want to remove active origin logical volume vg_test/testlv_rimage_0 with 1 snapshot(s)? [y/n]: n
Logical volume vg_test/testlv_rimage_0 not removed.
# lvs -a -o+devices,lv_uuid
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert Devices LV UUID
root vg_sys -wi-ao---- 5.00g /dev/vda2(0) fqp4Np-mLyL-y0OS-nufS-0hsT-7ijc-lGD9Za
swap vg_sys -wi-ao---- 2.00g /dev/vda2(2048) uNG1ht-eOKQ-y8Ji-yWSU-ZkZS-ZTIe-1M82mW
var vg_sys -wi-ao---- 3.00g /dev/vda2(1280) nDwCsP-vn38-6AUb-Dqer-VkLv-QHs3-CfVNCR
snap_leg_0 vg_test swi-a-s--- 52.00m [testlv_rimage_0] 0.00 /dev/vdb(26) f5AXqK-lDNe-hNrO-AW7G-yFvl-ctxU-fivKQf
testlv vg_test rwi-a-r--- 100.00m 100.00 testlv_rimage_0(0),testlv_rimage_1(0) Av5Y8Z-NBTH-tfN2-hCbW-Sur0-ebGU-SHBlsk
[testlv_rimage_0] vg_test owi-aor--- 100.00m /dev/vdb(1) OJyEep-Lgfm-B20o-4rfl-d38O-QByz-8oBa5z
[testlv_rimage_1] vg_test iwi-aor--- 100.00m /dev/vdc(1) J4AP4p-Mhfo-4oPn-SOFP-CGAi-riyL-CFr1fk
[testlv_rmeta_0] vg_test ewi-aor--- 4.00m /dev/vdb(0) Hgd58x-5Z2C-phkD-cIvl-KbPB-UWPz-THyZ3h
[testlv_rmeta_1] vg_test ewi-aor--- 4.00m /dev/vdc(0) mvehrT-Ofci-nWpm-A43m-iBpl-YW1U-Z1Z9x4
# lvremove --select lv_uuid=f5AXqK-lDNe-hNrO-AW7G-yFvl-ctxU-fivKQf
Do you really want to remove active origin logical volume vg_test/testlv_rimage_0 with 1 snapshot(s)? [y/n]: n
Logical volume vg_test/testlv_rimage_0 not removed.
-------------------- snip! --------------------

Alexander 'Leo' Bergolth

2017-10-23 16:40:17 UTC

Permalink

Post by Heinz Mauelshagen
LVM snapshots are meant to be used on the user visible raid1 LVs.
You found a bug allowing it to be used on its hidden legs.
Removing such per leg snapshot should be possible after the raid1 LV got
deactivated.

Hmm. This LV is currently in use. Any chance to remove it with the raid1
LV active?

Cheers,
--leo

Post by Heinz Mauelshagen

Post by Alexander 'Leo' Bergolth
Hi!
As the raid1 check of my two-drive LV recently reported a mismatch count

0, I did some forensics.

One experiment was to create a snapshot of one leg of the mirror and try
to mount the snapshot. This worked fine but unfortunately I cannot
remove the snapshot now!
-------------------- 8< --------------------
# lvremove vg_test/snap_leg_0
Do you really want to remove active origin logical volume
vg_test/testlv_rimage_0 with 1 snapshot(s)? [y/n]: n
   Logical volume vg_test/testlv_rimage_0 not removed.
-------------------- 8< --------------------
(I have added the full history of a test setup below.)
Could anyone provide a hint how to get rid of the snapshot?
(Maybe manually using dmsetup commands and metadata backup editing?)
Thanks,
--leo
-------------------- snip! --------------------
# lvcreate --type raid1 -m 1 -n testlv -L100m vg_test /dev/vdb /dev/vdc
   Logical volume "testlv" created.
# lvcreate -s -L50m -n snap_leg_0 /dev/mapper/vg_test-testlv_rimage_0
   Using default stripesize 64.00 KiB.
   Rounding up size to full physical extent 52.00 MiB
   Logical volume "snap_leg_0" created.
# lvremove vg_test/snap_leg_0
Do you really want to remove active origin logical volume
vg_test/testlv_rimage_0 with 1 snapshot(s)? [y/n]: n
   Logical volume vg_test/testlv_rimage_0 not removed.
# lvs -a -o+devices,lv_uuid
   LV                VG      Attr       LSize   Pool Origin
Data% Meta% Move Log Cpy%Sync Convert
Devices                               LV UUID
   root              vg_sys -wi-ao----
5.00g
/dev/vda2(0)
fqp4Np-mLyL-y0OS-nufS-0hsT-7ijc-lGD9Za
   swap              vg_sys -wi-ao----
2.00g
/dev/vda2(2048)
uNG1ht-eOKQ-y8Ji-yWSU-ZkZS-ZTIe-1M82mW
   var               vg_sys -wi-ao----
3.00g
/dev/vda2(1280)
nDwCsP-vn38-6AUb-Dqer-VkLv-QHs3-CfVNCR
   snap_leg_0        vg_test swi-a-s--- 52.00m      [testlv_rimage_0]
0.00
/dev/vdb(26)
f5AXqK-lDNe-hNrO-AW7G-yFvl-ctxU-fivKQf
   testlv            vg_test rwi-a-r---
100.00m                                               100.00
testlv_rimage_0(0),testlv_rimage_1(0)
Av5Y8Z-NBTH-tfN2-hCbW-Sur0-ebGU-SHBlsk
   [testlv_rimage_0] vg_test owi-aor---
100.00m
/dev/vdb(1)
OJyEep-Lgfm-B20o-4rfl-d38O-QByz-8oBa5z
   [testlv_rimage_1] vg_test iwi-aor---
100.00m
/dev/vdc(1)
J4AP4p-Mhfo-4oPn-SOFP-CGAi-riyL-CFr1fk
   [testlv_rmeta_0] vg_test ewi-aor---
4.00m
/dev/vdb(0)
Hgd58x-5Z2C-phkD-cIvl-KbPB-UWPz-THyZ3h
   [testlv_rmeta_1] vg_test ewi-aor---
4.00m
/dev/vdc(0)
mvehrT-Ofci-nWpm-A43m-iBpl-YW1U-Z1Z9x4
# lvremove --select lv_uuid=f5AXqK-lDNe-hNrO-AW7G-yFvl-ctxU-fivKQf
Do you really want to remove active origin logical volume
vg_test/testlv_rimage_0 with 1 snapshot(s)? [y/n]: n
   Logical volume vg_test/testlv_rimage_0 not removed.
-------------------- snip! --------------------

--
e-mail ::: Leo.Bergolth (at) wu.ac.at
fax ::: +43-1-31336-906050
location ::: IT-Services | Vienna University of Economics | Austria

Zdenek Kabelac

2017-10-24 11:03:15 UTC

Permalink

Post by Alexander 'Leo' Bergolth

Hmm. This LV is currently in use. Any chance to remove it with the raid1
LV active?
Cheers,

Hi

Not with lvm2 command.

I could probably imagine a way with several dmsetup commands but it looks
fairly easier to do it in some 'offline' time.

-

The one 'risky' idea could be to try 'vgcfgrestore' metadata before taking
snapshot then (forcibly) remove (dmsetup remove (-f) vg-snapshot) so
snapshot is no longer referencing origin.
(force should not be needed - but if some has opened snapshot...)

Then you should be able to use 'lvchange --refresh vg'

and finally drop 'left-over' cow volume with dmsetup remove.
(and make sure snapshot is already gone - if not - try remove again)

Regards

Zdenek

Alexander 'Leo' Bergolth

2017-10-25 08:47:13 UTC

Permalink

Post by Zdenek Kabelac

Post by Alexander 'Leo' Bergolth

Post by Heinz Mauelshagen
Removing such per leg snapshot should be possible after the raid1 LV got
deactivated.

Hmm. This LV is currently in use. Any chance to remove it with the raid1
LV active?

The one 'risky' idea could be to try 'vgcfgrestore' metadata before taking
snapshot then (forcibly) remove (dmsetup remove (-f) vg-snapshot) so
snapshot is no longer referencing origin.
(force should not be needed - but if some has opened snapshot...)
Then you should be able to use 'lvchange --refresh vg'
and finally drop 'left-over' cow volume with dmsetup remove.
(and make sure snapshot is already gone - if not - try remove again)

Thanks a lot!
I tried this on a test-box. Works perfect!

Cheers,
--leo

--
e-mail ::: Leo.Bergolth (at) wu.ac.at
fax ::: +43-1-31336-906050
location ::: IT-Services | Vienna University of Economics | Austria