[linux-lvm] Raid1 -> linear conversion = destroyed data

Discussion:

Phillip Susi

2014-12-04 23:07:37 UTC

I used lvconvert -m 1 --type raid1 to clone my root volume onto a new
drive, and then used --splitmirrors to split off the original volume
as a backup while I worked off the new drive for a few days. I then
decided the new drive was working well and used lvconvert -m 0 to
convert the volume back to linear. Instead of dropping the split off
original leg, lvm dropped the NEW copy and redirected all IO to the
OLD data, causing massive filesystem corruption.

This is on Ubuntu 14.10 using lvm 2.02.98 and linux 3.16.

Anatoly Pugachev

2014-12-05 10:12:04 UTC

Permalink

Post by Phillip Susi
I used lvconvert -m 1 --type raid1 to clone my root volume onto a new
drive, and then used --splitmirrors to split off the original volume
as a backup while I worked off the new drive for a few days. I then
decided the new drive was working well and used lvconvert -m 0 to
convert the volume back to linear. Instead of dropping the split off
original leg, lvm dropped the NEW copy and redirected all IO to the
OLD data, causing massive filesystem corruption.
This is on Ubuntu 14.10 using lvm 2.02.98 and linux 3.16.

Phillip,

can you please paste the commands you've used to convert the volume? I
don't have ubuntu , but rhel7 manual page for lvconvert, example section is
pretty clear on how to convert back to flat (-m0) volume and what physical
device you remove from logical volume, quote:

Converts mirror logical volume "vg00/lvmirror1" to linear, freeing
physical extents from /dev/sda:

lvconvert -m0 vg00/lvmirror1 /dev/sda

Phillip Susi

2014-12-07 21:24:55 UTC

Permalink

Post by Anatoly Pugachev
Phillip,
can you please paste the commands you've used to convert the
volume? I don't have ubuntu , but rhel7 manual page for lvconvert,
example section is pretty clear on how to convert back to flat
(-m0) volume and what physical device you remove from logical
Converts mirror logical volume "vg00/lvmirror1" to linear, freeing
lvconvert -m0 vg00/lvmirror1 /dev/sda

I simply used lvconvert -m0 vg/lv. I did not think I would need to
specify which leg should be kept; obviously it should be the one
currently in use rather than the out of date copy.

Anatoly Pugachev

2014-12-08 08:30:06 UTC

Permalink

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

I simply used lvconvert -m0 vg/lv. I did not think I would need to
specify which leg should be kept; obviously it should be the one
currently in use rather than the out of date copy.

I wonder how many more people would lost their data before there would
be changes to lvconvert -m0 to require specification of physicalVolume
instead of optional requirement.

Jason Warr

2014-12-08 20:59:59 UTC

Permalink

Post by Anatoly Pugachev

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

I simply used lvconvert -m0 vg/lv. I did not think I would need to
specify which leg should be kept; obviously it should be the one
currently in use rather than the out of date copy.

I wonder how many more people would lost their data before there would
be changes to lvconvert -m0 to require specification of physicalVolume
instead of optional requirement.

I may have missed part of the discussion but in order to fully understand
what is going on I'd like to know what arguments you used when you issued
the "lvconvert --splitmirrors" command. There are several options to it
that can influence what happens to the mirror members.

The best way I know of to get that is from the /etc/lvm/archive/ files for
the volume group in question. Those contain the maps for the volume
group. A new map version is created in that directory after every command
that modifies the volume group. Each map contains a "description" line
that shows the command that was run making the changes. The map does not
include the changes as it is meant to be used as a reference to the
configuration before the change.

If you only want to send the commands as pulled from the archive maps here
is an example from the system I have been trying to re-create this failure
scenario on.

[2:***@hostname /root]# grep Created /etc/lvm/archive/test01VG_0000*
/etc/lvm/archive/test01VG_00000-1787228376.vg:description = "Created
*before* executing 'vgcreate test01VG /dev/sdar /dev/sdas /dev/sdat
/dev/sdau /dev/sdav /dev/sdaw /dev/sdax'"
/etc/lvm/archive/test01VG_00001-1025082804.vg:description = "Created
*before* executing 'lvcreate -L50G -m1 --type raid1 test01VG -n test01LV'"
/etc/lvm/archive/test01VG_00002-178937097.vg:description = "Created
*before* executing 'lvconvert --splitmirrors 1 --name test01LV-M1
test01VG/test01LV'"

Phillip Susi

2014-12-11 01:15:05 UTC

Permalink

Post by Jason Warr
I may have missed part of the discussion but in order to fully
understand what is going on I'd like to know what arguments you
used when you issued the "lvconvert --splitmirrors" command. There
are several options to it that can influence what happens to the
mirror members.

The exact sequence of commands was:

lvconvert --type raid1 -m 1 faldara/utopic /dev/sdb1
lvconvert --splitmirrors 1 --trackchanges faldara/utopic --name
utopic-backup /dev/md0
lvconvert --splitmirrors 1 --trackchanges faldara/utopic --name
utopic-backup /dev/md0

The idea was to start using the new drive exclusively while keeping a
backup on the old drive.

Also I thought it was a bit odd that --splitmirrors accepted the
- --name argument, but silently ignored it, instead using an internal
auto generated name.

Jason Warr

2014-12-11 01:50:10 UTC

Permalink

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

lvconvert --type raid1 -m 1 faldara/utopic /dev/sdb1
lvconvert --splitmirrors 1 --trackchanges faldara/utopic --name
utopic-backup /dev/md0

It ignores the name because what it does with "--trackchanges" is rename
the mirror leg to a visible & read only version of the meta. Sample:

[5:***@goldmember /etc/sysconfig/network-scripts]# lvs -a -o
lv_name,lv_attr,lv_dm_path,devices,copy_per
LV Attr DMPath
Devices
test01LV rwi-aor--- /dev/mapper/test01VG-test01LV
test01LV_rimage_0(0),test01LV_rim
[test01LV_rimage_0] Iwi-aor--- /dev/mapper/test01VG-test01LV_rimage_0
/dev/sdas(12802)
[test01LV_rimage_1] Iwi-aor--- /dev/mapper/test01VG-test01LV_rimage_1
/dev/sdar(12802)
[test01LV_rmeta_0] ewi-aor--- /dev/mapper/test01VG-test01LV_rmeta_0
/dev/sdau(0)
[test01LV_rmeta_1] ewi-aor--- /dev/mapper/test01VG-test01LV_rmeta_1
/dev/sdat(0)

Would be like so after:

[5:***@goldmember /etc/sysconfig/network-scripts]# lvs -a -o
lv_name,lv_attr,lv_dm_path,devices,copy_per
LV Attr DMPath
Devices
test01LV rwi-aor--- /dev/mapper/test01VG-test01LV
test01LV_rimage_0(0),test01LV_rim
test01LV_rimage_0 Iri-a-r--- /dev/mapper/test01VG-test01LV_rimage_0
/dev/sdar(1)
[test01LV_rimage_1] iwi-aor--- /dev/mapper/test01VG-test01LV_rimage_1
/dev/sdas(12802)
[test01LV_rmeta_0] ewi-a-r--- /dev/mapper/test01VG-test01LV_rmeta_0
/dev/sdar(0)
[test01LV_rmeta_1] ewi-aor--- /dev/mapper/test01VG-test01LV_rmeta_1
/dev/sdas(12801)

lvconvert --splitmirrors 1 --trackchanges faldara/utopic --name
utopic-backup /dev/md0
The idea was to start using the new drive exclusively while keeping a
backup on the old drive.
Also I thought it was a bit odd that --splitmirrors accepted the
- --name argument, but silently ignored it, instead using an internal
auto generated name.

You can't track changes and rename the "backup" leg at the same time. If
you wanted to keep the backup leg after making sure the in use one was
good you would need to issue a new "--splitmirror"
command with a new volume name and *no* "--trackchanges" option.

Phillip Susi

2014-12-11 14:17:03 UTC

Permalink

Post by Jason Warr
It ignores the name because what it does with "--trackchanges" is
rename the mirror leg to a visible & read only version of the meta.

You mean it changes the flags to visible and read only. I don't see
why it can't rename it as well.

Post by Jason Warr
You can't track changes and rename the "backup" leg at the same
time. If you wanted to keep the backup leg after making sure the in
use one was good you would need to issue a new "--splitmirror"
command with a new volume name and *no* "--trackchanges" option.

Why not? A name is just a name; it does not *have* to be called
origin_rimage_0.

Jason Warr

2014-12-11 14:48:32 UTC

Permalink

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Post by Jason Warr
It ignores the name because what it does with "--trackchanges" is
rename the mirror leg to a visible & read only version of the meta.

You mean it changes the flags to visible and read only. I don't see
why it can't rename it as well.

Yes, visible and ro is what I meant.

Why not? A name is just a name; it does not *have* to be called
origin_rimage_0.

I don't disagree but it is the way the tool works. My best guess is the
name is ignored because when you issue "--trackchanges" it is not really
splitting off the leg, more of off lining it from the perspective of the
mirror. It was likely a developer decision for lvconvert to do it that
way when adding in RAID1 from the md stack.

The documentation for lvconvert could be more clear about it.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)
iQEcBAEBAgAGBQJUiadfAAoJENRVrw2cjl5Rt+kH/2edRv5pygm91ULcpZ9ehRuW
9bB0CpjyrLEp/f8bdqkcoyR2cpcuM2kUXBstvn7YLgyJnBdm8JNt6gZnanJLQ0gr
CXx6Tu2OQfp87NTbUDqJOLku6EaAQ053zQAoBgnIctFaCUGU+hC1L3iwdkYGnVwZ
uTdY0C3/t7ShPDvXqTrIqNugqOj7NFAptpENzxBavrkaIp5w47fvVHpWwFpwO3sU
CheSV+gpqHKlc6T6DfZlDCcCnZHKmMhXzs4EigNZIZFzM8INhn01mJ64tUJKNET7
eLvqIH8labOqAp4PVFido6Kgl6K/BKvGbzGJcqE9IR2V23yvzRHjretAGnFwed4=
=sGDQ
-----END PGP SIGNATURE-----

Phillip Susi

2014-12-15 21:53:59 UTC

Permalink

Post by Jason Warr
I may have missed part of the discussion but in order to fully
understand what is going on I'd like to know what arguments you
used when you issued the "lvconvert --splitmirrors" command.
There are several options to it that can influence what happens
to the mirror members.

lvconvert --type raid1 -m 1 faldara/utopic /dev/sdb1 lvconvert
--splitmirrors 1 --trackchanges faldara/utopic --name utopic-backup
/dev/md0 lvconvert --splitmirrors 1 --trackchanges faldara/utopic
--name utopic-backup /dev/md0
The idea was to start using the new drive exclusively while keeping
a backup on the old drive.

So is this on someone's radar now as a critical bug?

Phillip Susi

2014-12-11 01:16:28 UTC

Permalink

Post by Anatoly Pugachev
I wonder how many more people would lost their data before there
would be changes to lvconvert -m0 to require specification of
physicalVolume instead of optional requirement.

It should not be required. As long as both legs are in sync, it
doesn't matter which one is kept and which one is dropped. When one
is intentionally out of sync because it has been removed with
- --splitmirrors, then under no circumstances should that one be kept.