[linux-lvm] LV mirror extents on same physical device

[Julie Ashworth]

hi all,

just some background...
I installed a CentOS5 server in 2010 with a mdadm RAID1 containing 2 partitions (/dev/sda1 and /dev/sdb1). /dev/sda1 failed, and unfortuately, the mirror failed to sync with a new drive.

On this RAID1, I have one VG (VolGroup00) with 5 LVs.

I contacted the folks on the mdadm mailing list. The data on /dev/sdb is good, so they recommended that I convert the linear LVs to mirrors (to copy the data to a new drive).

I formatted a new drive (/dev/sde) identical to /dev/sd[ab], added it to a mdadm device (/dev/md2), which I then added to my VG:
mdadm --create --level=1 -n 2 /dev/md2 /dev/sde2 missing
pvcreate /dev/md2
vgextend VolGroup00 /dev/md2

I then converted all my volumes to mirrors:
lvconvert -m1 --mirrorlog=mirrored --alloc=anywhere /dev/VolGroup00/srvlv
lvconvert -m1 --mirrorlog=mirrored --alloc=anywhere /dev/VolGroup00/rootlv
lvconvert -m1 --mirrorlog=mirrored --alloc=anywhere /dev/VolGroup00/swaplv
lvconvert -m1 --mirrorlog=mirrored --alloc=anywhere /dev/VolGroup00/varlv
lvconvert -m1 --mirrorlog=mirrored --alloc=anywhere /dev/VolGroup00/tmplv

(in case you're wondering, when this completed, I added /dev/sda1 to /dev/md2, so /dev/md2 is a fuctional mdadm RAID1).

Now... the problem ;)....

When I run 'lvs' I see that the LV mirror extents were created on the same device (/dev/md1):

# lvs -a -o +devices
LV VG Attr LSize Origin Snap% Move Log Copy% Convert Devices
rootlv VolGroup00 mwi-ao 10.00G rootlv_mlog 100.00 rootlv_mimage_0(0),rootlv_mimage_1(0)
[rootlv_mimage_0] VolGroup00 iwi-ao 10.00G /dev/md1(0)
[rootlv_mimage_1] VolGroup00 iwi-ao 10.00G /dev/md1(4036)
[rootlv_mlog] VolGroup00 mwa-ao 32.00M 100.00 rootlv_mlog_mimage_0(0),rootlv_mlog_mimage_1(0)
[rootlv_mlog_mimage_0] VolGroup00 iwi-ao 32.00M /dev/md2(4)
[rootlv_mlog_mimage_1] VolGroup00 iwi-ao 32.00M /dev/md1(4356)


My long-term goal is to convert the mirror back to a linear volume using just /dev/md2, i.e.
lvconvert -m0 /dev/VolGroup00/rootlv /dev/md1
lvconvert -m0 /dev/VolGroup00/swaplv /dev/md1
lvconvert -m0 /dev/VolGroup00/srvlv /dev/md1
lvconvert -m0 /dev/VolGroup00/varlv /dev/md1
lvconvert -m0 /dev/VolGroup00/tmplv /dev/md1

... so it seems problematic that both extents are on /dev/md1.

I thought lvm was supposed to use both physical devices. Do you know why both extents were created on the same physical device?
Should I create another extent on /dev/md2 before I remove the /dev/md1 extent(s)? If so, can you give me an example of how to use lvconvert in this case?

Thanks in advance!
best,
Julie

[James Hawtin]

Ok I am not really sure what you are trying to do with madm, from what I
can see you want to move data from pv on md1 to a pv on md2, this is
really easy to do.

#pvcreate /dev/md2
#vgextend VolGroup00 /dev/md2
# Stop the old pv getting lvs created on it
pvchange -x n /dev/md1
# Whole thing in one go
pvmove -v /dev/md1 /dev/md2
# One at a time
pvmove -v -n rootlv /dev/md1 /dev/md2
vgreduce VolGroup00 /dev/md2

Your done.

James

[Julie Ashworth]

Post by James Hawtin
#pvcreate /dev/md2
#vgextend VolGroup00 /dev/md2
# Stop the old pv getting lvs created on it
pvchange -x n /dev/md1
# Whole thing in one go
pvmove -v /dev/md1 /dev/md2
# One at a time
pvmove -v -n rootlv /dev/md1 /dev/md2
vgreduce VolGroup00 /dev/md2

---end quoted text---

My LVs are already mirrored (w/ two extents on /dev/md1), so is pvmove really the right tool to move one of those extents?
best,
Julie

[John Stoffel]

Post by James Hawtin
#pvcreate /dev/md2
#vgextend VolGroup00 /dev/md2
# Stop the old pv getting lvs created on it
pvchange -x n /dev/md1
# Whole thing in one go
pvmove -v /dev/md1 /dev/md2
# One at a time
pvmove -v -n rootlv /dev/md1 /dev/md2
vgreduce VolGroup00 /dev/md2

Julie> ---end quoted text---

Julie> My LVs are already mirrored (w/ two extents on /dev/md1), so is
Julie> pvmove really the right tool to move one of those extents?
Julie> best,

Julie,

I think the problem is that you're confusing the Mirroring done at the
MD level, which is /dev/md#, and then the mirroring at the LVM level,
which is stacked on top of MD. So there's no need to mirror at both
levels. But let's double check your setup, so can you please reply
with the output of the following commands:

cat /proc/mounts
cat /proc/mdstat
pvs
vgs
lvs

This will clarify exactly what your setup it and how it looks.

John

[Julie Ashworth]

Post by John Stoffel
I think the problem is that you're confusing the Mirroring done at the
MD level, which is /dev/md#, and then the mirroring at the LVM level,
which is stacked on top of MD. So there's no need to mirror at both
levels. But let's double check your setup, so can you please reply

I'm not confused about mdadm vs. LVM mirror. I regret mentioning mdadm, since it seems to be confusing everyone. I should have only asked why the lvconvert command produced extents on the same physical device. At this point, I will just go ahead and test if
lvconvert -m0 /dev/VolGroup00/rootlv /dev/md1
will produce a functional linear volume on /dev/md2

best,
Julie

[Roger Heflin]

If I am reading what I can find correctly - -alloc=anywhere defeats
the mirror code from attempting to put on 2 different physical
devices.

The man pages seem to lack info on what other values there are for
that but disables device mirroring.

Post by Julie Ashworth
hi all,
just some background...
I installed a CentOS5 server in 2010 with a mdadm RAID1 containing 2 partitions (/dev/sda1 and /dev/sdb1). /dev/sda1 failed, and unfortuately, the mirror failed to sync with a new drive.
On this RAID1, I have one VG (VolGroup00) with 5 LVs.
I contacted the folks on the mdadm mailing list. The data on /dev/sdb is good, so they recommended that I convert the linear LVs to mirrors (to copy the data to a new drive).
mdadm --create --level=1 -n 2 /dev/md2 /dev/sde2 missing
pvcreate /dev/md2
vgextend VolGroup00 /dev/md2
lvconvert -m1 --mirrorlog=mirrored --alloc=anywhere /dev/VolGroup00/srvlv
lvconvert -m1 --mirrorlog=mirrored --alloc=anywhere /dev/VolGroup00/rootlv
lvconvert -m1 --mirrorlog=mirrored --alloc=anywhere /dev/VolGroup00/swaplv
lvconvert -m1 --mirrorlog=mirrored --alloc=anywhere /dev/VolGroup00/varlv
lvconvert -m1 --mirrorlog=mirrored --alloc=anywhere /dev/VolGroup00/tmplv
(in case you're wondering, when this completed, I added /dev/sda1 to /dev/md2, so /dev/md2 is a fuctional mdadm RAID1).
Now... the problem ;)....
# lvs -a -o +devices
LV VG Attr LSize Origin Snap% Move Log Copy% Convert Devices
rootlv VolGroup00 mwi-ao 10.00G rootlv_mlog 100.00 rootlv_mimage_0(0),rootlv_mimage_1(0)
[rootlv_mimage_0] VolGroup00 iwi-ao 10.00G /dev/md1(0)
[rootlv_mimage_1] VolGroup00 iwi-ao 10.00G /dev/md1(4036)
[rootlv_mlog] VolGroup00 mwa-ao 32.00M 100.00 rootlv_mlog_mimage_0(0),rootlv_mlog_mimage_1(0)
[rootlv_mlog_mimage_0] VolGroup00 iwi-ao 32.00M /dev/md2(4)
[rootlv_mlog_mimage_1] VolGroup00 iwi-ao 32.00M /dev/md1(4356)
My long-term goal is to convert the mirror back to a linear volume using just /dev/md2, i.e.
lvconvert -m0 /dev/VolGroup00/rootlv /dev/md1
lvconvert -m0 /dev/VolGroup00/swaplv /dev/md1
lvconvert -m0 /dev/VolGroup00/srvlv /dev/md1
lvconvert -m0 /dev/VolGroup00/varlv /dev/md1
lvconvert -m0 /dev/VolGroup00/tmplv /dev/md1
... so it seems problematic that both extents are on /dev/md1.
I thought lvm was supposed to use both physical devices. Do you know why both extents were created on the same physical device?
Should I create another extent on /dev/md2 before I remove the /dev/md1 extent(s)? If so, can you give me an example of how to use lvconvert in this case?
Thanks in advance!
best,
Julie
_______________________________________________
linux-lvm mailing list
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

[Roger Heflin]

without the anywhere option it indicates you need *4* devices, 2 for
data, and 2 for mirror log, using anywhere removes the more than 1
device for everything,.

Using this format should allow you to force the move of a copy from
one device to another, see the pvmove man page for more details.

pvmove /dev/sdb1:1000-1999 /dev/sdc1:0-999

I have had to do something similar on some other OS (not Linux) LVM
when it did incorrect things like this.

Post by Roger Heflin
If I am reading what I can find correctly - -alloc=anywhere defeats
the mirror code from attempting to put on 2 different physical
devices.
The man pages seem to lack info on what other values there are for
that but disables device mirroring.

Post by Julie Ashworth
hi all,
just some background...
I installed a CentOS5 server in 2010 with a mdadm RAID1 containing 2 partitions (/dev/sda1 and /dev/sdb1). /dev/sda1 failed, and unfortuately, the mirror failed to sync with a new drive.
On this RAID1, I have one VG (VolGroup00) with 5 LVs.
I contacted the folks on the mdadm mailing list. The data on /dev/sdb is good, so they recommended that I convert the linear LVs to mirrors (to copy the data to a new drive).
mdadm --create --level=1 -n 2 /dev/md2 /dev/sde2 missing
pvcreate /dev/md2
vgextend VolGroup00 /dev/md2
lvconvert -m1 --mirrorlog=mirrored --alloc=anywhere /dev/VolGroup00/srvlv
lvconvert -m1 --mirrorlog=mirrored --alloc=anywhere /dev/VolGroup00/rootlv
lvconvert -m1 --mirrorlog=mirrored --alloc=anywhere /dev/VolGroup00/swaplv
lvconvert -m1 --mirrorlog=mirrored --alloc=anywhere /dev/VolGroup00/varlv
lvconvert -m1 --mirrorlog=mirrored --alloc=anywhere /dev/VolGroup00/tmplv
(in case you're wondering, when this completed, I added /dev/sda1 to /dev/md2, so /dev/md2 is a fuctional mdadm RAID1).
Now... the problem ;)....
# lvs -a -o +devices
LV VG Attr LSize Origin Snap% Move Log Copy% Convert Devices
rootlv VolGroup00 mwi-ao 10.00G rootlv_mlog 100.00 rootlv_mimage_0(0),rootlv_mimage_1(0)
[rootlv_mimage_0] VolGroup00 iwi-ao 10.00G /dev/md1(0)
[rootlv_mimage_1] VolGroup00 iwi-ao 10.00G /dev/md1(4036)
[rootlv_mlog] VolGroup00 mwa-ao 32.00M 100.00 rootlv_mlog_mimage_0(0),rootlv_mlog_mimage_1(0)
[rootlv_mlog_mimage_0] VolGroup00 iwi-ao 32.00M /dev/md2(4)
[rootlv_mlog_mimage_1] VolGroup00 iwi-ao 32.00M /dev/md1(4356)
My long-term goal is to convert the mirror back to a linear volume using just /dev/md2, i.e.
lvconvert -m0 /dev/VolGroup00/rootlv /dev/md1
lvconvert -m0 /dev/VolGroup00/swaplv /dev/md1
lvconvert -m0 /dev/VolGroup00/srvlv /dev/md1
lvconvert -m0 /dev/VolGroup00/varlv /dev/md1
lvconvert -m0 /dev/VolGroup00/tmplv /dev/md1
... so it seems problematic that both extents are on /dev/md1.
I thought lvm was supposed to use both physical devices. Do you know why both extents were created on the same physical device?
Should I create another extent on /dev/md2 before I remove the /dev/md1 extent(s)? If so, can you give me an example of how to use lvconvert in this case?
Thanks in advance!
best,
Julie
_______________________________________________
linux-lvm mailing list
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

[Julie Ashworth]

Post by Roger Heflin
without the anywhere option it indicates you need *4* devices, 2 for
data, and 2 for mirror log, using anywhere removes the more than 1
device for everything,.

The -alloc=aywhere simply refers to the log. I'm not concerned about the log, I'm concered about the actual image extents, which seem to be both on /dev/md1.

Post by Roger Heflin
Using this format should allow you to force the move of a copy from
one device to another, see the pvmove man page for more details.
pvmove /dev/sdb1:1000-1999 /dev/sdc1:0-999

The end goal is to convert the mirror to a linear volume (on /dev/md2).
I'm still not convinced that pvmove is the right utility for mirrored extents. But thanks for the advice - I will try it if lvconvert -m0 fails. (this is a test server).

best,
Julie

[Julie Ashworth]

Post by Roger Heflin
If I am reading what I can find correctly - -alloc=anywhere defeats
the mirror code from attempting to put on 2 different physical
devices.

---end quoted text---

-alloc=anywhere just refers to the logs, which isn't important at this point.
best,
Julie

[Roger Heflin]

This indicates that it "worked" that way on some versions rhel5 but
does not work that way on rhel6 in a number of versions.

https://bugzilla.redhat.com/show_bug.cgi?id=594525

I would suspect depending on exactly what version you are using that
it is a bug.

I know up to at least 5.7 there are some bad LVM bugs when you are
doing abnormal things (mirroring, pvmoves with lv with multiple
extents), there are probably other bugs I don't know about.

I would suggest unmirroring it, and just doing a straight pvmove to
the new device, mirroring it in the middle would just appear to
complicate things.

Post by Julie Ashworth

Post by Roger Heflin
If I am reading what I can find correctly - -alloc=anywhere defeats
the mirror code from attempting to put on 2 different physical
devices.

---end quoted text---
-alloc=anywhere just refers to the logs, which isn't important at this point.
best,
Julie
_______________________________________________
linux-lvm mailing list
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

[Julie Ashworth]

Post by Roger Heflin
This indicates that it "worked" that way on some versions rhel5 but
does not work that way on rhel6 in a number of versions.
https://bugzilla.redhat.com/show_bug.cgi?id=594525
I would suspect depending on exactly what version you are using that
it is a bug.
I know up to at least 5.7 there are some bad LVM bugs when you are
doing abnormal things (mirroring, pvmoves with lv with multiple
extents), there are probably other bugs I don't know about.
I would suggest unmirroring it, and just doing a straight pvmove to
the new device, mirroring it in the middle would just appear to
complicate things.

---end quoted text---

Thank you.
It's possible to specify where the extents should be located, i.e.

The following command creates a mirrored logical volume with a single mirror. The volume is 500 megabytes in size, it is named mirrorlv, and it is carved out of volume group vg0. The first leg of the mirror is on device /dev/sda1, the second leg of the mirror is on device /dev/sdb1, and the mirror log is on /dev/sdc1.
lvcreate -L 500M -m1 -n mirrorlv vg0 /dev/sda1 /dev/sdb1 /dev/sdc1
(from http://linux-satya.blogspot.com/2010/09/logical-volume-manager.html)

I regret not doing this. Oh well. At this point, I haven't decided if I should convert to linear LV, or add aother mirror on /dev/md2.

Thanks again for the heads-up about the bug.
best,
Julie

[emmanuel segura]

i don't understand why you are using two mirror technologies together, i'm
using in lvm mirror without mdadm and never found problems

Post by Julie Ashworth
hi all,
just some background...
I installed a CentOS5 server in 2010 with a mdadm RAID1 containing 2
partitions (/dev/sda1 and /dev/sdb1). /dev/sda1 failed, and unfortuately,
the mirror failed to sync with a new drive.
On this RAID1, I have one VG (VolGroup00) with 5 LVs.
I contacted the folks on the mdadm mailing list. The data on /dev/sdb is
good, so they recommended that I convert the linear LVs to mirrors (to copy
the data to a new drive).
I formatted a new drive (/dev/sde) identical to /dev/sd[ab], added it to a
mdadm --create --level=1 -n 2 /dev/md2 /dev/sde2 missing
pvcreate /dev/md2
vgextend VolGroup00 /dev/md2
lvconvert -m1 --mirrorlog=mirrored --alloc=anywhere /dev/VolGroup00/srvlv
lvconvert -m1 --mirrorlog=mirrored --alloc=anywhere /dev/VolGroup00/rootlv
lvconvert -m1 --mirrorlog=mirrored --alloc=anywhere /dev/VolGroup00/swaplv
lvconvert -m1 --mirrorlog=mirrored --alloc=anywhere /dev/VolGroup00/varlv
lvconvert -m1 --mirrorlog=mirrored --alloc=anywhere /dev/VolGroup00/tmplv
(in case you're wondering, when this completed, I added /dev/sda1 to
/dev/md2, so /dev/md2 is a fuctional mdadm RAID1).
Now... the problem ;)....
When I run 'lvs' I see that the LV mirror extents were created on the same
# lvs -a -o +devices
LV VG Attr LSize Origin Snap% Move Log
Copy% Convert Devices
rootlv VolGroup00 mwi-ao 10.00G
rootlv_mlog 100.00 rootlv_mimage_0(0),rootlv_mimage_1(0)
[rootlv_mimage_0] VolGroup00 iwi-ao 10.00G
/dev/md1(0)
[rootlv_mimage_1] VolGroup00 iwi-ao 10.00G
/dev/md1(4036)
[rootlv_mlog] VolGroup00 mwa-ao 32.00M
100.00 rootlv_mlog_mimage_0(0),rootlv_mlog_mimage_1(0)
[rootlv_mlog_mimage_0] VolGroup00 iwi-ao 32.00M
/dev/md2(4)
[rootlv_mlog_mimage_1] VolGroup00 iwi-ao 32.00M
/dev/md1(4356)
My long-term goal is to convert the mirror back to a linear volume using
just /dev/md2, i.e.
lvconvert -m0 /dev/VolGroup00/rootlv /dev/md1
lvconvert -m0 /dev/VolGroup00/swaplv /dev/md1
lvconvert -m0 /dev/VolGroup00/srvlv /dev/md1
lvconvert -m0 /dev/VolGroup00/varlv /dev/md1
lvconvert -m0 /dev/VolGroup00/tmplv /dev/md1
... so it seems problematic that both extents are on /dev/md1.
I thought lvm was supposed to use both physical devices. Do you know why
both extents were created on the same physical device?
Should I create another extent on /dev/md2 before I remove the /dev/md1
extent(s)? If so, can you give me an example of how to use lvconvert in
this case?
Thanks in advance!
best,
Julie
_______________________________________________
linux-lvm mailing list
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

[Julie Ashworth]

Post by emmanuel segura
i don't understand why you are using two mirror technologies together, i'm
using in lvm mirror without mdadm and never found problems

---end quoted text---

hi Emmanuel,
I personally prefer to use mdadm RAID1, and I'm only using lvm mirroring to recover from an mdadm failure. If you want to learn more, I encourage you to join the mailing list for mdadm issues (linux-***@vger.kernel.org).
best,
Julie

[Julie Ashworth]

I doubt anybody else will have this problem, but just in case...
my solution was to upgrade lvm2, convert my mirrors to linear arrays, and recreate the mirrors. After the lvm2 upgrade, I didn't see allocation errors when creating the mirrored log.

lvconvert -m0 /dev/VolGroup00/lvname
lvconvert -m1 --mirrorlog=mirrored /dev/VolGroup00/lvname

best,
Julie

[emmanuel segura]

Hi Julie,

Thanks for your comment, but you mirror over mirror, i don't need to learn
anything like that :)

Post by Julie Ashworth
I doubt anybody else will have this problem, but just in case...
my solution was to upgrade lvm2, convert my mirrors to linear arrays, and
recreate the mirrors. After the lvm2 upgrade, I didn't see allocation
errors when creating the mirrored log.
lvconvert -m0 /dev/VolGroup00/lvname
lvconvert -m1 --mirrorlog=mirrored /dev/VolGroup00/lvname
best,
Julie
_______________________________________________
linux-lvm mailing list
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

[emmanuel segura]

Hi Julie,

Thanks for your comment, but you are using mirror over mirror, i don't need
to learn anything like that :)

Post by emmanuel segura
Hi Julie,
Thanks for your comment, but you mirror over mirror, i don't need to learn
anything like that :)
I doubt anybody else will have this problem, but just in case...

Post by Julie Ashworth
my solution was to upgrade lvm2, convert my mirrors to linear arrays, and
recreate the mirrors. After the lvm2 upgrade, I didn't see allocation
errors when creating the mirrored log.
lvconvert -m0 /dev/VolGroup00/lvname
lvconvert -m1 --mirrorlog=mirrored /dev/VolGroup00/lvname
best,
Julie
_______________________________________________
linux-lvm mailing list
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

--
esta es mi vida e me la vivo hasta que dios quiera