[linux-lvm] Simulating LVM Mirror Failure and recovery

[yogeen honnavar]

Dear users,

We have created a mirrored logical volume(lv1) in a volume group vg1. we have 2 disks /dev/sda and /dev/sdb. output of lvs command shows that mirror image_0 is stored on /dev/sda3 and image_1 on /dev/sdb3 and mirror log on /dev/sdb2. Copy % shows 100%. so mirroring is setup correctly.

now we wish to simulate mirror leg failure and recovery. as per the redhat lvm administrators guide we used the following command to trigger the failure of mirror leg1

dd if=/dev/zero of=/dev/vg1/lv1 count=10

as per the redhat guide this should result in mirror leg failure and any subsequent write activity to the mirror should convert the mirror into linear volume. but the output of lvs command still shows copy % as 100%. also data in filesystem of lv1 is erased and it is converted into read only filesystem.

can anyone please let us know the correct way to simulate lvm mirror failure and recovery.

thanks and regards
-yogeen


Add more friends to your messenger and enjoy! Go to http://messenger.yahoo.com/invite/

[Klaus Strebel]

Post by yogeen honnavar
Dear users,
We have created a mirrored logical volume(lv1) in a volume group vg1. we have 2 disks /dev/sda and /dev/sdb. output of lvs command shows that mirror image_0 is stored on /dev/sda3 and image_1 on /dev/sdb3 and mirror log on /dev/sdb2. Copy % shows 100%. so mirroring is setup correctly.
now we wish to simulate mirror leg failure and recovery. as per the redhat lvm administrators guide we used the following command to trigger the failure of mirror leg1
dd if=/dev/zero of=/dev/vg1/lv1 count=10
as per the redhat guide this should result in mirror leg failure and any subsequent write activity to the mirror should convert the mirror into linear volume. but the output of lvs command still shows copy % as 100%. also data in filesystem of lv1 is erased and it is converted into read only filesystem.
can anyone please let us know the correct way to simulate lvm mirror failure and recovery.
thanks and regards
-yogeen

Hi yogeen,

i guess you wanted to do a 'dd if=/dev/zero of=/dev/sdb3 count=10', with
your command, you filled the logical volume with zeros and that is
mirrored perfectly on both underlying devices - as it should ;-).

[vu pham]

Post by yogeen honnavar
Dear users,
We have created a mirrored logical volume(lv1) in a volume group vg1. we have 2 disks /dev/sda and /dev/sdb. output of lvs command shows that mirror image_0 is stored on /dev/sda3 and image_1 on /dev/sdb3 and mirror log on /dev/sdb2. Copy % shows 100%. so mirroring is setup correctly.
now we wish to simulate mirror leg failure and recovery. as per the redhat lvm administrators guide we used the following command to trigger the failure of mirror leg1
dd if=/dev/zero of=/dev/vg1/lv1 count=10
as per the redhat guide this should result in mirror leg failure and any subsequent write activity to the mirror should convert the mirror into linear volume. but the output of lvs command still shows copy % as 100%. also data in filesystem of lv1 is erased and it is converted into read only filesystem.
can anyone please let us know the correct way to simulate lvm mirror failure and recovery.
thanks and regards

I think to simulate error failure you have to write to the underlying
devices, not the mirrored device. When you write to the mirrored device,
lvl1 in this case, data is mirrored so there is no failure at all. So I
think you have to dd to either /dev/sda3 or /dev/sdb3.

Vu

[Alasdair G Kergon]

Post by yogeen honnavar
Dear users,
dd if=/dev/zero of=/dev/vg1/lv1 count=10
as per the redhat guide this should result in mirror leg failure and any

Can you provide the reference - where does it say this?

Alasdair

[Eric Brunson]

Post by Alasdair G Kergon

Post by yogeen honnavar
Dear users,
dd if=/dev/zero of=/dev/vg1/lv1 count=10
as per the redhat guide this should result in mirror leg failure and any

Can you provide the reference - where does it say this?
Alasdair

I believe this is the reference.

http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5/html/Cluster_Logical_Volume_Manager/mirrorrecover.html

Their example may be in error.

[Eric Brunson]

Post by Eric Brunson

Post by Alasdair G Kergon

Post by yogeen honnavar
Dear users,
dd if=/dev/zero of=/dev/vg1/lv1 count=10
as per the redhat guide this should result in mirror leg failure and any

Can you provide the reference - where does it say this?
Alasdair

I believe this is the reference.
http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5/html/Cluster_Logical_Volume_Manager/mirrorrecover.html
Their example may be in error.

Upon review, I think it's correct, but worded awkwardly:

In this example, the primary leg of the mirror |/dev/sda1| fails.
Any write activity to the mirrored volume causes LVM to detect the
failed mirror. When this occurs, LVM converts the mirror into a
single linear volume. In this case, to trigger the conversion, we
execute a |dd| command

[***@link-08 ~]#*|dd if=/dev/zero of=/dev/vg/groupfs count=10|*
10+0 records in
10+0 records out


I believe the dd is not to cause the failure, it's simply to generate
write activity to the volume group after the mirror leg has been caused
to fail through other means not documented.

Anyone else think that's the case?

e.

Post by Eric Brunson
_______________________________________________
linux-lvm mailing list
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

[Lupe Christoph]

Maybe it's because I'm not a native speaker, but I find it difficult to
understand this differently.

Post by Eric Brunson
In this example, the primary leg of the mirror |/dev/sda1| fails.

It fails. Not "we make it fail". No indication what makes it fail.

Post by Eric Brunson
Any write activity to the mirrored volume causes LVM to detect the
failed mirror. When this occurs, LVM converts the mirror into a
single linear volume. In this case, to trigger the conversion, we
execute a |dd| command

"we execute a |dd| command" "to trigger the *conversion*". Not to make
it fail.

Post by Eric Brunson
I believe the dd is not to cause the failure, it's simply to generate
write activity to the volume group after the mirror leg has been caused
to fail through other means not documented.

... as they said ...

So the dd command the OP sent was correct after all. He just
misunderstood its purpose.

Now back to our original topic - how do you make a mirror copy fail? ;-)

Lupe Christoph

[Don Bishop]

Hey Eric

Nice to see your still around. How's it all going?

best,
Don B

Post by Eric Brunson

Post by Eric Brunson

Post by Alasdair G Kergon

Post by yogeen honnavar
Dear users,
dd if=/dev/zero of=/dev/vg1/lv1 count=10
as per the redhat guide this should result in mirror leg failure and any

Can you provide the reference - where does it say this?
Alasdair

I believe this is the reference.
http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5/html/Cluster_Logical_Volume_Manager/mirrorrecover.html
Their example may be in error.

In this example, the primary leg of the mirror |/dev/sda1| fails.
Any write activity to the mirrored volume causes LVM to detect the
failed mirror. When this occurs, LVM converts the mirror into a
single linear volume. In this case, to trigger the conversion, we
execute a |dd| command
10+0 records in
10+0 records out
I believe the dd is not to cause the failure, it's simply to generate
write activity to the volume group after the mirror leg has been caused
to fail through other means not documented.
Anyone else think that's the case?
e.

Post by Eric Brunson
_______________________________________________
linux-lvm mailing list
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

_______________________________________________
linux-lvm mailing list
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

CONFIDENTIALITY NOTICE
This electronic mail transmission and any accompanying documents contain
information belonging to the sender that may be confidential and legally
privileged. This information is intended only for the use of the
individual or entity to whom this electronic mail transmission was sent
as indicated above. If you are not the intended recipient, any
disclosure, copying, distribution, or action taken in reliance on the
contents of the information contained in this transmissions is strictly
prohibited. If you have received this transmission in error, please
reply by e-mail to let me know and then permanently delete the message.
Thank you.

[Tomasz Chmielewski]

Post by vu pham

Post by yogeen honnavar
as per the redhat guide this should result in mirror leg failure and
any subsequent write activity to the mirror should convert the mirror
into linear volume. but the output of lvs command still shows copy %
as 100%. also data in filesystem of lv1 is erased and it is converted
into read only filesystem.
can anyone please let us know the correct way to simulate lvm mirror
failure and recovery.
thanks and regards

I think to simulate error failure you have to write to the underlying
devices, not the mirrored device. When you write to the mirrored device,
lvl1 in this case, data is mirrored so there is no failure at all. So I
think you have to dd to either /dev/sda3 or /dev/sdb3.

How will LVM distinguish between a valid and invalid copy in that case?

Are you sure that on reads, LVM will return data from the "good" copy,
not the one you just overwrote with "dd if=/dev/zero of=/dev/sdb
seek=xxx bs=1M count=10"?

[vu pham]

Post by Tomasz Chmielewski

Post by vu pham

Post by yogeen honnavar
as per the redhat guide this should result in mirror leg failure and
any subsequent write activity to the mirror should convert the mirror
into linear volume. but the output of lvs command still shows copy %
as 100%. also data in filesystem of lv1 is erased and it is converted
into read only filesystem.
can anyone please let us know the correct way to simulate lvm mirror
failure and recovery.
thanks and regards

I think to simulate error failure you have to write to the underlying
devices, not the mirrored device. When you write to the mirrored
device, lvl1 in this case, data is mirrored so there is no failure at
all. So I think you have to dd to either /dev/sda3 or /dev/sdb3.

How will LVM distinguish between a valid and invalid copy in that case?
Are you sure that on reads, LVM will return data from the "good" copy,
not the one you just overwrote with "dd if=/dev/zero of=/dev/sdb
seek=xxx bs=1M count=10"?

Good question. I thought that dd will destroy the lvm information at the
beginning of the partition which makes the leg broken. I made the
following test and the result is odd:

- created mirrored logical volumn mrlv1 on /dev/sdc1 and /dev/sdd1,
having /dev/sdd2 as log device
- mkfs.ext3 /dev/vg1/mrlv1, mount it and copy files onto it.
- dd if=/dev/zero of=/dev/sdd1
- read / write onto the mounted mirror lvm.

The output of lvs shows:
- lvm information on /dev/sdd2 is destroyed
- the 2nd mirror, which is /dev/sdd2, is *unknown device*
- the mirror is still ok

And no lvm log in /var/log/messages.

[***@xen3 ~]# lvs -a -o +devices
Couldn't find device with uuid 'ZUOCeZ-Ssb2-0fP6-n5YY-sz5n-TtXe-b7eL3a'.
Couldn't find device with uuid 'ZUOCeZ-Ssb2-0fP6-n5YY-sz5n-TtXe-b7eL3a'.
Couldn't find device with uuid 'ZUOCeZ-Ssb2-0fP6-n5YY-sz5n-TtXe-b7eL3a'.
Couldn't find device with uuid 'ZUOCeZ-Ssb2-0fP6-n5YY-sz5n-TtXe-b7eL3a'.
Couldn't find device with uuid 'ZUOCeZ-Ssb2-0fP6-n5YY-sz5n-TtXe-b7eL3a'.
Couldn't find device with uuid 'ZUOCeZ-Ssb2-0fP6-n5YY-sz5n-TtXe-b7eL3a'.
Couldn't find device with uuid 'ZUOCeZ-Ssb2-0fP6-n5YY-sz5n-TtXe-b7eL3a'.
LV VG Attr LSize Origin Snap% Move Log
[... deleted unrelated devices ...]

mrlv1 vg1 mwi-ao 200.00M mrlv1_mlog
100.00 mrlv1_mimage_0(0),mrlv1_mimage_1(0)
[mrlv1_mimage_0] vg1 iwi-ao 200.00M
/dev/sdc1(0)
[mrlv1_mimage_1] vg1 iwi-ao 200.00M
unknown device(0)
[mrlv1_mlog] vg1 lwi-ao 4.00M
/dev/sdd2(0)



Vu

[Lupe Christoph]

Post by yogeen honnavar
dd if=/dev/zero of=/dev/vg1/lv1 count=10
as per the redhat guide this should result in mirror leg failure and any subsequent write activity to the mirror should convert the mirror into linear volume. but the output of lvs command still shows copy % as 100%. also data in filesystem of lv1 is erased and it is converted into read only filesystem.

Apart from the problem with the of= parameter that was already pointed
out - why do you assume writing *anything* *anywhere* would simulate a
mirror failure? The most you do simulate with that is mirror
*corruption*.

To simulate a failure, you have to somehow disable a drive. Unplug a
cable, force it to spin down (assuming that it is not automatically spun
up again, I dunno), change the zoning, whatever is needed to prevent
access to the device.

Post by yogeen honnavar
can anyone please let us know the correct way to simulate lvm mirror failure and recovery.

Depends a lot on your hardware. E.g. the system I use for testing are
about 400 km away, so I'd rather not unplug anything...

HTH,
Lupe Christoph

[m***@us.ibm.com]

Post by Lupe Christoph

Post by yogeen honnavar
dd if=/dev/zero of=/dev/vg1/lv1 count=10
as per the redhat guide this should result in mirror leg failure and any

subsequent write activity to the mirror should convert the mirror into linear
volume. but the output of lvs command still shows copy % as 100%. also data in
filesystem of lv1 is erased and it is converted into read only filesystem.

Post by Lupe Christoph
Apart from the problem with the of= parameter that was already pointed
out - why do you assume writing *anything* *anywhere* would simulate a
mirror failure? The most you do simulate with that is mirror
*corruption*.

You are absolutely right.

Post by Lupe Christoph

Post by yogeen honnavar
can anyone please let us know the correct way to simulate lvm mirror failure
and recovery.

Depends a lot on your hardware. E.g. the system I use for testing are
about 400 km away, so I'd rather not unplug anything...

I usually run "echo 1 > /sys/block/sda/device/delete" to make /dev/sda
disappear and fail for LVM tests. You can fail the disk in a lot of ways
depending on your hardware, but this trick works with almost every
device!

[Takahiro Yasui]

Post by m***@us.ibm.com
I usually run "echo 1 > /sys/block/sda/device/delete" to make /dev/sda
disappear and fail for LVM tests. You can fail the disk in a lot of ways
depending on your hardware, but this trick works with almost every
device!

# echo offline > /sys/block/sda/device/state

might work as well. In this case, you can recover the failure by

# echo running > /sys/block/sda/device/state

Thanks,
Taka

[yogeen honnavar]

hi all,

Thanks for all the replies. as pointed out by Eric brunson i was referring to the following document

http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5/html/Cluster_Logical_Volume_Manager/mirrorrecover.html

i used the below command to simulate failed sda and then wrote some data to the mounted file system to trigger LVM conversion to linear mode.

# echo offline > /sys/block/sda/device/state

thanks to all.
regards
-yogeen

Subject: Re: [linux-lvm] Simulating LVM Mirror Failure and recovery
Date: Thursday, 16 April, 2009, 9:32 PM

Post by m***@us.ibm.com
I usually run "echo 1 >

/sys/block/sda/device/delete" to make /dev/sda

Post by m***@us.ibm.com
disappear and fail for LVM tests. You can fail the

disk in a lot of ways

Post by m***@us.ibm.com
depending on your hardware, but this trick works with

almost every

Post by m***@us.ibm.com
device!

# echo offline > /sys/block/sda/device/state
might work as well. In this case, you can recover the
failure by
# echo running > /sys/block/sda/device/state
Thanks,
Taka
_______________________________________________
linux-lvm mailing list
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Add more friends to your messenger and enjoy! Go to http://messenger.yahoo.com/invite/