[linux-lvm] LE to PE mapping

[Allen, Jack]

Hello:
I need to find a command on RHEL 5.8 that will show each LE to PE mapping for a LV. All the command I have looked at so far (lvs, pvs, lvdisplay and pvdisplay) using various option only show the start PE and how many.

I am dealing with an SAN read performance problem and some think it is the way the LE are mapped to the PE that is causing more I/O to take place on specific disk. The database application does reads and writes directly to the LV, there is no file system with files. The database block size is 8k and for the most part is very random across many LVs.

The configuration is a lot bigger than I am going to provide information here to make it a little simpler as to what I am trying to find out.

Let's assume 4 LUNs (PV) at 10G each.
The PVs are put in vg01 and 4 LVs are created using the following commands:

lvcreate -L2g -i4 -n lv00 vg01
lvcreate -L2g -i4 -n lv01 vg01
lvcreate -L2g -i4 -n lv02 vg01
lvcreate -L2g -i4 -n lv03 vg01

The strip size defaulted to 16K per the lvdisplay command. When doing the commands to list the LE and PE information it will show that lv00 starts at PE 0 on each PV and then what ever number of PEs it takes to make 2g across the PVs. Let's say 4 to make it simple, so that would be 16 in all. Now were the PE allocated like:

LE0 -> PV0 PE0
LE1 -> PV1 PE0
LE2 -> PV2 PE0
LE3 -> PV3 PE0
LE4 -> PV0 PE1
LE5 -> PV1 PE1
~~~ ~~ ~~~ ~~~
LE15 -> PV3 PE3

Or were they allocated like:

LE0 -> PV0 PE0
LE1 -> PV0 PE1
LE2 -> PV0 PE2
LE3 -> PV0 PE3
LE4 -> PV1 PE0
LE5 -> PV1 PE1
~~~ ~~ ~~~ ~~~
LE15 -> PV3 PE3

In this simple example it does not make a lot of difference, but say there were 512 PEs then it does as far as to which PV reads are done from with random reads. This is a database application with many users access different parts of the data is what makes it random and there could be many doing reads at the same time. During backups the blocks (PE) will be read sequential and if the allocation is done the second way then many reads will be done on the first PV and none on the other than the then second PV and non on the others and so on.

Something else I need some clarification on is the PE size of 4MB and the strip size of 16K. Are the PEs not allocated as shown above in either example?

Are the PE allocated in 4MB chunks?

Then what does the 16K do?


-------------------------
Jackson C. Allen
McKesson Provider Technologies
5995 Windward Parkway
Alpharetta, GA 30005
[phone]
***@McKesson.com<mailto:***@McKesson.com>
Confidentiality Notice: This e-mail message, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message.

[Peter Rajnoha]

Post by Allen, Jack
I need to find a command on RHEL 5.8 that will show each LE to
PE mapping for a LV. All the command I have looked at so far (lvs, pvs,
lvdisplay and pvdisplay) using various option only show the start PE and
how many.

Try lvdisplay --maps (or pvdisplay --maps for the other way round).

Post by Allen, Jack
Something else I need some clarification on is the PE size of
4MB and the strip size of 16K. Are the PEs not allocated as shown above
in either example?
Are the PE allocated in 4MB chunks?

Yes, it's 4MB extent size by default if not speficied otherwise
using vgcreate/vgchange -s.

Post by Allen, Jack
Then what does the 16K do?

That is stripe size - the data will be placed/rotated over disks
by this amount for each disk while the extent size is the basic unit
for mapping LV segments to the underlying disks. Of course, if you have,
for example, 2 stripes, you need an LV which is mapped to at least 2
different disks which also means at least 2 PEs to fulfill this condition
(otherwise, we couldn't map stripes onto different disks which would defeat
the essence of striping) - LVM does this rounding for you and
modifies the LV size automatically to make the stripe count and
LV mapping combination possible.

Post by Allen, Jack
-------------------------
Jackson C. Allen
McKesson Provider Technologies
5995 Windward Parkway
Alpharetta, GA 30005
[phone]
Confidentiality Notice: This e-mail message, including any attachments,
is for the sole use of the intended recipient(s) and may contain
confidential and privileged information. Any unauthorized review, use,
disclosure or distribution is prohibited. If you are not the intended
recipient, please contact the sender by reply e-mail and destroy all
copies of the original message.
_______________________________________________
linux-lvm mailing list
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

[Allen, Jack]

Peter:
Thanks for the reply, but that does not show what I am looking for. The system has 16 PVs and the LV is stripped at 16K.

lvdisplay --maps, shows:
--- Segments ---
Logical extent 0 to 1295:
Type striped
Stripes 16
Stripe size 16 KB
Stripe 0:
Physical volume /dev/mapper/mpath17
Physical extents 0 to 80
Stripe 1:
Physical volume /dev/mapper/mpath18
Physical extents 0 to 80
~~~ ~~~ ~~~ ~~~ ~~~ ~~~ ~~~ ~~~ ~~~ ~~~

This does not show exactly which LE/PE is where on each PV. I assume and did some test to determine if it was really stripped 16K on the first PV and then 16K on the next PV. But the output above does not indicate that. It could just as easily indicate that 1/16th (81) of the PE (8k blocks 0 to 162) were allocated on the first PV, and then (8k blocks 163 to 325) are on the second PV, etc. I wanted something that would show the LE/PE number to location on each PV.

I understand because the LV was the first thing allocated on the PV, that the first 81 PE on each were used. So it just shows the range instead of each LE/PE.

The output of pvdisplay --maps, just shows that PE extent 0 to 80 is LE 0 to 1295 range on each PV. So with the database application using 8K blocks I cannot really tell that 8K blocks 0 and 1 are on PV0 and 8k blocks 2 and 3 are on PV1, etc.. and then 8k blocks 32 and 33 are on PV0 just after 8k blocks 0 and 1.

I wanted something like:
Logical extent 0 to 1295:
Type striped
Stripes 16
Stripe size 16 KB
Stripe 0:
Physical volume /dev/mapper/mpath17
Physical extent 0
Physical extent 15
Physical extent 32
~~~~~~~~ ~~~~~~ ~~~~

-------------------------
Jackson C. Allen
McKesson Provider Technologies
5995 Windward Parkway
Alpharetta, GA 30005
[phone]
***@McKesson.com
Confidentiality Notice: This e-mail message, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential and privileged information.  Any unauthorized review, use, disclosure or distribution is prohibited.  If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message.






-----Original Message-----
From: Peter Rajnoha [mailto:***@redhat.com]
Sent: Monday, September 15, 2014 2:41 AM
To: Allen, Jack
Cc: LVM general discussion and development
Subject: Re: [linux-lvm] LE to PE mapping

Post by Allen, Jack
I need to find a command on RHEL 5.8 that will show each LE to
PE mapping for a LV. All the command I have looked at so far (lvs,
pvs, lvdisplay and pvdisplay) using various option only show the start
PE and how many.

Try lvdisplay --maps (or pvdisplay --maps for the other way round).

Post by Allen, Jack
Something else I need some clarification on is the PE size of
4MB and the strip size of 16K. Are the PEs not allocated as shown
above in either example?
Are the PE allocated in 4MB chunks?

Yes, it's 4MB extent size by default if not speficied otherwise using vgcreate/vgchange -s.

Post by Allen, Jack
Then what does the 16K do?

That is stripe size - the data will be placed/rotated over disks by this amount for each disk while the extent size is the basic unit for mapping LV segments to the underlying disks. Of course, if you have, for example, 2 stripes, you need an LV which is mapped to at least 2 different disks which also means at least 2 PEs to fulfill this condition (otherwise, we couldn't map stripes onto different disks which would defeat the essence of striping) - LVM does this rounding for you and modifies the LV size automatically to make the stripe count and LV mapping combination possible.

Post by Allen, Jack
-------------------------
Jackson C. Allen
McKesson Provider Technologies
5995 Windward Parkway
Alpharetta, GA 30005
[phone]
Confidentiality Notice: This e-mail message, including any
attachments, is for the sole use of the intended recipient(s) and may
contain confidential and privileged information. Any unauthorized
review, use, disclosure or distribution is prohibited. If you are not
the intended recipient, please contact the sender by reply e-mail and
destroy all copies of the original message.
_______________________________________________
linux-lvm mailing list
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

--
Peter

[Allen, Jack]

Hello:
I have only received 1 direct reply about this. Nobody else has anything to add?

Another question: Where is the mapping stored if I wanted to write a program to display it the way I want?

-------------------------
Jackson C. Allen
McKesson Provider Technologies
5995 Windward Parkway
Alpharetta, GA 30005
[phone]
***@McKesson.com<mailto:***@McKesson.com>
Confidentiality Notice: This e-mail message, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message.




From: linux-lvm-***@redhat.com [mailto:linux-lvm-***@redhat.com] On Behalf Of Allen, Jack
Sent: Friday, September 12, 2014 6:02 PM
To: linux-***@redhat.com
Subject: [linux-lvm] LE to PE mapping

Hello:
I need to find a command on RHEL 5.8 that will show each LE to PE mapping for a LV. All the command I have looked at so far (lvs, pvs, lvdisplay and pvdisplay) using various option only show the start PE and how many.

I am dealing with an SAN read performance problem and some think it is the way the LE are mapped to the PE that is causing more I/O to take place on specific disk. The database application does reads and writes directly to the LV, there is no file system with files. The database block size is 8k and for the most part is very random across many LVs.

The configuration is a lot bigger than I am going to provide information here to make it a little simpler as to what I am trying to find out.

Let's assume 4 LUNs (PV) at 10G each.
The PVs are put in vg01 and 4 LVs are created using the following commands:

lvcreate -L2g -i4 -n lv00 vg01
lvcreate -L2g -i4 -n lv01 vg01
lvcreate -L2g -i4 -n lv02 vg01
lvcreate -L2g -i4 -n lv03 vg01

The strip size defaulted to 16K per the lvdisplay command. When doing the commands to list the LE and PE information it will show that lv00 starts at PE 0 on each PV and then what ever number of PEs it takes to make 2g across the PVs. Let's say 4 to make it simple, so that would be 16 in all. Now were the PE allocated like:

LE0 -> PV0 PE0
LE1 -> PV1 PE0
LE2 -> PV2 PE0
LE3 -> PV3 PE0
LE4 -> PV0 PE1
LE5 -> PV1 PE1
~~~ ~~ ~~~ ~~~
LE15 -> PV3 PE3

Or were they allocated like:

LE0 -> PV0 PE0
LE1 -> PV0 PE1
LE2 -> PV0 PE2
LE3 -> PV0 PE3
LE4 -> PV1 PE0
LE5 -> PV1 PE1
~~~ ~~ ~~~ ~~~
LE15 -> PV3 PE3

In this simple example it does not make a lot of difference, but say there were 512 PEs then it does as far as to which PV reads are done from with random reads. This is a database application with many users access different parts of the data is what makes it random and there could be many doing reads at the same time. During backups the blocks (PE) will be read sequential and if the allocation is done the second way then many reads will be done on the first PV and none on the other than the then second PV and non on the others and so on.

Something else I need some clarification on is the PE size of 4MB and the strip size of 16K. Are the PEs not allocated as shown above in either example?

Are the PE allocated in 4MB chunks?

Then what does the 16K do?

-------------------------
Jackson C. Allen
McKesson Provider Technologies
5995 Windward Parkway
Alpharetta, GA 30005
[phone]
***@McKesson.com<mailto:***@McKesson.com>
Confidentiality Notice: This e-mail message, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message.

[Brassow Jonathan]

Post by Allen, Jack
I have only received 1 direct reply about this. Nobody else has anything to add?
Another question: Where is the mapping stored if I wanted to write a program to display it the way I want?

I don't think there is a way to do exactly what you want that I am aware of. You need to know three things: the order of the devices, the stripe size, and the overall size. You can compute what you want from that, but I don't think LVM will do that for you.

You can get those three fields from 'lvs', but I think you will have to do some scripting to get what you want from there.

As to the question of the mapping table... The mapping table is not stored anywhere. It is constructed from the LVs properties that are stored in the LVM metadata. You can, however, get the table used to construct the device by calling 'dmsetup table <DM device>'. The DM device is what /dev/<vg>/<lv> links to.

brassow