[linux-lvm] Performance penalty for 4k requests on thin provisioned volume

[Dale Stephenson]

Distribution: centos-release-7-3.1611.el7.centos.x86_64
Kernel: Linux 3.10.0-514.26.2.el7.x86_64
LVM: 2.02.166(2)-RHEL7 (2016-11-16)

Volume group consisted of an 8-drive SSD (500G drives) array, plus an additional SSD of the same size. The array had 64 k stripes.
Thin pool had -Zn option and 512k chunksize (full stripe), size 3T with metadata volume 16G. data was entirely on the 8-drive raid, metadata was entirely on the 9th drive.
Virtual volume “thin” was 300 GB. I also filled it with dd so that it would be fully provisioned before the test.
Volume “thick” was also 300GB, just an ordinary volume also entirely on the 8-drive array.

Four tests were run directlyagainst each volume using fio-2.2.8, random read, random write, sequential read, sequential write. Single thread, 4k blocksize, 90s run time.

The thin volume should much worse performance in this test:

Thin:
Random read : io=51607MB, bw=587168KB/s, iops=146792, runt= 90001msec
Random write: io=46530MB, bw=529406KB/s, iops=132351, runt= 90001msec
Sequential read : io=176561MB, bw=1961.7MB/s, iops=10462, runt= 90006msec
Sequential write: io=162451MB, bw=1804.1MB/s, iops=9626, runt= 90006msec

Thick:
Random read : io=88350MB, bw=981.68MB/s, iops=251303, runt= 90001msec
Random write: io=77905MB, bw=886372KB/s, iops=221592, runt= 90001msec
Sequential read : io=89095MB, bw=989.96MB/s, iops=253421, runt= 90001msec
Sequential write: io=77742MB, bw=884520KB/s, iops=221130, runt= 90001msec

As you can see, the number of iops for the thin-provisioned volume is dramatically less than for thick volumes. With the default 64k chunk size this is also true.

Running the same fio test with 512k request size shows similar performance for thick and thin volumes, so the maximum potential throughput of the thin volume seems broadly similar to thin volumes. But I’d like the maximum number of iops to at least be within shouting distance of thick volumes. Is there anything that can be done to improve that with either this version of lvm/device-mapper or later versions? I typically use xfs on top of volumes, and xfs is very fond of 4k requests.

Dale Stephenson

[Zdenek Kabelac]

Post by Dale Stephenson
Distribution: centos-release-7-3.1611.el7.centos.x86_64
Kernel: Linux 3.10.0-514.26.2.el7.x86_64
LVM: 2.02.166(2)-RHEL7 (2016-11-16)
Volume group consisted of an 8-drive SSD (500G drives) array, plus an additional SSD of the same size. The array had 64 k stripes.
Thin pool had -Zn option and 512k chunksize (full stripe), size 3T with metadata volume 16G. data was entirely on the 8-drive raid, metadata was entirely on the 9th drive.
Virtual volume “thin” was 300 GB. I also filled it with dd so that it would be fully provisioned before the test.
Volume “thick” was also 300GB, just an ordinary volume also entirely on the 8-drive array.
Four tests were run directlyagainst each volume using fio-2.2.8, random read, random write, sequential read, sequential write. Single thread, 4k blocksize, 90s run time.

Hi

Can you please provide output of:

lvs -a -o+stripes,stripesize,seg_pe_ranges

so we can see how is your stripe placed on devices ?

SSD typically do needs ideally write 512K chunks.
(something like 'lvcreate -LXXX -i8 -I512k vgname' )

Wouldn't be 'faster' to just concatenate 8 disks together instead of striping
- or stripe only across 2 disk - and then you concatenate 4 such striped areas...

64k stripes do not seem to look like ideal match in this case of 3 disk with
512K blocks

Regards

Zdenek

[Dale Stephenson]

Post by Zdenek Kabelac

Post by Dale Stephenson
Distribution: centos-release-7-3.1611.el7.centos.x86_64
Kernel: Linux 3.10.0-514.26.2.el7.x86_64
LVM: 2.02.166(2)-RHEL7 (2016-11-16)
Volume group consisted of an 8-drive SSD (500G drives) array, plus an additional SSD of the same size. The array had 64 k stripes.
Thin pool had -Zn option and 512k chunksize (full stripe), size 3T with metadata volume 16G. data was entirely on the 8-drive raid, metadata was entirely on the 9th drive.
Virtual volume “thin” was 300 GB. I also filled it with dd so that it would be fully provisioned before the test.
Volume “thick” was also 300GB, just an ordinary volume also entirely on the 8-drive array.
Four tests were run directlyagainst each volume using fio-2.2.8, random read, random write, sequential read, sequential write. Single thread, 4k blocksize, 90s run time.

Hi
lvs -a -o+stripes,stripesize,seg_pe_ranges
so we can see how is your stripe placed on devices ?

Sure, thank you for your help:
# lvs -a -o+stripes,stripesize,seg_pe_ranges
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert #Str Stripe PE Ranges
[lvol0_pmspare] volgr0 ewi------- 16.00g 1 0 /dev/md127:867328-871423
thick volgr0 -wi-a----- 300.00g 1 0 /dev/md127:790528-867327
thin volgr0 Vwi-a-t--- 300.00g thinpool 100.00 0 0
thinpool volgr0 twi-aot--- 3.00t 9.77 0.13 1 0 thinpool_tdata:0-786431
[thinpool_tdata] volgr0 Twi-ao---- 3.00t 1 0 /dev/md127:0-786431
[thinpool_tmeta] volgr0 ewi-ao---- 16.00g 1 0 /dev/sdb4:0-4095

md127 is an 8-drive RAID 0

As you can see, there’s no lvm striping; I rely on the software RAID underneath for that. Both thick and thin lvols are on the same PV.

Post by Zdenek Kabelac
SSD typically do needs ideally write 512K chunks.

I could create the md to use 512k chunks for RAID 0, but I wouldn’t expect that to have any impact on a single threaded test using 4k request size. Is there a hidden relationship that I’m unaware of?

Post by Zdenek Kabelac
(something like 'lvcreate -LXXX -i8 -I512k vgname’)

Would making lvm stripe on top of an md that already stripes confer any performance benefit in general, or for small (4k) requests in particular?

Post by Zdenek Kabelac
Wouldn't be 'faster' to just concatenate 8 disks together instead of striping - or stripe only across 2 disk - and then you concatenate 4 such striped areas…

For sustained throughput I would expect striping of 8 disks to blow away concatenation — however, for small requests I wouldn’t expect any advantage. On a non-redundant array, I would expect a single threaded test using 4k requests is going to end up reading/writing data from exactly one disk regardless of whether the underlying drives are concatenated or stripes.

Post by Zdenek Kabelac
64k stripes do not seem to look like ideal match in this case of 3 disk with 512K blocks

My mistake, I was sloppy with my terminology. My RAID 0 had a 64k *chunksize*, so it was a stripe of 64k chunks, not a 64k stripe. The stripe size was 512K, matching the thinpool chunk size. My understanding is that having the thin pool chunksize match the full stripe size of the underlying array is the best performing choice (at least for throughput).

What is the best choice for handling 4k request sizes?

Thank you,
Dale Stephenson

[Zdenek Kabelac]

Post by Dale Stephenson

Post by Zdenek Kabelac

Post by Dale Stephenson
Distribution: centos-release-7-3.1611.el7.centos.x86_64
Kernel: Linux 3.10.0-514.26.2.el7.x86_64
LVM: 2.02.166(2)-RHEL7 (2016-11-16)
Volume group consisted of an 8-drive SSD (500G drives) array, plus an additional SSD of the same size. The array had 64 k stripes.
Thin pool had -Zn option and 512k chunksize (full stripe), size 3T with metadata volume 16G. data was entirely on the 8-drive raid, metadata was entirely on the 9th drive.
Virtual volume “thin” was 300 GB. I also filled it with dd so that it would be fully provisioned before the test.
Volume “thick” was also 300GB, just an ordinary volume also entirely on the 8-drive array.
Four tests were run directlyagainst each volume using fio-2.2.8, random read, random write, sequential read, sequential write. Single thread, 4k blocksize, 90s run time.

Hi
lvs -a -o+stripes,stripesize,seg_pe_ranges
so we can see how is your stripe placed on devices ?

# lvs -a -o+stripes,stripesize,seg_pe_ranges
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert #Str Stripe PE Ranges
[lvol0_pmspare] volgr0 ewi------- 16.00g 1 0 /dev/md127:867328-871423
thick volgr0 -wi-a----- 300.00g 1 0 /dev/md127:790528-867327
thin volgr0 Vwi-a-t--- 300.00g thinpool 100.00 0 0
thinpool volgr0 twi-aot--- 3.00t 9.77 0.13 1 0 thinpool_tdata:0-786431
[thinpool_tdata] volgr0 Twi-ao---- 3.00t 1 0 /dev/md127:0-786431
[thinpool_tmeta] volgr0 ewi-ao---- 16.00g 1 0 /dev/sdb4:0-4095
md127 is an 8-drive RAID 0
As you can see, there’s no lvm striping; I rely on the software RAID underneath for that. Both thick and thin lvols are on the same PV.

Post by Zdenek Kabelac
SSD typically do needs ideally write 512K chunks.

I could create the md to use 512k chunks for RAID 0, but I wouldn’t expect that to have any impact on a single threaded test using 4k request size. Is there a hidden relationship that I’m unaware of?

Yep - it seems the setup in this case is the best fit.

If you can reevaluate different setups you may possibly get much higher
throughput.

My guess would be - the best targeting layout should be probably striping no
more then 2-3 disks and use bigger striping block.

And then just 'join' 'smaller' arrays together in lvm2 in 1 big LV.

Post by Dale Stephenson

Post by Zdenek Kabelac
(something like 'lvcreate -LXXX -i8 -I512k vgname’)

Would making lvm stripe on top of an md that already stripes confer any performance benefit in general, or for small (4k) requests in particular?

Rule #1 - try to avoid 'over-combining' things together.
- measure performance from 'bottom' upward in your device stack.
If the underlying devices gives poor speed - you can't make it better by any
super0smart disk-layout on top of it.

Post by Dale Stephenson

Post by Zdenek Kabelac
Wouldn't be 'faster' to just concatenate 8 disks together instead of striping - or stripe only across 2 disk - and then you concatenate 4 such striped areas…

For sustained throughput I would expect striping of 8 disks to blow away concatenation — however, for small requests I wouldn’t expect any advantage. On a non-redundant array, I would expect a single threaded test using 4k requests is going to end up reading/writing data from exactly one disk regardless of whether the underlying drives are concatenated or stripes.

It always depends which kind of load you expect the most.

I suspect spreading 4K blocks across 8 SSD is likely very far away from ideal
layout.

Any SSD is typically very bad with 4K blocks - it you want to 'spread' the
load on mores SSDs do not use less the 64K stripe chunks per SSD - this gives
you (8*64) 512K stripe size.

As for thin-pool chunksize - if you plan to use lots of snapshots - keep the
value lowest possible - 64K or 128K thin-pool chunksize.

But I'd still suggest to reevaluate/benchmark setup where you will use much
lower number of SSD for load spreading - and use bigger strip chunks per each
device. This should nicely improve performance in case of 'bigger' writes
and not that much slow things down with 4K loads....

Post by Dale Stephenson
What is the best choice for handling 4k request sizes?

Possibly NVMe can do a better job here.

Regards

Zdenek

[Zdenek Kabelac]

Post by Zdenek Kabelac

Post by Dale Stephenson
md127 is an 8-drive RAID 0
As you can see, there’s no lvm striping; I rely on the software RAID
underneath for that.  Both thick and thin lvols are on the same PV.

Post by Zdenek Kabelac
SSD typically do needs ideally write 512K chunks.

I could create the md to use 512k chunks for RAID 0, but I wouldn’t expect
that to have any impact on a single threaded test using 4k request size.  Is
there a hidden relationship that I’m unaware of?

Yep - it seems the setup in this case is the best fit.

Sorry my typo here - is NOT ;)


Zdenek

[Gionatan Danti]

Post by Zdenek Kabelac
Sorry my typo here - is NOT ;)
Zdenek

Hi Zdenek,
as the only variable is the LVM volume type (fat/thick vs thin), why the
thin volume is slower than the thick one?

I mean: all other things being equal, what is holding back the thin volume?

Thanks.

[Zdenek Kabelac]

Post by Gionatan Danti

Post by Zdenek Kabelac
Sorry my typo here - is NOT ;)
Zdenek

Hi Zdenek,
as the only variable is the LVM volume type (fat/thick vs thin), why the thin
volume is slower than the thick one?
I mean: all other things being equal, what is holding back the thin volume?

So few more question:

What is '/dev/sdb4' ? - is it also some fast SSD ?

([thinpool_tmeta] volgr0 ewi-ao---- 16.00g 1 0 /dev/sdb4:0-4095
- just checking to be sure your metadata device is not placed on rotational
storage device)...

What is your thin-pool chunk size - is it 64K ?
- if your raise thin-pool chunk size up - is it getting any better ?


Regards

Zdenek

[Dale Stephenson]

Post by Zdenek Kabelac

Post by Gionatan Danti

Post by Zdenek Kabelac
Sorry my typo here - is NOT ;)
Zdenek

Hi Zdenek,
as the only variable is the LVM volume type (fat/thick vs thin), why the thin volume is slower than the thick one?
I mean: all other things being equal, what is holding back the thin volume?

Gionatan has hit on the heart of my concern. I ordinarily expect minimal performance hit from remapping, and that’s clearly the case with the thick volume. But it’s not the case with thin, even though I’ve already fully provisioned it. Why is thin so much slower, and what can I do to make it faster?

Post by Zdenek Kabelac
What is '/dev/sdb4' ? - is it also some fast SSD ?

Correct, it’s an SSD identical to the ones used in the array.

Post by Zdenek Kabelac
- just checking to be sure your metadata device is not placed on rotational storage device)…

It is not in this case.

Post by Zdenek Kabelac
What is your thin-pool chunk size - is it 64K ?

In this case it is 512k, because I believed that to be the optimal chunk size for a 512k raid stripe.

Post by Zdenek Kabelac
- if your raise thin-pool chunk size up - is it getting any better ?

I have only tested direct to device at 64k and 512k, 512k performs better. It is not obvious to me why 512k *should* perform better when all the requests are only 4k, except that a given size of metadata would obviously map to a larger size. (That is, if no contiguous chunk indicators are used — in this particular case I would expect the thin provisioning to be completely contiguous.)

Thank you for your help.
Dale Stephenson

[Gionatan Danti]

Post by Zdenek Kabelac
Sorry my typo here - is NOT ;)
Zdenek

Hi Zdenek,
as the only variable is the LVM volume type (fat/thick vs thin), why the
thin volume is slower than the thick one?

I mean: all other things being equal, what is holding back the thin volume?

Thanks.

[Dale Stephenson]

Post by Dale Stephenson
I could create the md to use 512k chunks for RAID 0, but I wouldn’t expect that to have any impact on a single threaded test using 4k request size. Is there a hidden relationship that I’m unaware of?

If you can reevaluate different setups you may possibly get much higher throughput.

To be clear, while I would be interested in hearing about differing setups that would improve the 4k iops performance overall; I am most interested in getting the thin performance comparable to the thick performance. I did not expect such a large differential between the two types.

My guess would be - the best targeting layout should be probably striping no more then 2-3 disks and use bigger striping block.
And then just 'join' 'smaller' arrays together in lvm2 in 1 big LV.

Is this advice given because the drives involved are SSD? With rotational drives I would never expect concatenated small RAID 0 arrays to provide superior throughput to a single RAID 0. The best chunk size for sustained throughput depends on the typical transaction, theoretical best case would have typical request size equal to a full stripe.

I would not expect the raid arrangement (at least for RAID 0 or concatenated drives) to have a significant impact on 4k request performance, however. Too small to take advantage of striping. Is there a reason why the arrangement would make a difference for this size?

Post by Dale Stephenson

Post by Zdenek Kabelac
(something like 'lvcreate -LXXX -i8 -I512k vgname’)

Would making lvm stripe on top of an md that already stripes confer any performance benefit in general, or for small (4k) requests in particular?

Rule #1 - try to avoid 'over-combining' things together.
- measure performance from 'bottom' upward in your device stack.
If the underlying devices gives poor speed - you can't make it better by any super0smart disk-layout on top of it.

Fair enough. I have done exactly that — the performance of the underlying md RAID and thick volumes is virtually identical, as I expected. The thin performance is much, much lower on the same RAID. So it’s not the underlying layer that is causing the poor speed, it’s the thin volume.

Post by Dale Stephenson

Post by Zdenek Kabelac
Wouldn't be 'faster' to just concatenate 8 disks together instead of striping - or stripe only across 2 disk - and then you concatenate 4 such striped areas…

For sustained throughput I would expect striping of 8 disks to blow away concatenation — however, for small requests I wouldn’t expect any advantage. On a non-redundant array, I would expect a single threaded test using 4k requests is going to end up reading/writing data from exactly one disk regardless of whether the underlying drives are concatenated or stripes.

It always depends which kind of load you expect the most.
I suspect spreading 4K blocks across 8 SSD is likely very far away from ideal layout.

Very true. Trying to read/write a single 4k request across 8 drives, whether they are SSD or not, would require a 512-byte chunksize. Not only is that not possible with md, the overhead associated with generating 8 i/o requests would more than eat up any tiny gain from parallel i/o. (For rotational drives it’d be even worse, since you’d ensure the response time is governed by the slowest drive).

Which is why for a test like this, which is just using 4k requests in a single thread, I don’t really expect the arrangement of the RAID to have any significant effect. No matter what kind of drive you are using, and however your drives are arranged, I’d expect the 4k requests only to read or write to a single disk. So why is it so much slower on thin volumes than thick?

Any SSD is typically very bad with 4K blocks - it you want to 'spread' the load on mores SSDs do not use less the 64K stripe chunks per SSD - this gives you (8*64) 512K stripe size.

I am not using 4k “blocks” anywhere in the remapping layers. I am using a 4k *request* size. I want the highest possible number of iops for this request size; I know the throughput will stink compared to larger request sizes, but that’s to be expected.

As for thin-pool chunksize - if you plan to use lots of snapshots - keep the value lowest possible - 64K or 128K thin-pool chunksize.

Snapshots are the main attraction of thin provisioning to me, so 64k would be my preferred thin-pool chunksize for that reason. However, neither a 64k thin-pool chunksize nor a 512k thin-pool chunksize eliminates the large performance difference between thin and thick volumes for 4k requests.

But I'd still suggest to reevaluate/benchmark setup where you will use much lower number of SSD for load spreading - and use bigger strip chunks per each device. This should nicely improve performance in case of 'bigger' writes
and not that much slow things down with 4K loads….

Is the transactional latency so high compared to the data transfer itself that 3-drive SSD RAID 0 should outperform 8-drive SSD RAID 0 for a constant stripe size? Sounds like an interesting test to run, but I wouldn’t expect it to affect 4k requests.

Post by Dale Stephenson
What is the best choice for handling 4k request sizes?

Possibly NVMe can do a better job here.

What can be done to make *thin provisioning* handle 4k request sizes better? There may be hardware that performs better for 4k requests than the ones I’m using, but I’m not bothered by the lvm thick volume performance on it. What can I do to make the lvm thin volume perform as close to thick as possible?

Thank you,
Dale Stephenson