[linux-lvm] Any way to speed up activation of volumes with snapshots?

[Chris Friesen]

Hi,

I'm running a 3.10 kernel with LVM 2.02.95.

I'm running into a problem where activating snapshots can take quite a long
time, roughly one minute per 25GB of delta between the snapshot and the origin
volume. (See below for my test procedure.)

I realize that my kernel/LVM aren't exactly bleeding edge, and I wondering
whether more recent versions have done anything to speed up the activation
process (like maybe making it more lazy-loaded rather than reading in a bunch of
data up-front).

If anyone is aware of such improvements, I'd appreciate it if you could point me
at the appropriate changes.


For those that are interested, the test that I did was as follows:

lvcreate -L100G -n test1 vg
lvcreate -L 100G -s -n test1snapshot /dev/vg/test1
dd if=/dev/zero of=/dev/cinder-volumes/test1snapshot bs=1M count=95000
lvchange -an /dev/cinder-volumes/test1snapshot
time lvchange -ay /dev/cinder-volumes/test1snapshot

The final "lvchange" command took about 4m22s. This works out to 362MB/sec,
which is almost exactly the speed I got doing a "dd" from the volume to /dev/null.

Thanks,
Chris

[Chris Friesen]

My apologies, I got the version wrong. The tests below were actually from the
3.4 kernel and LVM 2.2.95. I'm doing tests on the newer versions now.

Chris

Post by Chris Friesen
Hi,
I'm running a 3.10 kernel with LVM 2.02.95.
I'm running into a problem where activating snapshots can take quite a long
time, roughly one minute per 25GB of delta between the snapshot and the origin
volume. (See below for my test procedure.)
I realize that my kernel/LVM aren't exactly bleeding edge, and I wondering
whether more recent versions have done anything to speed up the activation
process (like maybe making it more lazy-loaded rather than reading in a bunch of
data up-front).
If anyone is aware of such improvements, I'd appreciate it if you could point me
at the appropriate changes.
lvcreate -L100G -n test1 vg
lvcreate -L 100G -s -n test1snapshot /dev/vg/test1
dd if=/dev/zero of=/dev/cinder-volumes/test1snapshot bs=1M count=95000
lvchange -an /dev/cinder-volumes/test1snapshot
time lvchange -ay /dev/cinder-volumes/test1snapshot
The final "lvchange" command took about 4m22s. This works out to 362MB/sec,
which is almost exactly the speed I got doing a "dd" from the volume to /dev/null.
Thanks,
Chris
_______________________________________________
linux-lvm mailing list
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

[Zdenek Kabelac]

Post by Chris Friesen
Hi,
I'm running a 3.10 kernel with LVM 2.02.95.
I'm running into a problem where activating snapshots can take quite a long
time, roughly one minute per 25GB of delta between the snapshot and the origin
volume. (See below for my test procedure.)
I realize that my kernel/LVM aren't exactly bleeding edge, and I wondering
whether more recent versions have done anything to speed up the activation
process (like maybe making it more lazy-loaded rather than reading in a bunch
of data up-front).
If anyone is aware of such improvements, I'd appreciate it if you could point
me at the appropriate changes.

Hi

There is NO way to accelerate your existing setup, other then placing delta on
SSD - the format of snapshot was meant to be used 'temporarily' - i.e. until
you take backup of LV - but not for long-term multi-gigabyte case.

This is major mis-use of this 'snapshot' feature - which repeats over and over...

If you want to use long-living snapshots - you need to use thin-provisioning,
.

Post by Chris Friesen
The final "lvchange" command took about 4m22s. This works out to 362MB/sec,

I've seen users waiting even 1/2 hour - so you are still 'lucky' user ;)



Zdenek

[Chris Friesen]

Post by Zdenek Kabelac

Post by Chris Friesen
Hi,
I'm running a 3.10 kernel with LVM 2.02.95.
I'm running into a problem where activating snapshots can take quite a long
time, roughly one minute per 25GB of delta between the snapshot and the origin
volume. (See below for my test procedure.)
I realize that my kernel/LVM aren't exactly bleeding edge, and I wondering
whether more recent versions have done anything to speed up the activation
process (like maybe making it more lazy-loaded rather than reading in a bunch
of data up-front).
If anyone is aware of such improvements, I'd appreciate it if you could point
me at the appropriate changes.

Hi
There is NO way to accelerate your existing setup, other then placing delta on
SSD - the format of snapshot was meant to be used 'temporarily' - i.e. until you
take backup of LV - but not for long-term multi-gigabyte case.
This is major mis-use of this 'snapshot' feature - which repeats over and over...
If you want to use long-living snapshots - you need to use thin-provisioning,

Thanks for the quick response.

Presumably we would still have multi-GB of delta between the original volume and
the snapshot, so what would make the activation of a thin-provisioned snapshot
faster? Is the metadata stored differently?

Chris

[Zdenek Kabelac]

Post by Chris Friesen

Post by Zdenek Kabelac

Post by Chris Friesen
Hi,
I'm running a 3.10 kernel with LVM 2.02.95.
I'm running into a problem where activating snapshots can take quite a long
time, roughly one minute per 25GB of delta between the snapshot and the origin
volume. (See below for my test procedure.)
I realize that my kernel/LVM aren't exactly bleeding edge, and I wondering
whether more recent versions have done anything to speed up the activation
process (like maybe making it more lazy-loaded rather than reading in a bunch
of data up-front).
If anyone is aware of such improvements, I'd appreciate it if you could point
me at the appropriate changes.

Hi
There is NO way to accelerate your existing setup, other then placing delta on
SSD - the format of snapshot was meant to be used 'temporarily' - i.e. until you
take backup of LV - but not for long-term multi-gigabyte case.
This is major mis-use of this 'snapshot' feature - which repeats over and over...
If you want to use long-living snapshots - you need to use thin-provisioning,

Thanks for the quick response.
Presumably we would still have multi-GB of delta between the original volume
and the snapshot, so what would make the activation of a thin-provisioned
snapshot faster? Is the metadata stored differently?

Yes - metadata are on separate device(LV) and are using b-trees.
So now you could i.e. chain snap-of-snap-of-snap...
Also snaps do not need to be active with origin and many other goodies.

Drawbacks - all data have to be in same pool and share same space,
so you can't run 'just' 'snapshot' out-of-space - whole pool is out-of-space.
Also min chunk size is 64K (instead of 4K for old-snap)

Anyway - try and play and see what performs better and suit your needs.


Zdenek

[Zdenek Kabelac]

Post by Chris Friesen

Post by Zdenek Kabelac

Post by Chris Friesen
Hi,
I'm running a 3.10 kernel with LVM 2.02.95.
I'm running into a problem where activating snapshots can take quite a long
time, roughly one minute per 25GB of delta between the snapshot and the origin
volume. (See below for my test procedure.)
I realize that my kernel/LVM aren't exactly bleeding edge, and I wondering
whether more recent versions have done anything to speed up the activation
process (like maybe making it more lazy-loaded rather than reading in a bunch
of data up-front).
If anyone is aware of such improvements, I'd appreciate it if you could point
me at the appropriate changes.

Hi
There is NO way to accelerate your existing setup, other then placing delta on
SSD - the format of snapshot was meant to be used 'temporarily' - i.e. until you
take backup of LV - but not for long-term multi-gigabyte case.
This is major mis-use of this 'snapshot' feature - which repeats over and over...
If you want to use long-living snapshots - you need to use thin-provisioning,

Thanks for the quick response.
Presumably we would still have multi-GB of delta between the original volume
and the snapshot, so what would make the activation of a thin-provisioned
snapshot faster? Is the metadata stored differently?

If you don't mind dropping existing snapshot and creating new 'old-snapshot' -
using larger block size (i.e. 32K (or even more) instead of 4K) may improve
speed of disk reading noticeable (at the price of copying larger chunks when
modifying disk blocks - depends on use-case)....

And if you don't mind replacing your ancient kernel and trying some newer one
- there have been some 'read' order improvement for this old-snapshot target
made as well - so you may at least try if there will be any noticeable change
(actually would be interesting just to get report how much faster it boots
when you only exchange kernel for a boot - you don't have to use it - just
measure boot speed)

Zdenek

[Chris Friesen]

Post by Zdenek Kabelac
If you don't mind dropping existing snapshot and creating new 'old-snapshot' -
using larger block size (i.e. 32K (or even more) instead of 4K) may improve
speed of disk reading noticeable (at the price of copying larger chunks when
modifying disk blocks - depends on use-case)....

Okay, good info to have.

Post by Zdenek Kabelac
And if you don't mind replacing your ancient kernel and trying some newer one -
there have been some 'read' order improvement for this old-snapshot target made
as well - so you may at least try if there will be any noticeable change
(actually would be interesting just to get report how much faster it boots when
you only exchange kernel for a boot - you don't have to use it - just measure
boot speed)

This is essentially a custom distro, so changing just the kernel is a bit
complicated. However, I have a newer version that I want to test with which has
a newer kernel but the same version of LVM, so that should give some good data.

Chris

[Chris Friesen]

Post by Chris Friesen
Hi,
I'm running a 3.10 kernel with LVM 2.02.95.
I'm running into a problem where activating snapshots can take quite a long
time, roughly one minute per 25GB of delta between the snapshot and the origin
volume. (See below for my test procedure.)

As a side question, currently we have a script which runs "vgchange -ay <vg>",
which will activate the VG.

This script appears to activate the LVs within the VG serially rather than in
parallel. Does it do any processing to determine in which order to activate the
LVs?

The reason I ask is that we have an overall timeout on the activate phase, after
which we'll put any un-activated LVs into an error state. It would be a shame
to try to activate a snapshotted LV first, and then timeout without activating
any of the non-snapshotted LVs.

Chris

[Zdenek Kabelac]

Post by Chris Friesen

Post by Chris Friesen
Hi,
I'm running a 3.10 kernel with LVM 2.02.95.
I'm running into a problem where activating snapshots can take quite a long
time, roughly one minute per 25GB of delta between the snapshot and the origin
volume. (See below for my test procedure.)

As a side question, currently we have a script which runs "vgchange -ay <vg>",
which will activate the VG.
This script appears to activate the LVs within the VG serially rather than in
parallel. Does it do any processing to determine in which order to activate
the LVs?
The reason I ask is that we have an overall timeout on the activate phase,
after which we'll put any un-activated LVs into an error state. It would be a
shame to try to activate a snapshotted LV first, and then timeout without
activating any of the non-snapshotted LVs.

Hi

Sorry, but there is no 'parallel' activation from lvm2 command - as lvm2 is
not multithreadded app (and is quite far away from that).

What you can do with recent versions of lvm2 - you could 'mark' (with
lvchange --setskipactivation) some LVs to be skipped from normal activation.
Then you could activate such skipped LVs with 'lvchange -ay -K'.

As said lvm2 has not been designed for this case.
(btw during whole activation - your access to VG is actually limited only for
'read-only')

Zdenek

[Chris Friesen]

Post by Chris Friesen
As a side question, currently we have a script which runs "vgchange -ay <vg>",
which will activate the VG.
This script appears to activate the LVs within the VG serially rather than in
parallel. Does it do any processing to determine in which order to activate
the LVs?

What you can do with recent versions of lvm2 - you could 'mark' (with lvchange
--setskipactivation) some LVs to be skipped from normal activation.
Then you could activate such skipped LVs with 'lvchange -ay -K'.
As said lvm2 has not been designed for this case.
(btw during whole activation - your access to VG is actually limited only for
'read-only')

Right...we're planning on testing out thin provisioning but I have to figure out
a band-aid solution for products already in use.

Just to clarify...what does your "access to VG is actually limited only for
'read-only'" mean exactly?

Does it mean:

1) I only have read-only access to the LVs within the VG until all LVs in the VG
are activated?

or

2) I only have readonly access to the VG (so I can't add/remove/modify LVs) but
I have read/write access to the LVs within the VG.


Thanks,
Chris

[Zdenek Kabelac]

Post by Chris Friesen

Post by Chris Friesen
As a side question, currently we have a script which runs "vgchange -ay <vg>",
which will activate the VG.
This script appears to activate the LVs within the VG serially rather than in
parallel. Does it do any processing to determine in which order to activate
the LVs?

What you can do with recent versions of lvm2 - you could 'mark' (with lvchange
--setskipactivation) some LVs to be skipped from normal activation.
Then you could activate such skipped LVs with 'lvchange -ay -K'.
As said lvm2 has not been designed for this case.
(btw during whole activation - your access to VG is actually limited only for
'read-only')

Right...we're planning on testing out thin provisioning but I have to figure
out a band-aid solution for products already in use.
Just to clarify...what does your "access to VG is actually limited only for
'read-only'" mean exactly?
1) I only have read-only access to the LVs within the VG until all LVs in the
VG are activated?
or
2) I only have readonly access to the VG (so I can't add/remove/modify LVs)
but I have read/write access to the LVs within the VG.

2. is correct - you will not be able to modify VG content (as read-lock is
held for VG - and you need write-lock for updates)

Zdenek

[Chris Friesen]

Sorry, but there is no 'parallel' activation from lvm2 command - as lvm2 is not
multithreadded app (and is quite far away from that).

Just had a weird thought. What would happen if I ran the "vgchange -ay"
command, and then a few seconds later ran a bunch of "lvchange -ay" commands in
parallel?

Would that maybe let me activate non-snapshotted volumes even if the snapshotted
volumes took a long time?

Chris

[Zdenek Kabelac]

Post by Chris Friesen

Sorry, but there is no 'parallel' activation from lvm2 command - as lvm2 is not
multithreadded app (and is quite far away from that).

Just had a weird thought. What would happen if I ran the "vgchange -ay"
command, and then a few seconds later ran a bunch of "lvchange -ay" commands
in parallel?

Since all of them are 'activation' requests - you just end with success
and everything should be activated.

Much more interesting case is - if you run in parallel activation
and deactivation :) - then the result depends on last executed command.

Post by Chris Friesen
Would that maybe let me activate non-snapshotted volumes even if the
snapshotted volumes took a long time?

Yes - you could activate things in parallel this way - since every command has
separate udev cookie - so it waits on its set of LVs.

Zdenek