[linux-lvm] Strange dm-cache activity at startup

[Stefan Ring]

I have a fairly simple dm-cache setup with one PV on an HDD partition
as the origin block device, and the cache LVs live on a partition on
an SSD. The cache is in writethrough mode. Since I have no prior
experience with this, I basically followed the lvmcache man page when
creating this setup a few months ago.

Every morning at startup, the system immediately starts prolonged
activity on the devices involved in the cache setup. iostat shows
reading activity from the cache device and constant write activity on
the HDD. The HDD is fully utilized, and this goes on for more than an
hour. After that, everything goes into what I would consider the
"normal" state with only the IO activity going on that I actively
throw at it.

Is this normal/expected? What does it do? Why does it write back huge
amounts of data to the HDD after each system start?

Kernel is currently 3.17.8-200.fc20.x86_64 (Fedora)
Userland has been upgraded to fc22 meanwhile

origin:
└─sda6 8:6 0 894.6G 0 part
└─vg_zfs-lv_zfsdisk_corig 253:2 0 894.6G 0 lvm
└─vg_zfs-lv_zfsdisk 253:3 0 894.6G 0 lvm

cache:
└─sdb5 8:21 0 233.7G 0 part
├─vg_zfs-lv_cachedata_cdata 253:0 0 213.1G 0 lvm
│ └─vg_zfs-lv_zfsdisk 253:3 0 894.6G 0 lvm
└─vg_zfs-lv_cachedata_cmeta 253:1 0 9.4G 0 lvm
└─vg_zfs-lv_zfsdisk 253:3 0 894.6G 0 lvm

$ sudo dmsetup ls --tree
vg_zfs-lv_zfsdisk (253:3)
├─vg_zfs-lv_zfsdisk_corig (253:2)
│ └─ (8:6)
├─vg_zfs-lv_cachedata_cdata (253:0)
│ └─ (8:21)
└─vg_zfs-lv_cachedata_cmeta (253:1)
└─ (8:21)

$ sudo dmsetup table vg_zfs-lv_zfsdisk
0 1876041728 cache 253:1 253:0 253:2 128 1 writethrough mq 0

[Zdenek Kabelac]

Post by Stefan Ring
I have a fairly simple dm-cache setup with one PV on an HDD partition
as the origin block device, and the cache LVs live on a partition on
an SSD. The cache is in writethrough mode. Since I have no prior
experience with this, I basically followed the lvmcache man page when
creating this setup a few months ago.
Every morning at startup, the system immediately starts prolonged
activity on the devices involved in the cache setup. iostat shows
reading activity from the cache device and constant write activity on
the HDD. The HDD is fully utilized, and this goes on for more than an
hour. After that, everything goes into what I would consider the
"normal" state with only the IO activity going on that I actively
throw at it.
Is this normal/expected? What does it do? Why does it write back huge
amounts of data to the HDD after each system start?
Kernel is currently 3.17.8-200.fc20.x86_64 (Fedora)
Userland has been upgraded to fc22 meanwhile

Please first try to use the latest kernel.
Kernel 3.17 is almost a year old.

Lots of optimizations went in meanwhile.
It's pointless to hunt already fixed bug.

Zdenek

[Stefan Ring]

Post by Zdenek Kabelac
Please first try to use the latest kernel.
Kernel 3.17 is almost a year old.
Lots of optimizations went in meanwhile.
It's pointless to hunt already fixed bug.

No problem. I'll soon upgrade to 4.2 because I'm eager to try smq.
I've been held back by a zfsonlinux bug that has prevented an upgrade
and has only very recently been fixed.

[Zdenek Kabelac]

Post by Stefan Ring

Post by Zdenek Kabelac
Please first try to use the latest kernel.
Kernel 3.17 is almost a year old.
Lots of optimizations went in meanwhile.
It's pointless to hunt already fixed bug.

No problem. I'll soon upgrade to 4.2 because I'm eager to try smq.
I've been held back by a zfsonlinux bug that has prevented an upgrade
and has only very recently been fixed.

For playing with smq - you definitely need the latest lvm2 version as well.

Zdenek

[Joe Thornber]

Post by Stefan Ring
Is this normal/expected? What does it do? Why does it write back huge
amounts of data to the HDD after each system start?

No, this is not expected. The most likely cause is your system was
not shutdown cleanly so dm-cache has no choice but to assume all
blocks are dirty and need writing back. Make sure your volumes are
being deactivated as part of shutdown.

- Joe

[John Stoffel]

Post by Stefan Ring
Is this normal/expected? What does it do? Why does it write back huge
amounts of data to the HDD after each system start?

Joe> No, this is not expected. The most likely cause is your system was
Joe> not shutdown cleanly so dm-cache has no choice but to assume all
Joe> blocks are dirty and need writing back. Make sure your volumes are
Joe> being deactivated as part of shutdown.

How would you double check to make sure this happening properly in the
logs? I'm interested in using an SSD to cache my home file server
/home and /local directories to make NFS access faster... but I've
been hesitant to upgrade since I'd hate to lose all this data.

John

[Simon O'Riordan]

Avoid losing data by imaging the drive using Clonezilla. I do it all the
time.

Post by John Stoffel

Post by Stefan Ring
Is this normal/expected? What does it do? Why does it write back huge
amounts of data to the HDD after each system start?

Joe> No, this is not expected. The most likely cause is your system was
Joe> not shutdown cleanly so dm-cache has no choice but to assume all
Joe> blocks are dirty and need writing back. Make sure your volumes are
Joe> being deactivated as part of shutdown.
How would you double check to make sure this happening properly in the
logs? I'm interested in using an SSD to cache my home file server
/home and /local directories to make NFS access faster... but I've
been hesitant to upgrade since I'd hate to lose all this data.
John
_______________________________________________
linux-lvm mailing list
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

[John Stoffel]

Simon,
I have multiple terabytes of file storage at home. I'm looking to use
an SSD to speed up access to the more frequently used data. Don't
have the cash to upgrade to a full SSD environment.

Simon> Avoid losing data by imaging the drive using Clonezilla. I do it all the
Simon> time.

Post by John Stoffel

Post by Stefan Ring
Is this normal/expected? What does it do? Why does it write back huge
amounts of data to the HDD after each system start?

Joe> No, this is not expected. The most likely cause is your system was
Joe> not shutdown cleanly so dm-cache has no choice but to assume all
Joe> blocks are dirty and need writing back. Make sure your volumes are
Joe> being deactivated as part of shutdown.

Post by John Stoffel
How would you double check to make sure this happening properly in the
logs? I'm interested in using an SSD to cache my home file server
/home and /local directories to make NFS access faster... but I've
been hesitant to upgrade since I'd hate to lose all this data.
John
_______________________________________________
linux-lvm mailing list
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Simon> _______________________________________________
Simon> linux-lvm mailing list
Simon> linux-***@redhat.com
Simon> https://www.redhat.com/mailman/listinfo/linux-lvm
Simon> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

[Stefan Ring]

Post by Joe Thornber

Post by Stefan Ring
Is this normal/expected? What does it do? Why does it write back huge
amounts of data to the HDD after each system start?

No, this is not expected. The most likely cause is your system was
not shutdown cleanly so dm-cache has no choice but to assume all
blocks are dirty and need writing back. Make sure your volumes are
being deactivated as part of shutdown.

I would have hoped that the distro scripts would take care of this.
Apparently not, for whatever reason :(. I have since upgraded to
kernel 4.2, and the behavior has not changed. I'll try to reboot today
instead of shutdown and see if anything changes.

[Stefan Ring]

Post by Stefan Ring
I would have hoped that the distro scripts would take care of this.
Apparently not, for whatever reason :(. I have since upgraded to
kernel 4.2, and the behavior has not changed. I'll try to reboot today
instead of shutdown and see if anything changes.

Nothing has changed. However, I noticed this in dmesg during startup:

device-mapper: cache: You have created a cache device with a lot of
individual cache blocks (3491776)
All these mappings can consume a lot of kernel memory, and take some
time to read/write.
Please consider increasing the cache block size to reduce the overall
cache block count.

Now I'll have to see what to do about that.

[Ian Pilcher]

Post by Joe Thornber
No, this is not expected. The most likely cause is your system was
not shutdown cleanly so dm-cache has no choice but to assume all
blocks are dirty and need writing back. Make sure your volumes are
being deactivated as part of shutdown.

What exactly is required to cleanly shutdown an LVM cache or dm-cache
device?

AFAICT, systemd does a DM_DEV_REMOVE ioctl on all device mapper devices
during shutdown/reboot. Is this sufficient?

Thanks!

[Zdenek Kabelac]

Post by Ian Pilcher

Post by Joe Thornber
No, this is not expected. The most likely cause is your system was
not shutdown cleanly so dm-cache has no choice but to assume all
blocks are dirty and need writing back. Make sure your volumes are
being deactivated as part of shutdown.

What exactly is required to cleanly shutdown an LVM cache or dm-cache
device?
AFAICT, systemd does a DM_DEV_REMOVE ioctl on all device mapper devices
during shutdown/reboot. Is this sufficient?

ONLY if the give device are UNUSED.

If the volume is e.g. mounted - then 'remove' is irrelevant.

You can't remove device with open count != 0.
(check with: dmsetup info -c)

Zdenek