[linux-lvm] LVM cache/dm-cache questions.

[sergei]

Hi,

I have a few simple questions regarding relatively new feature LVM
cache/dm-cache.

From second hand information on internet, it appears that you cannot
resize cached volume.
Please correct me if I am wrong: to resize a cached volume, one needs to
lvremove the cache, resize and recreate the cache?

In scenario where cache device fails, how safe is the data on the cached
LVM?

Is it possible to make snapshots of the cached device?

Before I test these things I will need to upgrade distribution (as the
release I am on does not have the LVM cache), which means switching to
systemd (I would rather avoid, if LVM cache does not fail gracefully).

Thank you very much!

Sergei.

[Brassow Jonathan]

Hi,
I have a few simple questions regarding relatively new feature LVM cache/dm-cache.
From second hand information on internet, it appears that you cannot resize cached volume.
Please correct me if I am wrong: to resize a cached volume, one needs to lvremove the cache, resize and recreate the cache?

This is true at the moment. You are likely to be able to simply resize online in the future.

In scenario where cache device fails, how safe is the data on the cached LVM?

If you are using writethrough, quite safe. If you are using ‘writeback’, probably not (RAID for your cache in this case would probably be wise).

Is it possible to make snapshots of the cached device?

easy answer is “no”.

complicated answer includes a ‘yes’. You can cache a poolDataLV, then your thinLV and all its thinSnaps would be cached.

brassow

[Sergei Franco]

Thank you!

That clears up a few questions!


Regards.

Sergei.

Post by Brassow Jonathan

Hi,
I have a few simple questions regarding relatively new feature LVM cache/dm-cache.
From second hand information on internet, it appears that you cannot resize cached volume.
Please correct me if I am wrong: to resize a cached volume, one needs to lvremove the cache, resize and recreate the cache?

This is true at the moment. You are likely to be able to simply resize online in the future.

In scenario where cache device fails, how safe is the data on the cached LVM?

If you are using writethrough, quite safe. If you are using ‘writeback’, probably not (RAID for your cache in this case would probably be wise).

Is it possible to make snapshots of the cached device?

easy answer is “no”.
complicated answer includes a ‘yes’. You can cache a poolDataLV, then your thinLV and all its thinSnaps would be cached.
brassow
_______________________________________________
linux-lvm mailing list
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

[lejeczek]

question I'd like to ask is: how to LUKS encrypt cache pool LV ?

many thanks.
L.

Post by Brassow Jonathan

Hi,
I have a few simple questions regarding relatively new feature LVM cache/dm-cache.
From second hand information on internet, it appears that you cannot resize cached volume.
Please correct me if I am wrong: to resize a cached volume, one needs to lvremove the cache, resize and recreate the cache?

This is true at the moment. You are likely to be able to simply resize online in the future.

In scenario where cache device fails, how safe is the data on the cached LVM?

If you are using writethrough, quite safe. If you are using ‘writeback’, probably not (RAID for your cache in this case would probably be wise).

Is it possible to make snapshots of the cached device?

easy answer is “no”.
complicated answer includes a ‘yes’. You can cache a poolDataLV, then your thinLV and all its thinSnaps would be cached.
brassow
_______________________________________________
linux-lvm mailing list
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

[Xen]

Post by lejeczek
question I'd like to ask is: how to LUKS encrypt cache pool LV ?
many thanks.
L.

In general without being any expert here you will need to encrypt the
origin device (PV) on which it resides and likely also the PV of the
cache pool itself. It's no different from any other volume you would
use.

You can allow LUKS to pass-through discards if you open the volume with
--allow-discards on the cryptsetup prompt (command).

In case you have an SSD underneath. It was said that the cache itself
does not utilize discards just yet, but that it was in the planning.

[lejeczek]

well, I prefer to encrypt LV itself, and I'm trying the same
what worked always with my "regular" LVs, yet - "cache pools
LVs" fail to encrypt with: Command failed with code 22.

Post by Xen

Post by lejeczek
question I'd like to ask is: how to LUKS encrypt cache
pool LV ?
many thanks.
L.

In general without being any expert here you will need to
encrypt the origin device (PV) on which it resides and
likely also the PV of the cache pool itself. It's no
different from any other volume you would use.
You can allow LUKS to pass-through discards if you open
the volume with --allow-discards on the cryptsetup prompt
(command).
In case you have an SSD underneath. It was said that the
cache itself does not utilize discards just yet, but that
it was in the planning.
_______________________________________________
linux-lvm mailing list
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

[Xen]

well, I prefer to encrypt LV itself, and I'm trying the same what
worked always with my "regular" LVs, yet - "cache pools LVs" fail to
encrypt with: Command failed with code 22.

If you are going to encrypt an LV it will no longer be an LV but an
(opened) LUKS container.

[lejeczek]

whatever you might call it, it works, luks encrypting,
opening & mounting @boot - so I only wonder (which was my
question) why not cache pool LVs. Is it not supported...
would be great if a developer sees this question, I'm not
sure jut yet about filing a bug report.

Post by Xen

Post by lejeczek
well, I prefer to encrypt LV itself, and I'm trying the
same what
worked always with my "regular" LVs, yet - "cache pools
LVs" fail to
encrypt with: Command failed with code 22.

If you are going to encrypt an LV it will no longer be an
LV but an (opened) LUKS container.
_______________________________________________
linux-lvm mailing list
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

[Ondrej Kozina]

Post by lejeczek
whatever you might call it, it works, luks encrypting,
question) why not cache pool LVs. Is it not supported...
would be great if a developer sees this question, I'm not
sure jut yet about filing a bug report.

In general LVM2 doesn't auto-activate or interpret unknown device types.
LUKS header is considered unknown from LVM2 perspective. Simply put LVM2
doesn't understand LUKS header data. Not sure what you tried to do with
cache pool LV, but in my opinion any effort to encrypt (live or
detached) cache pool LV may end with severe data corruption...

As of now I think you have in general two options:

a) encrypt both PVs because obviously if you only encrypt the origin PV
you end up with decrypted plaintext data stored in cache pool. Probably
this is the exact scenario you were about to avoid?

Unfortunately a) is suboptimal with regard to performance since you'd
perform the encryption of data blocks twice.

Option b): encrypt the top level LV (the one constructed from both cache
and origin LV). This way ciphertext would be stored twice in cache PV
and origin PV but the encryption would be performed only once.

Regards
Ondrej

[lejeczek]

Post by Ondrej Kozina

Post by lejeczek
whatever you might call it, it works, luks encrypting,
question) why not cache pool LVs. Is it not supported...
would be great if a developer sees this question, I'm not
sure jut yet about filing a bug report.

In general LVM2 doesn't auto-activate or interpret unknown
device types. LUKS header is considered unknown from LVM2
perspective. Simply put LVM2 doesn't understand LUKS
header data. Not sure what you tried to do with cache pool
LV, but in my opinion any effort to encrypt (live or
detached) cache pool LV may end with severe data
corruption...
a) encrypt both PVs because obviously if you only encrypt
the origin PV you end up with decrypted plaintext data
stored in cache pool. Probably this is the exact scenario
you were about to avoid?
Unfortunately a) is suboptimal with regard to performance
since you'd perform the encryption of data blocks twice.
Option b): encrypt the top level LV (the one constructed
from both cache and origin LV). This way ciphertext would
be stored twice in cache PV and origin PV but the
encryption would be performed only once.

gee, guys, thanks Ondrej,
this I was saying from the beginning did not work - option b
- does not work. I can Not encrypt top level cache pool LV.
It does work with any other LV I have, but cache pool fails
(like I said earlier) with:

Command failed with code 22.

And me speculating on my own - whether it is a bug or just
limitation of implementation (Centos 7.2,
lvm2-2.02.130-5.el7_2.4.x86_64,
cryptsetup-1.6.7-1.el7.x86_64)- I thought instead I should
seek clarification.

Post by Ondrej Kozina
Regards
Ondrej
_______________________________________________
linux-lvm mailing list
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

[Ondrej Kozina]

Post by lejeczek

Post by Ondrej Kozina

Post by lejeczek
whatever you might call it, it works, luks encrypting,
question) why not cache pool LVs. Is it not supported...
would be great if a developer sees this question, I'm not
sure jut yet about filing a bug report.

In general LVM2 doesn't auto-activate or interpret unknown
device types. LUKS header is considered unknown from LVM2
perspective. Simply put LVM2 doesn't understand LUKS
header data. Not sure what you tried to do with cache pool
LV, but in my opinion any effort to encrypt (live or
detached) cache pool LV may end with severe data
corruption...
a) encrypt both PVs because obviously if you only encrypt
the origin PV you end up with decrypted plaintext data
stored in cache pool. Probably this is the exact scenario
you were about to avoid?
Unfortunately a) is suboptimal with regard to performance
since you'd perform the encryption of data blocks twice.
Option b): encrypt the top level LV (the one constructed
from both cache and origin LV). This way ciphertext would
be stored twice in cache PV and origin PV but the
encryption would be performed only once.

gee, guys, thanks Ondrej,
this I was saying from the beginning did not work - option b
- does not work. I can Not encrypt top level cache pool LV.
It does work with any other LV I have, but cache pool fails
Command failed with code 22.

Could you post here 'lsblk' and 'lvs' output together with exact
cryptsetup command you have executed to luksFormat your top level cached
LV? Please add also --debug option among your original cryptsetup options.

Regards
Ondrej

[lejeczek]

$ lvs -a h200front
LV VG Attr LSize Pool
Origin Data% Meta% Move Log Cpy%Sync Convert
0 h200front Cwi-aoC--- 9.10t
[cache0] [0_corig] 100.00 8.18 0.00
[0_corig] h200front rwi-aoC---
9.10t 100.00
[0_corig_rimage_0] h200front iwi-aor--- 1.82t
[0_corig_rimage_1] h200front iwi-aor--- 1.82t
[0_corig_rimage_2] h200front iwi-aor--- 1.82t
[0_corig_rimage_3] h200front iwi-aor--- 1.82t
[0_corig_rimage_4] h200front iwi-aor--- 1.82t
[0_corig_rimage_5] h200front iwi-aor--- 1.82t
[0_corig_rmeta_0] h200front ewi-aor--- 4.00m
[0_corig_rmeta_1] h200front ewi-aor--- 4.00m
[0_corig_rmeta_2] h200front ewi-aor--- 4.00m
[0_corig_rmeta_3] h200front ewi-aor--- 4.00m
[0_corig_rmeta_4] h200front ewi-aor--- 4.00m
[0_corig_rmeta_5] h200front ewi-aor--- 4.00m
[cache0] h200front Cwi---C---
220.00g 100.00 8.18 0.00
[cache0_cdata] h200front Cwi-aor---
220.00g 100.00
[cache0_cdata_rimage_0] h200front iwi-aor--- 220.00g
[cache0_cdata_rimage_1] h200front iwi-aor--- 220.00g
[cache0_cdata_rmeta_0] h200front ewi-aor--- 4.00m
[cache0_cdata_rmeta_1] h200front ewi-aor--- 4.00m
[cache0_cmeta] h200front ewi-aor---
512.00m 100.00
[cache0_cmeta_rimage_0] h200front iwi-aor--- 512.00m
[cache0_cmeta_rimage_1] h200front iwi-aor--- 512.00m
[cache0_cmeta_rmeta_0] h200front ewi-aor--- 4.00m
[cache0_cmeta_rmeta_1] h200front ewi-aor--- 4.00m
[lvol0_pmspare] h200front ewi------- 512.00m

I cannot debug now as for now I've given up the idea to
encrypt this LV, but I would say is should be easily
reproducible (maybe even waste of time looking at my setup)
I simply tried:

cryptsetup -v luksFormat /dev/mapper/h200front-0

and that works(ed) with "regurlar" LVs.

$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE
MOUNTPOINT
sda 8:0 0 238.5G 0 disk
├─sda1 8:1 0 1G 0 part
/boot
└─sda2 8:2 0 237.5G 0 part
├─SL-root 253:0 0 150G 0 lvm /
└─SL-home 253:1 0 10G 0 lvm
sdb 8:16 0 223.6G 0 disk
├─h200front-cache0_cdata_rmeta_0 253:4 0 4M 0 lvm
│ └─h200front-cache0_cdata 253:8 0 220G 0 lvm
│ └─h200front-0 253:29 0 9.1T 0 lvm
├─h200front-cache0_cdata_rimage_0 253:5 0 220G 0 lvm
│ └─h200front-cache0_cdata 253:8 0 220G 0 lvm
│ └─h200front-0 253:29 0 9.1T 0 lvm
├─h200front-cache0_cmeta_rmeta_0 253:9 0 4M 0 lvm
│ └─h200front-cache0_cmeta 253:13 0 512M 0 lvm
│ └─h200front-0 253:29 0 9.1T 0 lvm
└─h200front-cache0_cmeta_rimage_0 253:10 0 512M 0 lvm
└─h200front-cache0_cmeta 253:13 0 512M 0 lvm
└─h200front-0 253:29 0 9.1T 0 lvm
sdc 8:32 0 223.6G 0 disk
├─h200front-cache0_cdata_rmeta_1 253:6 0 4M 0 lvm
│ └─h200front-cache0_cdata 253:8 0 220G 0 lvm
│ └─h200front-0 253:29 0 9.1T 0 lvm
├─h200front-cache0_cdata_rimage_1 253:7 0 220G 0 lvm
│ └─h200front-cache0_cdata 253:8 0 220G 0 lvm
│ └─h200front-0 253:29 0 9.1T 0 lvm
├─h200front-cache0_cmeta_rmeta_1 253:11 0 4M 0 lvm
│ └─h200front-cache0_cmeta 253:13 0 512M 0 lvm
│ └─h200front-0 253:29 0 9.1T 0 lvm
└─h200front-cache0_cmeta_rimage_1 253:12 0 512M 0 lvm
└─h200front-cache0_cmeta 253:13 0 512M 0 lvm
└─h200front-0 253:29 0 9.1T 0 lvm
sdd 8:48 0 894.3G 0 disk
└─h300Int1-0 253:2 0 1.8T 0 lvm
└─h300Int1.0_crypt 253:27 0 1.8T 0
crypt /__.aLocalStorages/2
sde 8:64 0 894.3G 0 disk
└─h300Int1-0 253:2 0 1.8T 0 lvm
└─h300Int1.0_crypt 253:27 0 1.8T 0
crypt /__.aLocalStorages/2
sdf 8:80 0 447.1G 0 disk
└─h300Int0-0 253:3 0 1.3T 0 lvm
└─h300Int0.0_crypt 253:28 0 1.3T 0
crypt /__.aLocalStorages/1
sdg 8:96 0 447.1G 0 disk
└─h300Int0-0 253:3 0 1.3T 0 lvm
└─h300Int0.0_crypt 253:28 0 1.3T 0
crypt /__.aLocalStorages/1
sdh 8:112 0 447.1G 0 disk
└─h300Int0-0 253:3 0 1.3T 0 lvm
└─h300Int0.0_crypt 253:28 0 1.3T 0
crypt /__.aLocalStorages/1
sdi 8:128 0 1.8T 0 disk
├─h200front-0_corig_rmeta_0 253:14 0 4M 0 lvm
│ └─h200front-0_corig 253:26 0 9.1T 0 lvm
│ └─h200front-0 253:29 0 9.1T 0 lvm
└─h200front-0_corig_rimage_0 253:15 0 1.8T 0 lvm
└─h200front-0_corig 253:26 0 9.1T 0 lvm
└─h200front-0 253:29 0 9.1T 0 lvm
sdj 8:144 0 1.8T 0 disk
├─h200front-0_corig_rmeta_1 253:16 0 4M 0 lvm
│ └─h200front-0_corig 253:26 0 9.1T 0 lvm
│ └─h200front-0 253:29 0 9.1T 0 lvm
└─h200front-0_corig_rimage_1 253:17 0 1.8T 0 lvm
└─h200front-0_corig 253:26 0 9.1T 0 lvm
└─h200front-0 253:29 0 9.1T 0 lvm
sdk 8:160 0 1.8T 0 disk
├─h200front-0_corig_rmeta_2 253:18 0 4M 0 lvm
│ └─h200front-0_corig 253:26 0 9.1T 0 lvm
│ └─h200front-0 253:29 0 9.1T 0 lvm
└─h200front-0_corig_rimage_2 253:19 0 1.8T 0 lvm
└─h200front-0_corig 253:26 0 9.1T 0 lvm
└─h200front-0 253:29 0 9.1T 0 lvm
sdl 8:176 0 1.8T 0 disk
├─h200front-0_corig_rmeta_3 253:20 0 4M 0 lvm
│ └─h200front-0_corig 253:26 0 9.1T 0 lvm
│ └─h200front-0 253:29 0 9.1T 0 lvm
└─h200front-0_corig_rimage_3 253:21 0 1.8T 0 lvm
└─h200front-0_corig 253:26 0 9.1T 0 lvm
└─h200front-0 253:29 0 9.1T 0 lvm
sdm 8:192 0 1.8T 0 disk
├─h200front-0_corig_rmeta_4 253:22 0 4M 0 lvm
│ └─h200front-0_corig 253:26 0 9.1T 0 lvm
│ └─h200front-0 253:29 0 9.1T 0 lvm
└─h200front-0_corig_rimage_4 253:23 0 1.8T 0 lvm
└─h200front-0_corig 253:26 0 9.1T 0 lvm
└─h200front-0 253:29 0 9.1T 0 lvm
sdn 8:208 0 1.8T 0 disk
├─h200front-0_corig_rmeta_5 253:24 0 4M 0 lvm
│ └─h200front-0_corig 253:26 0 9.1T 0 lvm
│ └─h200front-0 253:29 0 9.1T 0 lvm
└─h200front-0_corig_rimage_5 253:25 0 1.8T 0 lvm
└─h200front-0_corig 253:26 0 9.1T 0 lvm
└─h200front-0 253:29 0 9.1T 0 lvm
sdo 8:224 0 9.1T 0 disk
/__.aLocalStorages/3

LV is working fine, being a host for all sorts of data, only
cryptsetup does not work on it.
many thanks.

Post by Ondrej Kozina

Post by lejeczek

Post by Ondrej Kozina

Post by lejeczek
whatever you might call it, it works, luks encrypting,
question) why not cache pool LVs. Is it not supported...
would be great if a developer sees this question, I'm not
sure jut yet about filing a bug report.

In general LVM2 doesn't auto-activate or interpret unknown
device types. LUKS header is considered unknown from LVM2
perspective. Simply put LVM2 doesn't understand LUKS
header data. Not sure what you tried to do with cache pool
LV, but in my opinion any effort to encrypt (live or
detached) cache pool LV may end with severe data
corruption...
a) encrypt both PVs because obviously if you only encrypt
the origin PV you end up with decrypted plaintext data
stored in cache pool. Probably this is the exact scenario
you were about to avoid?
Unfortunately a) is suboptimal with regard to performance
since you'd perform the encryption of data blocks twice.
Option b): encrypt the top level LV (the one constructed
from both cache and origin LV). This way ciphertext would
be stored twice in cache PV and origin PV but the
encryption would be performed only once.

gee, guys, thanks Ondrej,
this I was saying from the beginning did not work - option b
- does not work. I can Not encrypt top level cache pool LV.
It does work with any other LV I have, but cache pool fails
Command failed with code 22.

Could you post here 'lsblk' and 'lvs' output together with
exact cryptsetup command you have executed to luksFormat
your top level cached LV? Please add also --debug option
among your original cryptsetup options.
Regards
Ondrej
_______________________________________________
linux-lvm mailing list
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

[Ondrej Kozina]

Thank you for the lsblk and lvs output. At least now we know that raid
is somehow involved in the setup...

Post by lejeczek
I cannot debug now as for now I've given up the idea to
encrypt this LV, but I would say is should be easily
reproducible (maybe even waste of time looking at my setup)

Well, without the debug output from cryptsetup I can't be of any more
help, I'm sorry. It's like starring into crystal ball guessing what may
have gone wrong. The -22 error code is general "wrong parameters"
answer. I don't even know if cryptsetup has got into code path where
it's supposed to open the device for write.

But let me give it one last shot. Did you by any chance end up with 22
right after the prompt:

"Are you sure? (Type uppercase yes):"

If so, are you sure that your answer was 'YES' instead of lowercase 'yes'?

Regards
Ondrej

[lejeczek]

now when you say that - (I) believe it or not - I think it's
possible. I cannot test it, for now the LV is in production
but! I few days ago introduced an external usb keboard which
has not LED indicators at all and now when I play with it it
seems there might be problem on my f24, maybe the
drivers(?). Gee, open source with all its greatness and the
luck of officially Christened, own hardware(laptop) can be a
reason of great frustration sometimes.
I'll put together another cache pool some days and then
shall provide clarification to my own frustration.

Post by Ondrej Kozina
Thank you for the lsblk and lvs output. At least now we
know that raid is somehow involved in the setup...

Post by lejeczek
I cannot debug now as for now I've given up the idea to
encrypt this LV, but I would say is should be easily
reproducible (maybe even waste of time looking at my setup)

Well, without the debug output from cryptsetup I can't be
of any more help, I'm sorry. It's like starring into
crystal ball guessing what may have gone wrong. The -22
error code is general "wrong parameters" answer. I don't
even know if cryptsetup has got into code path where it's
supposed to open the device for write.
But let me give it one last shot. Did you by any chance
"Are you sure? (Type uppercase yes):"
If so, are you sure that your answer was 'YES' instead of
lowercase 'yes'?
Regards
Ondrej
_______________________________________________
linux-lvm mailing list
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

[Xen]

I cannot debug now as for now I've given up the idea to encrypt this
LV, but I would say is should be easily reproducible (maybe even
waste of time looking at my setup)

I can definitely say I have encrypted a cached volume without issue.

I have had a disk with two cached volumes, if you want to know, both
caches originating from the same SSD.

Main caches (main origin volume) were on a HDD in a simple regular
non-thin LVM.

[Xen]

whatever you might call it, it works, luks encrypting, opening &
cache pool LVs. Is it not supported...
would be great if a developer sees this question, I'm not sure jut yet
about filing a bug report.

Ondrej has it down.

You can only encrypt the combined volume, not the individual parts,
unless you encrypt those at the PV level.