[linux-lvm] [dm-crypt] LVM on LUKS: volumes missing

[Ondrej Kozina]

Hi fauno,

provided the driver was unlocked successfully it seems unrelated to
cryptsetup/LUKS to me. Could we move the discussion to lvm mail list?

Hi, maybe I'm too shocked but I couldn't find anything on this issue :)
I have a fully encrypted HD using the LVM on LUKS method from
ArchWiki[^0], with the LUKS header and key file on an external device.
Today I started having some disk failures (root remounted ro, xfs
partition giving errors), and after I decided to reboot to run fsck, I
can't find anything.

Could you paste here output of pvscan -vvvv --cache
/dev/mapper/insert_the_unlocked_device_name? Together with your
/etc/lvm/lvm.conf file? If it's a device with rootfs we're talking about
you will most probably have to extract /etc/lvm/lvm.conf file from
initramfs image.

Well, generally, if disk sectors accommodating PV header are damaged,
lvm2 won't recognise the device...

When the encrypted partition is opened, I don't see any errors, not even
on dmesg, but LVM can't find any volume. They're just missing.
Is there anything I can do? Thanks!
FWIW I had the same issue with another HD a few months back, though it
didn't had physical errors. It didn't had anything important so I
wasn't worried.

Ok, try same approach as above for this drive.

Regards
Ondrej

[fauno]

Post by Ondrej Kozina
Hi fauno,
provided the driver was unlocked successfully it seems unrelated to
cryptsetup/LUKS to me. Could we move the discussion to lvm mail list?

ok :)

Post by Ondrej Kozina

Hi, maybe I'm too shocked but I couldn't find anything on this issue :)
I have a fully encrypted HD using the LVM on LUKS method from
ArchWiki[^0], with the LUKS header and key file on an external device.
Today I started having some disk failures (root remounted ro, xfs
partition giving errors), and after I decided to reboot to run fsck, I
can't find anything.

Could you paste here output of pvscan -vvvv --cache
/dev/mapper/insert_the_unlocked_device_name?

#libdm-config.c:863 Setting activation/monitoring to 0
#lvmcmdline.c:1093 Processing: pvscan -vvvv --cache
/dev/mapper/system
#lvmcmdline.c:1096 O_DIRECT will be used
#libdm-config.c:799 Setting global/locking_type to 1
#libdm-config.c:799 Setting global/wait_for_locks to 1
#locking/locking.c:242 File-based locking selected.
#libdm-config.c:768 Setting global/locking_dir to /run/lock/lvm
#libdm-config.c:863 Setting global/prioritise_write_locks to 1
#locking/file_locking.c:236 Locking /run/lock/lvm/P_global RB
#locking/file_locking.c:141 _do_flock /run/lock/lvm/P_global:aux WB
#locking/file_locking.c:51 _undo_flock /run/lock/lvm/P_global:aux
#locking/file_locking.c:141 _do_flock /run/lock/lvm/P_global RB
#pvscan.c:150 Using physical volume(s) on command line
#device/dev-cache.c:347 /dev/mapper/system: Added to device cache
#locking/file_locking.c:74 Unlocking /run/lock/lvm/P_global
#locking/file_locking.c:51 _undo_flock /run/lock/lvm/P_global
Command failed with status code 5.

Post by Ondrej Kozina
Together with your
/etc/lvm/lvm.conf file? If it's a device with rootfs we're talking about
you will most probably have to extract /etc/lvm/lvm.conf file from
initramfs image.

this is the lvm.conf from the initramfs (removed comments)


config {
checks = 1
abort_on_errors = 0
profile_dir = "/etc/lvm/profile"
}
devices {
dir = "/dev"
scan = [ "/dev" ]
obtain_device_list_from_udev = 1
external_device_info_source = "none"
cache_dir = "/etc/lvm/cache"
cache_file_prefix = ""
write_cache_state = 1
sysfs_scan = 1
multipath_component_detection = 1
md_component_detection = 1
fw_raid_component_detection = 0
md_chunk_alignment = 1
data_alignment_detection = 1
data_alignment = 0
data_alignment_offset_detection = 1
ignore_suspended_devices = 0
ignore_lvm_mirrors = 1
disable_after_error_count = 0
require_restorefile_with_uuid = 1
pv_min_size = 2048
issue_discards = 0
}
allocation {
maximise_cling = 1
use_blkid_wiping = 1
wipe_signatures_when_zeroing_new_lvs = 1
mirror_logs_require_separate_pvs = 0
cache_pool_metadata_require_separate_pvs = 0
thin_pool_metadata_require_separate_pvs = 0
}
log {
verbose = 0
silent = 0
syslog = 1
overwrite = 0
level = 0
indent = 1
command_names = 0
prefix = " "
activation = 0
debug_classes = [ "memory", "devices", "activation", "allocation",
"lvmetad", "metadata", "cache", "locking", "lvmpolld" ]
}
backup {
backup = 1
backup_dir = "/etc/lvm/backup"
archive = 1
archive_dir = "/etc/lvm/archive"
retain_min = 10
retain_days = 30
}
shell {
history_size = 100
}
global {
umask = 077
test = 0
units = "h"
si_unit_consistency = 1
suffix = 1
activation = 1
proc = "/proc"
etc = "/etc"
locking_type = 1
wait_for_locks = 1
fallback_to_clustered_locking = 1
fallback_to_local_locking = 1
locking_dir = "/run/lock/lvm"
prioritise_write_locks = 1
abort_on_internal_errors = 0
detect_internal_vg_cache_corruption = 0
metadata_read_only = 0
mirror_segtype_default = "raid1"
raid10_segtype_default = "raid10"
sparse_segtype_default = "thin"
use_lvmetad = 1
use_lvmlockd = 0
system_id_source = "none"
use_lvmpolld = 0
}
activation {
checks = 0
udev_sync = 1
udev_rules = 1
verify_udev_operations = 0
retry_deactivation = 1
missing_stripe_filler = "error"
use_linear_target = 1
reserved_stack = 64
reserved_memory = 8192
process_priority = -18
raid_region_size = 512
readahead = "auto"
raid_fault_policy = "warn"
mirror_image_fault_policy = "remove"
mirror_log_fault_policy = "allocate"
snapshot_autoextend_threshold = 100
snapshot_autoextend_percent = 20
thin_pool_autoextend_threshold = 100
thin_pool_autoextend_percent = 20
use_mlockall = 0
monitoring = 1
polling_interval = 15
activation_mode = "degraded"
}
dmeventd {
mirror_library = "libdevmapper-event-lvm2mirror.so"
snapshot_library = "libdevmapper-event-lvm2snapshot.so"
thin_library = "libdevmapper-event-lvm2thin.so"
}

Post by Ondrej Kozina
Well, generally, if disk sectors accommodating PV header are damaged,
lvm2 won't recognise the device...

:(

[Ondrej Kozina]

Post by fauno
#libdm-config.c:863 Setting activation/monitoring to 0
#lvmcmdline.c:1093 Processing: pvscan -vvvv --cache
/dev/mapper/system
#lvmcmdline.c:1096 O_DIRECT will be used
#libdm-config.c:799 Setting global/locking_type to 1
#libdm-config.c:799 Setting global/wait_for_locks to 1
#locking/locking.c:242 File-based locking selected.
#libdm-config.c:768 Setting global/locking_dir to /run/lock/lvm
#libdm-config.c:863 Setting global/prioritise_write_locks to 1
#locking/file_locking.c:236 Locking /run/lock/lvm/P_global RB
#locking/file_locking.c:141 _do_flock /run/lock/lvm/P_global:aux WB
#locking/file_locking.c:51 _undo_flock /run/lock/lvm/P_global:aux
#locking/file_locking.c:141 _do_flock /run/lock/lvm/P_global RB
#pvscan.c:150 Using physical volume(s) on command line
#device/dev-cache.c:347 /dev/mapper/system: Added to device cache
#locking/file_locking.c:74 Unlocking /run/lock/lvm/P_global
#locking/file_locking.c:51 _undo_flock /run/lock/lvm/P_global
Command failed with status code 5.

This is very strange pvscan -vvvv output. Reading upstream lvm2 code, it
doesn't make much sense to me. Most notably I can't see full error path.
Could it be that pvscan -vvvv output was cut off somehow?

Regards
O.

[fauno]

Post by Ondrej Kozina
This is very strange pvscan -vvvv output. Reading upstream lvm2 code, it
doesn't make much sense to me. Most notably I can't see full error path.
Could it be that pvscan -vvvv output was cut off somehow?
Regards
O.

that's exactly as it was printed

[fauno]

Post by fauno

Post by Ondrej Kozina
This is very strange pvscan -vvvv output. Reading upstream lvm2 code, it
doesn't make much sense to me. Most notably I can't see full error path.
Could it be that pvscan -vvvv output was cut off somehow?
Regards
O.

that's exactly as it was printed

fwiw, on dm-crypt list i was suggested to read the opened device with
`strings` and `hexedit`, but i couldn't find any lvm2 info

[Zdenek Kabelac]

Post by fauno

Post by Ondrej Kozina
This is very strange pvscan -vvvv output. Reading upstream lvm2 code, it
doesn't make much sense to me. Most notably I can't see full error path.
Could it be that pvscan -vvvv output was cut off somehow?
Regards
O.

that's exactly as it was printed

Hi

This was not a full lvm2 -vvvv trace output - unless you would be using
a modified lvm2 source code base.

'pvscan -vvvv --cache device &>out'


Also please provide full lvm2 package version eventually reference to package
with sources (and extra patches) (link).

Regards


Zdenek

[fauno]

Post by Zdenek Kabelac
Hi
This was not a full lvm2 -vvvv trace output - unless you would be using
a modified lvm2 source code base.
'pvscan -vvvv --cache device &>out'
Also please provide full lvm2 package version eventually reference to package
with sources (and extra patches) (link).

i can't confirm the version now, but it was lvm2 from trisquel belenos
(ubuntu trusty, lvm2 2.02.98 it seems[^0]), the hd was running archlinux
with latest lvm2 from repos. i can try later with a more modern lvm2.

the pvscan output was exactly that though, i just selected it from the
terminal i was using. see it finished with error 5, perhaps that's why
it's incomplete?


thanks!


[^0]: http://packages.trisquel.info/belenos/lvm2

[Zdenek Kabelac]

Post by fauno

Post by Zdenek Kabelac
Hi
This was not a full lvm2 -vvvv trace output - unless you would be using
a modified lvm2 source code base.
'pvscan -vvvv --cache device &>out'
Also please provide full lvm2 package version eventually reference to package
with sources (and extra patches) (link).

i can't confirm the version now, but it was lvm2 from trisquel belenos
(ubuntu trusty, lvm2 2.02.98 it seems[^0]), the hd was running archlinux
with latest lvm2 from repos. i can try later with a more modern lvm2.
the pvscan output was exactly that though, i just selected it from the
terminal i was using. see it finished with error 5, perhaps that's why
it's incomplete?
thanks!

So in your case any usage of lvmetad seems rather irrelevant.
(it's been freshly deployed daemon in that age for testing...)

It's unclear where have you get use_lvmpolld in your config file
since I'm sure it's not been in 2.02.98 (introduced around 2.02.120)

So are you mixing usage of various lvm2 versions ?


What does 'vgscan -vvvv' shows on decrypted partition?

Also please provide output of:

dmsetup table
dmsetup status

And yeah 2.02.98 is somewhat out-of-date....
(Make sure your systems' & initramdisk version are somewhat matching)

Regards

Zdenek