[linux-lvm] [lvmlockd] recovery lvmlockd after kill_vg

[Damon Wang]

Hi,

AFAIK once sanlock can not access lease storage, it will run
"kill_vg" to lvmlockd, and the standard process should be deactivate
logical volumes and drop vg locks.

But sometimes the storage will recovery after kill_vg(and before we
deactivate or drop lock), and then it will prints "storage failed for
sanlock leases" on lvm commands like this:

[***@dev1-2 ~]# vgck 71b1110c97bd48aaa25366e2dc11f65f
WARNING: Not using lvmetad because config setting use_lvmetad=0.
WARNING: To avoid corruption, rescan devices to make changes visible
(pvscan --cache).
VG 71b1110c97bd48aaa25366e2dc11f65f lock skipped: storage failed for
sanlock leases
Reading VG 71b1110c97bd48aaa25366e2dc11f65f without a lock.

so what should I do to recovery this, (better) without affect
volumes in using?

I find a way but it seems very tricky: save "lvmlockctl -i" output,
run lvmlockctl -r vg and then activate volumes as the previous output.

Do we have an "official" way to handle this? Since it is pretty
common that when I find lvmlockd failed, the storage has already
recovered.

Thanks,
Damon Wang

[David Teigland]

Post by Damon Wang
Hi,
AFAIK once sanlock can not access lease storage, it will run
"kill_vg" to lvmlockd, and the standard process should be deactivate
logical volumes and drop vg locks.
But sometimes the storage will recovery after kill_vg(and before we
deactivate or drop lock), and then it will prints "storage failed for
WARNING: Not using lvmetad because config setting use_lvmetad=0.
WARNING: To avoid corruption, rescan devices to make changes visible
(pvscan --cache).
VG 71b1110c97bd48aaa25366e2dc11f65f lock skipped: storage failed for
sanlock leases
Reading VG 71b1110c97bd48aaa25366e2dc11f65f without a lock.
so what should I do to recovery this, (better) without affect
volumes in using?
I find a way but it seems very tricky: save "lvmlockctl -i" output,
run lvmlockctl -r vg and then activate volumes as the previous output.
Do we have an "official" way to handle this? Since it is pretty
common that when I find lvmlockd failed, the storage has already
recovered.

Hi, to figure out that workaround, you've probably already read the
section of the lvmlockd man page: "sanlock lease storage failure", which
gives some background about what's happening and why. What the man page
is missing is some help about false failure detections like you're seeing.

It sounds like io delays from your storage are a little longer than
sanlock is allowing for. With the default 10 sec io timeout, sanlock will
initiate recovery (kill_vg in lvmlockd) after 80 seconds of no successful
io from the storage. After this, it decides the storage has failed. If
it's not failed, just slow, then the proper way to handle that is to
increase the timeouts. (Or perhaps try to configure the storage to avoid
such lengthy delays.) Once a failure is detected and recovery is begun,
there's not an official way to back out of it.

You can increase the sanlock io timeout with lvmlockd -o <seconds>.
sanlock multiplies that by 8 to get the total length of time before
starting recovery. I'd look at how long your temporary storage outages
last and set io_timeout so that 8*io_timeout will cover it.

Dave

[Damon Wang]

This post might be inappropriate. Click to display it.

[David Teigland]

Post by Damon Wang
Thank you for your reply, I have another question under such circumstances.
I usually run "vgck" to check weather vg is good, but sometimes it
seems it stuck, and leave a VGLK on sanlock. (I'm sure io error will
cause it, but sometimes not because io error)
Then i'll try use sanlock client release -r xxx to release it, but it
also sometimes not work.(be stuck)
Then I may lvmlockctl -r to drop vg lockspace, but it still may stuck,
and I'm io is ok when it stuck
This usually happens on multipath storage, I consider multipath will
queue some io is blamed, but not sure.
Any idea?

First, you might be able to avoid this issue by doing the check using
something other than an lvm command, or perhaps and lvm command configured
to avoid taking locks (the --nolocking option in vgs/pvs/lvs). What's
appropriate depends on specifically what you want to know from the check.

I still haven't fixed the issue you found earlier, which sounds like it
could be the same or related to what you're describing now.
https://www.redhat.com/archives/linux-lvm/2018-July/msg00011.html

As for manually cleaning up a stray lock using sanlock client, there may
be some limits on the situations that works in, I don't recall off hand.
You should try using the -p <pid> option with client release to match the
pid of lvmlockd.

Configuring multipath to fail more quickly instead of queueing might give
you a better chance of cleaning things up.

Dave

[David Teigland]

as the manual says, it should deactivate volumes and drop lockspace as
quick as possible,

A year ago we discussed a more automated solution for forcing a VG offline
when its sanlock lockspace was shut down:

https://www.redhat.com/archives/lvm-devel/2017-September/msg00011.html

The idea was to forcibly shut down LVs (using dmsetup wipe_table) in the
VG when the kill_vg happened, then to automatically do the 'lvmlockctl
--drop' when the LVs were safely shut down. There were some loose ends
around integrating this solution that I never sorted out, so it's remained
on my todo list.

TTY can get a message, but it's not a good way to listen or monitor, so I
run vgck periodically and parse its stdout and stderr, once "sanlock lease
storage failure" or
something unusual happens, an alert will be triggered and I'll do some
check(I hope all this process can be automatically).

If you are specifically interested in detecting this lease timeout
condition, there are some sanlock commands that can give you this info.
You can also detect ahead of time that a VG's lockspace is getting close
the threshold. I can get back to you with more specific fields to look
at, but for now take a look at 'sanlock client renewal' and some of the
internal details that are printed by 'sanlock client status -D'.

Dave

[Damon Wang]

It's really help, I noticed "sanlock client renewal" returns practical
information, but not noticed "sanlock client status -D", the later is
what I exactly want, thanks!

Damon

Post by David Teigland

as the manual says, it should deactivate volumes and drop lockspace as
quick as possible,

A year ago we discussed a more automated solution for forcing a VG offline
https://www.redhat.com/archives/lvm-devel/2017-September/msg00011.html
The idea was to forcibly shut down LVs (using dmsetup wipe_table) in the
VG when the kill_vg happened, then to automatically do the 'lvmlockctl
--drop' when the LVs were safely shut down. There were some loose ends
around integrating this solution that I never sorted out, so it's remained
on my todo list.

TTY can get a message, but it's not a good way to listen or monitor, so I
run vgck periodically and parse its stdout and stderr, once "sanlock lease
storage failure" or
something unusual happens, an alert will be triggered and I'll do some
check(I hope all this process can be automatically).

If you are specifically interested in detecting this lease timeout
condition, there are some sanlock commands that can give you this info.
You can also detect ahead of time that a VG's lockspace is getting close
the threshold. I can get back to you with more specific fields to look
at, but for now take a look at 'sanlock client renewal' and some of the
internal details that are printed by 'sanlock client status -D'.
Dave