[linux-lvm] Distributed Locking of LVM

[Kalyana sundaram]

We have a shared iscsi disk at all our boxes, with one vg. We are using
clvm to manage locks to create extend activate delete list.
CLVM is tedious to manage especially especially due to reboot if locking is
struck somewhere. Instead we are thinking of modifying file_locking.c to
get lock with distributed store like zookeeper or redis and on success run
the above command, on failure backoff and try again
The external ocking code will sit at _file_lock_resource, case LCK_VG.
Do you think its risky. Is there some other way people handle it?

[Digimer]

Post by Kalyana sundaram
We have a shared iscsi disk at all our boxes, with one vg. We are using
clvm to manage locks to create extend activate delete list.
CLVM is tedious to manage especially especially due to reboot if locking
is struck somewhere. Instead we are thinking of modifying file_locking.c
to get lock with distributed store like zookeeper or redis and on
success run the above command, on failure backoff and try again
The external ocking code will sit at _file_lock_resource, case LCK_VG.
Do you think its risky. Is there some other way people handle it?

I think you're trying to work around a problem, instead of solving it.
The only time locking blocks with DLM is if a node entered an unknown
state and fencing wasn't setup or failed (in which case hanging is the
least bad option available).

A lot of thought and design went into DLM / clvmd. I would be cautious
trying to reinvent a wheel this complex. Instead, do you have any
indication why blocking occurs? Do you have fencing setup and working?
What cluster stack/version/OS are you using?

You might want to post on the clusterlabs list, too.

Users: .... http://lists.clusterlabs.org/mailman/listinfo/users
Developers: http://lists.clusterlabs.org/mailman/listinfo/developers

[Zdenek Kabelac]

We have a shared iscsi disk at all our boxes, with one vg. We are using clvm
to manage locks to create extend activate delete list.
CLVM is tedious to manage especially especially due to reboot if locking is
struck somewhere. Instead we are thinking of modifying file_locking.c to get
lock with distributed store like zookeeper or redis and on success run the
above command, on failure backoff and try again
The external ocking code will sit at _file_lock_resource, case LCK_VG.
Do you think its risky. Is there some other way people handle it?

Hi

Have you considered usage of 'lvmlockd' and it's sanlock interface
(where locks are maintained on shared storage device).

Regards

Zdenek

[David Teigland]

Post by Zdenek Kabelac

Post by Kalyana sundaram
We have a shared iscsi disk at all our boxes, with one vg. We are using
clvm to manage locks to create extend activate delete list.
CLVM is tedious to manage especially especially due to reboot if locking
is struck somewhere. Instead we are thinking of modifying file_locking.c
to get lock with distributed store like zookeeper or redis and on
success run the above command, on failure backoff and try again
The external ocking code will sit at _file_lock_resource, case LCK_VG.
Do you think its risky. Is there some other way people handle it?

Hi
Have you considered usage of 'lvmlockd' and it's sanlock interface
(where locks are maintained on shared storage device).

And lvmlockd can use different lock managers. Currently it supports
sanlock (where the locks are kept in the VG's storage and networking is
not used), or the dlm (which uses the network and requires corosync). You
also could try adding another lock manager option to lvmlockd.

See here for more info:
http://man7.org/linux/man-pages/man8/lvmlockd.8.html

I think people will find lvmlockd works much better than clvm.
https://www.redhat.com/archives/linux-lvm/2016-November/msg00022.html

[Kalyana sundaram]

Thanks all people
I understand reboot/fencing is mandatory
I hope the visibility might be better in external locking tool like redis
With lvmlockd I find no deb available for ubuntu, and documentations for
clvm to handle an issue is difficult to find
Could somebody redirect me for a good documentation on clvm?
Let me compile lvmlockd and give it a shot

Post by Kalyana sundaram

Post by Zdenek Kabelac

Post by Kalyana sundaram
We have a shared iscsi disk at all our boxes, with one vg. We are using
clvm to manage locks to create extend activate delete list.
CLVM is tedious to manage especially especially due to reboot if

locking

Post by Zdenek Kabelac

Post by Kalyana sundaram
is struck somewhere. Instead we are thinking of modifying

file_locking.c

Post by Zdenek Kabelac

Post by Kalyana sundaram
to get lock with distributed store like zookeeper or redis and on
success run the above command, on failure backoff and try again
The external ocking code will sit at _file_lock_resource, case LCK_VG.
Do you think its risky. Is there some other way people handle it?

Hi
Have you considered usage of 'lvmlockd' and it's sanlock interface
(where locks are maintained on shared storage device).

And lvmlockd can use different lock managers. Currently it supports
sanlock (where the locks are kept in the VG's storage and networking is
not used), or the dlm (which uses the network and requires corosync). You
also could try adding another lock manager option to lvmlockd.
http://man7.org/linux/man-pages/man8/lvmlockd.8.html
I think people will find lvmlockd works much better than clvm.
https://www.redhat.com/archives/linux-lvm/2016-November/msg00022.html

[Zdenek Kabelac]

Post by Kalyana sundaram
Thanks all people
I understand reboot/fencing is mandatory
I hope the visibility might be better in external locking tool like redis
With lvmlockd I find no deb available for ubuntu, and documentations for clvm
to handle an issue is difficult to find

Unfortunately Debian based distros are not 'the best fit' for anything lvm2
related and it's not lvm2 fault ;) and it's pretty hard to fix....

Maybe you can try some other distro ?
Or eventually building things from source ?


Regards


Zdenek

[John Stoffel]

Post by Kalyana sundaram
Thanks all people
I understand reboot/fencing is mandatory
I hope the visibility might be better in external locking tool like redis
With lvmlockd I find no deb available for ubuntu, and documentations for clvm
to handle an issue is difficult to find

Zdenek> Unfortunately Debian based distros are not 'the best fit' for
Zdenek> anything lvm2 related and it's not lvm2 fault ;) and it's
Zdenek> pretty hard to fix....

Can you please elaborate on this? I use Debian distros by
preference. Is it because they've changed stuff to fit the debian
ideals too much?

John

[Peter Rajnoha]

Post by John Stoffel

Post by Kalyana sundaram
Thanks all people
I understand reboot/fencing is mandatory
I hope the visibility might be better in external locking tool like redis
With lvmlockd I find no deb available for ubuntu, and documentations for clvm
to handle an issue is difficult to find

Zdenek> Unfortunately Debian based distros are not 'the best fit' for
Zdenek> anything lvm2 related and it's not lvm2 fault ;) and it's
Zdenek> pretty hard to fix....
Can you please elaborate on this? I use Debian distros by
preference. Is it because they've changed stuff to fit the debian
ideals too much?

It's more about the attitude of the lvm2 package maintenance in Debian.

There were times when certain things didn't work quite well because
there were changes done apart from upstream release and they were not
correct, causing various problems (e.g. Debian used changed udev rules
for some time which had some issues and even if upstream warned about
these issues right away, it took some time to fix them - by not using
these extra changes apart from upstream). Also, there were certain
decisions done for Debian's lvm2 package in the past like removing the
clvmd altogether from the package, not taking into account there might
be active users still and not providing any alternative properly (the
clvmd is now back).

So it's all about these issues and decisions which make lvm2 in Debian
not that trustworthy, at least not from upstream's perspective.

Things may have changed and improved these days, but upstream is now
very cautious when it comes to recommending Debian for lvm2 usage (that
also implies Ubuntu usage as the packages are somewhat inherited from
Debian there).