[linux-lvm] The node was fenced in the cluster when cmirrord was enabled on LVM2.2.02.120

[Gang He]

Hello List,

I got a bug report from the customer, which said the node was fenced in the cluster when they enabled cmirrord.
Before the node was fenced, we can see some log printed as below,

2018-09-25T12:55:26.555018+02:00 qu1ci11 cmirrord[6253]: cpg_mcast_joined error: 2
2018-09-25T12:55:31.604832+02:00 qu1ci11 sbd[2865]: warning: inquisitor_child: /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi2-0-0-2 requested a reset
2018-09-25T12:55:31.608112+02:00 qu1ci11 sbd[2865]: emerg: do_exit: Rebooting system: reboot
2018-09-25T12:55:33.202189+02:00 qu1ci11 kernel: [ 4750.932328] device-mapper: dm-log-userspace: [LYuPIux2] Request timed out: [15/93273] - retrying
2018-09-25T12:55:35.186091+02:00 qu1ci11 kernel: [ 4752.916268] device-mapper: dm-log-userspace: [LYuPIux2] Request timed out: [9/93274] - retrying
2018-09-25T12:55:41.382129+02:00 qu1ci11 kernel: [ 4759.112231] device-mapper: dm-log-userspace: [LYuPIux2] Request timed out: [15/93275] - retrying
2018-09-25T12:55:41.382157+02:00 qu1ci11 kernel: [ 4759.116237] device-mapper: dm-log-userspace: [LYuPIux2] Request timed out: [15/93276] - retrying
2018-09-25T12:55:41.534092+02:00 qu1ci11 kernel: [ 4759.264201] device-mapper: dm-log-userspace: [LYuPIux2] Request timed out: [15/93278] - retrying
2018-09-25T12:55:41.534117+02:00 qu1ci11 kernel: [ 4759.264274] device-mapper: dm-log-userspace: [LYuPIux2] Request timed out: [15/93279] - retrying
2018-09-25T12:55:41.534119+02:00 qu1ci11 kernel: [ 4759.264278] device-mapper: dm-log-userspace: [LYuPIux2] Request timed out: [15/93277] - retrying
...

2018-09-25T12:56:26.439557+02:00 qu1ci11 lrmd[3795]: warning: rsc_VG_ASCS_monitor_60000 process (PID 4467) timed out
2018-09-25T12:56:26.439974+02:00 qu1ci11 lrmd[3795]: warning: rsc_VG_ASCS_monitor_60000:4467 - timed out after 60000ms
2018-09-25T12:56:26.534104+02:00 qu1ci11 kernel: [ 4804.264240] device-mapper: dm-log-userspace: [LYuPIux2] Request timed out: [15/93321] - retrying
2018-09-25T12:56:26.534122+02:00 qu1ci11 kernel: [ 4804.264287] device-mapper: dm-log-userspace: [LYuPIux2] Request timed out: [15/93320] - retrying
2018-09-25T12:56:26.534124+02:00 qu1ci11 kernel: [ 4804.264311] device-mapper: dm-log-userspace: [LYuPIux2] Request timed out: [15/93322] - retrying

Did you guys encounter the similar issue before? I can find the similar bug report at http://lists.linux-ha.org/pipermail/linux-ha/2014-December/048427.html
If you know the root cause, please let me know.


Thanks
Gang

[Gang He]

Hello Guys,

Did you see this problem before?
it looks there was a similar problem from the Redhat website https://access.redhat.com/solutions/1421123#.


Thanks
Gang

Post by Gang He

On 2018/10/19 at 17:05, in message

Hello List,
I got a bug report from the customer, which said the node was fenced in the
cluster when they enabled cmirrord.
Before the node was fenced, we can see some log printed as below,
2018-09-25T12:55:26.555018+02:00 qu1ci11 cmirrord[6253]: cpg_mcast_joined error: 2
inquisitor_child: /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi2-0-0-2
requested a reset
Rebooting system: reboot
dm-log-userspace: [LYuPIux2] Request timed out: [15/93273] - retrying
dm-log-userspace: [LYuPIux2] Request timed out: [9/93274] - retrying
dm-log-userspace: [LYuPIux2] Request timed out: [15/93275] - retrying
dm-log-userspace: [LYuPIux2] Request timed out: [15/93276] - retrying
dm-log-userspace: [LYuPIux2] Request timed out: [15/93278] - retrying
dm-log-userspace: [LYuPIux2] Request timed out: [15/93279] - retrying
dm-log-userspace: [LYuPIux2] Request timed out: [15/93277] - retrying
...
rsc_VG_ASCS_monitor_60000 process (PID 4467) timed out
rsc_VG_ASCS_monitor_60000:4467 - timed out after 60000ms
dm-log-userspace: [LYuPIux2] Request timed out: [15/93321] - retrying
dm-log-userspace: [LYuPIux2] Request timed out: [15/93320] - retrying
dm-log-userspace: [LYuPIux2] Request timed out: [15/93322] - retrying
Did you guys encounter the similar issue before? I can find the similar bug
report at http://lists.linux-ha.org/pipermail/linux-ha/2014-December/048427.html
If you know the root cause, please let me know.
Thanks
Gang
_______________________________________________
linux-lvm mailing list
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/