Discussion:
[linux-lvm] lvmetad crashes with “memory smash” during lvm parallel operations
Shivarama Krishnan, Gururaj (Nokia - IN/Bangalore)
2017-07-20 08:56:48 UTC
Permalink
Hi,

We went through the code on the this issue and able to point out that the structure memblock in
unprotected from parallel threads.

void dm_bounds_check_debug(void)
{
struct memblock *mb = _head;
while (mb) {
size_t i;
char *ptr = ((char *) (mb + 1)) + mb->length;
for (i = 0; i < sizeof(unsigned long); i++)
if (*ptr++ != (char) mb->id)
assert(!"Memory smash");

mb = mb->next;
}
}

So, if in case another thread try to access remove an entry and at the same time of query of the
list, the lvmetad will crash. It is good have a lock introduced to memoryblock structure.

With Regards,
Gururaj S
"We share We Grow"
Hi ,
Greetings.
We are facing an issue with LVMETAD crashing during the parallel lvm operations
(vgscan/pvscan/lvscan/lvs/vgs)
PKG_NAME=LVM2
PKG_VERSION=2.02.168
PKG_URL=LVM2.2.02.168.tgz
PKG_MD5=00209bdd6befe9a7330f162909313ae8
PKG_DATE=2016-12-01
This issue is also seen in LVM2-2.02.171.
The issue is reproduced on the MIPS Architecture HARDWARE with SCSI DISKS.
Device: TOSHIBA AL13SEB600 Version: 4701
Serial number: 54R0A04HFYP6
Device type: disk
Transport protocol: SAS
# modinfo mptsas
filename: /lib/modules/2.6.34.15
g75d10fa/kernel/drivers/message/fusion/mptsas.ko
version: 3.04.14
license: GPL
description: Fusion MPT SAS Host driver
author: LSI Corporation
srcversion: AF0ADD21F2C0510BB52D860
alias: pci:v00001000d00000062sv*sd*bc*sc*i*
alias: pci:v00001000d00000058sv*sd*bc*sc*i*
alias: pci:v00001000d00000056sv*sd*bc*sc*i*
alias: pci:v00001000d00000054sv*sd*bc*sc*i*
alias: pci:v00001000d00000050sv*sd*bc*sc*i*
depends: mptscsih,mptbase
vermagic: 2.6.34.15
64BIT
parm: mpt_pt_clear: Clear persistency table: enable=1 (default=MPTSCSIH_PT_CLEAR=0)
(int)
parm: max_lun: max lun, default=16895 (int)
So we enabled the debug flag to see the lvmetad operation . I could see that the vg_list
operation
is overlapping the from the other thread and causing the leak in linked list and consequent
"memory
smash" for LVMETAD.
Jul 17 15:04:14 NODE1 lvmeta.sh[4623]: token_update end len 3 pid 5763 new token
filter:3239235440
Jul 17 15:04:14 NODE1 lvmeta.sh[4623]: 5763 global info flags none reason none token
filter:3239235440 update_pid 0
Jul 17 15:04:14 NODE1 lvmeta.sh[4623]: 5763 global info flags none reason none token
filter:3239235440 update_pid 0
Jul 18 12:06:46 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:06:46 NODE1 lvmeta.sh[4623]: pv_list
Jul 18 12:06:47 NODE1 lvmeta.sh[4623]: vg_lookup vgid wxfglR-s3f8-yc3z-sg8K-JcBi-o0Qy-Ic8z0w
name
VG_NODE1
Jul 18 12:06:47 NODE1 lvmeta.sh[4623]: vg_clear_outdated_pvs vgid wxfglR-s3f8-yc3z-sg8K-JcBi-
o0Qy-
Ic8z0w
Jul 18 12:06:48 NODE1 lvmeta.sh[4623]: PV /dev/sda4 VG VG_NODE1 lvm2 [<106.71 GiB /
47.75
GiB free]
Jul 18 12:06:48 NODE1 lvmeta.sh[4623]: vg_lookup vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2
name
VG_System
Jul 18 12:06:48 NODE1 lvmeta.sh[4623]: vg_clear_outdated_pvs vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-
Q49h-
5LWne2
Jul 18 12:06:48 NODE1 lvmeta.sh[4623]: PV /dev/sda3 VG VG_System lvm2 [29.29 GiB /
9.60
GiB free]
Jul 18 12:06:48 NODE1 lvmeta.sh[4623]: Total: 2 [136.00 GiB] / in use: 2 [136.00 GiB] / in no
0 [0 ]
Jul 18 12:07:01 NODE1 lvmeta.sh[4623]: 9936 global info flags none reason none token
filter:3239235440 update_pid 0
Jul 18 12:07:01 NODE1 lvmeta.sh[4623]: 9936 global info flags none reason none token
filter:3239235440 update_pid 0
Jul 18 12:07:01 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: 9977 global info flags none reason none token
filter:3239235440 update_pid 0
Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: 9977 global info flags none reason none token
filter:3239235440 update_pid 0
Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: vg_lookup vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2
name
VG_System
Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: 10003 global info flags none reason none token
filter:3239235440 update_pid 0
Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: 10003 global info flags none reason none token
filter:3239235440 update_pid 0
Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: 10030 global info flags none reason none token
filter:3239235440 update_pid 0
Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: 10030 global info flags none reason none token
filter:3239235440 update_pid 0
Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: vg_lookup vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2
name
VG_System
Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: 10055 global info flags none reason none token
filter:3239235440 update_pid 0
Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: 10055 global info flags none reason none token
filter:3239235440 update_pid 0
Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: vg_clear_outdated_pvs vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-
Q49h-
5LWne2
Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: 10083 global info flags none reason none token
filter:3239235440 update_pid 0
Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: 10083 global info flags none reason none token
filter:3239235440 update_pid 0
Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: vg_lookup vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2
name
VG_System
Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: 10102 global info flags none reason none token
filter:3239235440 update_pid 0
Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: 10102 global info flags none reason none token
filter:3239235440 update_pid 0
Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: vg_clear_outdated_pvs vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-
Q49h-
5LWne2
Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: 10129 global info flags none reason none token
filter:3239235440 update_pid 0
Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: 10129 global info flags none reason none token
filter:3239235440 update_pid 0
Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: vg_lookup vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2
name
VG_System
Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: 10153 global info flags none reason none token
filter:3239235440 update_pid 0
Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: 10153 global info flags none reason none token
filter:3239235440 update_pid 0
Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: 10177 global info flags none reason none token
filter:3239235440 update_pid 0
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: 10177 global info flags none reason none token
filter:3239235440 update_pid 0
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: vg_lookup vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2
name
VG_System
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: vg_clear_outdated_pvs vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-
Q49h-
5LWne2
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: 10200 global info flags none reason none token
filter:3239235440 update_pid 0
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: 10200 global info flags none reason none token
filter:3239235440 update_pid 0
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: vg_clear_outdated_pvs vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-
Q49h-
5LWne2
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: 10225 global info flags none reason none token
filter:3239235440 update_pid 0
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: 10225 global info flags none reason none token
filter:3239235440 update_pid 0
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: vg_lookup vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2
name
VG_System
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: 10250 global info flags none reason none token
filter:3239235440 update_pid 0
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: 10250 global info flags none reason none token
filter:3239235440 update_pid 0
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: vg_clear_outdated_pvs vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-
Q49h-
5LWne2
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: 10276 global info flags none reason none token
filter:3239235440 update_pid 0
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: 10276 global info flags none reason none token
filter:3239235440 update_pid 0
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: vg_lookup vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2
name
VG_System
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: 10297 global info flags none reason none token
filter:3239235440 update_pid 0
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: 10297 global info flags none reason none token
filter:3239235440 update_pid 0
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: 10324 global info flags none reason none token
filter:3239235440 update_pid 0
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: 10324 global info flags none reason none token
filter:3239235440 update_pid 0
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: vg_clear_outdated_pvs vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-
Q49h-
5LWne2
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: vg_lookup vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2
name
VG_System
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: 10344 global info flags none reason none token
filter:3239235440 update_pid 0
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: 10344 global info flags none reason none token
filter:3239235440 update_pid 0
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: 10360 global info flags none reason none token
filter:3239235440 update_pid 0
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: 10360 global info flags none reason none token
filter:3239235440 update_pid 0
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: vg_lookup vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2
name
VG_System
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: 10375 global info flags none reason none token
filter:3239235440 update_pid 0
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: 10375 global info flags none reason none token
filter:3239235440 update_pid 0
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: vg_lookup vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2
name
VG_System
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: vg_clear_outdated_pvs vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-
Q49h-
5LWne2
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: 10395 global info flags none reason none token
filter:3239235440 update_pid 0
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: 10395 global info flags none reason none token
filter:3239235440 update_pid 0
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: 10424 global info flags none reason none token
filter:3239235440 update_pid 0
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: 10424 global info flags none reason none token
filter:3239235440 update_pid 0
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: 10426 global info flags none reason none token
filter:3239235440 update_pid 0
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: 10426 global info flags none reason none token
filter:3239235440 update_pid 0
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: vg_lookup vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2
name
VG_System
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: vg_lookup vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2
name
VG_System
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: vg_clear_outdated_pvs vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-
Q49h-
5LWne2
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: vg_clear_outdated_pvs vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-
Q49h-
5LWne2
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: 10438 global info flags none reason none token
filter:3239235440 update_pid 0
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: vg_clear_outdated_pvs vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-
Q49h-
5LWne2
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: 10438 global info flags none reason none token
filter:3239235440 update_pid 0
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: vg_lookup vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2
name
VG_System
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: 10446 global info flags none reason none token
filter:3239235440 update_pid 0
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: 10446 global info flags none reason none token
filter:3239235440 update_pid 0
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: 10456 global info flags none reason none token
filter:3239235440 update_pid 0
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: 10456 global info flags none reason none token
filter:3239235440 update_pid 0
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: vg_clear_outdated_pvs vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-
Q49h-
5LWne2
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: vg_clear_outdated_pvs vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-
Q49h-
5LWne2
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: 10467 global info flags none reason none token
filter:3239235440 update_pid 0
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: 10467 global info flags none reason none token
filter:3239235440 update_pid 0
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: vg_clear_outdated_pvs vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-
Q49h-
5LWne2
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: 10484 global info flags none reason none token
filter:3239235440 update_pid 0
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: 10484 global info flags none reason none token
filter:3239235440 update_pid 0
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: vg_lookup vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2
name
VG_System
Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: 10502 global info flags none reason none token
filter:3239235440 update_pid 0
Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: 10502 global info flags none reason none token
filter:3239235440 update_pid 0
Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: 10536 global info flags none reason none token
filter:3239235440 update_pid 0
Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: 10536 global info flags none reason none token
filter:3239235440 update_pid 0
Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: 10551 global info flags none reason none token
filter:3239235440 update_pid 0
Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: 10551 global info flags none reason none token
filter:3239235440 update_pid 0
Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: vg_clear_outdated_pvs vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-
Q49h-
5LWne2
Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: set info VG name none uuid ziVnMV-iS3r-PMBW-VVKO-pE9F-
Q49h-
5LWne2 cache_version 351 new_version 352
Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: 10574 global info flags none reason none token
filter:3239235440 update_pid 0
Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: 10574 global info flags none reason none token
filter:3239235440 update_pid 0
Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: set info VG name none uuid ziVnMV-iS3r-PMBW-VVKO-pE9F-
Q49h-
5LWne2 cache_version 351 new_version 353
Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: 10590 global info flags none reason none token
filter:3239235440 update_pid 0
Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: 10590 global info flags none reason none token
filter:3239235440 update_pid 0
Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: vg_update vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2
name
VG_System
Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: update_metadata begin arg_vgid ziVnMV-iS3r-PMBW-VVKO-
pE9F-
Q49h-5LWne2 arg_name VG_System pvid none
Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: update_metadata existing vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-
Q49h-5LWne2 and existing name VG_System
Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: update_metadata for VG_System ziVnMV-iS3r-PMBW-VVKO-pE9F-
Q49h-5LWne2 from 351 to 353
Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: moving PV tpUO22-Ra0R-sXa4-bFH4-eFH1-z2Dr-Nf46dV to VG
#orphan
Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: moving PV tpUO22-Ra0R-sXa4-bFH4-eFH1-z2Dr-Nf46dV to VG
ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2
Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: update_metadata is done for VG_System ziVnMV-iS3r-PMBW-
VVKO-
pE9F-Q49h-5LWne2
Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: pv_found pvid tpUO22-Ra0R-sXa4-bFH4-eFH1-z2Dr-Nf46dV on
device 2051 matches existing
Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: vg_lookup vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2
name
VG_System
Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: 10649 global info flags none reason none token
filter:3239235440 update_pid 0
Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: *** Error in `/usr/bin/lvmetad': malloc(): doubl
e
linked list corrupted: 0x000000ffdc0008c0 ***
Assertion `!"Memory smash"' failed.
Jul 18 12:07:06 NODE1 lvmeta.sh[4623]: /usr/bin/lvmeta.sh: line 5: 5761
Aborted (core dumped) /usr/bin/lvmetad -f -s /run/lvm/lvmetad.socket -l debug
For us it is reproducible every time when we restart the my host. During but it crashes, since the
systemd startup script request parallel vgs/lvs to load to the software.
for i in `seq 1 100`; do lvs & done
This results in crash of lvmetad. on the mentioned version.
Attaching the gdb information (We have full gdb information available in the attachment)
#0 __GI_raise (sig=<optimized out>) at ../sysdeps/unix/sysv/linux/raise.c:58
#1 0x000000fff3f53b2c in __GI_abort () at abort.c:89
#2 0x000000fff3f4910c in __assert_fail_base (fmt=0xfff4081c60 "%s%s%s:%u: %s%sAssertion `%s'
failed.\n%n",
assertion=0xfff4179be8 "!\"Memory smash\"", file=0xfff4179a70 "mm/dbg_malloc.c",
line=<optimized out>, function=<optimized out>) at assert.c:92
#3 0x000000fff3f491dc in __GI___assert_fail (assertion=0xfff4179be8 "!\"Memory smash\"",
file=0xfff4179a70 "mm/dbg_malloc.c", line=<optimized out>,
function=0xfff4179c08 "dm_bounds_check_debug") at assert.c:101
#4 0x000000fff415eab4 in dm_bounds_check_debug () from /usr/lib64/libdevmapper.so.1.02
#5 0x000000fff415ef04 in dm_bounds_check_wrapper () from /usr/lib64/libdevmapper.so.1.02
#6 0x000000fff415e0f0 in dm_malloc_aux_debug () from /usr/lib64/libdevmapper.so.1.02
#7 0x000000fff415ecb4 in dm_malloc_wrapper () from /usr/lib64/libdevmapper.so.1.02
#8 0x000000fff4159d2c in dm_vasprintf () from /usr/lib64/libdevmapper.so.1.02
#9 0x000000fff4159f68 in dm_asprintf () from /usr/lib64/libdevmapper.so.1.02
#10 0x000000aab47155fc in buffer_append_vf ()
#11 0x000000aab4712a80 in daemon_reply_simple ()
#12 0x000000aab470ec88 in _ftext ()
Backtrace stopped: frame did not save the PC
Can you help us on this .
With Regards,
Gururaj S
"We share We grow"
David Teigland
2017-07-24 16:58:29 UTC
Permalink
Hi ,
Greetings.
We are facing an issue with LVMETAD crashing during the parallel lvm operations
(vgscan/pvscan/lvscan/lvs/vgs)
PKG_NAME=LVM2
PKG_VERSION=2.02.168
PKG_URL=LVM2.2.02.168.tgz
PKG_MD5=00209bdd6befe9a7330f162909313ae8
PKG_DATE=2016-12-01
This issue is also seen in LVM2-2.02.171.
Please check if this fix is applied to your version:
https://sourceware.org/git/?p=lvm2.git;a=commit;h=c459f23565e86eb9a8a116995cda7dcaed858b9f

AFAICT all the problems you sent are the fault of the memory debugging
code, not the original problem. You need to compile lvmetad with
debugging, but not memory debugging. Then gdb should give relevant
backtraces.

Dave

Loading...