[linux-lvm] Volume group inconsistent

[Marcos Dutra]

Hi everybody,

Yesterday some storage discs of my VG was with missing status
/dev/mapper/plt-01 platinum
lvm2 a-m 500.00g 24.95g
/dev/mapper/plt-02 platinum
lvm2 a-m 500.00g 12.93g

I tried recover the discs and I used the command:
# vgextend --restoremissing platinum /dev/mapper/plt-01

It worked, removing missing status, but I did a big mistake, i tried to run
this command with multiple discs:
# vgextend --restoremissing platinum /dev/mapper/plt-0[2-9]

After that when I ran pvs command the output shows me checksum error:
/dev/mapper/plt-04: Checksum error
/dev/mapper/plt-06: Checksum error
/dev/mapper/plt-01: Checksum error
Couldn't read volume group metadata.
/dev/mapper/plt-02: Checksum error
Couldn't read volume group metadata.
/dev/mapper/plt-03: Checksum error

I tried restore metadata without success:
# vgcfgrestore -f platinum_02336-1600706364.vg platinum
/dev/mapper/plt-03: Checksum error
/dev/mapper/plt-22: Checksum error
/dev/mapper/plt-19: Checksum error
/dev/mapper/plt-16: Checksum error
/dev/mapper/plt-13: Checksum error
/dev/mapper/plt-11: Checksum error
/dev/mapper/plt-09: Checksum error
/dev/mapper/plt-07: Checksum error
/dev/mapper/plt-01: Checksum error
/dev/mapper/plt-05: Checksum error
/dev/mapper/plt-25: Checksum error
/dev/mapper/plt-23: Checksum error
/dev/mapper/plt-20: Checksum error
/dev/mapper/plt-17: Checksum error
/dev/mapper/plt-14: Checksum error
/dev/mapper/plt-02: Checksum error
/dev/mapper/plt-24: Checksum error
/dev/mapper/plt-21: Checksum error
/dev/mapper/plt-18: Checksum error
/dev/mapper/plt-12: Checksum error
/dev/mapper/plt-10: Checksum error
/dev/mapper/plt-08: Checksum error
/dev/mapper/plt-04: Checksum error
/dev/mapper/plt-06: Checksum error
Cannot restore Volume Group platinum with 24 PVs marked as missing.
Restore failed.

Thanks for help me.

[Zdenek Kabelac]

Post by Marcos Dutra
Hi everybody,
Yesterday some storage discs of my VG was with missing status
/dev/mapper/plt-01 platinum lvm2
a-m 500.00g 24.95g
/dev/mapper/plt-02 platinum
lvm2 a-m 500.00g 12.93g
# vgextend --restoremissing platinum /dev/mapper/plt-01
It worked, removing missing status, but I did a big mistake, i tried to run
# vgextend --restoremissing platinum /dev/mapper/plt-0[2-9]
/dev/mapper/plt-04: Checksum error
/dev/mapper/plt-06: Checksum error
/dev/mapper/plt-01: Checksum error
Couldn't read volume group metadata.
/dev/mapper/plt-02: Checksum error
Couldn't read volume group metadata.
/dev/mapper/plt-03: Checksum error
# vgcfgrestore -f platinum_02336-1600706364.vg
<http://platinum_02336-1600706364.vg> platinum

Without having access to your current metadata - it's hard to give
proper advice.


So please attach 'vgcfgbackup' of your current setup.

Have you tried to 'extend' your existing vg 'plt' with referencing LVs from
itself???
Is 'plt' VG ?

Zdenek

[Marcos Dutra]

Hi Zdenek,

I attached 2 files, one with missing status and another my backup today. I
restored my vg from backup but I still have problem, if I create some new
lv volumes, vg status come back with missing status, I don't know why.
Since saturday night I don't create any volumes in this vg and I don't have
any missing status until now, but I would like come back to create new lv
in this volume.
My server is centos-6.6 with lastest updates.

Thanks

Marcos Dutra

Post by Zdenek Kabelac

Post by Marcos Dutra
Hi everybody,
Yesterday some storage discs of my VG was with missing status
/dev/mapper/plt-01 platinum
lvm2
a-m 500.00g 24.95g
/dev/mapper/plt-02 platinum
lvm2 a-m 500.00g 12.93g
# vgextend --restoremissing platinum /dev/mapper/plt-01
It worked, removing missing status, but I did a big mistake, i tried to run
# vgextend --restoremissing platinum /dev/mapper/plt-0[2-9]
/dev/mapper/plt-04: Checksum error
/dev/mapper/plt-06: Checksum error
/dev/mapper/plt-01: Checksum error
Couldn't read volume group metadata.
/dev/mapper/plt-02: Checksum error
Couldn't read volume group metadata.
/dev/mapper/plt-03: Checksum error
# vgcfgrestore -f platinum_02336-1600706364.vg
<http://platinum_02336-1600706364.vg> platinum

Without having access to your current metadata - it's hard to give
proper advice.
So please attach 'vgcfgbackup' of your current setup.
Have you tried to 'extend' your existing vg 'plt' with referencing LVs
from itself???
Is 'plt' VG ?
Zdenek
_______________________________________________
linux-lvm mailing list
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

[Zdenek Kabelac]

Post by Marcos Dutra
Hi Zdenek,
I attached 2 files, one with missing status and another my backup today. I
restored my vg from backup but I still have problem, if I create some new lv
volumes, vg status come back with missing status, I don't know why. Since
saturday night I don't create any volumes in this vg and I don't have any
missing status until now, but I would like come back to create new lv in this
volume.
My server is centos-6.6 with lastest updates.

What are those /dev/mapper/plt-XX devices ?

It seems like the lvm2 command cannot write to this place properly?

Checksum error looks like serious problem.

From attached output - all your PVs were properly markes as not missing,
however the lvm2 cannot read back proper crc code.

So what are those 0.5GB devices - are they properly writable ?
(especially in the disk header)

Note - there are special type of block devices like DASD devices
that must be first properly configured by IBM storage manager to provide
regular devices.

Regards

Zdenek

[Marcos Dutra]

Zdenek,

My multipath devices have an alias to plt-XX instead of the default name
mpathXX.

I can create new volumes with lvcreate, but after 3 or 4 new volumes my VG
pass to inconsistent status because 24 disks stay in missing status, I
don't know why. Do you think that the lvm cannot write properly?

My disks are provided by IBM storage DS8000 but I don't have access the
configuration. Those devices was working properly until last friday since
august 2014.

When I restore my backup with vgcfgrestore, my VG return active without
problem, then I believe I don't have problens with write properly. Do you
have any idea or a test to do? I cannot shut off this VG now because it's
active and used by virtual servers, I have a big problem now.

I created a new VG to create new volumes but I'd like fix this one.

Thanks

Post by Zdenek Kabelac

Post by Marcos Dutra
Hi Zdenek,
I attached 2 files, one with missing status and another my backup today. I
restored my vg from backup but I still have problem, if I create some new lv
volumes, vg status come back with missing status, I don't know why. Since
saturday night I don't create any volumes in this vg and I don't have any
missing status until now, but I would like come back to create new lv in this
volume.
My server is centos-6.6 with lastest updates.

What are those /dev/mapper/plt-XX devices ?
It seems like the lvm2 command cannot write to this place properly?
Checksum error looks like serious problem.
From attached output - all your PVs were properly markes as not missing,
however the lvm2 cannot read back proper crc code.
So what are those 0.5GB devices - are they properly writable ?
(especially in the disk header)
Note - there are special type of block devices like DASD devices
that must be first properly configured by IBM storage manager to provide
regular devices.
Regards
Zdenek
_______________________________________________
linux-lvm mailing list
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

[service hofman]

This post might be inappropriate. Click to display it.

[Marcos Dutra]

Thanks again Eric!

You're wright, I have 64 pv each one 500GB hdd. I have problems with my pv
starting by pv0 until pv23, I had seen after dd lvm doesn't recognize the
devices of these pvs, status "unknown device", others devices I don't have
the same problem. I cut this output from dd of hdisk plt-85 = pv64

Device with problem

-----

pv0 {
id = "N7lOu9-Ahjn-lf19-WVFW-xglN-yfL6-KOOw6O"
device = "unknown device"

status = ["ALLOCATABLE"]
flags = ["MISSING"]
dev_size = 1048576000
pe_start = 2048
pe_count = 127999
}

-----

Device without problem

-----

pv24 {
id = "D8aEQf-2CVS-oyVE-t40G-kOd7-eUe5-SkCe8V"
device = "/dev/mapper/mpathr"

status = ["ALLOCATABLE"]
flags = []
dev_size = 1048576000
pe_start = 2048
pe_count = 127999
}

-----


But the output from another hdisk plt-01 = pv0 I have different view.

-----

pv0 {
id = "N7lOu9-Ahjn-lf19-WVFW-xglN-yfL6-KOOw6O"
device = "/dev/mapper/plt-01"

status = ["ALLOCATABLE"]
flags = ["MISSING"]
dev_size = 1048576000
pe_start = 2048
pe_count = 127999
}

-----

I'm getting crazy lol...

Thank you

Marcos Dutra

Post by service hofman
----
pv64 {
id = "TFSqaf-Wd6b-9w2r-8ZLp-rVN9-5851-0qNX0z"
device = "/dev/mapper/plt-85" # Hint only
status = ["ALLOCATABLE"]
flags = []
dev_size = 1048576000 # 500 Gigabytes
pe_start = 2048
pe_count = 127999 # 499.996 Gigabytes
}
}
----
pvXX this is One for Team or for [hdd] /dev/sdX
this look like you have 64hdd for 500Gb one.
and first look MBR 0-2Mb
dd if=/dev/sdX of=/boot/sdX.mbr bs=512 count=4000
in side sdX.mbr is lvm.vg
if is this same [ platinum_after_missing.vg ]
you found all record /etc/lvm/*
rest is easy.
device is inode, but not lv.
[ -b "/dev/mapper/plt-01" ] && echo "True " || echo "False"
-----
logical_volumes {
volume-a2d55040-a597-419b-95b6-403f5b287773 {
id = "GA9jj9-hooD-Au8T-23Bw-LNox-LkWF-G1TUG6"
status = ["READ", "WRITE", "VISIBLE"]
flags = []
creation_host = "brasilia"
creation_time = 1409946331 # 2014-09-05
16:45:31 -0300
segment_count = 1
segment1 {
start_extent = 0
extent_count = 1024 # 4 Gigabytes
type = "striped"
stripe_count = 1 # linear
stripes = [
"pv10", 30730
]
}
}
-----
segment1 == lv like /dev/sdX1
pv10==[hdd X], offset[sdX1] == [ -->[xxx] ]
but every one is start == 0[+offset]
30730*4096=[offset]
by recovery us dd if=/dev/sdX1 bs=4096 count=(1024*1000) seek=($offset)
this not LVM only IBM LLVM
LINUX use LVM. kernel.org
IBM use AIX kernel
ComputerAlarmTechnologySystem©
http://plus.google.com/105013137826111836446
[phone]
_______________________________________________
linux-lvm mailing list
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

[service hofman]

Say something about yourself
would you like to do something,
more for others.
are you able to help you,

the problem is not knowing.

but the time spent.
as to what should be excellent .

what you are doing nothing,
what you can do for others.
LVM is a good start.
but much remains to be done.
look on the transport speed ,
your hdd . what you are doing is pointless.

why ?
a transport speed of the system.
my to 6Gb/s copy file
hdd has 4Gb/s copy file


? check speed system and hdd.
and tell me that I could change

Now, ...
Before you change anything,
See how it works. Do not test .
You have always help Adding sutfix --help
or use the performance
prefix man

And always cheak
/usr/share/doc/ and read all the book
Install every doc book


Ok.
By test recovery Hdd is Not LVM
try not give to mount for system LVM,
Because you lost again ....

First
Look
Install hdparm
hdparm -h --help.

Ask for speed.

Hdparm -Tt
Hdparm -iI

fdisk /dev/sdX

pvs -o pv_all,gv_all,lv_all,seg_all

[Marcos Dutra]

Thank you Eric,

Your help are welcome. I'm not an expert on LVM, but if I know something to
solve the problem, I will help them. I participated on Debian-users list,
squid, qmail and ldap, now I'm migrating my technical knowledge to LVM,
openstack and distributed file system like ceph, it's a new world for me.

I'm studying my case yet, all disks are over SAN IBM storage, so hdparm in
my case will not work at all. Unfortunately I don't have any support on OS
system (Centos), if I find something new I will post here.

Thank you

Marcos Dutra

Post by service hofman
Say something about yourself
would you like to do something,
more for others.
are you able to help you,
the problem is not knowing.
but the time spent.
as to what should be excellent .
what you are doing nothing,
what you can do for others.
LVM is a good start.
but much remains to be done.
look on the transport speed ,
your hdd . what you are doing is pointless.
why ?
a transport speed of the system.
my to 6Gb/s copy file
hdd has 4Gb/s copy file
? check speed system and hdd.
and tell me that I could change
Now, ...
Before you change anything,
See how it works. Do not test .
You have always help Adding sutfix --help
or use the performance
prefix man
And always cheak
/usr/share/doc/ and read all the book
Install every doc book
Ok.
By test recovery Hdd is Not LVM
try not give to mount for system LVM,
Because you lost again ....
First
Look
Install hdparm
hdparm -h --help.
Ask for speed.
Hdparm -Tt
Hdparm -iI
fdisk /dev/sdX
pvs -o pv_all,gv_all,lv_all,seg_all
--
ComputerAlarmTechnologySystem©
http://plus.google.com/105013137826111836446
[phone]
_______________________________________________
linux-lvm mailing list
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

[service hofman]

Hi

dd if=/dev/sda of=/boor/sda.mbr bs=512 count=4000

mcedit -e /boot/sda.mbr
or
hexedit


look lvm list in bootrecord

eric hofman

***@gmail.com

[Marcos Dutra]

Hi Eric,

I ran the dd command and I saw the flags "MISSING" yet, if I run pvs, I
don't see MISSING flag.

# pvs /dev/mapper/plt-25
PV VG Fmt Attr PSize PFree
/dev/mapper/plt-25 platinum lvm2 a-- 500.00g 0

The same disk after dd command (dd if=/dev/mapper/plt-25 of=plt-25.br
bs=512 count=4000)

pv23 {
id = "CEPd6z-Bklc-BNsq-pqcn-SMhu-jdVS-6GIPI6"
device = "/dev/mapper/plt-25"

status = ["ALLOCATABLE"]
flags = ["MISSING"]
dev_size = 1048576000
pe_start = 2048
pe_count = 127999
}

Now I know why I receive the "MISSING" flag after create new volume, how
can I fix it? Do you have any idea?

Before I restored my lvm backup with vgcfgrestore, I can see now it didn't
work properly.

Thank you

Marcos Dutra

Post by service hofman
Hi
dd if=/dev/sda of=/boor/sda.mbr bs=512 count=4000
mcedit -e /boot/sda.mbr
or
hexedit
look lvm list in bootrecord
eric hofman
--
ComputerAlarmTechnologySystem©
http://plus.google.com/105013137826111836446
[phone]
_______________________________________________
linux-lvm mailing list
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/