[linux-lvm] Data percentage too large after thin_dump --repair/thin_restore

[Timur Alperovich]

Hi there,

I'm using LVM 2.02.98 and encountered a metadata corruption issue. To
recover from it, I attempted to perform the following steps:
1. thin_check /dev/mapper/pool_tmeta
2. thin_dump /dev/mapper/pool_tmeta > /tmp/metadata
3. dd if=/dev/zero of=/dev/mapper/pool_tmeta
4. thin_restore -i /tmp/metadata -o /dev/mapper/pool_tmeta

All of the above steps have succeeded, however, when attempting to list the
_metadata\_percent_ field, I get an error:

***@thin_vm:~$ sudo lvs group/pool -o +metadata_percent
Data percentage too large.
dm_report_object: report function failed for field metadata_percent
LV VG Attr LSize Pool Origin Data% Move Log Copy%
Convert Meta%
pool group twi-a-tz- 383.88g 34.56


The pool itself can now be activated, `thin_check` passes, and all the data
appears to be intact.

Is this a known issue and is there a workaround? I need to be able to
examine the _metadata\_percent_ field to make sure we don't exhaust the
metadata space.

Thank you,
Timur

[Zdenek Kabelac]

Post by Timur Alperovich
Hi there,
I'm using LVM 2.02.98 and encountered a metadata corruption issue. To recover
1. thin_check /dev/mapper/pool_tmeta
2. thin_dump /dev/mapper/pool_tmeta > /tmp/metadata

Hi

NEVER EVER use _tmeta device from running active thin-pool volume.
It's the very same case like you would be running 'fsck' on
a mounted filesystem.

Post by Timur Alperovich
3. dd if=/dev/zero of=/dev/mapper/pool_tmeta
4. thin_restore -i /tmp/metadata -o /dev/mapper/pool_tmeta
All of the above steps have succeeded, however, when attempting to list the

I'm surprised you've not got kernel OOPS after such brutal destruction
of life metadata device (i.e. almost equal to zeroing your root volume).

Post by Timur Alperovich
Is this a known issue and is there a workaround? I need to be able to examine
the _metadata\_percent_ field to make sure we don't exhaust the metadata space.

Normal way -

lvconvert --repair vg/pool


If this is not working - then you can 'swap' metadata out of your thin-pool
using following sequence-

- make sure pool is not active.
- build temporary local LV (lvcreate -l1 vg -n temp)
- swap this LV with metadata of to-be-repaired pool
(lvconvert --thinpool vg/fixpool --poolmetadata temp)
- activate 'temp' LV now with pool's metadata
(lvchange -ay vg/temp)
- repair metadata
(you may need other 'bigger' volume to restore fixed metadata)
(i.e. thin_restore -i /dev/vg/temp -o /dev/vg/biggertemp)
- thin_check restored volume
- thin_dump - and compare vgcfgrestore & this dump are at the same
transaction_id state (look at lvm2 metadata for thin pool)
- deactivate again related volumes
- swap repaired LV back
(lvconvert --thinpool vg/fixpool --poolmetadata repairedtemp)
- try to active repaired thin pool
- remove unneeded volumes from vg....


Zdenek

[Timur Alperovich]

Thanks for the reply Zdenek!

I realized that the biggest confusion for me is that I'm using LVM2 tools
2.02.98-6 (which is the latest version in Ubuntu) and the swapping support
was added in 2.02.98-9 (I'm curious why they didn't pick the latest of
2.02.98).

Post by Timur Alperovich
Hi there,

Post by Timur Alperovich
I'm using LVM 2.02.98 and encountered a metadata corruption issue. To recover
1. thin_check /dev/mapper/pool_tmeta
2. thin_dump /dev/mapper/pool_tmeta > /tmp/metadata

Hi
NEVER EVER use _tmeta device from running active thin-pool volume.
It's the very same case like you would be running 'fsck' on
a mounted filesystem.

3. dd if=/dev/zero of=/dev/mapper/pool_tmeta

Post by Timur Alperovich

Post by Timur Alperovich
4. thin_restore -i /tmp/metadata -o /dev/mapper/pool_tmeta
All of the above steps have succeeded, however, when attempting to list the

I'm surprised you've not got kernel OOPS after such brutal destruction
of life metadata device (i.e. almost equal to zeroing your root volume).
Is this a known issue and is there a workaround? I need to be able to

Post by Timur Alperovich
examine
the _metadata\_percent_ field to make sure we don't exhaust the metadata space.

Normal way -
lvconvert --repair vg/pool

I believe this was added in 2.02.102. Unfortunately, upgrading LVM2 is a
bit tricky on Ubuntu (will need to build the package from source).

Post by Timur Alperovich
If this is not working - then you can 'swap' metadata out of your thin-pool
using following sequence-
- make sure pool is not active.
- build temporary local LV (lvcreate -l1 vg -n temp)
- swap this LV with metadata of to-be-repaired pool
(lvconvert --thinpool vg/fixpool --poolmetadata temp)

This command fails, as --poolmetadata is not a valid option. I found that
the swapping was not supported until 2.02.98-9 and I'm using 2.02.98-6 (in
2.02.98-9 changelog: "Add lvconvert support to swap thin pool metadata
volume."). Is there a way to do this without the swapping support? I guess
I should be able to do it with dmsetup? I'm not sure how painful that would
be.

I'll look into upgrading the version, as well (which is probably the right
thing to do).

- activate 'temp' LV now with pool's metadata

(lvchange -ay vg/temp)

Post by Timur Alperovich
- repair metadata
(you may need other 'bigger' volume to restore fixed metadata)
(i.e. thin_restore -i /dev/vg/temp -o /dev/vg/biggertemp)
- thin_check restored volume
- thin_dump - and compare vgcfgrestore & this dump are at the same
transaction_id state (look at lvm2 metadata for thin pool)
- deactivate again related volumes
- swap repaired LV back
(lvconvert --thinpool vg/fixpool --poolmetadata repairedtemp)
- try to active repaired thin pool
- remove unneeded volumes from vg....
Zdenek
_______________________________________________
linux-lvm mailing list
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Timur

[Zdenek Kabelac]

Post by Timur Alperovich
Thanks for the reply Zdenek!
I realized that the biggest confusion for me is that I'm using LVM2 tools
2.02.98-6 (which is the latest version in Ubuntu) and the swapping support was
added in 2.02.98-9 (I'm curious why they didn't pick the latest of 2.02.98).
Hi there,
I'm using LVM 2.02.98 and encountered a metadata corruption issue. To
recover
1. thin_check /dev/mapper/pool_tmeta
2. thin_dump /dev/mapper/pool_tmeta > /tmp/metadata
Hi
NEVER EVER use _tmeta device from running active thin-pool volume.
It's the very same case like you would be running 'fsck' on
a mounted filesystem.
3. dd if=/dev/zero of=/dev/mapper/pool_tmeta
4. thin_restore -i /tmp/metadata -o /dev/mapper/pool_tmeta
All of the above steps have succeeded, however, when attempting to list the
I'm surprised you've not got kernel OOPS after such brutal destruction
of life metadata device (i.e. almost equal to zeroing your root volume).
Is this a known issue and is there a workaround? I need to be able to
examine
the _metadata\_percent_ field to make sure we don't exhaust the
metadata space.
Normal way -
lvconvert --repair vg/pool
I believe this was added in 2.02.102. Unfortunately, upgrading LVM2 is a bit
tricky on Ubuntu (will need to build the package from source).
If this is not working - then you can 'swap' metadata out of your thin-pool
using following sequence-
- make sure pool is not active.
- build temporary local LV (lvcreate -l1 vg -n temp)
- swap this LV with metadata of to-be-repaired pool
(lvconvert --thinpool vg/fixpool --poolmetadata temp)
This command fails, as --poolmetadata is not a valid option. I found that the
swapping was not supported until 2.02.98-9 and I'm using 2.02.98-6 (in
2.02.98-9 changelog: "Add lvconvert support to swap thin pool metadata
volume."). Is there a way to do this without the swapping support? I guess I
should be able to do it with dmsetup? I'm not sure how painful that would be.
I'll look into upgrading the version, as well (which is probably the right
thing to do).

IMHO by far the compilation of upstream git repo is the simplest way how
to do it - everything else is much much more complicated.

You could probably active thin-pool - and via dmsetup remove running pool
target - which should left active _tmeta device - and then you should be able
to play with this - thought it will be very tricky to not make a mistake
with this type of work.

Zdenek