[linux-lvm] Backup superblock for thin provision?

[Gionatan Danti]

Hi all,
using thin provisioning in production machines (using it mostly for its
fast snapshot support, rather than for thin provision / storage
overcommit by itself), I wonder what to do if a critical metadata
corruption, as the loss of the superblock, should happen.

Filesystems generally have some backup copy of the superblock; should
the primary one fail, another copy can be used.

So I have the following questions:
- how about thin LVM? Has it a backup superblock somewhere?
- how can the metadata be reliable backupped without shutting down the
volume?
- more generally, how to deal with metadata backup? Does vgcfgrestore
works for thin volumes?

Thank you all.

[Mike Snitzer]

On Fri, May 13 2016 at 6:39am -0400,

Post by Gionatan Danti
Hi all,
using thin provisioning in production machines (using it mostly for
its fast snapshot support, rather than for thin provision / storage
overcommit by itself), I wonder what to do if a critical metadata
corruption, as the loss of the superblock, should happen.
Filesystems generally have some backup copy of the superblock;
should the primary one fail, another copy can be used.
- how about thin LVM? Has it a backup superblock somewhere?
- how can the metadata be reliable backupped without shutting down
the volume?
- more generally, how to deal with metadata backup? Does
vgcfgrestore works for thin volumes?
Thank you all.

There is more to the thinp metadata than just the metadata superblock.

The DM thin-pool's metadata device was purposely split out from the data
device to allow for additional metadata fault protection using RAID.

I'll defer to the LVM developers for if/how LVM can be used to have
thinp metadata redundancy even if you don't have multiple devices to be
able to use a conventional RAID device.

[Zdenek Kabelac]

Post by Mike Snitzer
On Fri, May 13 2016 at 6:39am -0400,

Post by Gionatan Danti
Hi all,
using thin provisioning in production machines (using it mostly for
its fast snapshot support, rather than for thin provision / storage
overcommit by itself), I wonder what to do if a critical metadata
corruption, as the loss of the superblock, should happen.
Filesystems generally have some backup copy of the superblock;
should the primary one fail, another copy can be used.
- how about thin LVM? Has it a backup superblock somewhere?
- how can the metadata be reliable backupped without shutting down
the volume?
- more generally, how to deal with metadata backup? Does
vgcfgrestore works for thin volumes?
Thank you all.

There is more to the thinp metadata than just the metadata superblock.
The DM thin-pool's metadata device was purposely split out from the data
device to allow for additional metadata fault protection using RAID.
I'll defer to the LVM developers for if/how LVM can be used to have
thinp metadata redundancy even if you don't have multiple devices to be
able to use a conventional RAID device.

There is always the option to take 'metadata' snapshot and just
thin_dump content of metadata to a file (located in some 'safe' place)

However validation of 'restore' if there are some 'snapshots' is questionable
as the b-tree describing mapped blocks may change significantly
so 'rescued' content may than reference lots of bad blocks.

If you want to just protect 'superblock' against disk fault - usage
of 'raid' could be an option - but ATM there are some 'related' costs with
management of 'stacked' device tree.

Regards

Zdenek

[Gionatan Danti]

Post by Zdenek Kabelac
There is always the option to take 'metadata' snapshot and just
thin_dump content of metadata to a file (located in some 'safe' place)

Something as "dmsetup reserve_metadata_snap", thin_dump, "dmsetup
release_metadata_snap" ? If so, I'm already using for testing purpose ;)

Post by Zdenek Kabelac
However validation of 'restore' if there are some 'snapshots' is
questionable as the b-tree describing mapped blocks may change
significantly
so 'rescued' content may than reference lots of bad blocks.

Sure, with snapshots the situation is surely more complex, as due to the
CoW the old metadata can point to stale data, right?

Post by Zdenek Kabelac
If you want to just protect 'superblock' against disk fault - usage
of 'raid' could be an option - but ATM there are some 'related' costs
with management of 'stacked' device tree.

Both my tmeta and tdata reside on a RAID6 array provided by a
BBU-protected RAID card, so I should be safe here.

[Gionatan Danti]

Post by Mike Snitzer
There is more to the thinp metadata than just the metadata superblock.
The DM thin-pool's metadata device was purposely split out from the data
device to allow for additional metadata fault protection using RAID.
I'll defer to the LVM developers for if/how LVM can be used to have
thinp metadata redundancy even if you don't have multiple devices to be
able to use a conventional RAID device.

Sure - I'm using thinp on a RAID6 volume, with two disks redundancy.

But what about an user error, or a lvm-related bug, which invalidate the
superblock? In this case, RAID is of no help. On the other side, a
superblock backup copy can be used to restore the system without major
data loss.

More in general, how do you deal with metadata backups?
Thanks.