[linux-lvm] Rebuilding ext4 filesystem on an LV

[Ken Bass]

In short:

- originally created LV with several physical drives (5)
- formatted LV for ext4
- one physical drive got many sectors trashed. Those included the partition
and metadata.
- did the following to recover the LV (VolGroupX-LogVolX):
---pvcreate --uuid <original uuid> /dev/sdX (successfully created)
---vgcfgrestore VolGroupX (restsored volume group VolGroupX)
---vgscan (found volume group "VolGroupX" using metadata type lvm2)
--- vgchange -ay VolGroupX ( note response:
" device-mapper: reload ioctl failed: Invalid argument.
1 logical volume(s) in volume group "VolGroupX" now active")

At this pont, /dev/mapper/VolGroupX-LogVolX exists, but I can't run any
utilities on it, since it can't find any superblock. For example:
________________
[***@Elmer ken] # e2fsck /dev/mapper/VolGroupX-LogVolX
e2fsck 1.41.9 (22-Aug-2009)
e2fsck: Invalid argument while trying to open /dev/mapper/VolGroupX-LogVolX

The superblock could not be read or does not describe a correct ext2
filesystem. If the device is valid and it really contains an ext2
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
e2fsck -b 8193 <device>

[***@Elmer ken] #
_________________________
I really want to recover at least the data on the 4 other drives, and any
data left from this drive as well.

Any help will be greatly appreciated.

TIA

ken

[Bryan Whitehead]

Dig deep into your memory (or just try different combos) of running
mkfs.ext4 -n /dev/yourVG/yourLV

-n is like a dry run so you can look for the alternate superblock locations.

Using the above info, run fsck.ext4 -b <alt location> /dev/yourVG/yourLV

Even if it can find an alternate superblock, I'm not confident the
fsck would be able to help much…

That said, I think you might be out of luck. Go ahead and give it a
shot, I'm curious how much data you'll get.

Post by Ken Bass
- originally created LV with several physical drives (5)
- formatted LV for ext4
- one physical drive got many sectors trashed. Those included the partition
and metadata.
---pvcreate --uuid <original uuid>  /dev/sdX  (successfully created)
---vgcfgrestore VolGroupX (restsored volume group VolGroupX)
---vgscan  (found volume group "VolGroupX" using metadata type lvm2)
" device-mapper: reload ioctl failed: Invalid argument.
 1 logical volume(s) in volume group "VolGroupX" now active")
At this pont, /dev/mapper/VolGroupX-LogVolX exists, but I can't run any
________________
e2fsck 1.41.9 (22-Aug-2009)
e2fsck: Invalid argument while trying to open /dev/mapper/VolGroupX-LogVolX
The superblock could not be read or does not describe a correct ext2
filesystem.  If the device is valid and it really contains an ext2
filesystem (and not swap or ufs or something else), then the superblock
    e2fsck -b 8193 <device>
_________________________
I really want to recover at least the data on the 4 other drives, and any
data left from this drive as well.
Any help will be greatly appreciated.
TIA
ken
_______________________________________________
linux-lvm mailing list
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

[Ken Bass]

Post by Bryan Whitehead
Dig deep into your memory (or just try different combos) of running
mkfs.ext4 -n /dev/yourVG/yourLV
-n is like a dry run so you can look for the alternate superblock locations.
Using the above info, run fsck.ext4 -b <alt location> /dev/yourVG/yourLV

Tried all of that, with many different offsets. No luck.

Post by Bryan Whitehead
Even if it can find an alternate superblock, I'm not confident the
fsck would be able to help much…

Do you mean in general, or just for recovery of the trashed drive? I didn't
have much hope for the trashed drive, but I was hoping to recover what was
on the other drives.

Post by Bryan Whitehead
That said, I think you might be out of luck. Go ahead and give it a
shot, I'm curious how much data you'll get.

Me too :-).

But what bothers me is this message I get when I run e2fsck (which runs
fsck.ext4):
"e2fsck: Invalid argument while trying to open
/dev/mapper/VolGroupX-LogVolX"

Why am I getting that?

thx for the response.

ken

[Bryan Whitehead]

Post by Ken Bass
But what bothers me is this message I get when I run e2fsck (which runs
"e2fsck: Invalid argument while trying to open
/dev/mapper/VolGroupX-LogVolX"
Why am I getting that?
thx for the response.

The lv is available right?

It would be great to see the output from lvdisplay and vgdisplay.

-Bryan

[Ken Bass]

Post by Bryan Whitehead
The lv is available right?
It would be great to see the output from lvdisplay and vgdisplay.
-Bryan

___________________________________
[***@Elmer ken]# lvdisplay /dev/VolGroupX/LogVolX
--- Logical volume ---
LV Name /dev/VolGroupX/LogVolX
VG Name VolGroupW
LV UUID eBQgqM-RHra-ruaK-CcEt-stnX-U9yR-Uz4UH5
LV Write Access read/write
LV Status available
# open 0
LV Size 442.80 GB
Current LE 113358
Segments 5
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:11

[***@Elmer ken]# vgdisplay VolGroupX
--- Volume group ---
VG Name VolGroupX
System ID
Format lvm2
Metadata Areas 5
Metadata Sequence No 28
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 1
Open LV 0
Max PV 0
Cur PV 5
Act PV 5
VG Size 442.80 GB
PE Size 4.00 MB
Total PE 113358
Alloc PE / Size 113358 / 442.80 GB
Free PE / Size 0 / 0
VG UUID 2CXUUY-j6nu-07qP-kKd1-ZkBJ-hNwf-iunkwK

___________________________________

I would think I should be able to find a backup superblock among the
remaining drives. I tried running mke2fs -n to get a list of possible
superblocks, but when I ran e2fsck with those values, I got the ioctl error,
and then the no superblock found message.

Is there any way to scan each of the individual drives for data, outside of
LVM? Just another desparate thought.

thx again.

ken

[Stuart D. Gathman]

Post by Ken Bass
But what bothers me is this message I get when I run e2fsck (which runs
"e2fsck: Invalid argument while trying to open
/dev/mapper/VolGroupX-LogVolX"
Why am I getting that?

Not an expert, just a user, but I suspect that the ioerr is because the
metadata for that LV is still pointing to the missing PV. (I know you replaced
it, but that doesn't update the LV.) I also noticed that the lvdisplay
you posted says LogVolX belongs to VolGroupW, not VolGroupX. Perhaps you
sanitized the names and that is a typo - otherwise, there is something screwy.

I suspect recovering the remaining data will require skipping the (huge)
chuck of LV that was on the missing PV. Or maybe the experts here
will recommend editing the metadata to move the extent from the missing
PV to the new PV. The huge chunk will now have random data from the
new drive. I would have initialized the new drive to zero (using the
security erase) before using (maybe you did).

[Ken Bass]

Post by Stuart D. Gathman

Post by Ken Bass
But what bothers me is this message I get when I run e2fsck (which runs
"e2fsck: Invalid argument while trying to open
/dev/mapper/VolGroupX-LogVolX"
Why am I getting that?

Not an expert, just a user, but I suspect that the ioerr is because the
metadata for that LV is still pointing to the missing PV. (I know you replaced
it, but that doesn't update the LV.) I also noticed that the lvdisplay
you posted says LogVolX belongs to VolGroupW, not VolGroupX. Perhaps you
sanitized the names and that is a typo - otherwise, there is something screwy.

Yes, I guess my sanitizing wasn't 100% sterile :-(.

Post by Stuart D. Gathman
I suspect recovering the remaining data will require skipping the (huge)
chuck of LV that was on the missing PV. Or maybe the experts here
will recommend editing the metadata to move the extent from the missing
PV to the new PV.

That is what I am really trying to do. As I said early on, I was hoping to
find a backup superblock on the remaining part of the LV that might help
restore some of the data.. I have also looked at the metadata files, and was
wondering if I could manually rearrange the PVs. Any expert opinion on that?


FWIW: I found some utilities that supposedly will scan a drive with an
ext2/3 filesystem and recover files from it - although it only runs on
windows. It's free, so maybe worth a try.

thx to all for the help.

ken

[Malahal Naineni]

Post by Stuart D. Gathman

Post by Ken Bass
But what bothers me is this message I get when I run e2fsck (which runs
"e2fsck: Invalid argument while trying to open
/dev/mapper/VolGroupX-LogVolX"
Why am I getting that?

Not an expert, just a user, but I suspect that the ioerr is because
the metadata for that LV is still pointing to the missing PV. (I know
you replaced it, but that doesn't update the LV.)

If everything went right, he should have the new PV in the LV, but looks
like something failed while activating the LV. Run 'dmsetup table' to
find out if the LV currently has new PV or old PV. You need
device-mapper knowledge for this.

Post by Stuart D. Gathman
I suspect recovering the remaining data will require skipping the (huge)
chuck of LV that was on the missing PV. Or maybe the experts here
will recommend editing the metadata to move the extent from the missing
PV to the new PV. The huge chunk will now have random data from the
new drive. I would have initialized the new drive to zero (using the
security erase) before using (maybe you did).

The best course of action would have been 'reading what ever data
available' from old disk to new disk. If the disk is absolutely
thrashed, then zero'ing the new PV may be better.

[Ken Bass]

Post by Malahal Naineni
If everything went right, he should have the new PV in the LV, but looks
like something failed while activating the LV. Run 'dmsetup table' to
find out if the LV currently has new PV or old PV. You need
device-mapper knowledge for this.
I will have to try that out. Will let you know.
The best course of action would have been 'reading what ever data
available' from old disk to new disk. If the disk is absolutely
thrashed, then zero'ing the new PV may be better.

I did try to read anything I could out of the old disk. I do believe there
is still some data on it that wasn't trashed. The problem is how to find it
and reconstruct it without any of the LVM or ext4 filesystem information
(as I said, I couldn't find any superblocks). Are there any utilities to do
something like that?

Then there are the remaining drives that weren't affected. Unfortunately
without the first drive, I can't get to anything on the others. Or can I?
I've tried several things, but I haven't found anything that works. Are
there any utilities for that?

I found an app that will scan a disk looking for lost partitions, and then
looking for various filesystems within them - and it does recognize ext2/3.
It will then recover files at that point. But it runs on windows (won't work
with wine), and it is taking several days to scan the entire drive (100G).
Also, there could be several hundred files, and trying to identify all of
those (possibly) recovered files would be excrutiating :-( .

As always, thx everyone for your help.

ken

[Malahal Naineni]

Post by Ken Bass
I did try to read anything I could out of the old disk. I do believe there
is still some data on it that wasn't trashed. The problem is how to find
it and reconstruct it without any of the LVM or ext4 filesystem�
information (as I said, I couldn't find any superblocks). Are there any
utilities to do something like that?

I don't know much about ext2/3/4 file systems but I imagine debugfs may
help you. I never used it myself though.

Thanks, Malahal.

[Stuart D. Gathman]

Post by Ken Bass

Post by Malahal Naineni
The best course of action would have been 'reading what ever data
available' from old disk to new disk. If the disk is absolutely
thrashed, then zero'ing the new PV may be better.

I did try to read anything I could out of the old disk. I do believe there
is still some data on it that wasn't trashed. The problem is how to find it
and reconstruct it without any of the LVM or ext4 filesystem information
(as I said, I couldn't find any superblocks). Are there any utilities to do
something like that?

Method 1:
If you can, have *all* the drives connected, the failed drive, and the new
drive. Restore LVM metadata to include the failed drive again. Now add the
new drive and use pvmove to migrate data that can be read to the new
drive. (I've never used pvmove for this purpose - does it handle IO
errors and continue?)

Method 2:
With both new and failed drive connected, use dd_rescue or equivalent to
copy (most/some of) the failed drive to the new drive. Remove failed
drive and bring up VG. If UUID was not readable on failed drive, use
pvcreate to restore it on new drive.

[Stuart D. Gathman]

Post by Stuart D. Gathman
With both new and failed drive connected, use dd_rescue or equivalent to
copy (most/some of) the failed drive to the new drive. Remove failed
drive and bring up VG. If UUID was not readable on failed drive, use
pvcreate to restore it on new drive.

I should mention that an external case-free USB adapter is the best way
to attempt recovery of failed drives. You can even run the IDE/SATA/Power
cable to a box in the freezer (and the USB adapter can sit on top of the
freezer and a long USB cable go to your computer) - which often helps recover
data from failing drives.

[Ken Bass]

Post by Stuart D. Gathman

Post by Stuart D. Gathman
With both new and failed drive connected, use dd_rescue or equivalent to
copy (most/some of) the failed drive to the new drive. Remove failed
drive and bring up VG. If UUID was not readable on failed drive, use
pvcreate to restore it on new drive.

(Question: does new drive need to be same physical size? can Or can I just
create a new partition with the old size - new drive is bigger.)

The real problem seems to be that the LV appears to the LVM as not
formatted. I am guessing this is because not only the partition got trashed,
but also the superblock(s) data. I have tried to find backup superblocks,
but I keep getting the ioctl error, even from mke2fs. I will try dd_rescue -
I have to install it, since it is not by default.

I added a new drive to the LV, and tried pvmove, but, I got the ioctl error
again with that.

I should mention that an external case-free USB adapter is the best way

Post by Stuart D. Gathman
to attempt recovery of failed drives. You can even run the IDE/SATA/Power
cable to a box in the freezer (and the USB adapter can sit on top of the
freezer and a long USB cable go to your computer) - which often helps recover
data from failing drives.

If the mountain won't come to Mohammed... I can always just get a large bag
of ice and (carefully wrapped up in waterproof cover) put the drive into it.
Will try that also :-).

FWIW: I did run dmsetup _table, and from /var/log/messages:

____________
Jul 12 09:17:25 Elmer kernel: device-mapper: ioctl: error adding target to
table
Jul 12 09:24:16 Elmer kernel: device-mapper: table: device 253:3 too small
for target
____________

Couple of thoughts:
* Using dd_rescue, is there a utility that can read through the physical
blocks on the drive, searching for something that looks like a superblock?

* Or, is it possible to physically disconnect the bad drive, bring up the
remainder of the LV with the hope of recovering whatever data was on those
drives. Att this point, I am resigned to the fact that the first drive is
dead or on its last gasp (SMARTD says something like: "Danger (Will
Robinson!) drive failure imminent. Back up data immediately.)" But the other
drives in the LV should have something on them still. Right?

As always thanks for the help.

Post by Stuart D. Gathman
Business Management Systems Inc. Phone: [phone] Fax: [phone]
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

"All consuming fires of hell" - very loose translation from Mozart's
unfinished Mass for the Dead :-)

ken