[linux-lvm] Bad disk?

Discussion:

Mauricio Tavares

2010-11-10 14:39:00 UTC

Yesterday I added a hard drive (to put extra stuff on it) to my ubuntu
10.10 box and created a LVM in it. Then copied some files to it and
restarted the machine to see if it would mount into the right
mountpoint. It didn't. So I decided to see if it was there (vg in
question is export):

***@strangepork:~$ sudo vgscan
Reading all physical volumes. This may take a while...
/dev/dm-0: read failed after 0 of 4096 at 429496664064: Input/output
error
/dev/dm-0: read failed after 0 of 4096 at 429496721408: Input/output
error
/dev/dm-0: read failed after 0 of 4096 at 0: Input/output error
/dev/dm-0: read failed after 0 of 4096 at 4096: Input/output error
/dev/dm-0: read failed after 0 of 4096 at 0: Input/output error
Found volume group "export" using metadata type lvm2
Found volume group "root" using metadata type lvm2
***@strangepork:~$

Those dm-0 messages do not make me happy. dmesg and vgchange make me
think the problem is on the new drive:

[ 268.024593] scsi 0:0:0:0: Direct-Access ATA ST3500320NS
SN04 PQ: 0 ANSI: 5
[ 268.024900] sd 0:0:0:0: [sdc] 976773168 512-byte logical blocks: (500
GB/465 GiB)
[ 268.024918] sd 0:0:0:0: Attached scsi generic sg2 type 0
[ 268.024996] sd 0:0:0:0: [sdc] Write Protect is off
[ 268.025003] sd 0:0:0:0: [sdc] Mode Sense: 00 3a 00 00
[ 268.025046] sd 0:0:0:0: [sdc] Write cache: enabled, read cache:
enabled, doesn't support DPO or FUA
[ 268.025377] sdc: sdc1
[ 268.049853] sd 0:0:0:0: [sdc] Attached SCSI disk
[ 335.467482] quiet_error: 3 callbacks suppressed
[ 335.467492] Buffer I/O error on device dm-0, logical block 104857584
[ 335.467540] Buffer I/O error on device dm-0, logical block 104857584
[ 335.467589] Buffer I/O error on device dm-0, logical block 104857598
[ 335.467615] Buffer I/O error on device dm-0, logical block 104857598
[ 335.467647] Buffer I/O error on device dm-0, logical block 0
[ 335.467671] Buffer I/O error on device dm-0, logical block 0
[ 335.467703] Buffer I/O error on device dm-0, logical block 1
[ 335.467734] Buffer I/O error on device dm-0, logical block 104857599
[ 335.467762] Buffer I/O error on device dm-0, logical block 104857599
[ 335.467788] Buffer I/O error on device dm-0, logical block 104857599

***@strangepork:~$ sudo vgchange -a y
/dev/dm-0: read failed after 0 of 4096 at 0: Input/output error
1 logical volume(s) in volume group "export" now active
4 logical volume(s) in volume group "root" now active
***@strangepork:~$

Glancing through http://readlist.com/lists/centos.org/centos/1/5704.html
I wondered if I had the wrong file system descriptor:

***@strangepork:~$ sudo sfdisk -l /dev/sdc

Disk /dev/sdc: 60801 cylinders, 255 heads, 63 sectors/track
Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0

Device Boot Start End #cyls #blocks Id System
/dev/sdc1 0+ 60800 60801- 488384001 8e Linux LVM
/dev/sdc2 0 - 0 0 0 Empty
/dev/sdc3 0 - 0 0 0 Empty
/dev/sdc4 0 - 0 0 0 Empty
***@strangepork:~$

I guess that rules that out. Does that mean the disk is bad? If so, is
there a way to retrieve the data I put in it?

Phillip Susi

2010-11-10 15:11:54 UTC

Permalink

Post by Mauricio Tavares
Yesterday I added a hard drive (to put extra stuff on it) to my ubuntu
10.10 box and created a LVM in it. Then copied some files to it and
restarted the machine to see if it would mount into the right
mountpoint. It didn't. So I decided to see if it was there (vg in

None of the errors you posted mention a physical disk at all, so there
isn't anything wrong with them. To find out what dm-0 is you need to
poke around with dmsetup.

Mauricio Tavares

2010-11-10 17:56:24 UTC

Permalink

Post by Phillip Susi

None of the errors you posted mention a physical disk at all, so there
isn't anything wrong with them. To find out what dm-0 is you need to
poke around with dmsetup.

Now that might be interesting since I honestly do not know how to use
it (and my google-fu seems weak today). But this is a bit more info I found:

***@strangepork:~$ sudo pvs -a
/dev/dm-0: read failed after 0 of 4096 at 0: Input/output error
/dev/dm-0: read failed after 0 of 4096 at 429496664064: Input/output
error
/dev/dm-0: read failed after 0 of 4096 at 429496721408: Input/output
error
/dev/dm-0: read failed after 0 of 4096 at 0: Input/output error
/dev/dm-0: read failed after 0 of 4096 at 4096: Input/output error
/dev/dm-0: read failed after 0 of 4096 at 0: Input/output error
PV VG Fmt Attr PSize PFree
/dev/dm-0 -- 0 0
/dev/dm-1 root lvm2 a- 277.05g 192.55g
/dev/dm-2 -- 0 0
/dev/dm-3 -- 0 0
/dev/dm-4 -- 0 0
/dev/dm-5 -- 0 0
/dev/ram0 -- 0 0
/dev/ram1 -- 0 0
/dev/ram10 -- 0 0
/dev/ram11 -- 0 0
/dev/ram12 -- 0 0
/dev/ram13 -- 0 0
/dev/ram14 -- 0 0
/dev/ram15 -- 0 0
/dev/ram2 -- 0 0
/dev/ram3 -- 0 0
/dev/ram4 -- 0 0
/dev/ram5 -- 0 0
/dev/ram6 -- 0 0
/dev/ram7 -- 0 0
/dev/ram8 -- 0 0
/dev/ram9 -- 0 0
/dev/root -- 0 0
/dev/sda1 -- 0 0
/dev/sda3 -- 0 0
/dev/sdc1 export lvm2 a- 465.76g 65.76g
***@strangepork:~$

FYI, the root vg in inside /dev/sda3, which is an encrypted partition.
How can I find which drive /dev/dm-0 is?

Stuart D. Gathman

2010-11-10 17:41:48 UTC

Permalink

Those dm-0 messages do not make me happy. dmesg and vgchange make me think the

Those dm-0 messages are probably a logical error. For instance, a snapshot
that is full would give those errors. You need to tell us what dm-0 is
mapped to. Look in /dev/mapper for starters.

0 ANSI: 5
[ 268.024900] sd 0:0:0:0: [sdc] 976773168 512-byte logical blocks: (500
GB/465 GiB)
[ 268.024918] sd 0:0:0:0: Attached scsi generic sg2 type 0
[ 268.024996] sd 0:0:0:0: [sdc] Write Protect is off
[ 268.025003] sd 0:0:0:0: [sdc] Mode Sense: 00 3a 00 00
[ 268.025046] sd 0:0:0:0: [sdc] Write cache: enabled, read cache: enabled,
doesn't support DPO or FUA
[ 268.025377] sdc: sdc1
[ 268.049853] sd 0:0:0:0: [sdc] Attached SCSI disk

This is normal for your new disk.

[ 335.467482] quiet_error: 3 callbacks suppressed
[ 335.467492] Buffer I/O error on device dm-0, logical block 104857584
[ 335.467540] Buffer I/O error on device dm-0, logical block 104857584
[ 335.467589] Buffer I/O error on device dm-0, logical block 104857598
[ 335.467615] Buffer I/O error on device dm-0, logical block 104857598
[ 335.467647] Buffer I/O error on device dm-0, logical block 0
[ 335.467671] Buffer I/O error on device dm-0, logical block 0
[ 335.467703] Buffer I/O error on device dm-0, logical block 1
[ 335.467734] Buffer I/O error on device dm-0, logical block 104857599
[ 335.467762] Buffer I/O error on device dm-0, logical block 104857599
[ 335.467788] Buffer I/O error on device dm-0, logical block 104857599

Again, this is on dm-0, not sdc.

Disk /dev/sdc: 60801 cylinders, 255 heads, 63 sectors/track
Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0
Device Boot Start End #cyls #blocks Id System
/dev/sdc1 0+ 60800 60801- 488384001 8e Linux LVM
/dev/sdc2 0 - 0 0 0 Empty
/dev/sdc3 0 - 0 0 0 Empty
/dev/sdc4 0 - 0 0 0 Empty

This is normal, not sure what is has to do with filesystem desciptors.

Tell us exactly what you mean by "put a LVM on it". Did you run
pvcreate? vgcreate? lvcreate? You might find the output of "pvs"
enlightening. That will tell us what PVs you have created.
And list /dev/mapper so we know what dm-0 is, and include the output of "lvs".

--
Stuart D. Gathman <***@bmsi.com>
Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

Mauricio Tavares

2010-11-10 20:35:04 UTC

Permalink

Post by Stuart D. Gathman

Those dm-0 messages do not make me happy. dmesg and vgchange make me think the

Those dm-0 messages are probably a logical error. For instance, a snapshot
that is full would give those errors. You need to tell us what dm-0 is
mapped to. Look in /dev/mapper for starters.

Sorry for that: I did not think it would be there. But, as you said, it is:

lrwxrwxrwx 1 root root 7 2010-11-10 09:30 export-vms -> ../dm-0

the export vg is in /dev/sdc1.

Post by Stuart D. Gathman

This is normal for your new disk.

Again, this is on dm-0, not sdc.

This is normal, not sure what is has to do with filesystem desciptors.
Tell us exactly what you mean by "put a LVM on it". Did you run
pvcreate? vgcreate? lvcreate? You might find the output of "pvs"
enlightening. That will tell us what PVs you have created.
And list /dev/mapper so we know what dm-0 is, and include the output of "lvs".

Let me put this way, I thought I did. I mean, after creating the
partition, setting it to LVM (8e), then running

pvcreate /dev/sdc1
vgcreate export /dev/sdc1
lvcreate -L 400G --name vms export

I used mkfs.ext4 to create partition (on /dev/mapper/export-vms) and off
I went. Do you think I missed a step?

Stuart D. Gathman

2010-11-11 16:46:37 UTC

Permalink

Post by Mauricio Tavares

Post by Stuart D. Gathman
Tell us exactly what you mean by "put a LVM on it". Did you run
pvcreate? vgcreate? lvcreate? You might find the output of "pvs"
enlightening. That will tell us what PVs you have created.
And list /dev/mapper so we know what dm-0 is, and include the output of "lvs".

Let me put this way, I thought I did. I mean, after creating the
partition, setting it to LVM (8e), then running
pvcreate /dev/sdc1
vgcreate export /dev/sdc1
lvcreate -L 400G --name vms export
I used mkfs.ext4 to create partition (on /dev/mapper/export-vms) and off I
went. Do you think I missed a step?

Great. Now include output of "lvs"

BTW, if you really suspect a disk error, test for it directly.
E.g., you can run

# dd if=/dev/sdc1 of=/dev/null bs=256k

to read through the partition or

# smartctl -t long /dev/sdc

To initiate a long self test of the disk (need smartmontools installed).

A brand new disk that flunks self test is indeed defective.

However, for real physical I/O errors, there would be errors logged
in /var/log/messages referencing sdc (as opposed to dm-0), so I still
think it is a logical error.

Mauricio Tavares

2010-11-11 18:39:28 UTC

Permalink

Post by Stuart D. Gathman

Post by Mauricio Tavares

Let me put this way, I thought I did. I mean, after creating the
partition, setting it to LVM (8e), then running
pvcreate /dev/sdc1
vgcreate export /dev/sdc1
lvcreate -L 400G --name vms export
I used mkfs.ext4 to create partition (on /dev/mapper/export-vms) and off I
went. Do you think I missed a step?

Great. Now include output of "lvs"

***@strangepork:~$ sudo lvs export
/dev/dm-0: read failed after 0 of 4096 at 0: Input/output error
LV VG Attr LSize Origin Snap% Move Log Copy% Convert
vms export -wi-a- 400.00g

Post by Stuart D. Gathman
BTW, if you really suspect a disk error, test for it directly.
E.g., you can run
# dd if=/dev/sdc1 of=/dev/null bs=256k
to read through the partition or
# smartctl -t long /dev/sdc
To initiate a long self test of the disk (need smartmontools installed).
A brand new disk that flunks self test is indeed defective.
However, for real physical I/O errors, there would be errors logged
in /var/log/messages referencing sdc (as opposed to dm-0), so I still
think it is a logical error.

Ray Morris

2010-11-10 18:58:01 UTC

Permalink

Post by Mauricio Tavares
How can I find which drive /dev/dm-0 is?

Try:
cat /sys/block/dm-0/dm/name

On my system, I created dm2device so I can tell which snapshot is
full or whatever device has a problem:

# cat /usr/local/bin/dm2device
#!/bin/sh

cat /sys/block/$1/dm/name

--
Ray Morris
***@bettercgi.com

Strongbox - The next generation in site security:
http://www.bettercgi.com/strongbox/

Throttlebox - Intelligent Bandwidth Control
http://www.bettercgi.com/throttlebox/

Strongbox / Throttlebox affiliate program:
http://www.bettercgi.com/affiliates/user/register.php

Post by Mauricio Tavares

Post by Phillip Susi

Post by Mauricio Tavares
Yesterday I added a hard drive (to put extra stuff on it) to my
ubuntu
10.10 box and created a LVM in it. Then copied some files to it and
restarted the machine to see if it would mount into the right
mountpoint. It didn't. So I decided to see if it was there (vg in

None of the errors you posted mention a physical disk at all, so
there
isn't anything wrong with them. To find out what dm-0 is you need to
poke around with dmsetup.

Now that might be interesting since I honestly do not know how
to use it (and my google-fu seems weak today). But this is a bit more
/dev/dm-0: read failed after 0 of 4096 at 0: Input/output error
Input/output error
Input/output error
/dev/dm-0: read failed after 0 of 4096 at 0: Input/output error
/dev/dm-0: read failed after 0 of 4096 at 4096: Input/output error
/dev/dm-0: read failed after 0 of 4096 at 0: Input/output error
PV VG Fmt Attr PSize PFree
/dev/dm-0 -- 0 0
/dev/dm-1 root lvm2 a- 277.05g 192.55g
/dev/dm-2 -- 0 0
/dev/dm-3 -- 0 0
/dev/dm-4 -- 0 0
/dev/dm-5 -- 0 0
/dev/ram0 -- 0 0
/dev/ram1 -- 0 0
/dev/ram10 -- 0 0
/dev/ram11 -- 0 0
/dev/ram12 -- 0 0
/dev/ram13 -- 0 0
/dev/ram14 -- 0 0
/dev/ram15 -- 0 0
/dev/ram2 -- 0 0
/dev/ram3 -- 0 0
/dev/ram4 -- 0 0
/dev/ram5 -- 0 0
/dev/ram6 -- 0 0
/dev/ram7 -- 0 0
/dev/ram8 -- 0 0
/dev/ram9 -- 0 0
/dev/root -- 0 0
/dev/sda1 -- 0 0
/dev/sda3 -- 0 0
/dev/sdc1 export lvm2 a- 465.76g 65.76g
FYI, the root vg in inside /dev/sda3, which is an encrypted
partition. How can I find which drive /dev/dm-0 is?
_______________________________________________
linux-lvm mailing list
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Mauricio Tavares

2010-11-10 20:29:00 UTC

Permalink

Post by Ray Morris

Post by Mauricio Tavares
How can I find which drive /dev/dm-0 is?

cat /sys/block/dm-0/dm/name
On my system, I created dm2device so I can tell which snapshot is
# cat /usr/local/bin/dm2device
#!/bin/sh
cat /sys/block/$1/dm/name

Thanks!

cat /sys/block/dm-0/dm/name returned export-vms, which is /dev/sdc1.

Post by Ray Morris
--
Ray Morris
http://www.bettercgi.com/strongbox/
Throttlebox - Intelligent Bandwidth Control
http://www.bettercgi.com/throttlebox/
http://www.bettercgi.com/affiliates/user/register.php

Post by Mauricio Tavares

Post by Phillip Susi

None of the errors you posted mention a physical disk at all, so there
isn't anything wrong with them. To find out what dm-0 is you need to
poke around with dmsetup.

Now that might be interesting since I honestly do not know how to use
it (and my google-fu seems weak today). But this is a bit more info I
/dev/dm-0: read failed after 0 of 4096 at 0: Input/output error
/dev/dm-0: read failed after 0 of 4096 at 429496664064: Input/output
error
/dev/dm-0: read failed after 0 of 4096 at 429496721408: Input/output
error
/dev/dm-0: read failed after 0 of 4096 at 0: Input/output error
/dev/dm-0: read failed after 0 of 4096 at 4096: Input/output error
/dev/dm-0: read failed after 0 of 4096 at 0: Input/output error
PV VG Fmt Attr PSize PFree
/dev/dm-0 -- 0 0
/dev/dm-1 root lvm2 a- 277.05g 192.55g
/dev/dm-2 -- 0 0
/dev/dm-3 -- 0 0
/dev/dm-4 -- 0 0
/dev/dm-5 -- 0 0
/dev/ram0 -- 0 0
/dev/ram1 -- 0 0
/dev/ram10 -- 0 0
/dev/ram11 -- 0 0
/dev/ram12 -- 0 0
/dev/ram13 -- 0 0
/dev/ram14 -- 0 0
/dev/ram15 -- 0 0
/dev/ram2 -- 0 0
/dev/ram3 -- 0 0
/dev/ram4 -- 0 0
/dev/ram5 -- 0 0
/dev/ram6 -- 0 0
/dev/ram7 -- 0 0
/dev/ram8 -- 0 0
/dev/ram9 -- 0 0
/dev/root -- 0 0
/dev/sda1 -- 0 0
/dev/sda3 -- 0 0
/dev/sdc1 export lvm2 a- 465.76g 65.76g
FYI, the root vg in inside /dev/sda3, which is an encrypted partition.
How can I find which drive /dev/dm-0 is?
_______________________________________________
linux-lvm mailing list
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

_______________________________________________
linux-lvm mailing list
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/