[linux-lvm] Confusing error: No space left on device

Discussion:

Jason Pyeron

2015-02-17 16:20:58 UTC

This is likely a problem between keyboard and chair, but it has got me confused all the same.

I was having issues with rsync, so I switched to dd and the same problem persists.

Any suggestions?

History says "lvcreate --size 8589934592b --name ciphershed-dmz-http vg_five66"

***@five-66 ~
# dd of=/dev/vg_five66/ciphershed-dmz-http bs=512 if=/dev/zero
dd: writing `/dev/vg_five66/ciphershed-dmz-http': No space left on device
8580489+0 records in
8580488+0 records out
4393209856 bytes (4.4 GB) copied, 11.1884 s, 393 MB/s

***@five-66 ~
# lvdisplay /dev/vg_five66/ciphershed-dmz-http
--- Logical volume ---
LV Path /dev/vg_five66/ciphershed-dmz-http
LV Name ciphershed-dmz-http
VG Name vg_five66
LV UUID 3yQHr9-WpdJ-76eL-4lEZ-hv0X-HP3N-GgqOvN
LV Write Access read/write
LV Creation host, time five-66.pdinc.us, 2015-02-17 05:05:17 -0500
LV Status available
# open 0
LV Size 8.00 GiB
Current LE 2048
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:3

***@five-66 ~
# vgdisplay
--- Volume group ---
VG Name vg_five66
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 36
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 5
Open LV 2
Max PV 0
Cur PV 1
Act PV 1
VG Size 1.82 TiB
PE Size 4.00 MiB
Total PE 476806
Alloc PE / Size 19866 / 77.60 GiB
Free PE / Size 456940 / 1.74 TiB
VG UUID drT68w-cRLR-ZcmV-VYFz-haaK-Djd4-S8q9EV

***@five-66 ~
# rpm -qf `which vgdisplay`
lvm2-2.02.111-2.el6_6.1.x86_64

***@five-66 ~
# uname -a
Linux five-66.pdinc.us 2.6.32-504.8.1.el6.x86_64 #1 SMP Wed Jan 28 21:11:36 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
- -
- Jason Pyeron PD Inc. http://www.pdinc.us -
- Principal Consultant 10 West 24th Street #100 -
- +1 (443) 269-1555 x333 Baltimore, Maryland 21218 -
- -
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
This message is copyright PD Inc, subject to license 20080407P00.

Jason Pyeron

2015-02-17 16:52:43 UTC

Permalink

-----Original Message-----
From: Jason Pyeron
Sent: Tuesday, February 17, 2015 11:21
This is likely a problem between keyboard and chair, but it
has got me confused all the same.
I was having issues with rsync, so I switched to dd and the
same problem persists.
Any suggestions?
History says "lvcreate --size 8589934592b --name
ciphershed-dmz-http vg_five66"
# dd of=/dev/vg_five66/ciphershed-dmz-http bs=512 if=/dev/zero
dd: writing `/dev/vg_five66/ciphershed-dmz-http': No space
left on device
8580489+0 records in
8580488+0 records out
4393209856 bytes (4.4 GB) copied, 11.1884 s, 393 MB/s
# lvdisplay /dev/vg_five66/ciphershed-dmz-http
--- Logical volume ---
LV Path /dev/vg_five66/ciphershed-dmz-http
LV Name ciphershed-dmz-http
VG Name vg_five66
LV UUID 3yQHr9-WpdJ-76eL-4lEZ-hv0X-HP3N-GgqOvN
LV Write Access read/write
LV Creation host, time five-66.pdinc.us, 2015-02-17 05:05:17 -0500
LV Status available
# open 0
LV Size 8.00 GiB
Current LE 2048
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:3
# vgdisplay
--- Volume group ---
VG Name vg_five66
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 36
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 5
Open LV 2
Max PV 0
Cur PV 1
Act PV 1
VG Size 1.82 TiB
PE Size 4.00 MiB
Total PE 476806
Alloc PE / Size 19866 / 77.60 GiB
Free PE / Size 456940 / 1.74 TiB
VG UUID drT68w-cRLR-ZcmV-VYFz-haaK-Djd4-S8q9EV
# rpm -qf `which vgdisplay`
lvm2-2.02.111-2.el6_6.1.x86_64
# uname -a
Linux five-66.pdinc.us 2.6.32-504.8.1.el6.x86_64 #1 SMP Wed
Jan 28 21:11:36 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

# lvremove /dev/vg_five66/ciphershed-dmz-http
Do you really want to remove active logical volume ciphershed-dmz-http? [y/n]: y
Logical volume "ciphershed-dmz-http" successfully removed

***@five-66 ~
# lvcreate --size 8589934592b --name ciphershed-dmz-http vg_five66
Logical volume "ciphershed-dmz-http" created

***@five-66 ~
# dd of=/dev/vg_five66/ciphershed-dmz-http bs=512 if=/dev/zero
dd: writing `/dev/vg_five66/ciphershed-dmz-http': No space left on device
16777217+0 records in
16777216+0 records out
8589934592 bytes (8.6 GB) copied, 254.765 s, 33.7 MB/s

***@five-66 ~
# lvdisplay /dev/vg_five66/ciphershed-dmz-http
--- Logical volume ---
LV Path /dev/vg_five66/ciphershed-dmz-http
LV Name ciphershed-dmz-http
VG Name vg_five66
LV UUID slRXQ8-I3pi-2uKA-QSkd-TAwU-7deR-cKxOj9
LV Write Access read/write
LV Creation host, time five-66.pdinc.us, 2015-02-17 06:46:39 -0500
LV Status available
# open 0
LV Size 8.00 GiB
Current LE 2048
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:3

Now I am concerned, removing and recreating fixed the issue?

--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
- -
- Jason Pyeron PD Inc. http://www.pdinc.us -
- Principal Consultant 10 West 24th Street #100 -
- +1 (443) 269-1555 x333 Baltimore, Maryland 21218 -
- -
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
This message is copyright PD Inc, subject to license 20080407P00.

Jason Pyeron

2015-02-17 17:08:33 UTC

Permalink

-----Original Message-----
From: Jason Pyeron
Sent: Tuesday, February 17, 2015 11:53

<snip/>

It is back.

***@five-66 ~
# rsync --inplace --progress server:/var/lib/libvirt/images/ciphershed-dmz-http.img /dev/vg_five66/ciphershed-dmz-http -vv
opening connection using: ssh server rsync --server --sender -vve.Ls --inplace . /var/lib/libvirt/images/ciphershed-dmz-http.img
delta-transmission enabled
ciphershed-dmz-http.img
4285333400 49% 113.37MB/s 0:00:37
rsync: write failed on "/dev/vg_five66/ciphershed-dmz-http": No space left on device (28)
rsync error: error in file IO (code 11) at receiver.c(301) [receiver=3.0.6]
rsync: connection unexpectedly closed (48 bytes received so far) [generator]
rsync error: error in rsync protocol data stream (code 12) at io.c(600) [generator=3.0.6]

***@five-66 ~
# dd of=/dev/vg_five66/ciphershed-dmz-http bs=512 if=/dev/zero
dd: writing `/dev/vg_five66/ciphershed-dmz-http': No space left on device
8580489+0 records in
8580488+0 records out
4393209856 bytes (4.4 GB) copied, 11.2402 s, 391 MB/s

-Jason

--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
- -
- Jason Pyeron PD Inc. http://www.pdinc.us -
- Principal Consultant 10 West 24th Street #100 -
- +1 (443) 269-1555 x333 Baltimore, Maryland 21218 -
- -
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
This message is copyright PD Inc, subject to license 20080407P00.

Roger Heflin

2015-02-17 17:29:59 UTC

Permalink

Look in /etc/lvm/archive there should be large amounts of details on
exactly how the lv was built and exactly what commands were done to it
that could have changed things from the original config.

Post by Jason Pyeron

-----Original Message-----
From: Jason Pyeron
Sent: Tuesday, February 17, 2015 11:53

<snip/>
It is back.
# rsync --inplace --progress server:/var/lib/libvirt/images/ciphershed-dmz-http.img /dev/vg_five66/ciphershed-dmz-http -vv
opening connection using: ssh server rsync --server --sender -vve.Ls --inplace . /var/lib/libvirt/images/ciphershed-dmz-http.img
delta-transmission enabled
ciphershed-dmz-http.img
4285333400 49% 113.37MB/s 0:00:37
rsync: write failed on "/dev/vg_five66/ciphershed-dmz-http": No space left on device (28)
rsync error: error in file IO (code 11) at receiver.c(301) [receiver=3.0.6]
rsync: connection unexpectedly closed (48 bytes received so far) [generator]
rsync error: error in rsync protocol data stream (code 12) at io.c(600) [generator=3.0.6]
# dd of=/dev/vg_five66/ciphershed-dmz-http bs=512 if=/dev/zero
dd: writing `/dev/vg_five66/ciphershed-dmz-http': No space left on device
8580489+0 records in
8580488+0 records out
4393209856 bytes (4.4 GB) copied, 11.2402 s, 391 MB/s
-Jason
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
- -
- Jason Pyeron PD Inc. http://www.pdinc.us -
- Principal Consultant 10 West 24th Street #100 -
- +1 (443) 269-1555 x333 Baltimore, Maryland 21218 -
- -
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
This message is copyright PD Inc, subject to license 20080407P00.
_______________________________________________
linux-lvm mailing list
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Jason Pyeron

2015-02-17 19:11:07 UTC

Permalink

-----Original Message-----
From: Roger Heflin
Sent: Tuesday, February 17, 2015 12:30
Look in /etc/lvm/archive there should be large amounts of details on
exactly how the lv was built and exactly what commands were done to it
that could have changed things from the original config.

Nothing that I see is unexpected, this dir is good to know about. It looks like Stuart Gathman pointed me in the right direction.

On Tue, Feb 17, 2015 at 11:08 AM, Jason Pyeron

Post by Jason Pyeron

-----Original Message-----
From: Jason Pyeron
Sent: Tuesday, February 17, 2015 11:53

05:05:17 -0500

Post by Jason Pyeron

LV Status available
# open 0
LV Size 8.00 GiB
Current LE 2048
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:3
# vgdisplay
--- Volume group ---
VG Name vg_five66
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 36
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 5
Open LV 2
Max PV 0
Cur PV 1
Act PV 1
VG Size 1.82 TiB
PE Size 4.00 MiB
Total PE 476806
Alloc PE / Size 19866 / 77.60 GiB
Free PE / Size 456940 / 1.74 TiB
VG UUID drT68w-cRLR-ZcmV-VYFz-haaK-Djd4-S8q9EV
# rpm -qf `which vgdisplay`
lvm2-2.02.111-2.el6_6.1.x86_64
# uname -a
Linux five-66.pdinc.us 2.6.32-504.8.1.el6.x86_64 #1 SMP Wed
Jan 28 21:11:36 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

<snip/>
It is back.
# rsync --inplace --progress

server:/var/lib/libvirt/images/ciphershed-dmz-http.img
/dev/vg_five66/ciphershed-dmz-http -vv

Post by Jason Pyeron
opening connection using: ssh server rsync --server

--sender -vve.Ls --inplace .
/var/lib/libvirt/images/ciphershed-dmz-http.img

Post by Jason Pyeron
delta-transmission enabled
ciphershed-dmz-http.img
4285333400 49% 113.37MB/s 0:00:37
rsync: write failed on

"/dev/vg_five66/ciphershed-dmz-http": No space left on device (28)

Post by Jason Pyeron
rsync error: error in file IO (code 11) at receiver.c(301)

[receiver=3.0.6]

Post by Jason Pyeron
rsync: connection unexpectedly closed (48 bytes received so

far) [generator]

Post by Jason Pyeron
rsync error: error in rsync protocol data stream (code 12)

at io.c(600) [generator=3.0.6]

Post by Jason Pyeron
# dd of=/dev/vg_five66/ciphershed-dmz-http bs=512 if=/dev/zero
dd: writing `/dev/vg_five66/ciphershed-dmz-http': No space

left on device

Post by Jason Pyeron
8580489+0 records in
8580488+0 records out
4393209856 bytes (4.4 GB) copied, 11.2402 s, 391 MB/s

--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
- -
- Jason Pyeron PD Inc. http://www.pdinc.us -
- Principal Consultant 10 West 24th Street #100 -
- +1 (443) 269-1555 x333 Baltimore, Maryland 21218 -
- -
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
This message is copyright PD Inc, subject to license 20080407P00.

Stuart Gathman

2015-02-17 18:53:44 UTC

Permalink

Post by Jason Pyeron
This is likely a problem between keyboard and chair, but it has got me confused all the same.
I was having issues with rsync, so I switched to dd and the same problem persists.
Any suggestions?
History says "lvcreate --size 8589934592b --name ciphershed-dmz-http vg_five66

"No space left on device" is a filesystem error. This happens to me
occasionally when the /dev/... path does not actually exist for various
reasons, and the output is going into the root filesystem instead.

In your case, the "varying reason" is rsync. Rsync removes
/dev/vg_five66/ciphershed-dmz-http, and creates a regular file - which
it proceeds to fill with data from your source lv until running out of
space in your root fs. When you removed and recreated the LV, this
removed the rsync handiwork and your subsequent dd worked again. But
when you ran rsync again - bye bye link to LV device node.

There is a patch to rsync for copying devices nodes - but the /dev/...
pathnames are often symlinks.
There are utilities for copying block devices with rsync protocol - I
have a python script that I have used around somewhere, but I can't put
my fingers on it at the moment.

Jason Pyeron

2015-02-17 20:26:02 UTC

Permalink

-----Original Message-----
From: Stuart Gathman
Sent: Tuesday, February 17, 2015 13:54

Post by Jason Pyeron
This is likely a problem between keyboard and chair, but it

has got me confused all the same.

Post by Jason Pyeron
I was having issues with rsync, so I switched to dd and the

same problem persists.

Post by Jason Pyeron
Any suggestions?
History says "lvcreate --size 8589934592b --name

ciphershed-dmz-http vg_five66
"No space left on device" is a filesystem error. This happens to me
occasionally when the /dev/... path does not actually exist
for various
reasons, and the output is going into the root filesystem instead.

I knew the problem was between the keyboard and chair.

In your case, the "varying reason" is rsync. Rsync removes
/dev/vg_five66/ciphershed-dmz-http, and creates a regular
file - which
it proceeds to fill with data from your source lv until
running out of
space in your root fs. When you removed and recreated the LV, this
removed the rsync handiwork and your subsequent dd worked again. But
when you ran rsync again - bye bye link to LV device node.

I thought I was being smart: rsync --inplace --copy-devices --partial --compress --progress ...

Turns out I need: rsync --write-devices ...

Which is not in RHEL6, so off to patch from https://git.samba.org/?p=rsync-patches.git;a=blob;f=write-devices.diff

There is a patch to rsync for copying devices nodes - but the
/dev/...
pathnames are often symlinks.

Rsync to /dev/dm-2 has the same issue:

if (statret == 0 && !S_ISREG(sx.st.st_mode)) {
if (delete_item(fname, sx.st.st_mode, del_opts | DEL_FOR_FILE) != 0)
...

The above patch changes it to:

if (statret == 0 && !(S_ISREG(sx.st.st_mode) || (write_devices && IS_DEVICE(sx.st.st_mode)))) {
...

There are utilities for copying block devices with rsync protocol - I
have a python script that I have used around somewhere, but I
can't put
my fingers on it at the moment.

Thanks!

Here is the RHEL6 rsync srpm (patch) and results, for all the others and Google.

https://bugzilla.redhat.com/show_bug.cgi?id=1193654

-Jason

Continue reading on narkive:

Search results for '[linux-lvm] Confusing error: No space left on device' (Questions and Answers)

replies

USB help plz!!!!?

started 2008-04-07 23:49:36 UTC

computers & internet

replies

how to solve "Out of Memory Error"?

started 2009-08-17 08:10:32 UTC

desktops

replies

Hard Drive failure Critical Error and System Restore Error, please help!?