[linux-lvm] [BUG?] OOM executing blkdiscardon a thin lv

[Qu Wenruo]

Hi,

[Bug]
When I try to discard the whole thin lv using blkdiscard, an OOM occurs
and I lost my ssh connection.
TTY login also fails to start, although kernel and network still looks
alive but no way to login into the system.
Have to reboot.

Not sure if this is a bug in blkdiscard or lvm.

[lvs output]
LV VG Attr LSize Pool Origin Data% Move
Log Cpy%Sync Convert
disk1 test_vg Vwi-a-tz-- 1.00t thinpool 0.77
thinpool test_vg twi-a-tz-- 30.00g 26.28

[Reproducer]
1. Make a thin pool with 30G space.
2. Create 1T thin lv from the thin pool created in 1.
3. Do some operation to occupy some space in thin lv
4. Exec blkdiscard on the thin lv

[Related package version]
kernel-3.17.3-200.fc20.x86_64
lvm2-2.02.106-1.fc20.x86_64
util-linux-2.24.2-1.fc20.x86_64

[How to discard thin lv]
BTW, what's the proper method to discard all allocated space for a thin lv?
Lvremove and lvcreate?!

Thanks,
Qu

[Mike Snitzer]

On Wed, Nov 26 2014 at 2:54am -0500,

Post by Qu Wenruo
Hi,
[Bug]
When I try to discard the whole thin lv using blkdiscard, an OOM
occurs and I lost my ssh connection.
TTY login also fails to start, although kernel and network still
looks alive but no way to login into the system.
Have to reboot.
Not sure if this is a bug in blkdiscard or lvm.

It is a bug in the kernel's blkdev_issue_discard() for allowing too many
inflight bios at once. We probably should add an upper bound for how
many inflight discard bios blkdev_issue_discard() will allow at any one
time.

Reason why so many discard bios are being issued is that the DM thin
device currently only allows a discard as large as the DM thin-pool's
configured blocksize. So if you're using something like a 64K or
whatever for the thin-pool blocksize then divide your 1TB thin lv by
that value and you can quickly see why it takes so may discard bios to
cover the entire logical address space of the 1TB thin device.

Some good news is that Joe has been working on range (multi block)
discard support to allow for very large discards to be issued to DM thin
devices. He is focusing on getting this working on dm-cache at the
moment but once that is complete bringing it to dm-thin is next. It is
a stretch goal to have that land in time for 3.19 merge window (but it
is quickly approaching so it may need to wait for 3.20).

Post by Qu Wenruo
[lvs output]
LV VG Attr LSize Pool Origin Data% Move
Log Cpy%Sync Convert
disk1 test_vg Vwi-a-tz-- 1.00t thinpool 0.77
thinpool test_vg twi-a-tz-- 30.00g 26.28
[Reproducer]
1. Make a thin pool with 30G space.
2. Create 1T thin lv from the thin pool created in 1.
3. Do some operation to occupy some space in thin lv
4. Exec blkdiscard on the thin lv
[Related package version]
kernel-3.17.3-200.fc20.x86_64
lvm2-2.02.106-1.fc20.x86_64
util-linux-2.24.2-1.fc20.x86_64
[How to discard thin lv]
BTW, what's the proper method to discard all allocated space for a thin lv?
Lvremove and lvcreate?!

Moving forward it'll be a new tool that Joe wrote called thin_trim:
https://github.com/jthornber/thin-provisioning-tools/commit/8e921580554ed91e84bb68ea32a8c2ea47ad6ff3

But that code hasn't been fully tested and included in the
device-mapper-persistent-data package yet.

[Qu Wenruo]

-------- Original Message --------
Subject: Re: [BUG?] OOM executing blkdiscard on a thin lv
From: Mike Snitzer <***@redhat.com>
To: Qu Wenruo <***@cn.fujitsu.com>
Date: 2014年11月26日 23:06

Post by Mike Snitzer
On Wed, Nov 26 2014 at 2:54am -0500,

Post by Qu Wenruo
Hi,
[Bug]
When I try to discard the whole thin lv using blkdiscard, an OOM
occurs and I lost my ssh connection.
TTY login also fails to start, although kernel and network still
looks alive but no way to login into the system.
Have to reboot.
Not sure if this is a bug in blkdiscard or lvm.

It is a bug in the kernel's blkdev_issue_discard() for allowing too many
inflight bios at once. We probably should add an upper bound for how
many inflight discard bios blkdev_issue_discard() will allow at any one
time.
Reason why so many discard bios are being issued is that the DM thin
device currently only allows a discard as large as the DM thin-pool's
configured blocksize. So if you're using something like a 64K or
whatever for the thin-pool blocksize then divide your 1TB thin lv by
that value and you can quickly see why it takes so may discard bios to
cover the entire logical address space of the 1TB thin device.
Some good news is that Joe has been working on range (multi block)
discard support to allow for very large discards to be issued to DM thin
devices. He is focusing on getting this working on dm-cache at the
moment but once that is complete bringing it to dm-thin is next. It is
a stretch goal to have that land in time for 3.19 merge window (but it
is quickly approaching so it may need to wait for 3.20).

Thanks for the explain, got it.
I'll wait several kernel releases and try it again.

Post by Mike Snitzer

Post by Qu Wenruo
[lvs output]
LV VG Attr LSize Pool Origin Data% Move
Log Cpy%Sync Convert
disk1 test_vg Vwi-a-tz-- 1.00t thinpool 0.77
thinpool test_vg twi-a-tz-- 30.00g 26.28
[Reproducer]
1. Make a thin pool with 30G space.
2. Create 1T thin lv from the thin pool created in 1.
3. Do some operation to occupy some space in thin lv
4. Exec blkdiscard on the thin lv
[Related package version]
kernel-3.17.3-200.fc20.x86_64
lvm2-2.02.106-1.fc20.x86_64
util-linux-2.24.2-1.fc20.x86_64
[How to discard thin lv]
BTW, what's the proper method to discard all allocated space for a thin lv?
Lvremove and lvcreate?!

https://github.com/jthornber/thin-provisioning-tools/commit/8e921580554ed91e84bb68ea32a8c2ea47ad6ff3
But that code hasn't been fully tested and included in the
device-mapper-persistent-data package yet.

OK, so I'll use the lvremove and lvcreate until the tool gets mature.

Thanks,
Qu