[linux-lvm] thin: 235:2 pool target (735616 blocks) too small: expected 809216

[Brian J. Murrell]

I have a Fedora 23 system running (presumably, since I can't boot it to
be 100% sure) 2.02.132 of LVM.  It has ceased to boot and reports:

device-mapper: resume ioctl on (253:2) failed: Invalid argument
Unable to resume laptop-pool00-tpool (253:2)
thin: Data device (dm-1) dicard unsupported: Disabling discard passdown.
thin: 235:2 pool target (735616 blocks) too small: expected 809216
table: 253:2: thin-pool preresume failed, error = -22

Any ideas what the problem could be?  I can't imagine why all of a
sudden the pool target would be smaller than it should be.

What further information can I provide to help debug (and very
hopefully repair) this system?

Cheers,
b.

[Zdenek Kabelac]

Post by Brian J. Murrell
I have a Fedora 23 system running (presumably, since I can't boot it to
device-mapper: resume ioctl on (253:2) failed: Invalid argument
Unable to resume laptop-pool00-tpool (253:2)
thin: Data device (dm-1) dicard unsupported: Disabling discard passdown.
thin: 235:2 pool target (735616 blocks) too small: expected 809216
table: 253:2: thin-pool preresume failed, error = -22
Any ideas what the problem could be? I can't imagine why all of a
sudden the pool target would be smaller than it should be.
What further information can I provide to help debug (and very
hopefully repair) this system?

Hi

Well - could you post somewhere for download 1st. MB of your
PV device?

dd if=/dev/sdX of=/tmp/upload_me bs=1M count=1

Have you already tried to restore in some way ?

Regards


Zdenek

[Brian J. Murrell]

This post might be inappropriate. Click to display it.

[Zdenek Kabelac]

Post by Zdenek Kabelac
Hi

Sorry for the delay. It can be a bugger to get an environment on a
laptop that won't boot USB devices to a state where you can access the
PV and store something from it.

Post by Zdenek Kabelac
Well - could you post somewhere for download 1st. MB of your
PV device?
dd if=/dev/sdX of=/tmp/upload_me bs=1M count=1

http://www.interlinx.bc.ca/~brian/lvm-pv.img

Post by Zdenek Kabelac
Have you already tried to restore in some way ?

I have not tried anything yet for fear of making things worse ("angels
tread" and all that), so this is a blank slate from the point of it
first failing.

Hi

So it seems your machine has crashed (you probably know better) during
thin-pool resize operation.

Unfortunately fc23 has lvm2 version 2.02.132 - and version 2.02.133 has
improvement patch to make the resize more resistent (but still not as good as
I wish to be).

So what happened - lvm2 resized _tdata LV - tried to resumed it - and
it has failed along this path - however since the 'resize' is ATM a single
transaction - the lvm2 rollback reverted to previous size - yet thin-pool
already managed to remembered 'new' bigger size.

So please take a look at your logs (if you have some) if there
is something suspicious to be mentioned (thought if you machined
has freezed, hardly any log will be available).

To get access to your thin-pool - I'm attaching restored metadata content from
your disk header with 'bigger' _tdata volume.

To restore use:

'vgcfgrestore -f back --force brianr-laptop'

It would be really interesting to know the reason of failure - but I can
understand you could hardly obtain.

Regards


Zdenek

[Brian J. Murrell]

Hi

Hi.

So it seems your  machine has crashed (you probably know better)
during
thin-pool resize operation.

So, this is actually a family member's machine and he was using it at
the time so I don't know, firsthand what happened.

He says he was just adding some plugin to some (Windows) program in
Wine.  Doesn't seem like that should come anywhere near resizing so I
will ask him again.

Unfortunately fc23 has lvm2 version 2.02.132 - and version 2.02.133
has 
improvement patch to make the resize more resistent (but still not as
good as 
I wish to be).

Incremental improvements are improvements all the same.  :-)

So what happened -  lvm2 resized _tdata LV - tried to resumed it -
and
it has failed along this path - however since the 'resize' is ATM a
single 
transaction - the lvm2 rollback reverted to previous size - yet thin-
pool
already managed to remembered 'new' bigger size.

Ahhh.

So please take a look at your logs (if you have some) if there
is something suspicious to be mentioned (thought if you machined
has freezed, hardly any log will be available).

I will take a look once I can get the system back up and running.

To get access to your thin-pool - I'm attaching restored metadata
content from 
your disk header with 'bigger' _tdata volume.

There was no attachment.

'vgcfgrestore -f back  --force brianr-laptop'

I can do that in Fedora's "rescue" environment?  Probably "lvm
vgcfgrestore -f back  --force brianr-laptop" instead.

It would be really interesting to know the reason of failure - but I
can 
understand you could hardly obtain.

I will see if there is anything in the logs and get back to you.  It's
the least I can do in return for the help you have provided.  Much
appreciated.

Cheers,
b.

[Zdenek Kabelac]

Post by Brian J. Murrell

Post by Zdenek Kabelac
Hi

Hi.

Post by Zdenek Kabelac
So it seems your machine has crashed (you probably know better) during
thin-pool resize operation.

So, this is actually a family member's machine and he was using it at
the time so I don't know, firsthand what happened.
He says he was just adding some plugin to some (Windows) program in
Wine. Doesn't seem like that should come anywhere near resizing so I
will ask him again.

Yep that might explain 'missing' surrouding details...
Basically any write might result in thin-pool running out of threshold
and requireing new space - what's unclear is the later failure which
would be really good to know....

Post by Brian J. Murrell

Post by Zdenek Kabelac
Unfortunately fc23 has lvm2 version 2.02.132 - and version 2.02.133 has
improvement patch to make the resize more resistent (but still not as good as
I wish to be).

Incremental improvements are improvements all the same. :-)

Post by Zdenek Kabelac
So what happened - lvm2 resized _tdata LV - tried to resumed it - and
it has failed along this path - however since the 'resize' is ATM a single
transaction - the lvm2 rollback reverted to previous size - yet thin-
pool
already managed to remembered 'new' bigger size.

Ahhh.

Post by Zdenek Kabelac
So please take a look at your logs (if you have some) if there
is something suspicious to be mentioned (thought if you machined
has freezed, hardly any log will be available).

I will take a look once I can get the system back up and running.

Post by Zdenek Kabelac
To get access to your thin-pool - I'm attaching restored metadata content from
your disk header with 'bigger' _tdata volume.

There was no attachment.

Ops - so once again ;)

Post by Brian J. Murrell

Post by Zdenek Kabelac
'vgcfgrestore -f back --force brianr-laptop'

I can do that in Fedora's "rescue" environment? Probably "lvm
vgcfgrestore -f back --force brianr-laptop" instead.

Post by Zdenek Kabelac
It would be really interesting to know the reason of failure - but I can
understand you could hardly obtain.

I will see if there is anything in the logs and get back to you. It's
the least I can do in return for the help you have provided. Much
appreciated.
Cheers,

Zdenek

[Zdenek Kabelac]

Post by Zdenek Kabelac
Yep that might explain 'missing' surrouding details...
Basically any write might result in thin-pool running out of
threshold
and requireing new space

Ahhh. I was wondering if the resize in question could be some kind of
implicit resizing done by the "hidden" thin-pool maintenance. I'm
fairly sure he knows zilch about LVM and resizing. :-)

When lvm.conf has defined thin_pool_autoextend_threshold <100% the pool is
automatically monitored and resized when threshold is crossed.

Post by Zdenek Kabelac
- what's unclear is the later failure which
would be really good to know....

Is there anything in particular I can look for in the logs that would
indicate when this implicit resizing was happening to look for other

If you spot any 'device-mapper' errors on May 25 in kernel log....

activity that might have caused a problem?

Post by Zdenek Kabelac
Ops - so once again ;)

Cheers. :-)
b.

Zdenek

[Brian J. Murrell]

 
When lvm.conf  has defined thin_pool_autoextend_threshold <100% the
pool is 
automatically monitored and resized when threshold is crossed.

Indeed.

If you spot any 'device-mapper' errors on May 25 in kernel log....

Nothing.  Not a single device-mapper message since May 19 and that was
only:

May 19 17:00:23 laptop kernel: device-mapper: thin: Data device (dm-1) discard unsupported: Disabling discard passdown.

during a normal boot.

For what it's worth, I have the machine up and running so your help and
advise were spot on.  Thanks again so much.

Cheers,
b.