[linux-lvm] LVM snapshot setup for mysql-zrm backups problem

[Philip Amadeo Saeli]

I'm not sure where is the best place to ask this so, since I am using
LVM, I am asking here in the hope that someone can at least direct me to
the appropriate place to ask this. This is my first time setting up
LVM snapshots for mysql-zrm backups.

CentOS-6.6
MySQL-5.6.22
MySQL-zrm-3.0 (EPEL).

Short version:

I am unable to tell if the MySQL-zrm LVM snapshots are indeed taking
place, or if the backup is being taken directly from the main database
LVs. The log file info isn't giving me enough info to tell one way or
another.

Some more info:

I am setting up a database server with all of the database files on LVM
LVs and am using mysql-zrm for logical backups using LVM snapshots to
reduce the database lock times.

I have followed all of the various setup instructions very closely, and
am using the default "lvm-snapshot.pl" plugin. However, the only log
file lines mentioning snapshot are the following:

mysql:backup:INFO: Running snapshot verification using command /usr/share/mysql-zrm/plugins/lvm-snapshot.pl --action verify-config 2>/home/tmp/xH8KKpsF2O

There are no other log entries following this one that seem to have
anything to do with snapshots. There are no errors. The backups
succeed. Both "--verbose" and "--noquiet" are enabled.

Here are the salient entries from the mysql-zrm.conf file:

backup-level=0
backup-mode=logical
lvm-snapshot=20G
snapshot-plugin="/usr/share/mysql-zrm/plugins/lvm-snapshot.pl"
backup-type=regular
destination=/home/mysql-zrm
retention-policy=4W
compress=1

Any info or help would be much appreciated!

Thanks,

--Phil

[Marian Csontos]

Post by Philip Amadeo Saeli
I'm not sure where is the best place to ask this so, since I am using

MySQL list, perhaps?

What does `lvs` say? Is there any snapshot of the LV where your database
lives? Copy if to backup destination and try to check the content - I
presume there is a tool to check the backup...

Just a word of warning:

if the script is using old-style snapshots, you want to copy and delete
it ASAP, or you will see severely degraded performance, and much worse
once you have multiple snapshots in the system.

Also once the space allocated for snapshot gets full, the snapshot
becomes invalid, so hurry up!

-- Martian

Post by Philip Amadeo Saeli
LVM, I am asking here in the hope that someone can at least direct me to
the appropriate place to ask this. This is my first time setting up
LVM snapshots for mysql-zrm backups.
CentOS-6.6
MySQL-5.6.22
MySQL-zrm-3.0 (EPEL).
I am unable to tell if the MySQL-zrm LVM snapshots are indeed taking
place, or if the backup is being taken directly from the main database
LVs. The log file info isn't giving me enough info to tell one way or
another.
I am setting up a database server with all of the database files on LVM
LVs and am using mysql-zrm for logical backups using LVM snapshots to
reduce the database lock times.
I have followed all of the various setup instructions very closely, and
am using the default "lvm-snapshot.pl" plugin. However, the only log
mysql:backup:INFO: Running snapshot verification using command /usr/share/mysql-zrm/plugins/lvm-snapshot.pl --action verify-config 2>/home/tmp/xH8KKpsF2O
There are no other log entries following this one that seem to have
anything to do with snapshots. There are no errors. The backups
succeed. Both "--verbose" and "--noquiet" are enabled.
backup-level=0
backup-mode=logical
lvm-snapshot=20G
snapshot-plugin="/usr/share/mysql-zrm/plugins/lvm-snapshot.pl"
backup-type=regular
destination=/home/mysql-zrm
retention-policy=4W
compress=1
Any info or help would be much appreciated!
Thanks,
--Phil

[Philip Amadeo Saeli]

Thanks for the response. More info and questions following.

Post by Marian Csontos
What does `lvs` say? Is there any snapshot of the LV where your
database lives? Copy if to backup destination and try to check the
content - I presume there is a tool to check the backup...

The backups appear to be OK. However, my suspicion is that they're
being made from the main database LVs rather than from a freshly-made
snapshot.

I am able to manually make a snapshot LV and it looks OK. I have also
upped the LVM log level so I can better monitor any LVM errors or
warnings.

However, I have another question:

When I run any of the various LVM cmds with the "-v" flag, I get a
"DEGRADED MODE. Incomplete RAID LVs will be processed." message in the
output. I find this surprising since none of the LVM vols on that
system are RAID vols. They are simply single PV vols created for the
sole purpose of enabling snapshots for consistent point-in-time database
backups. Any ideas why it is printing that?

Post by Marian Csontos
if the script is using old-style snapshots, you want to copy and
delete it ASAP, or you will see severely degraded performance, and
much worse once you have multiple snapshots in the system.

What do you mean by "old-style" snapshots? Does that mean non-thin
snapshots? Or something else? What type of degraded performance? Any
references?

Note that the snapshots I'm using are ephemeral and are to exist only
long enough for the actual database backup operation to complete.

Post by Marian Csontos
Also once the space allocated for snapshot gets full, the snapshot
becomes invalid, so hurry up!
-- Martian

Thanks!

--Phil

Post by Marian Csontos

Post by Philip Amadeo Saeli
the appropriate place to ask this. This is my first time setting up
LVM snapshots for mysql-zrm backups.
CentOS-6.6
MySQL-5.6.22
MySQL-zrm-3.0 (EPEL).
I am unable to tell if the MySQL-zrm LVM snapshots are indeed taking
place, or if the backup is being taken directly from the main database
LVs. The log file info isn't giving me enough info to tell one way or
another.
I am setting up a database server with all of the database files on LVM
LVs and am using mysql-zrm for logical backups using LVM snapshots to
reduce the database lock times.
I have followed all of the various setup instructions very closely, and
am using the default "lvm-snapshot.pl" plugin. However, the only log
mysql:backup:INFO: Running snapshot verification using command /usr/share/mysql-zrm/plugins/lvm-snapshot.pl --action verify-config 2>/home/tmp/xH8KKpsF2O
There are no other log entries following this one that seem to have
anything to do with snapshots. There are no errors. The backups
succeed. Both "--verbose" and "--noquiet" are enabled.
backup-level=0
backup-mode=logical
lvm-snapshot=20G
snapshot-plugin="/usr/share/mysql-zrm/plugins/lvm-snapshot.pl"
backup-type=regular
destination=/home/mysql-zrm
retention-policy=4W
compress=1
Any info or help would be much appreciated!
Thanks,
--Phil

[Marian Csontos]

Post by Philip Amadeo Saeli
Thanks for the response. More info and questions following.

Post by Marian Csontos
What does `lvs` say? Is there any snapshot of the LV where your
database lives? Copy if to backup destination and try to check the
content - I presume there is a tool to check the backup...

The backups appear to be OK. However, my suspicion is that they're
being made from the main database LVs rather than from a freshly-made
snapshot.

That's exactly question for mysql-list.

You could monitor the snapshot with for example iotop or something to
see if it is being read from but...

Post by Philip Amadeo Saeli
I am able to manually make a snapshot LV and it looks OK. I have also
upped the LVM log level so I can better monitor any LVM errors or
warnings.
When I run any of the various LVM cmds with the "-v" flag, I get a
"DEGRADED MODE. Incomplete RAID LVs will be processed." message in the
output. I find this surprising since none of the LVM vols on that
system are RAID vols. They are simply single PV vols created for the
sole purpose of enabling snapshots for consistent point-in-time database
backups. Any ideas why it is printing that?

Post the output of the command with -vvvv, please.
And may be lvmdump.

Post by Philip Amadeo Saeli

Post by Marian Csontos
if the script is using old-style snapshots, you want to copy and
delete it ASAP, or you will see severely degraded performance, and
much worse once you have multiple snapshots in the system.

What do you mean by "old-style" snapshots? Does that mean non-thin
snapshots? Or something else?

Yes, non-thin snapshots.

Post by Philip Amadeo Saeli
What type of degraded performance? Any
references?

For example I were unable to squeeze more than few hundred IOPS on
rather high-end hardware from old-snapshots (but it was on RHEL6 kernel
missing few recent performance optimizations.)

And the more snapshots of the origin LV you have the write performance
is divided by the number of them as the original value must be written
to each of them. (And there is no optimization for that. So remember to
remove the snapshots early.)

HTH

Marian

Post by Philip Amadeo Saeli
Note that the snapshots I'm using are ephemeral and are to exist only
long enough for the actual database backup operation to complete.

Post by Marian Csontos
Also once the space allocated for snapshot gets full, the snapshot
becomes invalid, so hurry up!
-- Martian

Thanks!
--Phil

Post by Marian Csontos

Post by Philip Amadeo Saeli
the appropriate place to ask this. This is my first time setting up
LVM snapshots for mysql-zrm backups.
CentOS-6.6
MySQL-5.6.22
MySQL-zrm-3.0 (EPEL).
I am unable to tell if the MySQL-zrm LVM snapshots are indeed taking
place, or if the backup is being taken directly from the main database
LVs. The log file info isn't giving me enough info to tell one way or
another.
I am setting up a database server with all of the database files on LVM
LVs and am using mysql-zrm for logical backups using LVM snapshots to
reduce the database lock times.
I have followed all of the various setup instructions very closely, and
am using the default "lvm-snapshot.pl" plugin. However, the only log
mysql:backup:INFO: Running snapshot verification using command /usr/share/mysql-zrm/plugins/lvm-snapshot.pl --action verify-config 2>/home/tmp/xH8KKpsF2O
There are no other log entries following this one that seem to have
anything to do with snapshots. There are no errors. The backups
succeed. Both "--verbose" and "--noquiet" are enabled.
backup-level=0
backup-mode=logical
lvm-snapshot=20G
snapshot-plugin="/usr/share/mysql-zrm/plugins/lvm-snapshot.pl"
backup-type=regular
destination=/home/mysql-zrm
retention-policy=4W
compress=1
Any info or help would be much appreciated!
Thanks,
--Phil