Skip to content

Commit abb0ae5

Browse files
gsmolkgsmolk
committed
[Issue #66] Incremental restore
1 parent 53b7fe9 commit abb0ae5

15 files changed

+2802
-117
lines changed

‎doc/pgprobackup.xml

+146-2
Original file line numberDiff line numberDiff line change
@@ -196,6 +196,12 @@ doc/src/sgml/pgprobackup.sgml
196196
backups than by replaying WAL files.
197197
</para>
198198
</listitem>
199+
<listitem>
200+
<para>
201+
Incremental restore: speed up restore from backup by reusing
202+
valid unchanged pages available in PGDATA.
203+
</para>
204+
</listitem>
199205
<listitem>
200206
<para>
201207
Validation: automatic data consistency checks and on-demand
@@ -1698,6 +1704,105 @@ pg_probackup restore -B <replaceable>backup_dir</replaceable> --instance <replac
16981704
speed up the recovery.
16991705
</para>
17001706
</note>
1707+
<refsect3 id="pbk-incremental-restore">
1708+
<title>Incremental Restore</title>
1709+
<para>
1710+
The speed of restore from backup can be significantly improved
1711+
by replacing only invalid and changed pages in already
1712+
existing PostgreSQL data directory using
1713+
<link linkend="pbk-incremental-restore-options">incremental
1714+
restore options</link> with the <xref linkend="pbk-restore"/>
1715+
command.
1716+
</para>
1717+
<para>
1718+
To restore the database cluster from a backup in incremental mode,
1719+
run the <xref linkend="pbk-restore"/> command with the following options:
1720+
</para>
1721+
<programlisting>
1722+
pg_probackup restore -B <replaceable>backup_dir</replaceable> --instance <replaceable>instance_name</replaceable> -D <replaceable>data_dir</replaceable> -I <replaceable>incremental_mode</replaceable>
1723+
</programlisting>
1724+
<para>
1725+
Where <replaceable>incremental_mode</replaceable> can take one of the
1726+
following values:
1727+
</para>
1728+
<itemizedlist spacing="compact">
1729+
<listitem>
1730+
<para>
1731+
CHECKSUM — read all data files in the data directory, validate
1732+
header and checksum in every page and replace only invalid
1733+
pages and those with checksum and LSN not matching with
1734+
corresponding page in backup. This is the simplest,
1735+
the most fool-proof incremental mode.
1736+
</para>
1737+
</listitem>
1738+
<listitem>
1739+
<para>
1740+
LSN — read the <replaceable>pg_control</replaceable> in the
1741+
data directory to obtain redo LSN and redo TLI, which allows
1742+
to determine a point in history(shiftpoint), where data directory
1743+
state shifted from backup chain history. If shiftpoint is not within
1744+
reach of backup chain history, then restore is aborted.
1745+
If shiftpoint is within reach of backup chain history, then read
1746+
all data files in the data directory, validate header and checksum in
1747+
every page and replace only invalid pages and those with LSN greater
1748+
than shiftpoint.
1749+
This mode offer a far greater speed up compared to CHECKSUM, but rely
1750+
on two conditions to be met. First,
1751+
<ulink url="https://postgrespro.com/docs/postgresql/current/runtime-config-preset.html#GUC-DATA-CHECKSUMS">
1752+
data checksums</ulink> parameter must be enabled in data directory (to avoid corruption
1753+
due to hint bits). This condition will be checked at the start of
1754+
incremental restore and the operation will be aborted if checksums are disabled.
1755+
Second, the <replaceable>pg_control</replaceable> file must be
1756+
synched with state of data directory. This condition cannot checked
1757+
at the start of restore, so it is a user responsibility to ensure
1758+
that pg_control contain valid information. Because of that is not
1759+
recommended to use LSN mode in any situation, where pg_control has
1760+
been tampered with:
1761+
after <replaceable>pg_resetxlog</replaceable> execution,
1762+
after restore from backup without recovery been run, etc.
1763+
</para>
1764+
</listitem>
1765+
<listitem>
1766+
<para>
1767+
NONE — regular restore without any incremental optimizations.
1768+
</para>
1769+
</listitem>
1770+
</itemizedlist>
1771+
1772+
<para>
1773+
Suppose you want to return an old master as replica after switchover
1774+
using incremental restore in LSN mode:
1775+
</para>
1776+
<programlisting>
1777+
=============================================================================================================================================
1778+
Instance Version ID Recovery Time Mode WAL Mode TLI Time Data WAL Zratio Start LSN Stop LSN Status
1779+
=============================================================================================================================================
1780+
node 12 QBRNBP 2020-06-11 17:40:58+03 DELTA ARCHIVE 16/15 40s 194MB 16MB 8.26 15/2C000028 15/2D000128 OK
1781+
node 12 QBRIDX 2020-06-11 15:51:42+03 PAGE ARCHIVE 15/15 11s 18MB 16MB 5.10 14/DC000028 14/DD0000B8 OK
1782+
node 12 QBRIAJ 2020-06-11 15:51:08+03 PAGE ARCHIVE 15/15 20s 141MB 96MB 6.22 14/D4BABFE0 14/DA9871D0 OK
1783+
node 12 QBRHT8 2020-06-11 15:45:56+03 FULL ARCHIVE 15/0 2m:11s 1371MB 416MB 10.93 14/9D000028 14/B782E9A0 OK
1784+
1785+
pg_probackup restore -B /backup --instance node -R -I lsn
1786+
INFO: Running incremental restore into nonempty directory: "/var/lib/pgsql/12/data"
1787+
INFO: Destination directory redo point 15/2E000028 on tli 16 is within reach of backup QBRIDX with Stop LSN 14/DD0000B8 on tli 15, incremental restore in 'lsn' mode is possible
1788+
INFO: shift LSN: 14/DD0000B8
1789+
INFO: Restoring the database from backup at 2020-06-11 15:39:08+03
1790+
INFO: Extracting the content of destination directory for incremental restore
1791+
INFO: Destination directory content extracted, time elapsed: 1s
1792+
INFO: Removing redundant files in destination directory
1793+
INFO: Redundant files are removed, time elapsed: 1s
1794+
INFO: Start restoring backup files. PGDATA size: 15GB
1795+
INFO: Backup files are restored. Transfered bytes: 1693MB, time elapsed: 43s
1796+
INFO: Restore incremental ratio (less is better): 11% (1693MB/15GB)
1797+
INFO: Restore of backup QBRHT8 completed.
1798+
</programlisting>
1799+
<note>
1800+
<para>
1801+
Incremental restore is possible only for backups with
1802+
<literal>program_version</literal> equal or greater than 2.3.0.
1803+
</para>
1804+
</note>
1805+
</refsect3>
17011806
<refsect3 id="pbk-partial-restore">
17021807
<title>Partial Restore</title>
17031808
<para>
@@ -1715,7 +1820,7 @@ pg_probackup restore -B <replaceable>backup_dir</replaceable> --instance <replac
17151820
</para>
17161821
<programlisting>
17171822
pg_probackup restore -B <replaceable>backup_dir</replaceable> --instance <replaceable>instance_name</replaceable> --db-include=<replaceable>database_name</replaceable>
1718-
</programlisting>
1823+
</programlisting>
17191824
<para>
17201825
The <option>--db-include</option> option can be specified
17211826
multiple times. For example, to restore only databases
@@ -3863,7 +3968,7 @@ pg_probackup restore -B <replaceable>backup_dir</replaceable> --instance <replac
38633968
options</link>,
38643969
<link linkend="pbk-remote-wal-archive-options">remote WAL archive
38653970
options</link>, <link linkend="pbk-logging-opts">logging
3866-
options</link>, <link linkend="pbk-partial-restore">partial
3971+
options</link>, <link linkend="pbk-partial-restore-options">partial
38673972
restore options</link>, and <link linkend="pbk-common-options">common
38683973
options</link> can be used.
38693974
</para>
@@ -5075,6 +5180,45 @@ pg_probackup archive-get -B <replaceable>backup_dir</replaceable> --instance <re
50755180
</variablelist>
50765181
</para>
50775182
</refsect3>
5183+
<refsect3 id="pbk-incremental-restore-options">
5184+
<title>Incremental Restore Options</title>
5185+
<para>
5186+
This section describes the options for incremental cluster restore.
5187+
These options can be used with the
5188+
<xref linkend="pbk-restore"/> command.
5189+
</para>
5190+
<para>
5191+
<variablelist>
5192+
<varlistentry>
5193+
<term><option>-I <replaceable>incremental_mode</replaceable></option></term>
5194+
<term><option>--incremental-mode=<replaceable>incremental_mode</replaceable></option></term>
5195+
<listitem>
5196+
<para>
5197+
Specifies the incremental mode to be used. Possible values are:
5198+
5199+
<itemizedlist spacing="compact">
5200+
<listitem>
5201+
<para>
5202+
<literal>CHECKSUM</literal> — replace only pages with mismatched checksum and LSN.
5203+
</para>
5204+
</listitem>
5205+
<listitem>
5206+
<para>
5207+
<literal>LSN</literal> — replace only pages with LSN greater than point of divergence.
5208+
</para>
5209+
</listitem>
5210+
<listitem>
5211+
<para>
5212+
<literal>NONE</literal> — regular restore.
5213+
</para>
5214+
</listitem>
5215+
</itemizedlist>
5216+
</para>
5217+
</listitem>
5218+
</varlistentry>
5219+
</variablelist>
5220+
</para>
5221+
</refsect3>
50785222
<refsect3 id="pbk-partial-restore-options">
50795223
<title>Partial Restore Options</title>
50805224
<para>

‎src/catalog.c

+2-2
Original file line numberDiff line numberDiff line change
@@ -1048,7 +1048,7 @@ catalog_get_timelines(InstanceConfig *instance)
10481048
TimeLineHistoryEntry *tln;
10491049

10501050
sscanf(file->name, "%08X.history", &tli);
1051-
timelines = read_timeline_history(arclog_path, tli);
1051+
timelines = read_timeline_history(arclog_path, tli, true);
10521052

10531053
if (!tlinfo || tlinfo->tli != tli)
10541054
{
@@ -1902,7 +1902,7 @@ write_backup_filelist(pgBackup *backup, parray *files, const char *root,
19021902
if (file->linked)
19031903
len += sprintf(line+len, ",\"linked\":\"%s\"", file->linked);
19041904

1905-
if (file->n_blocks != BLOCKNUM_INVALID)
1905+
if (file->n_blocks > 0)
19061906
len += sprintf(line+len, ",\"n_blocks\":\"%i\"", file->n_blocks);
19071907

19081908
sprintf(line+len, "}\n");

0 commit comments

Comments
 (0)