Skip to content

Commit b188658

Browse files
author
Commitfest Bot
committed
[CF 5111] v20251030 - Synchronization of sequences to subscriber
This branch was automatically generated by a robot using patches from an email thread registered at: https://commitfest.postgresql.org/patch/5111 The branch will be overwritten each time a new patch version is posted to the thread, and also periodically to check for bitrot caused by changes on the master branch. Patch(es): https://www.postgresql.org/message-id/CALDaNm3HTiMB549A_or_gLctioD+AzXejbBYQN4bRidt3Y3=8Q@mail.gmail.com Author(s): vigneshwaran C
2 parents c106ef0 + cc3ac5a commit b188658

File tree

31 files changed

+1546
-192
lines changed

31 files changed

+1546
-192
lines changed

‎doc/src/sgml/catalogs.sgml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6568,7 +6568,7 @@ SCRAM-SHA-256$<replaceable>&lt;iteration count&gt;</replaceable>:<replaceable>&l
65686568
(references <link linkend="catalog-pg-class"><structname>pg_class</structname></link>.<structfield>oid</structfield>)
65696569
</para>
65706570
<para>
6571-
Reference to relation
6571+
Reference to table or sequence
65726572
</para></entry>
65736573
</row>
65746574

‎doc/src/sgml/config.sgml‎

Lines changed: 9 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -5191,9 +5191,9 @@ ANY <replaceable class="parameter">num_sync</replaceable> ( <replaceable class="
51915191
is taken into account.
51925192
</para>
51935193
<para>
5194-
In logical replication, this parameter also limits how often a failing
5195-
replication apply worker or table synchronization worker will be
5196-
respawned.
5194+
In logical replication, this parameter also limits how quickly a
5195+
failing replication apply worker, table synchronization worker, or
5196+
sequence synchronization worker will be respawned.
51975197
</para>
51985198
</listitem>
51995199
</varlistentry>
@@ -5334,8 +5334,8 @@ ANY <replaceable class="parameter">num_sync</replaceable> ( <replaceable class="
53345334
<listitem>
53355335
<para>
53365336
Specifies maximum number of logical replication workers. This includes
5337-
leader apply workers, parallel apply workers, and table synchronization
5338-
workers.
5337+
leader apply workers, parallel apply workers, table synchronization
5338+
workers and a sequence synchronization worker.
53395339
</para>
53405340
<para>
53415341
Logical replication workers are taken from the pool defined by
@@ -5358,10 +5358,12 @@ ANY <replaceable class="parameter">num_sync</replaceable> ( <replaceable class="
53585358
<para>
53595359
Maximum number of synchronization workers per subscription. This
53605360
parameter controls the amount of parallelism of the initial data copy
5361-
during the subscription initialization or when new tables are added.
5361+
during the subscription initialization or when new tables or sequences
5362+
are added.
53625363
</para>
53635364
<para>
5364-
Currently, there can be only one synchronization worker per table.
5365+
Currently, there can be only one table synchronization worker per table
5366+
and one sequence synchronization worker to synchronize all sequences.
53655367
</para>
53665368
<para>
53675369
The synchronization workers are taken from the pool defined by

‎doc/src/sgml/func/func-sequence.sgml‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -143,6 +143,30 @@ SELECT setval('myseq', 42, false); <lineannotation>Next <function>nextval</fu
143143
or <literal>SELECT</literal> privilege on the last used sequence.
144144
</para></entry>
145145
</row>
146+
147+
<row>
148+
<entry role="func_table_entry"><para role="func_signature">
149+
<indexterm>
150+
<primary>pg_get_sequence_data</primary>
151+
</indexterm>
152+
<function>pg_get_sequence_data</function> ( <type>regclass</type> )
153+
<returnvalue>record</returnvalue>
154+
( <parameter>last_value</parameter> <type>bigint</type>,
155+
<parameter>is_called</parameter> <type>bool</type>,
156+
<parameter>page_lsn</parameter> <type>pg_lsn</type> )
157+
</para>
158+
<para>
159+
Returns information about the sequence. <literal>last_value</literal>
160+
indicates last sequence value set in sequence by nextval or setval,
161+
<literal>is_called</literal> indicates whether the sequence has been
162+
used, and <literal>page_lsn</literal> is the LSN corresponding to the
163+
most recent WAL record that modified this sequence relation.
164+
</para>
165+
<para>
166+
This function requires <literal>USAGE</literal>
167+
or <literal>SELECT</literal> privilege on the sequence.
168+
</para></entry>
169+
</row>
146170
</tbody>
147171
</tgroup>
148172
</table>

‎doc/src/sgml/logical-replication.sgml‎

Lines changed: 215 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -113,7 +113,9 @@
113113
Publications may currently only contain tables or sequences. Objects must be
114114
added explicitly, except when a publication is created using
115115
<literal>FOR TABLES IN SCHEMA</literal>, <literal>FOR ALL TABLES</literal>,
116-
or <literal>FOR ALL SEQUENCES</literal>.
116+
or <literal>FOR ALL SEQUENCES</literal>. Unlike tables, the state of
117+
sequences can be synchronized at any time. For more information, see
118+
<xref linkend="logical-replication-sequences"/>.
117119
</para>
118120

119121
<para>
@@ -1745,6 +1747,200 @@ Publications:
17451747
</note>
17461748
</sect1>
17471749

1750+
<sect1 id="logical-replication-sequences">
1751+
<title>Replicating Sequences</title>
1752+
1753+
<para>
1754+
To synchronize sequences from a publisher to a subscriber, first publish
1755+
them using <link linkend="sql-createpublication-params-for-all-sequences">
1756+
<command>CREATE PUBLICATION ... FOR ALL SEQUENCES</command></link> and then
1757+
on the subscriber:
1758+
</para>
1759+
1760+
<para>
1761+
<itemizedlist>
1762+
<listitem>
1763+
<para>
1764+
use <link linkend="sql-createsubscription"><command>CREATE SUBSCRIPTION</command></link>
1765+
to initially synchronize the published sequences.
1766+
</para>
1767+
</listitem>
1768+
<listitem>
1769+
<para>
1770+
use <link linkend="sql-altersubscription-params-refresh-publication">
1771+
<command>ALTER SUBSCRIPTION ... REFRESH PUBLICATION</command></link>
1772+
to synchronize only newly added sequences.
1773+
</para>
1774+
</listitem>
1775+
<listitem>
1776+
<para>
1777+
use <link linkend="sql-altersubscription-params-refresh-sequences">
1778+
<command>ALTER SUBSCRIPTION ... REFRESH SEQUENCES</command></link>
1779+
to re-synchronize all sequences.
1780+
</para>
1781+
</listitem>
1782+
</itemizedlist>
1783+
</para>
1784+
1785+
<para>
1786+
A new <firstterm>sequence synchronization worker</firstterm> will be started
1787+
after executing any of the above subscriber commands, and will exit once the
1788+
sequences are synchronized.
1789+
</para>
1790+
<para>
1791+
The ability to launch a sequence synchronization worker is limited by the
1792+
<link linkend="guc-max-sync-workers-per-subscription">
1793+
<varname>max_sync_workers_per_subscription</varname></link>
1794+
configuration.
1795+
</para>
1796+
1797+
<sect2 id="sequence-definition-mismatches">
1798+
<title>Sequence Definition Mismatches</title>
1799+
<para>
1800+
During sequence synchronization, the sequence definitions of the publisher
1801+
and the subscriber are compared. An error is logged listing all differing
1802+
sequences before the process exits. The apply worker detects this failure
1803+
and repeatedly respawns the sequence synchronization worker to retry until
1804+
all differences are resolved. See also
1805+
<link linkend="guc-wal-retrieve-retry-interval"><varname>wal_retrieve_retry_interval</varname></link>.
1806+
</para>
1807+
<para>
1808+
To resolve this, use
1809+
<link linkend="sql-altersequence"><command>ALTER SEQUENCE</command></link>
1810+
to align the subscriber's sequence parameters with those of the publisher.
1811+
</para>
1812+
</sect2>
1813+
1814+
<sect2 id="sequences-out-of-sync">
1815+
<title>Refreshing Stale Sequences</title>
1816+
<para>
1817+
Subscriber side sequence values may frequently become out of sync due to
1818+
updates on the publisher.
1819+
</para>
1820+
<para>
1821+
To verify, compare the sequence values between the publisher and
1822+
subscriber, and if necessary, execute
1823+
<link linkend="sql-altersubscription-params-refresh-sequences">
1824+
<command>ALTER SUBSCRIPTION ... REFRESH SEQUENCES</command></link>.
1825+
</para>
1826+
</sect2>
1827+
1828+
<sect2 id="logical-replication-sequences-examples">
1829+
<title>Examples</title>
1830+
1831+
<para>
1832+
Create some sequences on the publisher.
1833+
<programlisting>
1834+
test_pub=# CREATE SEQUENCE s1 START WITH 10 INCREMENT BY 1;
1835+
CREATE SEQUENCE
1836+
test_pub=# CREATE SEQUENCE s2 START WITH 100 INCREMENT BY 10;
1837+
CREATE SEQUENCE
1838+
</programlisting></para>
1839+
1840+
<para>
1841+
Create the same sequences on the subscriber.
1842+
<programlisting>
1843+
test_sub=# CREATE SEQUENCE s1 START WITH 10 INCREMENT BY 1
1844+
CREATE SEQUENCE
1845+
test_sub=# CREATE SEQUENCE s2 START WITH 100 INCREMENT BY 10;
1846+
CREATE SEQUENCE
1847+
</programlisting></para>
1848+
1849+
<para>
1850+
Update the sequences at the publisher side a few times.
1851+
<programlisting>
1852+
test_pub=# SELECT nextval('s1');
1853+
nextval
1854+
---------
1855+
10
1856+
(1 row)
1857+
test_pub=# SELECT nextval('s1');
1858+
nextval
1859+
---------
1860+
11
1861+
(1 row)
1862+
test_pub=# SELECT nextval('s2');
1863+
nextval
1864+
---------
1865+
100
1866+
(1 row)
1867+
test_pub=# SELECT nextval('s2');
1868+
nextval
1869+
---------
1870+
110
1871+
(1 row)
1872+
</programlisting></para>
1873+
1874+
<para>
1875+
Create a publication for the sequences.
1876+
<programlisting>
1877+
test_pub=# CREATE PUBLICATION pub1 FOR ALL SEQUENCES;
1878+
CREATE PUBLICATION
1879+
</programlisting></para>
1880+
1881+
<para>
1882+
Subscribe to the publication.
1883+
<programlisting>
1884+
test_sub=# CREATE SUBSCRIPTION sub1
1885+
test_sub-# CONNECTION 'host=localhost dbname=test_pub application_name=sub1'
1886+
test_sub-# PUBLICATION pub1;
1887+
CREATE SUBSCRIPTION
1888+
</programlisting></para>
1889+
1890+
<para>
1891+
Observe that initial sequence values are synchronized.
1892+
<programlisting>
1893+
test_sub=# SELECT * FROM s1;
1894+
last_value | log_cnt | is_called
1895+
------------+---------+-----------
1896+
11 | 31 | t
1897+
(1 row)
1898+
1899+
test_sub=# SELECT * FROM s2;
1900+
last_value | log_cnt | is_called
1901+
------------+---------+-----------
1902+
110 | 31 | t
1903+
(1 row)
1904+
</programlisting></para>
1905+
1906+
<para>
1907+
Update the sequences at the publisher side.
1908+
<programlisting>
1909+
test_pub=# SELECT nextval('s1');
1910+
nextval
1911+
---------
1912+
12
1913+
(1 row)
1914+
test_pub=# SELECT nextval('s2');
1915+
nextval
1916+
---------
1917+
120
1918+
(1 row)
1919+
</programlisting></para>
1920+
1921+
<para>
1922+
Re-synchronize all the sequences on the subscriber using
1923+
<link linkend="sql-altersubscription-params-refresh-sequences">
1924+
<command>ALTER SUBSCRIPTION ... REFRESH SEQUENCES</command></link>.
1925+
<programlisting>
1926+
test_sub=# ALTER SUBSCRIPTION sub1 REFRESH SEQUENCES;
1927+
ALTER SUBSCRIPTION
1928+
1929+
test_sub=# SELECT * FROM s1;
1930+
last_value | log_cnt | is_called
1931+
------------+---------+-----------
1932+
12 | 30 | t
1933+
(1 row)
1934+
1935+
test_sub=# SELECT * FROM s2
1936+
last_value | log_cnt | is_called
1937+
------------+---------+-----------
1938+
120 | 30 | t
1939+
(1 row)
1940+
</programlisting></para>
1941+
</sect2>
1942+
</sect1>
1943+
17481944
<sect1 id="logical-replication-conflicts">
17491945
<title>Conflicts</title>
17501946

@@ -2090,16 +2286,19 @@ CONTEXT: processing remote data for replication origin "pg_16395" during "INSER
20902286

20912287
<listitem>
20922288
<para>
2093-
Sequence data is not replicated. The data in serial or identity columns
2094-
backed by sequences will of course be replicated as part of the table,
2095-
but the sequence itself would still show the start value on the
2096-
subscriber. If the subscriber is used as a read-only database, then this
2097-
should typically not be a problem. If, however, some kind of switchover
2098-
or failover to the subscriber database is intended, then the sequences
2099-
would need to be updated to the latest values, either by copying the
2100-
current data from the publisher (perhaps
2101-
using <command>pg_dump</command>) or by determining a sufficiently high
2102-
value from the tables themselves.
2289+
Incremental sequence changes are not replicated. Although the data in
2290+
serial or identity columns backed by sequences will be replicated as part
2291+
of the table, the sequences themselves do not replicate ongoing changes.
2292+
On the subscriber, a sequence will retain the last value it synchronized
2293+
from the publisher. If the subscriber is used as a read-only database,
2294+
then this should typically not be a problem. If, however, some kind of
2295+
switchover or failover to the subscriber database is intended, then the
2296+
sequences would need to be updated to the latest values, either by
2297+
executing <link linkend="sql-altersubscription-params-refresh-sequences">
2298+
<command>ALTER SUBSCRIPTION ... REFRESH SEQUENCES</command></link>
2299+
or by copying the current data from the publisher (perhaps using
2300+
<command>pg_dump</command>) or by determining a sufficiently high value
2301+
from the tables themselves.
21032302
</para>
21042303
</listitem>
21052304

@@ -2423,8 +2622,8 @@ CONTEXT: processing remote data for replication origin "pg_16395" during "INSER
24232622
<para>
24242623
<link linkend="guc-max-logical-replication-workers"><varname>max_logical_replication_workers</varname></link>
24252624
must be set to at least the number of subscriptions (for leader apply
2426-
workers), plus some reserve for the table synchronization workers and
2427-
parallel apply workers.
2625+
workers), plus some reserve for the parallel apply workers, table synchronization workers, and a sequence
2626+
synchronization worker.
24282627
</para>
24292628

24302629
<para>
@@ -2437,8 +2636,9 @@ CONTEXT: processing remote data for replication origin "pg_16395" during "INSER
24372636

24382637
<para>
24392638
<link linkend="guc-max-sync-workers-per-subscription"><varname>max_sync_workers_per_subscription</varname></link>
2440-
controls the amount of parallelism of the initial data copy during the
2441-
subscription initialization or when new tables are added.
2639+
controls how many tables can be synchronized in parallel during
2640+
subscription initialization or when new tables are added. One additional
2641+
worker is also needed for sequence synchronization.
24422642
</para>
24432643

24442644
<para>

‎doc/src/sgml/monitoring.sgml‎

Lines changed: 12 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2045,8 +2045,9 @@ description | Waiting for a newly initialized WAL file to reach durable storage
20452045
</para>
20462046
<para>
20472047
Type of the subscription worker process. Possible types are
2048-
<literal>apply</literal>, <literal>parallel apply</literal>, and
2049-
<literal>table synchronization</literal>.
2048+
<literal>apply</literal>, <literal>parallel apply</literal>,
2049+
<literal>table synchronization</literal>, and
2050+
<literal>sequence synchronization</literal>.
20502051
</para></entry>
20512052
</row>
20522053

@@ -2192,6 +2193,15 @@ description | Waiting for a newly initialized WAL file to reach durable storage
21922193
</para></entry>
21932194
</row>
21942195

2196+
<row>
2197+
<entry role="catalog_table_entry"><para role="column_definition">
2198+
<structfield>sequence_sync_error_count</structfield> <type>bigint</type>
2199+
</para>
2200+
<para>
2201+
Number of times an error occurred during the sequence synchronization
2202+
</para></entry>
2203+
</row>
2204+
21952205
<row>
21962206
<entry role="catalog_table_entry"><para role="column_definition">
21972207
<structfield>sync_error_count</structfield> <type>bigint</type>

0 commit comments

Comments
 (0)