Skip to content

[feature] incremental restore #66

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
gsmolk opened this issue May 7, 2019 · 8 comments
Closed

[feature] incremental restore #66

gsmolk opened this issue May 7, 2019 · 8 comments
Labels
enhancement
Milestone
2.4.0

Comments

@gsmolk
Copy link
Contributor

gsmolk commented May 7, 2019 •
edited
Loading

It would be useful to scan PGDATA for changed blocks and replace only them instead of copying whole instance anew.
New option:

-I | --incremental-mode=checksum | lsn | none
@gsmolk gsmolk changed the title [feature] differential restore [feature] incremental restore Mar 5, 2020
@Extortioner01
Copy link

Добрый день!
В нашей конфигурации бэкап снимается с физической реплики. Теперь давайте представим, что реплика безнадёжно отстала - мониторинг не сработал, файлы wal, необходимые для того, чтобы реплика догнала мастер отсутствуют (понятно, что настроены всякие archive_command, мониторинг и прочие вещи, которые практически исключают такой исход, но тем не менее беда случилась и реплика отстала).
На реплике мы можем посмотреть START LSN и STOP LSN последнего доступного бэкапа.
Можем ли мы используя pg_probackup снять инкрементальный бэкап (видимо, в режиме DELTA) с мастера, начиная с известного нам LSN, после чего подложить его на реплику и каким-то образом догнать состояние реплики до мастера?

@uspen
Copy link

uspen commented Apr 7, 2020

Здравствуйте! Спасибо за отличный продукт!

Очень хотелось бы иметь возможность восстановить backup в непустой кластер. Наподобие --delta у pgbackrest

Задача утилиты резервного копирования не только качественно делать backup, но и, иногда более важно, быстро его восстановить

@gsmolk
Copy link
Contributor Author

gsmolk commented Apr 7, 2020

@uspen, @Extortioner01, cпасибо за ваш фидбэк.
Инкрементальное восстановление будет в версии 2.4.0.

@gsmolk gsmolk added this to the 2.4.0 milestone Apr 7, 2020
@gsmolk gsmolk mentioned this issue May 21, 2020
Closed
@gsmolk gsmolk pinned this issue May 25, 2020
gsmolk added a commit that referenced this issue May 25, 2020
@gsmolk
[Issue #66] Incremental restore, initial commit
b696669
gsmolk added a commit that referenced this issue May 27, 2020
@gsmolk
[Issue #66] add sanity checks for incremental restore
87252a8
@gsmolk
Copy link
Contributor Author

gsmolk commented May 28, 2020

First results:

simple  restore:
INFO: Backup files are restored. Transfered bytes: 7533MB, time elapsed: 58s
INFO: Restore overwriting ratio (less is better): 100% (7533MB/7501MB)

incremental:
INFO: Backup files are restored. Transfered bytes: 120kB, time elapsed: 22s
INFO: Restore overwriting ratio (less is better): 0% (120kB/7501MB)

incremental-lsn:
INFO: Backup files are restored. Transfered bytes: 16MB, time elapsed: 8s
INFO: Restore overwriting ratio (less is better): 0% (16MB/7501MB)

@gsmolk
Copy link
Contributor Author

gsmolk commented May 29, 2020

More results:

incremental
INFO: Backup files are restored. Transfered bytes: 1074MB, time elapsed: 28s
INFO: Restore incremental ratio (less is better): 14% (1074MB/7513MB)

incremental-lsn:
INFO: Backup files are restored. Transfered bytes: 1078MB, time elapsed: 12s
INFO: Restore incremental ratio (less is better): 14% (1078MB/7501MB)

gsmolk added a commit that referenced this issue May 30, 2020
@gsmolk
[Issue #66] Add lsn-based incremental restore
45995f0
gsmolk added a commit that referenced this issue Jun 5, 2020
@gsmolk
[Issue #66] support external directories, handle timeline mismatch
269ca37
gsmolk added a commit that referenced this issue Jun 10, 2020
@gsmolk
[Issue #66] added "-I | --incremenal-mode" option
7bdc682
gsmolk added a commit that referenced this issue Jun 10, 2020
@gsmolk
[Issue #66] update help
b2830e9
gsmolk added a commit that referenced this issue Jun 10, 2020
@gsmolk
[Issue #66] test fix
62db492
gsmolk added a commit that referenced this issue Jun 10, 2020
@gsmolk
[Issue #66] rename horizonLsn to shift_lsn
ff6e11e
gsmolk added a commit that referenced this issue Jun 11, 2020
@gsmolk
[Issue #66] fix segfault when restoring full backup, rewrite help
8a65fc8
gsmolk added a commit that referenced this issue Jun 11, 2020
@gsmolk
[Issue #66] new tests
9bc310a
gsmolk added a commit that referenced this issue Jun 11, 2020
@gsmolk
[Issue #66] nitpicking
d9b7dee
gsmolk added a commit that referenced this issue Jun 11, 2020
@gsmolk
[Issue #66] add switchover tests
2c39666
gsmolk added a commit that referenced this issue Jun 12, 2020
@gsmolk
[Issue #66] update help
da082fb
gsmolk added a commit that referenced this issue Jun 12, 2020
@gsmolk
[Issue #66] fix help
41d4013
gsmolk added a commit that referenced this issue Jun 12, 2020
@gsmolk
[Issue #66] Documentation
496605b
gsmolk added a commit that referenced this issue Jun 13, 2020
@gsmolk
[Issue #66] for LSN mode data_checksums must be enabled
7506825
gsmolk added a commit that referenced this issue Jun 13, 2020
@gsmolk
[Issue #66] tests for hint bits
2cb4326
gsmolk added a commit that referenced this issue Jun 13, 2020
@gsmolk
[Issue #66] documentation update
bfdabf3
@gsmolk
Copy link
Contributor Author

gsmolk commented Jun 18, 2020

Merged.

@gsmolk gsmolk closed this as completed Jun 18, 2020
@gsmolk gsmolk mentioned this issue Jun 18, 2020
Closed
@gsmolk gsmolk unpinned this issue Jun 18, 2020
gsmolk added a commit that referenced this issue Jun 22, 2020
@gsmolk
[Issue #66] Incremental restore
abb0ae5
gsmolk added a commit that referenced this issue Jun 23, 2020
@gsmolk
[Issue #66] Correctly handle zero-sized nonedata files during increme…
f5e7e33 
…ntal restore
@gsmolk
Copy link
Contributor Author

gsmolk commented Jun 24, 2020 •
edited
Loading

rsync vs incremental restore, 15 GB:

rsync  9m45s
vs
pbk 57s. Transfered bytes: 962MB, incremental ratio (less is better): 6% (962MB/15GB)

@gsmolk gsmolk mentioned this issue Apr 27, 2021
Closed
@gsmolk
Copy link
Contributor Author

gsmolk commented Apr 27, 2021

@Extortioner01

Можем ли мы используя pg_probackup снять инкрементальный бэкап (видимо, в режиме DELTA) с мастера, начиная с известного нам LSN, после чего подложить его на реплику и каким-то образом догнать состояние реплики до мастера?

Эта фича пилится в рамках #277

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement
Projects
None yet
Milestone
2.4.0
Development

No branches or pull requests
3 participants
@uspen @gsmolk @Extortioner01