From b072922592a35864a5887e7e9820fe02224109db Mon Sep 17 00:00:00 2001 From: "release-controller[bot]" Date: Fri, 19 Apr 2024 04:48:09 +0000 Subject: [PATCH 01/47] 3.12.2 release --- docs/getting-started.md | 2 +- docs/requirements.md | 18 +++++++++++------- 2 files changed, 12 insertions(+), 8 deletions(-) diff --git a/docs/getting-started.md b/docs/getting-started.md index ac85e5073..02f2cbe9b 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -1,6 +1,6 @@ # Getting started - 1. [Download the latest version of backup-utils][1] and extract the repository using `tar`: + 1. [Download the latest release version][1] and extract the repository using `tar`: `tar -xzvf /path/to/github-backup-utils-vMAJOR.MINOR.PATCH.tar.gz` diff --git a/docs/requirements.md b/docs/requirements.md index 6fa54e567..b2a3d282a 100644 --- a/docs/requirements.md +++ b/docs/requirements.md @@ -34,6 +34,10 @@ However, if your rsync package has backported the CVE fix without backporting th Option #3 is required if your operating system's package manager does not have access to rsync v3.2.5 or later (e.g. Ubuntu Focal). +Please note that some operating systems have their own versioning scheme for packages (including `rsync`). +If your backup host is using one of these operating systems, you will not be able to rely on a version check to determine whether you are +affected by the `rsync` performance degredation described above. + ## Storage requirements Storage requirements vary based on current Git repository disk usage and growth @@ -65,23 +69,23 @@ Please avoid using an NFS mount for the data directory (where backup data is sto Starting with Backup Utilities v2.13.0, version support is inline with that of the [GitHub Enterprise Server upgrade requirements][8] and as such, support is limited to three versions of GitHub Enterprise Server: the version that corresponds with the version -of Backup Utilities, and the two versions prior to it. +of Backup Utilities, and the two releases prior to it. For example, Backup Utilities v2.13.0 can be used to backup and restore all patch -versions from 2.11.0 to the latest patch version of GitHub Enterprise Server 2.13. +releases from 2.11.0 to the latest patch release of GitHub Enterprise Server 2.13. Backup Utilities v2.14.0 will be released when GitHub Enterprise Server 2.14.0 is released -and will then be used to backup all versions of GitHub Enterprise Server from 2.12.0 -to the latest patch version of GitHub Enterprise Server 2.14. +and will then be used to backup all releases of GitHub Enterprise Server from 2.12.0 +to the latest patch release of GitHub Enterprise Server 2.14. Backup Utilities v2.11.4 and earlier offer support for GitHub Enterprise Server 2.10 -and earlier versions up to GitHub Enterprise Server 2.2.0. Backup Utilities v2.11.0 and earlier +and earlier releases up to GitHub Enterprise Server 2.2.0. Backup Utilities v2.11.0 and earlier offer support for GitHub Enterprise Server 2.1.0 and earlier. -**Note**: You can restore a snapshot that's at most two feature versions behind +**Note**: You can restore a snapshot that's at most two feature releases behind the restore target's version of GitHub Enterprise Server. For example, to restore a snapshot of GitHub Enterprise Server 2.11, the target GitHub Enterprise Server appliance must be running GitHub Enterprise Server 2.12.x or 2.13.x. You can't restore a snapshot from -2.10 to 2.13, because that's three versions ahead. +2.10 to 2.13, because that's three releases ahead. **Note**: You _cannot_ restore a backup created from a newer version of GitHub Enterprise Server to an older version. For example, an attempt to restore a snapshot of GitHub Enterprise Server 2.21 to a GitHub Enterprise Server 2.20 environment will fail with an error of `Error: Snapshot can not be restored to an older release of GitHub Enterprise Server.`. From 375e01d33700eec25cdda0c6ed5756a26e4c39bf Mon Sep 17 00:00:00 2001 From: "release-controller[bot]" Date: Wed, 8 May 2024 23:59:31 +0000 Subject: [PATCH 02/47] 3.12.3 release From 653dd1e86de1d45f9aa79247b670697fad1f8a49 Mon Sep 17 00:00:00 2001 From: "release-controller[bot]" Date: Fri, 17 May 2024 12:43:04 +0000 Subject: [PATCH 03/47] 3.13.0 release --- docs/getting-started.md | 2 +- docs/requirements.md | 14 +++++++------- docs/scheduling-backups.md | 20 +++++++++++++++----- 3 files changed, 23 insertions(+), 13 deletions(-) diff --git a/docs/getting-started.md b/docs/getting-started.md index 02f2cbe9b..ac85e5073 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -1,6 +1,6 @@ # Getting started - 1. [Download the latest release version][1] and extract the repository using `tar`: + 1. [Download the latest version of backup-utils][1] and extract the repository using `tar`: `tar -xzvf /path/to/github-backup-utils-vMAJOR.MINOR.PATCH.tar.gz` diff --git a/docs/requirements.md b/docs/requirements.md index b2a3d282a..c87a0775a 100644 --- a/docs/requirements.md +++ b/docs/requirements.md @@ -69,23 +69,23 @@ Please avoid using an NFS mount for the data directory (where backup data is sto Starting with Backup Utilities v2.13.0, version support is inline with that of the [GitHub Enterprise Server upgrade requirements][8] and as such, support is limited to three versions of GitHub Enterprise Server: the version that corresponds with the version -of Backup Utilities, and the two releases prior to it. +of Backup Utilities, and the two versions prior to it. For example, Backup Utilities v2.13.0 can be used to backup and restore all patch -releases from 2.11.0 to the latest patch release of GitHub Enterprise Server 2.13. +versions from 2.11.0 to the latest patch version of GitHub Enterprise Server 2.13. Backup Utilities v2.14.0 will be released when GitHub Enterprise Server 2.14.0 is released -and will then be used to backup all releases of GitHub Enterprise Server from 2.12.0 -to the latest patch release of GitHub Enterprise Server 2.14. +and will then be used to backup all versions of GitHub Enterprise Server from 2.12.0 +to the latest patch version of GitHub Enterprise Server 2.14. Backup Utilities v2.11.4 and earlier offer support for GitHub Enterprise Server 2.10 -and earlier releases up to GitHub Enterprise Server 2.2.0. Backup Utilities v2.11.0 and earlier +and earlier versions up to GitHub Enterprise Server 2.2.0. Backup Utilities v2.11.0 and earlier offer support for GitHub Enterprise Server 2.1.0 and earlier. -**Note**: You can restore a snapshot that's at most two feature releases behind +**Note**: You can restore a snapshot that's at most two feature versions behind the restore target's version of GitHub Enterprise Server. For example, to restore a snapshot of GitHub Enterprise Server 2.11, the target GitHub Enterprise Server appliance must be running GitHub Enterprise Server 2.12.x or 2.13.x. You can't restore a snapshot from -2.10 to 2.13, because that's three releases ahead. +2.10 to 2.13, because that's three versions ahead. **Note**: You _cannot_ restore a backup created from a newer version of GitHub Enterprise Server to an older version. For example, an attempt to restore a snapshot of GitHub Enterprise Server 2.21 to a GitHub Enterprise Server 2.20 environment will fail with an error of `Error: Snapshot can not be restored to an older release of GitHub Enterprise Server.`. diff --git a/docs/scheduling-backups.md b/docs/scheduling-backups.md index 0e00a5d67..3a7b0d12c 100644 --- a/docs/scheduling-backups.md +++ b/docs/scheduling-backups.md @@ -3,7 +3,16 @@ Regular backups should be scheduled using `cron(8)` or similar command scheduling service on the backup host. The backup frequency will dictate the worst case [recovery point objective (RPO)][1] in your backup plan. We recommend -hourly backups at the least. +hourly backups as a starting point. + +It's important to consider the duration of each backup operation on the +GitHub Enterprise Server (GHES) appliance. Backups of large datasets or +over slow network links can take more than an hour. Additionally, +maintenance queues are paused during a portion of a backup runs. +We recommend scheduling backups to allow sufficient time for jobs +waiting in maintenance queues to process between backup runs + +Only one backup may be in progress at a time. ## Example scheduling of backups @@ -19,7 +28,8 @@ storage. To schedule hourly backup snapshots with verbose informational output written to a log file and errors generating an email: -``` + +```shell MAILTO=admin@example.com 0 * * * * /opt/backup-utils/bin/ghe-backup -v 1>>/opt/backup-utils/backup.log 2>&1 @@ -27,13 +37,13 @@ MAILTO=admin@example.com To schedule nightly backup snapshots instead, use: -``` +```shell MAILTO=admin@example.com 0 0 * * * /opt/backup-utils/bin/ghe-backup -v 1>>/opt/backup-utils/backup.log 2>&1 ``` -## Example snapshot pruning +## Example snapshot pruning By default all expired and incomplete snapshots are deleted at the end of the main backup process `ghe-backup`. If pruning these snapshots takes a long time you can @@ -44,7 +54,7 @@ If this option is enabled you will need to schedule the pruning script `ghe-prun To schedule daily snapshot pruning, use: -``` +```shell MAILTO=admin@example.com 0 3 * * * /opt/backup-utils/share/github-backup-utils/ghe-prune-snapshots 1>>/opt/backup-utils/prune-snapshots.log 2>&1 From dc623bf03117c08cd34e5fe275765a648382b779 Mon Sep 17 00:00:00 2001 From: Hao Jiang <45571951+jianghao0718@users.noreply.github.com> Date: Tue, 28 May 2024 10:02:23 -0600 Subject: [PATCH 04/47] Add backup.config-example --- backup.config-example | 132 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 132 insertions(+) create mode 100644 backup.config-example diff --git a/backup.config-example b/backup.config-example new file mode 100644 index 000000000..89a0917a3 --- /dev/null +++ b/backup.config-example @@ -0,0 +1,132 @@ +# GitHub Enterprise Server backup configuration file + +# The hostname of the GitHub Enterprise Server appliance to back up. The host +# must be reachable via SSH from the backup host. +GHE_HOSTNAME="github.example.com" + +# Path to where backup data is stored. By default this is the "data" +# directory next to this file but can be set to an absolute path +# elsewhere for backing up to a separate partition / mount point. +GHE_DATA_DIR="data" + +# The number of backup snapshots to retain. Old snapshots are pruned after each +# successful ghe-backup run. This option should be tuned based on the frequency +# of scheduled backup runs. If backups are scheduled hourly, snapshots will be +# available for the past N hours; if backups are scheduled daily, snapshots will +# be available for the past N days ... +GHE_NUM_SNAPSHOTS=10 + +# Pruning snapshots can be scheduled outside of the backup process. +# If set to 'yes', snapshots will not be pruned by ghe-backup. +# Instead, ghe-pruning-snapshots will need to be invoked separately via cron +#GHE_PRUNING_SCHEDULED=yes + +# If --incremental is used to generate incremental MySQL backups with ghe-backup, +# then you need to specify how many cycles of full and incremental backups will be +# performed before the next full backup is created. +# For example, if `GHE_INCREMENTAL_BACKUP_MAX` is set to 14, backup-utils will +# run 1 full backup and then 13 incremental backups before performing another full backup on the next cycle. +#GHE_INCREMENTAL_MAX_BACKUPS=14 + +# If GHE_ROUTE_VERIFICATION is set to true then ghe-repository-backup and +# ghe-storage-backup will issue a warning if the repositories and objects in +# the backup do not match the pre-backup inventory of routes. +#GHE_ROUTE_VERIFICATION=false + +# If GHE_SKIP_CHECKS is set to true (or if --skip-checks is used with ghe-backup) then ghe-host-check +# disk space validation and software version checks on the backup-host will be disabled. +#GHE_SKIP_CHECKS=false + +# Cluster filesystem to check if it's writable as part of ghe-host-check +# By default it is /data/user/tmp but can be updated if needed +#GHE_FILE_SYSTEM_WRITE_CHECK="/data/user/tmp" + +# The hostname of the GitHub appliance to restore. If you've set up a separate +# GitHub appliance to act as a standby for recovery, specify its IP or hostname +# here. The host to restore to may also be specified directly when running +# ghe-restore so use of this variable isn't strictly required. +# +#GHE_RESTORE_HOST="github-standby.example.com" + +# If set to 'yes', ghe-restore will omit the restore of audit logs. +# +#GHE_RESTORE_SKIP_AUDIT_LOGS=no + +# When verbose output is enabled with `-v`, it's written to stdout by default. If +# you'd prefer it to be written to a separate file, set this option. +# +#GHE_VERBOSE_LOG="/var/log/backup-verbose.log" + +# Any extra options passed to the SSH command. +# In a single instance environment, nothing is required by default. +# In a clustering environment, "-i abs-path-to-ssh-private-key" is required. +# +#GHE_EXTRA_SSH_OPTS="" + +# Any extra options passed to the rsync command. Nothing required by default. +# +#GHE_EXTRA_RSYNC_OPTS="" + +# If set to 'yes', rsync will be set to use compression during backups and restores transfers. Defaults to 'no'. +# +#GHE_RSYNC_COMPRESSION_ENABLED=yes + +# If enabled and set to 'no', rsync warning message during backups will be suppressed. +#RSYNC_WARNING=no + + +# If set to 'yes', logging output will be colorized. +# +#OUTPUT_COLOR=no + +# If set to 'no', GHE_DATA_DIR will not be created automatically +# and restore/backup will exit 8 +# +#GHE_CREATE_DATA_DIR=yes + +# If set to 'yes', git fsck will run on the repositories +# and print some additional info. +# +# WARNING: do not enable this, only useful for debugging/development +#GHE_BACKUP_FSCK=no + +# Cadence of MSSQL backups +# ,, all in minutes +# e.g. +# - Full backup every week (10080 minutes) +# - Differential backup every day (1440 minutes) +# - Transactionlog backup every 15 minutes +# +#GHE_MSSQL_BACKUP_CADENCE=10080,1440,15 + +# If set to 'yes', ghe-backup jobs will run in parallel. Defaults to 'no'. +# +#GHE_PARALLEL_ENABLED=yes + +# Sets the maximum number of jobs to run in parallel. Defaults to the number +# of available processing units on the machine. +# +#GHE_PARALLEL_MAX_JOBS=2 + +# Sets the maximum number of rsync jobs to run in parallel. Defaults to the +# configured GHE_PARALLEL_MAX_JOBS, or the number of available processing +# units on the machine. +# +# GHE_PARALLEL_RSYNC_MAX_JOBS=3 + +# When jobs are running in parallel wait as needed to avoid starting new jobs +# when the system's load average is not below the specified percentage. Defaults to +# unrestricted. +# +#GHE_PARALLEL_MAX_LOAD=50 + +# When running an external mysql database, run this script to trigger a MySQL backup +# rather than attempting to backup via backup-utils directly. +#EXTERNAL_DATABASE_BACKUP_SCRIPT="/bin/false" + +# When running an external mysql database, run this script to trigger a MySQL restore +# rather than attempting to backup via backup-utils directly. +#EXTERNAL_DATABASE_RESTORE_SCRIPT="/bin/false" + +# If set to 'yes', Pages data will be included in backup and restore. Defaults to 'yes' +#GHE_BACKUP_PAGES=no From 7861ad9752da1542028273fe02cbb599fc234ff0 Mon Sep 17 00:00:00 2001 From: Hao Jiang Date: Tue, 28 May 2024 10:15:32 -0600 Subject: [PATCH 05/47] Revert "Add backup.config-example" This reverts commit dc623bf03117c08cd34e5fe275765a648382b779. --- backup.config-example | 132 ------------------------------------------ 1 file changed, 132 deletions(-) delete mode 100644 backup.config-example diff --git a/backup.config-example b/backup.config-example deleted file mode 100644 index 89a0917a3..000000000 --- a/backup.config-example +++ /dev/null @@ -1,132 +0,0 @@ -# GitHub Enterprise Server backup configuration file - -# The hostname of the GitHub Enterprise Server appliance to back up. The host -# must be reachable via SSH from the backup host. -GHE_HOSTNAME="github.example.com" - -# Path to where backup data is stored. By default this is the "data" -# directory next to this file but can be set to an absolute path -# elsewhere for backing up to a separate partition / mount point. -GHE_DATA_DIR="data" - -# The number of backup snapshots to retain. Old snapshots are pruned after each -# successful ghe-backup run. This option should be tuned based on the frequency -# of scheduled backup runs. If backups are scheduled hourly, snapshots will be -# available for the past N hours; if backups are scheduled daily, snapshots will -# be available for the past N days ... -GHE_NUM_SNAPSHOTS=10 - -# Pruning snapshots can be scheduled outside of the backup process. -# If set to 'yes', snapshots will not be pruned by ghe-backup. -# Instead, ghe-pruning-snapshots will need to be invoked separately via cron -#GHE_PRUNING_SCHEDULED=yes - -# If --incremental is used to generate incremental MySQL backups with ghe-backup, -# then you need to specify how many cycles of full and incremental backups will be -# performed before the next full backup is created. -# For example, if `GHE_INCREMENTAL_BACKUP_MAX` is set to 14, backup-utils will -# run 1 full backup and then 13 incremental backups before performing another full backup on the next cycle. -#GHE_INCREMENTAL_MAX_BACKUPS=14 - -# If GHE_ROUTE_VERIFICATION is set to true then ghe-repository-backup and -# ghe-storage-backup will issue a warning if the repositories and objects in -# the backup do not match the pre-backup inventory of routes. -#GHE_ROUTE_VERIFICATION=false - -# If GHE_SKIP_CHECKS is set to true (or if --skip-checks is used with ghe-backup) then ghe-host-check -# disk space validation and software version checks on the backup-host will be disabled. -#GHE_SKIP_CHECKS=false - -# Cluster filesystem to check if it's writable as part of ghe-host-check -# By default it is /data/user/tmp but can be updated if needed -#GHE_FILE_SYSTEM_WRITE_CHECK="/data/user/tmp" - -# The hostname of the GitHub appliance to restore. If you've set up a separate -# GitHub appliance to act as a standby for recovery, specify its IP or hostname -# here. The host to restore to may also be specified directly when running -# ghe-restore so use of this variable isn't strictly required. -# -#GHE_RESTORE_HOST="github-standby.example.com" - -# If set to 'yes', ghe-restore will omit the restore of audit logs. -# -#GHE_RESTORE_SKIP_AUDIT_LOGS=no - -# When verbose output is enabled with `-v`, it's written to stdout by default. If -# you'd prefer it to be written to a separate file, set this option. -# -#GHE_VERBOSE_LOG="/var/log/backup-verbose.log" - -# Any extra options passed to the SSH command. -# In a single instance environment, nothing is required by default. -# In a clustering environment, "-i abs-path-to-ssh-private-key" is required. -# -#GHE_EXTRA_SSH_OPTS="" - -# Any extra options passed to the rsync command. Nothing required by default. -# -#GHE_EXTRA_RSYNC_OPTS="" - -# If set to 'yes', rsync will be set to use compression during backups and restores transfers. Defaults to 'no'. -# -#GHE_RSYNC_COMPRESSION_ENABLED=yes - -# If enabled and set to 'no', rsync warning message during backups will be suppressed. -#RSYNC_WARNING=no - - -# If set to 'yes', logging output will be colorized. -# -#OUTPUT_COLOR=no - -# If set to 'no', GHE_DATA_DIR will not be created automatically -# and restore/backup will exit 8 -# -#GHE_CREATE_DATA_DIR=yes - -# If set to 'yes', git fsck will run on the repositories -# and print some additional info. -# -# WARNING: do not enable this, only useful for debugging/development -#GHE_BACKUP_FSCK=no - -# Cadence of MSSQL backups -# ,, all in minutes -# e.g. -# - Full backup every week (10080 minutes) -# - Differential backup every day (1440 minutes) -# - Transactionlog backup every 15 minutes -# -#GHE_MSSQL_BACKUP_CADENCE=10080,1440,15 - -# If set to 'yes', ghe-backup jobs will run in parallel. Defaults to 'no'. -# -#GHE_PARALLEL_ENABLED=yes - -# Sets the maximum number of jobs to run in parallel. Defaults to the number -# of available processing units on the machine. -# -#GHE_PARALLEL_MAX_JOBS=2 - -# Sets the maximum number of rsync jobs to run in parallel. Defaults to the -# configured GHE_PARALLEL_MAX_JOBS, or the number of available processing -# units on the machine. -# -# GHE_PARALLEL_RSYNC_MAX_JOBS=3 - -# When jobs are running in parallel wait as needed to avoid starting new jobs -# when the system's load average is not below the specified percentage. Defaults to -# unrestricted. -# -#GHE_PARALLEL_MAX_LOAD=50 - -# When running an external mysql database, run this script to trigger a MySQL backup -# rather than attempting to backup via backup-utils directly. -#EXTERNAL_DATABASE_BACKUP_SCRIPT="/bin/false" - -# When running an external mysql database, run this script to trigger a MySQL restore -# rather than attempting to backup via backup-utils directly. -#EXTERNAL_DATABASE_RESTORE_SCRIPT="/bin/false" - -# If set to 'yes', Pages data will be included in backup and restore. Defaults to 'yes' -#GHE_BACKUP_PAGES=no From 86250466764ce38ab7cee37d055d62b59adb0820 Mon Sep 17 00:00:00 2001 From: "release-controller[bot]" Date: Tue, 18 Jun 2024 17:04:12 +0000 Subject: [PATCH 06/47] 3.13.0 release From 33868f0893ee4e29f6d976016e7151fd06d4642e Mon Sep 17 00:00:00 2001 From: amildahl Date: Tue, 13 Aug 2024 10:43:35 +0000 Subject: [PATCH 07/47] 3.13.99 release From d0653a0f125b8603d169833d46c3b9b1a426dd19 Mon Sep 17 00:00:00 2001 From: amildahl Date: Tue, 13 Aug 2024 12:01:44 +0000 Subject: [PATCH 08/47] 3.12.4 release --- docs/getting-started.md | 2 +- docs/requirements.md | 14 +++++++------- docs/scheduling-backups.md | 20 +++++--------------- 3 files changed, 13 insertions(+), 23 deletions(-) diff --git a/docs/getting-started.md b/docs/getting-started.md index ac85e5073..02f2cbe9b 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -1,6 +1,6 @@ # Getting started - 1. [Download the latest version of backup-utils][1] and extract the repository using `tar`: + 1. [Download the latest release version][1] and extract the repository using `tar`: `tar -xzvf /path/to/github-backup-utils-vMAJOR.MINOR.PATCH.tar.gz` diff --git a/docs/requirements.md b/docs/requirements.md index c87a0775a..b2a3d282a 100644 --- a/docs/requirements.md +++ b/docs/requirements.md @@ -69,23 +69,23 @@ Please avoid using an NFS mount for the data directory (where backup data is sto Starting with Backup Utilities v2.13.0, version support is inline with that of the [GitHub Enterprise Server upgrade requirements][8] and as such, support is limited to three versions of GitHub Enterprise Server: the version that corresponds with the version -of Backup Utilities, and the two versions prior to it. +of Backup Utilities, and the two releases prior to it. For example, Backup Utilities v2.13.0 can be used to backup and restore all patch -versions from 2.11.0 to the latest patch version of GitHub Enterprise Server 2.13. +releases from 2.11.0 to the latest patch release of GitHub Enterprise Server 2.13. Backup Utilities v2.14.0 will be released when GitHub Enterprise Server 2.14.0 is released -and will then be used to backup all versions of GitHub Enterprise Server from 2.12.0 -to the latest patch version of GitHub Enterprise Server 2.14. +and will then be used to backup all releases of GitHub Enterprise Server from 2.12.0 +to the latest patch release of GitHub Enterprise Server 2.14. Backup Utilities v2.11.4 and earlier offer support for GitHub Enterprise Server 2.10 -and earlier versions up to GitHub Enterprise Server 2.2.0. Backup Utilities v2.11.0 and earlier +and earlier releases up to GitHub Enterprise Server 2.2.0. Backup Utilities v2.11.0 and earlier offer support for GitHub Enterprise Server 2.1.0 and earlier. -**Note**: You can restore a snapshot that's at most two feature versions behind +**Note**: You can restore a snapshot that's at most two feature releases behind the restore target's version of GitHub Enterprise Server. For example, to restore a snapshot of GitHub Enterprise Server 2.11, the target GitHub Enterprise Server appliance must be running GitHub Enterprise Server 2.12.x or 2.13.x. You can't restore a snapshot from -2.10 to 2.13, because that's three versions ahead. +2.10 to 2.13, because that's three releases ahead. **Note**: You _cannot_ restore a backup created from a newer version of GitHub Enterprise Server to an older version. For example, an attempt to restore a snapshot of GitHub Enterprise Server 2.21 to a GitHub Enterprise Server 2.20 environment will fail with an error of `Error: Snapshot can not be restored to an older release of GitHub Enterprise Server.`. diff --git a/docs/scheduling-backups.md b/docs/scheduling-backups.md index 3a7b0d12c..0e00a5d67 100644 --- a/docs/scheduling-backups.md +++ b/docs/scheduling-backups.md @@ -3,16 +3,7 @@ Regular backups should be scheduled using `cron(8)` or similar command scheduling service on the backup host. The backup frequency will dictate the worst case [recovery point objective (RPO)][1] in your backup plan. We recommend -hourly backups as a starting point. - -It's important to consider the duration of each backup operation on the -GitHub Enterprise Server (GHES) appliance. Backups of large datasets or -over slow network links can take more than an hour. Additionally, -maintenance queues are paused during a portion of a backup runs. -We recommend scheduling backups to allow sufficient time for jobs -waiting in maintenance queues to process between backup runs - -Only one backup may be in progress at a time. +hourly backups at the least. ## Example scheduling of backups @@ -28,8 +19,7 @@ storage. To schedule hourly backup snapshots with verbose informational output written to a log file and errors generating an email: - -```shell +``` MAILTO=admin@example.com 0 * * * * /opt/backup-utils/bin/ghe-backup -v 1>>/opt/backup-utils/backup.log 2>&1 @@ -37,13 +27,13 @@ MAILTO=admin@example.com To schedule nightly backup snapshots instead, use: -```shell +``` MAILTO=admin@example.com 0 0 * * * /opt/backup-utils/bin/ghe-backup -v 1>>/opt/backup-utils/backup.log 2>&1 ``` -## Example snapshot pruning +## Example snapshot pruning By default all expired and incomplete snapshots are deleted at the end of the main backup process `ghe-backup`. If pruning these snapshots takes a long time you can @@ -54,7 +44,7 @@ If this option is enabled you will need to schedule the pruning script `ghe-prun To schedule daily snapshot pruning, use: -```shell +``` MAILTO=admin@example.com 0 3 * * * /opt/backup-utils/share/github-backup-utils/ghe-prune-snapshots 1>>/opt/backup-utils/prune-snapshots.log 2>&1 From 70509cf3d4cf50d9f4e17b0873af6b8b3cc7aa72 Mon Sep 17 00:00:00 2001 From: amildahl Date: Tue, 13 Aug 2024 12:02:32 +0000 Subject: [PATCH 09/47] 3.13.1 release --- docs/getting-started.md | 2 +- docs/requirements.md | 14 +++++++------- docs/scheduling-backups.md | 20 +++++++++++++++----- 3 files changed, 23 insertions(+), 13 deletions(-) diff --git a/docs/getting-started.md b/docs/getting-started.md index 02f2cbe9b..ac85e5073 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -1,6 +1,6 @@ # Getting started - 1. [Download the latest release version][1] and extract the repository using `tar`: + 1. [Download the latest version of backup-utils][1] and extract the repository using `tar`: `tar -xzvf /path/to/github-backup-utils-vMAJOR.MINOR.PATCH.tar.gz` diff --git a/docs/requirements.md b/docs/requirements.md index b2a3d282a..c87a0775a 100644 --- a/docs/requirements.md +++ b/docs/requirements.md @@ -69,23 +69,23 @@ Please avoid using an NFS mount for the data directory (where backup data is sto Starting with Backup Utilities v2.13.0, version support is inline with that of the [GitHub Enterprise Server upgrade requirements][8] and as such, support is limited to three versions of GitHub Enterprise Server: the version that corresponds with the version -of Backup Utilities, and the two releases prior to it. +of Backup Utilities, and the two versions prior to it. For example, Backup Utilities v2.13.0 can be used to backup and restore all patch -releases from 2.11.0 to the latest patch release of GitHub Enterprise Server 2.13. +versions from 2.11.0 to the latest patch version of GitHub Enterprise Server 2.13. Backup Utilities v2.14.0 will be released when GitHub Enterprise Server 2.14.0 is released -and will then be used to backup all releases of GitHub Enterprise Server from 2.12.0 -to the latest patch release of GitHub Enterprise Server 2.14. +and will then be used to backup all versions of GitHub Enterprise Server from 2.12.0 +to the latest patch version of GitHub Enterprise Server 2.14. Backup Utilities v2.11.4 and earlier offer support for GitHub Enterprise Server 2.10 -and earlier releases up to GitHub Enterprise Server 2.2.0. Backup Utilities v2.11.0 and earlier +and earlier versions up to GitHub Enterprise Server 2.2.0. Backup Utilities v2.11.0 and earlier offer support for GitHub Enterprise Server 2.1.0 and earlier. -**Note**: You can restore a snapshot that's at most two feature releases behind +**Note**: You can restore a snapshot that's at most two feature versions behind the restore target's version of GitHub Enterprise Server. For example, to restore a snapshot of GitHub Enterprise Server 2.11, the target GitHub Enterprise Server appliance must be running GitHub Enterprise Server 2.12.x or 2.13.x. You can't restore a snapshot from -2.10 to 2.13, because that's three releases ahead. +2.10 to 2.13, because that's three versions ahead. **Note**: You _cannot_ restore a backup created from a newer version of GitHub Enterprise Server to an older version. For example, an attempt to restore a snapshot of GitHub Enterprise Server 2.21 to a GitHub Enterprise Server 2.20 environment will fail with an error of `Error: Snapshot can not be restored to an older release of GitHub Enterprise Server.`. diff --git a/docs/scheduling-backups.md b/docs/scheduling-backups.md index 0e00a5d67..3a7b0d12c 100644 --- a/docs/scheduling-backups.md +++ b/docs/scheduling-backups.md @@ -3,7 +3,16 @@ Regular backups should be scheduled using `cron(8)` or similar command scheduling service on the backup host. The backup frequency will dictate the worst case [recovery point objective (RPO)][1] in your backup plan. We recommend -hourly backups at the least. +hourly backups as a starting point. + +It's important to consider the duration of each backup operation on the +GitHub Enterprise Server (GHES) appliance. Backups of large datasets or +over slow network links can take more than an hour. Additionally, +maintenance queues are paused during a portion of a backup runs. +We recommend scheduling backups to allow sufficient time for jobs +waiting in maintenance queues to process between backup runs + +Only one backup may be in progress at a time. ## Example scheduling of backups @@ -19,7 +28,8 @@ storage. To schedule hourly backup snapshots with verbose informational output written to a log file and errors generating an email: -``` + +```shell MAILTO=admin@example.com 0 * * * * /opt/backup-utils/bin/ghe-backup -v 1>>/opt/backup-utils/backup.log 2>&1 @@ -27,13 +37,13 @@ MAILTO=admin@example.com To schedule nightly backup snapshots instead, use: -``` +```shell MAILTO=admin@example.com 0 0 * * * /opt/backup-utils/bin/ghe-backup -v 1>>/opt/backup-utils/backup.log 2>&1 ``` -## Example snapshot pruning +## Example snapshot pruning By default all expired and incomplete snapshots are deleted at the end of the main backup process `ghe-backup`. If pruning these snapshots takes a long time you can @@ -44,7 +54,7 @@ If this option is enabled you will need to schedule the pruning script `ghe-prun To schedule daily snapshot pruning, use: -``` +```shell MAILTO=admin@example.com 0 3 * * * /opt/backup-utils/share/github-backup-utils/ghe-prune-snapshots 1>>/opt/backup-utils/prune-snapshots.log 2>&1 From 147c003ea7f52114c50c9021d328339544b7991a Mon Sep 17 00:00:00 2001 From: "release-controller[bot]" Date: Wed, 14 Aug 2024 21:18:04 +0000 Subject: [PATCH 10/47] 3.14.0 release From 36819b417ce8b9f2d4428368c076531c2f360adf Mon Sep 17 00:00:00 2001 From: "release-controller[bot]" Date: Tue, 24 Sep 2024 16:37:01 +0000 Subject: [PATCH 11/47] 3.12.5 release --- docs/getting-started.md | 2 +- docs/requirements.md | 14 +++++++------- docs/scheduling-backups.md | 20 +++++--------------- 3 files changed, 13 insertions(+), 23 deletions(-) diff --git a/docs/getting-started.md b/docs/getting-started.md index ac85e5073..02f2cbe9b 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -1,6 +1,6 @@ # Getting started - 1. [Download the latest version of backup-utils][1] and extract the repository using `tar`: + 1. [Download the latest release version][1] and extract the repository using `tar`: `tar -xzvf /path/to/github-backup-utils-vMAJOR.MINOR.PATCH.tar.gz` diff --git a/docs/requirements.md b/docs/requirements.md index c87a0775a..b2a3d282a 100644 --- a/docs/requirements.md +++ b/docs/requirements.md @@ -69,23 +69,23 @@ Please avoid using an NFS mount for the data directory (where backup data is sto Starting with Backup Utilities v2.13.0, version support is inline with that of the [GitHub Enterprise Server upgrade requirements][8] and as such, support is limited to three versions of GitHub Enterprise Server: the version that corresponds with the version -of Backup Utilities, and the two versions prior to it. +of Backup Utilities, and the two releases prior to it. For example, Backup Utilities v2.13.0 can be used to backup and restore all patch -versions from 2.11.0 to the latest patch version of GitHub Enterprise Server 2.13. +releases from 2.11.0 to the latest patch release of GitHub Enterprise Server 2.13. Backup Utilities v2.14.0 will be released when GitHub Enterprise Server 2.14.0 is released -and will then be used to backup all versions of GitHub Enterprise Server from 2.12.0 -to the latest patch version of GitHub Enterprise Server 2.14. +and will then be used to backup all releases of GitHub Enterprise Server from 2.12.0 +to the latest patch release of GitHub Enterprise Server 2.14. Backup Utilities v2.11.4 and earlier offer support for GitHub Enterprise Server 2.10 -and earlier versions up to GitHub Enterprise Server 2.2.0. Backup Utilities v2.11.0 and earlier +and earlier releases up to GitHub Enterprise Server 2.2.0. Backup Utilities v2.11.0 and earlier offer support for GitHub Enterprise Server 2.1.0 and earlier. -**Note**: You can restore a snapshot that's at most two feature versions behind +**Note**: You can restore a snapshot that's at most two feature releases behind the restore target's version of GitHub Enterprise Server. For example, to restore a snapshot of GitHub Enterprise Server 2.11, the target GitHub Enterprise Server appliance must be running GitHub Enterprise Server 2.12.x or 2.13.x. You can't restore a snapshot from -2.10 to 2.13, because that's three versions ahead. +2.10 to 2.13, because that's three releases ahead. **Note**: You _cannot_ restore a backup created from a newer version of GitHub Enterprise Server to an older version. For example, an attempt to restore a snapshot of GitHub Enterprise Server 2.21 to a GitHub Enterprise Server 2.20 environment will fail with an error of `Error: Snapshot can not be restored to an older release of GitHub Enterprise Server.`. diff --git a/docs/scheduling-backups.md b/docs/scheduling-backups.md index 3a7b0d12c..0e00a5d67 100644 --- a/docs/scheduling-backups.md +++ b/docs/scheduling-backups.md @@ -3,16 +3,7 @@ Regular backups should be scheduled using `cron(8)` or similar command scheduling service on the backup host. The backup frequency will dictate the worst case [recovery point objective (RPO)][1] in your backup plan. We recommend -hourly backups as a starting point. - -It's important to consider the duration of each backup operation on the -GitHub Enterprise Server (GHES) appliance. Backups of large datasets or -over slow network links can take more than an hour. Additionally, -maintenance queues are paused during a portion of a backup runs. -We recommend scheduling backups to allow sufficient time for jobs -waiting in maintenance queues to process between backup runs - -Only one backup may be in progress at a time. +hourly backups at the least. ## Example scheduling of backups @@ -28,8 +19,7 @@ storage. To schedule hourly backup snapshots with verbose informational output written to a log file and errors generating an email: - -```shell +``` MAILTO=admin@example.com 0 * * * * /opt/backup-utils/bin/ghe-backup -v 1>>/opt/backup-utils/backup.log 2>&1 @@ -37,13 +27,13 @@ MAILTO=admin@example.com To schedule nightly backup snapshots instead, use: -```shell +``` MAILTO=admin@example.com 0 0 * * * /opt/backup-utils/bin/ghe-backup -v 1>>/opt/backup-utils/backup.log 2>&1 ``` -## Example snapshot pruning +## Example snapshot pruning By default all expired and incomplete snapshots are deleted at the end of the main backup process `ghe-backup`. If pruning these snapshots takes a long time you can @@ -54,7 +44,7 @@ If this option is enabled you will need to schedule the pruning script `ghe-prun To schedule daily snapshot pruning, use: -```shell +``` MAILTO=admin@example.com 0 3 * * * /opt/backup-utils/share/github-backup-utils/ghe-prune-snapshots 1>>/opt/backup-utils/prune-snapshots.log 2>&1 From e45439229923296ec425ccdecf24179692dbe61f Mon Sep 17 00:00:00 2001 From: "release-controller[bot]" Date: Tue, 24 Sep 2024 16:37:56 +0000 Subject: [PATCH 12/47] 3.14.1 release --- docs/getting-started.md | 2 +- docs/requirements.md | 14 +++++++------- docs/scheduling-backups.md | 20 +++++++++++++++----- 3 files changed, 23 insertions(+), 13 deletions(-) diff --git a/docs/getting-started.md b/docs/getting-started.md index 02f2cbe9b..ac85e5073 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -1,6 +1,6 @@ # Getting started - 1. [Download the latest release version][1] and extract the repository using `tar`: + 1. [Download the latest version of backup-utils][1] and extract the repository using `tar`: `tar -xzvf /path/to/github-backup-utils-vMAJOR.MINOR.PATCH.tar.gz` diff --git a/docs/requirements.md b/docs/requirements.md index b2a3d282a..c87a0775a 100644 --- a/docs/requirements.md +++ b/docs/requirements.md @@ -69,23 +69,23 @@ Please avoid using an NFS mount for the data directory (where backup data is sto Starting with Backup Utilities v2.13.0, version support is inline with that of the [GitHub Enterprise Server upgrade requirements][8] and as such, support is limited to three versions of GitHub Enterprise Server: the version that corresponds with the version -of Backup Utilities, and the two releases prior to it. +of Backup Utilities, and the two versions prior to it. For example, Backup Utilities v2.13.0 can be used to backup and restore all patch -releases from 2.11.0 to the latest patch release of GitHub Enterprise Server 2.13. +versions from 2.11.0 to the latest patch version of GitHub Enterprise Server 2.13. Backup Utilities v2.14.0 will be released when GitHub Enterprise Server 2.14.0 is released -and will then be used to backup all releases of GitHub Enterprise Server from 2.12.0 -to the latest patch release of GitHub Enterprise Server 2.14. +and will then be used to backup all versions of GitHub Enterprise Server from 2.12.0 +to the latest patch version of GitHub Enterprise Server 2.14. Backup Utilities v2.11.4 and earlier offer support for GitHub Enterprise Server 2.10 -and earlier releases up to GitHub Enterprise Server 2.2.0. Backup Utilities v2.11.0 and earlier +and earlier versions up to GitHub Enterprise Server 2.2.0. Backup Utilities v2.11.0 and earlier offer support for GitHub Enterprise Server 2.1.0 and earlier. -**Note**: You can restore a snapshot that's at most two feature releases behind +**Note**: You can restore a snapshot that's at most two feature versions behind the restore target's version of GitHub Enterprise Server. For example, to restore a snapshot of GitHub Enterprise Server 2.11, the target GitHub Enterprise Server appliance must be running GitHub Enterprise Server 2.12.x or 2.13.x. You can't restore a snapshot from -2.10 to 2.13, because that's three releases ahead. +2.10 to 2.13, because that's three versions ahead. **Note**: You _cannot_ restore a backup created from a newer version of GitHub Enterprise Server to an older version. For example, an attempt to restore a snapshot of GitHub Enterprise Server 2.21 to a GitHub Enterprise Server 2.20 environment will fail with an error of `Error: Snapshot can not be restored to an older release of GitHub Enterprise Server.`. diff --git a/docs/scheduling-backups.md b/docs/scheduling-backups.md index 0e00a5d67..3a7b0d12c 100644 --- a/docs/scheduling-backups.md +++ b/docs/scheduling-backups.md @@ -3,7 +3,16 @@ Regular backups should be scheduled using `cron(8)` or similar command scheduling service on the backup host. The backup frequency will dictate the worst case [recovery point objective (RPO)][1] in your backup plan. We recommend -hourly backups at the least. +hourly backups as a starting point. + +It's important to consider the duration of each backup operation on the +GitHub Enterprise Server (GHES) appliance. Backups of large datasets or +over slow network links can take more than an hour. Additionally, +maintenance queues are paused during a portion of a backup runs. +We recommend scheduling backups to allow sufficient time for jobs +waiting in maintenance queues to process between backup runs + +Only one backup may be in progress at a time. ## Example scheduling of backups @@ -19,7 +28,8 @@ storage. To schedule hourly backup snapshots with verbose informational output written to a log file and errors generating an email: -``` + +```shell MAILTO=admin@example.com 0 * * * * /opt/backup-utils/bin/ghe-backup -v 1>>/opt/backup-utils/backup.log 2>&1 @@ -27,13 +37,13 @@ MAILTO=admin@example.com To schedule nightly backup snapshots instead, use: -``` +```shell MAILTO=admin@example.com 0 0 * * * /opt/backup-utils/bin/ghe-backup -v 1>>/opt/backup-utils/backup.log 2>&1 ``` -## Example snapshot pruning +## Example snapshot pruning By default all expired and incomplete snapshots are deleted at the end of the main backup process `ghe-backup`. If pruning these snapshots takes a long time you can @@ -44,7 +54,7 @@ If this option is enabled you will need to schedule the pruning script `ghe-prun To schedule daily snapshot pruning, use: -``` +```shell MAILTO=admin@example.com 0 3 * * * /opt/backup-utils/share/github-backup-utils/ghe-prune-snapshots 1>>/opt/backup-utils/prune-snapshots.log 2>&1 From dc75df4a4683d97ec3a677cfb962e0ca9d91db09 Mon Sep 17 00:00:00 2001 From: "release-controller[bot]" Date: Tue, 24 Sep 2024 16:38:38 +0000 Subject: [PATCH 13/47] 3.13.2 release From fdc9b0128a9888b2ae831b2393313a972c51abaa Mon Sep 17 00:00:00 2001 From: "release-controller[bot]" Date: Tue, 12 Nov 2024 19:11:42 +0000 Subject: [PATCH 14/47] 3.15.0 release From fe047a6ddd32f421803513896b060b501fcb450f Mon Sep 17 00:00:00 2001 From: Devin Dooley Date: Wed, 18 Dec 2024 13:51:17 -0800 Subject: [PATCH 15/47] 3.13.3 release From ba3aa670a21c278e9b4dd35ec03a395d37a2dd44 Mon Sep 17 00:00:00 2001 From: Devin Dooley Date: Wed, 18 Dec 2024 13:51:33 -0800 Subject: [PATCH 16/47] 3.14.2 release From 9a4b34d7530488076728086bb7f83e9ad557aa38 Mon Sep 17 00:00:00 2001 From: Devin Dooley Date: Wed, 18 Dec 2024 13:51:38 -0800 Subject: [PATCH 17/47] 3.15.1 release From 407beeb90abd25e115d5c81cacefe7774d5bacde Mon Sep 17 00:00:00 2001 From: "release-controller[bot]" Date: Wed, 18 Dec 2024 22:01:46 +0000 Subject: [PATCH 18/47] 3.14.2 release From 5c13db29e53eb90435d40b81ca3bfcdf8d6d4d69 Mon Sep 17 00:00:00 2001 From: "release-controller[bot]" Date: Wed, 18 Dec 2024 22:01:56 +0000 Subject: [PATCH 19/47] 3.13.3 release From e34699540f9ac04fe26bfad846c490e428b0039c Mon Sep 17 00:00:00 2001 From: timreimherr Date: Tue, 7 Jan 2025 18:54:46 +0000 Subject: [PATCH 20/47] 3.15.1 release From a2924f0d5d7c76368347f3d1745ffc5fe24d3928 Mon Sep 17 00:00:00 2001 From: timreimherr Date: Tue, 7 Jan 2025 18:56:25 +0000 Subject: [PATCH 21/47] 3.14.2 release From 782b06cadb9f4d36924869050ea064de70e35166 Mon Sep 17 00:00:00 2001 From: timreimherr Date: Tue, 7 Jan 2025 18:56:40 +0000 Subject: [PATCH 22/47] 3.13.3 release From 91ed56287f1c1d271b6904a2866cbc8042ff004a Mon Sep 17 00:00:00 2001 From: Casey Tucker Date: Wed, 15 Jan 2025 14:49:43 -0800 Subject: [PATCH 23/47] Update getting-started.md This file is no longer included in this repository. Since it actually is included in tarball referenced in the first step, we can remove the hyperlink. --- docs/getting-started.md | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/docs/getting-started.md b/docs/getting-started.md index ac85e5073..1a04799c9 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -7,7 +7,7 @@ **Note**: you will need to use [Backup Utilities v2.11.x][2] or the `legacy` branch to backup and restore GitHub Enterprise Server 2.10 and earlier. - 2. Copy the [`backup.config-example`][3] file to `backup.config` and modify as + 2. Copy the `backup.config-example` file to `backup.config` and modify as necessary. The `GHE_HOSTNAME` value must be set to the primary GitHub Enterprise Server hostname. Additional options are available and documented in the configuration file but none are required for basic backup functionality. @@ -28,7 +28,7 @@ with the `-i ` SSH option. 3. Add the backup host's SSH public key to the GitHub Enterprise Server appliance, in order to grant it administrative shell access. - See [Accessing the GitHub Enterprise Server administrative shell (SSH)][4] for instructions. + See [Accessing the GitHub Enterprise Server administrative shell (SSH)][3] for instructions. 4. Run `bin/ghe-host-check` to verify SSH connectivity with the GitHub appliance. @@ -37,5 +37,4 @@ [1]: https://github.com/github/backup-utils/releases [2]: https://github.com/github/backup-utils/releases/tag/v2.11.4 -[3]: https://github.com/github/enterprise-backup-site/blob/master/backup.config-example -[4]: https://docs.github.com/enterprise-server/admin/configuration/configuring-your-enterprise/accessing-the-administrative-shell-ssh +[3]: https://docs.github.com/enterprise-server/admin/configuration/configuring-your-enterprise/accessing-the-administrative-shell-ssh From 7a64e2d4582423c1c361740aba13e6591cc7f165 Mon Sep 17 00:00:00 2001 From: "release-controller[bot]" <110195724+release-controller[bot]@users.noreply.github.com> Date: Tue, 25 Feb 2025 20:25:24 +0000 Subject: [PATCH 24/47] 3.16.0 release --- docs/backup.config-example | 147 +++++++++++++++++++++++++++++++++++++ docs/getting-started.md | 7 +- docs/usage.md | 2 +- 3 files changed, 152 insertions(+), 4 deletions(-) create mode 100644 docs/backup.config-example diff --git a/docs/backup.config-example b/docs/backup.config-example new file mode 100644 index 000000000..552d04b01 --- /dev/null +++ b/docs/backup.config-example @@ -0,0 +1,147 @@ +# GitHub Enterprise Server backup configuration file + +# The hostname of the GitHub Enterprise Server appliance to back up. The host +# must be reachable via SSH from the backup host. +GHE_HOSTNAME="github.example.com" + +# Path to where backup data is stored. By default this is the "data" +# directory next to this file but can be set to an absolute path +# elsewhere for backing up to a separate partition / mount point. +GHE_DATA_DIR="data" + +# The number of backup snapshots to retain. Old snapshots are pruned after each +# successful ghe-backup run. This option should be tuned based on the frequency +# of scheduled backup runs. If backups are scheduled hourly, snapshots will be +# available for the past N hours; if backups are scheduled daily, snapshots will +# be available for the past N days ... +GHE_NUM_SNAPSHOTS=10 + +# Pruning snapshots can be scheduled outside of the backup process. +# If set to 'yes', snapshots will not be pruned by ghe-backup. +# Instead, ghe-prune-snapshots will need to be invoked separately via cron +#GHE_PRUNING_SCHEDULED=yes + +# If --incremental is used to generate incremental MySQL backups with ghe-backup, +# then you need to specify how many cycles of full and incremental backups will be +# performed before the next full backup is created. +# For example, if `GHE_INCREMENTAL_BACKUP_MAX` is set to 14, backup-utils will +# run 1 full backup and then 13 incremental backups before performing another full backup on the next cycle. +#GHE_INCREMENTAL_MAX_BACKUPS=14 + +# If GHE_ROUTE_VERIFICATION is set to true then ghe-repository-backup and +# ghe-storage-backup will issue a warning if the repositories and objects in +# the backup do not match the pre-backup inventory of routes. +#GHE_ROUTE_VERIFICATION=false + +# If GHE_MANAGE_CONSOLE_PW_RESTORE is set to false then management-console password +# will not be restored from backed-up snapshot data, it is restored by default +#GHE_MANAGE_CONSOLE_PW_RESTORE=true + +# If GHE_SKIP_CHECKS is set to true (or if --skip-checks is used with ghe-backup) then ghe-host-check +# disk space validation and software version checks on the backup-host will be disabled. +#GHE_SKIP_CHECKS=false + +# Cluster filesystem to check if it's writable as part of ghe-host-check +# By default it is /data/user/tmp but can be updated if needed +#GHE_FILE_SYSTEM_WRITE_CHECK="/data/user/tmp" + +# The hostname of the GitHub appliance to restore. If you've set up a separate +# GitHub appliance to act as a standby for recovery, specify its IP or hostname +# here. The host to restore to may also be specified directly when running +# ghe-restore so use of this variable isn't strictly required. +# +#GHE_RESTORE_HOST="github-standby.example.com" + +# If set to 'yes', ghe-restore will omit the restore of audit logs. +# +#GHE_RESTORE_SKIP_AUDIT_LOGS=no + +# If set to 'yes', backup and restore of Elasticsearch indices will be skipped +# +#GHE_SKIP_SEARCH_INDICES=no + +# When verbose output is enabled with `-v`, it's written to stdout by default. If +# you'd prefer it to be written to a separate file, set this option. +# +#GHE_VERBOSE_LOG="/var/log/backup-verbose.log" + +# Any extra options passed to the SSH command. +# In a single instance environment, nothing is required by default. +# In a clustering environment, "-i abs-path-to-ssh-private-key" is required. +# +#GHE_EXTRA_SSH_OPTS="" +# +# All backup processes are ran with the lowest priority for scheduling by default. +# To change throttling behaviour/allow higher priority for backup processes, set higher values for following variables. +# default value for GHENICE=nice -n 19 +# default value for GHE_IONICE=ionice -c 3 +#GHE_NICE="" +#GHE_IONICE="" + +# Any extra options passed to the rsync command. Nothing required by default. +# +#GHE_EXTRA_RSYNC_OPTS="" + +# If set to 'yes', rsync will be set to use compression during backups and restores transfers. Defaults to 'no'. +# +#GHE_RSYNC_COMPRESSION_ENABLED=yes + +# If enabled and set to 'no', rsync warning message during backups will be suppressed. +#RSYNC_WARNING=no + + +# If set to 'yes', logging output will be colorized. +# +#OUTPUT_COLOR=no + +# If set to 'no', GHE_DATA_DIR will not be created automatically +# and restore/backup will exit 8 +# +#GHE_CREATE_DATA_DIR=yes + +# If set to 'yes', git fsck will run on the repositories +# and print some additional info. +# +# WARNING: do not enable this, only useful for debugging/development +#GHE_BACKUP_FSCK=no + +# Cadence of MSSQL backups +# ,, all in minutes +# e.g. +# - Full backup every week (10080 minutes) +# - Differential backup every day (1440 minutes) +# - Transactionlog backup every 15 minutes +# +#GHE_MSSQL_BACKUP_CADENCE=10080,1440,15 + +# If set to 'yes', ghe-backup jobs will run in parallel. Defaults to 'no'. +# +#GHE_PARALLEL_ENABLED=yes + +# Sets the maximum number of jobs to run in parallel. Defaults to the number +# of available processing units on the machine. +# +#GHE_PARALLEL_MAX_JOBS=2 + +# Sets the maximum number of rsync jobs to run in parallel. Defaults to the +# configured GHE_PARALLEL_MAX_JOBS, or the number of available processing +# units on the machine. +# +# GHE_PARALLEL_RSYNC_MAX_JOBS=3 + +# When jobs are running in parallel wait as needed to avoid starting new jobs +# when the system's load average is not below the specified percentage. Defaults to +# unrestricted. +# +#GHE_PARALLEL_MAX_LOAD=50 + +# When running an external mysql database, run this script to trigger a MySQL backup +# rather than attempting to backup via backup-utils directly. +#EXTERNAL_DATABASE_BACKUP_SCRIPT="/bin/false" + +# When running an external mysql database, run this script to trigger a MySQL restore +# rather than attempting to backup via backup-utils directly. +#EXTERNAL_DATABASE_RESTORE_SCRIPT="/bin/false" + +# If set to 'yes', Pages data will be included in backup and restore. Defaults to 'yes' +#GHE_BACKUP_PAGES=no diff --git a/docs/getting-started.md b/docs/getting-started.md index 1a04799c9..a272c796e 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -7,7 +7,7 @@ **Note**: you will need to use [Backup Utilities v2.11.x][2] or the `legacy` branch to backup and restore GitHub Enterprise Server 2.10 and earlier. - 2. Copy the `backup.config-example` file to `backup.config` and modify as + 2. Copy the [`backup.config-example`][3] file to `backup.config` and modify as necessary. The `GHE_HOSTNAME` value must be set to the primary GitHub Enterprise Server hostname. Additional options are available and documented in the configuration file but none are required for basic backup functionality. @@ -28,7 +28,7 @@ with the `-i ` SSH option. 3. Add the backup host's SSH public key to the GitHub Enterprise Server appliance, in order to grant it administrative shell access. - See [Accessing the GitHub Enterprise Server administrative shell (SSH)][3] for instructions. + See [Accessing the GitHub Enterprise Server administrative shell (SSH)][4] for instructions. 4. Run `bin/ghe-host-check` to verify SSH connectivity with the GitHub appliance. @@ -37,4 +37,5 @@ [1]: https://github.com/github/backup-utils/releases [2]: https://github.com/github/backup-utils/releases/tag/v2.11.4 -[3]: https://docs.github.com/enterprise-server/admin/configuration/configuring-your-enterprise/accessing-the-administrative-shell-ssh +[3]: https://github.com/github/enterprise-backup-site/blob/master/docs/backup.config-example +[4]: https://docs.github.com/enterprise-server/admin/configuration/configuring-your-enterprise/accessing-the-administrative-shell-ssh diff --git a/docs/usage.md b/docs/usage.md index 4324b58cc..33960eca7 100644 --- a/docs/usage.md +++ b/docs/usage.md @@ -13,7 +13,7 @@ These commands are run on the host you [installed][1] Backup Utilities on. You can supply your own configuration file or use the example configuration file as a template where you can set up your environment for backing up and restoring. -An example configuration file with documentation on possible settings can found in [backup.config-example](../backup.config-example). +An example configuration file with documentation on possible settings can found in [backup.config-example](backup.config-example). There are a number of command-line options that can also be passed to the `ghe-restore` command. Of particular note, if you use an external MySQL service but are restoring from a snapshot prior to enabling this, or vice versa, you must migrate the MySQL data outside of the context of backup-utils first, then pass the `--skip-mysql` flag to `ghe-restore`. From 63464e766f37176cfec4ee5d9a5b89a092816cd4 Mon Sep 17 00:00:00 2001 From: "release-controller[bot]" <110195724+release-controller[bot]@users.noreply.github.com> Date: Thu, 20 Mar 2025 23:29:40 +0000 Subject: [PATCH 25/47] 3.14.3 release --- docs/backup.config-example | 147 ------------------------------------- docs/getting-started.md | 2 +- docs/usage.md | 2 +- 3 files changed, 2 insertions(+), 149 deletions(-) delete mode 100644 docs/backup.config-example diff --git a/docs/backup.config-example b/docs/backup.config-example deleted file mode 100644 index 552d04b01..000000000 --- a/docs/backup.config-example +++ /dev/null @@ -1,147 +0,0 @@ -# GitHub Enterprise Server backup configuration file - -# The hostname of the GitHub Enterprise Server appliance to back up. The host -# must be reachable via SSH from the backup host. -GHE_HOSTNAME="github.example.com" - -# Path to where backup data is stored. By default this is the "data" -# directory next to this file but can be set to an absolute path -# elsewhere for backing up to a separate partition / mount point. -GHE_DATA_DIR="data" - -# The number of backup snapshots to retain. Old snapshots are pruned after each -# successful ghe-backup run. This option should be tuned based on the frequency -# of scheduled backup runs. If backups are scheduled hourly, snapshots will be -# available for the past N hours; if backups are scheduled daily, snapshots will -# be available for the past N days ... -GHE_NUM_SNAPSHOTS=10 - -# Pruning snapshots can be scheduled outside of the backup process. -# If set to 'yes', snapshots will not be pruned by ghe-backup. -# Instead, ghe-prune-snapshots will need to be invoked separately via cron -#GHE_PRUNING_SCHEDULED=yes - -# If --incremental is used to generate incremental MySQL backups with ghe-backup, -# then you need to specify how many cycles of full and incremental backups will be -# performed before the next full backup is created. -# For example, if `GHE_INCREMENTAL_BACKUP_MAX` is set to 14, backup-utils will -# run 1 full backup and then 13 incremental backups before performing another full backup on the next cycle. -#GHE_INCREMENTAL_MAX_BACKUPS=14 - -# If GHE_ROUTE_VERIFICATION is set to true then ghe-repository-backup and -# ghe-storage-backup will issue a warning if the repositories and objects in -# the backup do not match the pre-backup inventory of routes. -#GHE_ROUTE_VERIFICATION=false - -# If GHE_MANAGE_CONSOLE_PW_RESTORE is set to false then management-console password -# will not be restored from backed-up snapshot data, it is restored by default -#GHE_MANAGE_CONSOLE_PW_RESTORE=true - -# If GHE_SKIP_CHECKS is set to true (or if --skip-checks is used with ghe-backup) then ghe-host-check -# disk space validation and software version checks on the backup-host will be disabled. -#GHE_SKIP_CHECKS=false - -# Cluster filesystem to check if it's writable as part of ghe-host-check -# By default it is /data/user/tmp but can be updated if needed -#GHE_FILE_SYSTEM_WRITE_CHECK="/data/user/tmp" - -# The hostname of the GitHub appliance to restore. If you've set up a separate -# GitHub appliance to act as a standby for recovery, specify its IP or hostname -# here. The host to restore to may also be specified directly when running -# ghe-restore so use of this variable isn't strictly required. -# -#GHE_RESTORE_HOST="github-standby.example.com" - -# If set to 'yes', ghe-restore will omit the restore of audit logs. -# -#GHE_RESTORE_SKIP_AUDIT_LOGS=no - -# If set to 'yes', backup and restore of Elasticsearch indices will be skipped -# -#GHE_SKIP_SEARCH_INDICES=no - -# When verbose output is enabled with `-v`, it's written to stdout by default. If -# you'd prefer it to be written to a separate file, set this option. -# -#GHE_VERBOSE_LOG="/var/log/backup-verbose.log" - -# Any extra options passed to the SSH command. -# In a single instance environment, nothing is required by default. -# In a clustering environment, "-i abs-path-to-ssh-private-key" is required. -# -#GHE_EXTRA_SSH_OPTS="" -# -# All backup processes are ran with the lowest priority for scheduling by default. -# To change throttling behaviour/allow higher priority for backup processes, set higher values for following variables. -# default value for GHENICE=nice -n 19 -# default value for GHE_IONICE=ionice -c 3 -#GHE_NICE="" -#GHE_IONICE="" - -# Any extra options passed to the rsync command. Nothing required by default. -# -#GHE_EXTRA_RSYNC_OPTS="" - -# If set to 'yes', rsync will be set to use compression during backups and restores transfers. Defaults to 'no'. -# -#GHE_RSYNC_COMPRESSION_ENABLED=yes - -# If enabled and set to 'no', rsync warning message during backups will be suppressed. -#RSYNC_WARNING=no - - -# If set to 'yes', logging output will be colorized. -# -#OUTPUT_COLOR=no - -# If set to 'no', GHE_DATA_DIR will not be created automatically -# and restore/backup will exit 8 -# -#GHE_CREATE_DATA_DIR=yes - -# If set to 'yes', git fsck will run on the repositories -# and print some additional info. -# -# WARNING: do not enable this, only useful for debugging/development -#GHE_BACKUP_FSCK=no - -# Cadence of MSSQL backups -# ,, all in minutes -# e.g. -# - Full backup every week (10080 minutes) -# - Differential backup every day (1440 minutes) -# - Transactionlog backup every 15 minutes -# -#GHE_MSSQL_BACKUP_CADENCE=10080,1440,15 - -# If set to 'yes', ghe-backup jobs will run in parallel. Defaults to 'no'. -# -#GHE_PARALLEL_ENABLED=yes - -# Sets the maximum number of jobs to run in parallel. Defaults to the number -# of available processing units on the machine. -# -#GHE_PARALLEL_MAX_JOBS=2 - -# Sets the maximum number of rsync jobs to run in parallel. Defaults to the -# configured GHE_PARALLEL_MAX_JOBS, or the number of available processing -# units on the machine. -# -# GHE_PARALLEL_RSYNC_MAX_JOBS=3 - -# When jobs are running in parallel wait as needed to avoid starting new jobs -# when the system's load average is not below the specified percentage. Defaults to -# unrestricted. -# -#GHE_PARALLEL_MAX_LOAD=50 - -# When running an external mysql database, run this script to trigger a MySQL backup -# rather than attempting to backup via backup-utils directly. -#EXTERNAL_DATABASE_BACKUP_SCRIPT="/bin/false" - -# When running an external mysql database, run this script to trigger a MySQL restore -# rather than attempting to backup via backup-utils directly. -#EXTERNAL_DATABASE_RESTORE_SCRIPT="/bin/false" - -# If set to 'yes', Pages data will be included in backup and restore. Defaults to 'yes' -#GHE_BACKUP_PAGES=no diff --git a/docs/getting-started.md b/docs/getting-started.md index a272c796e..ac85e5073 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -37,5 +37,5 @@ [1]: https://github.com/github/backup-utils/releases [2]: https://github.com/github/backup-utils/releases/tag/v2.11.4 -[3]: https://github.com/github/enterprise-backup-site/blob/master/docs/backup.config-example +[3]: https://github.com/github/enterprise-backup-site/blob/master/backup.config-example [4]: https://docs.github.com/enterprise-server/admin/configuration/configuring-your-enterprise/accessing-the-administrative-shell-ssh diff --git a/docs/usage.md b/docs/usage.md index 33960eca7..4324b58cc 100644 --- a/docs/usage.md +++ b/docs/usage.md @@ -13,7 +13,7 @@ These commands are run on the host you [installed][1] Backup Utilities on. You can supply your own configuration file or use the example configuration file as a template where you can set up your environment for backing up and restoring. -An example configuration file with documentation on possible settings can found in [backup.config-example](backup.config-example). +An example configuration file with documentation on possible settings can found in [backup.config-example](../backup.config-example). There are a number of command-line options that can also be passed to the `ghe-restore` command. Of particular note, if you use an external MySQL service but are restoring from a snapshot prior to enabling this, or vice versa, you must migrate the MySQL data outside of the context of backup-utils first, then pass the `--skip-mysql` flag to `ghe-restore`. From 6832553bc59893b2ec313dfdea0621b2879e2d11 Mon Sep 17 00:00:00 2001 From: "release-controller[bot]" <110195724+release-controller[bot]@users.noreply.github.com> Date: Thu, 20 Mar 2025 23:30:25 +0000 Subject: [PATCH 26/47] 3.15.2 release From 772f231094ad0a1ed773476656fae49783167015 Mon Sep 17 00:00:00 2001 From: "release-controller[bot]" <110195724+release-controller[bot]@users.noreply.github.com> Date: Thu, 20 Mar 2025 23:31:05 +0000 Subject: [PATCH 27/47] 3.16.1 release --- docs/backup.config-example | 147 +++++++++++++++++++++++++++++++++++++ docs/getting-started.md | 2 +- docs/usage.md | 2 +- 3 files changed, 149 insertions(+), 2 deletions(-) create mode 100644 docs/backup.config-example diff --git a/docs/backup.config-example b/docs/backup.config-example new file mode 100644 index 000000000..552d04b01 --- /dev/null +++ b/docs/backup.config-example @@ -0,0 +1,147 @@ +# GitHub Enterprise Server backup configuration file + +# The hostname of the GitHub Enterprise Server appliance to back up. The host +# must be reachable via SSH from the backup host. +GHE_HOSTNAME="github.example.com" + +# Path to where backup data is stored. By default this is the "data" +# directory next to this file but can be set to an absolute path +# elsewhere for backing up to a separate partition / mount point. +GHE_DATA_DIR="data" + +# The number of backup snapshots to retain. Old snapshots are pruned after each +# successful ghe-backup run. This option should be tuned based on the frequency +# of scheduled backup runs. If backups are scheduled hourly, snapshots will be +# available for the past N hours; if backups are scheduled daily, snapshots will +# be available for the past N days ... +GHE_NUM_SNAPSHOTS=10 + +# Pruning snapshots can be scheduled outside of the backup process. +# If set to 'yes', snapshots will not be pruned by ghe-backup. +# Instead, ghe-prune-snapshots will need to be invoked separately via cron +#GHE_PRUNING_SCHEDULED=yes + +# If --incremental is used to generate incremental MySQL backups with ghe-backup, +# then you need to specify how many cycles of full and incremental backups will be +# performed before the next full backup is created. +# For example, if `GHE_INCREMENTAL_BACKUP_MAX` is set to 14, backup-utils will +# run 1 full backup and then 13 incremental backups before performing another full backup on the next cycle. +#GHE_INCREMENTAL_MAX_BACKUPS=14 + +# If GHE_ROUTE_VERIFICATION is set to true then ghe-repository-backup and +# ghe-storage-backup will issue a warning if the repositories and objects in +# the backup do not match the pre-backup inventory of routes. +#GHE_ROUTE_VERIFICATION=false + +# If GHE_MANAGE_CONSOLE_PW_RESTORE is set to false then management-console password +# will not be restored from backed-up snapshot data, it is restored by default +#GHE_MANAGE_CONSOLE_PW_RESTORE=true + +# If GHE_SKIP_CHECKS is set to true (or if --skip-checks is used with ghe-backup) then ghe-host-check +# disk space validation and software version checks on the backup-host will be disabled. +#GHE_SKIP_CHECKS=false + +# Cluster filesystem to check if it's writable as part of ghe-host-check +# By default it is /data/user/tmp but can be updated if needed +#GHE_FILE_SYSTEM_WRITE_CHECK="/data/user/tmp" + +# The hostname of the GitHub appliance to restore. If you've set up a separate +# GitHub appliance to act as a standby for recovery, specify its IP or hostname +# here. The host to restore to may also be specified directly when running +# ghe-restore so use of this variable isn't strictly required. +# +#GHE_RESTORE_HOST="github-standby.example.com" + +# If set to 'yes', ghe-restore will omit the restore of audit logs. +# +#GHE_RESTORE_SKIP_AUDIT_LOGS=no + +# If set to 'yes', backup and restore of Elasticsearch indices will be skipped +# +#GHE_SKIP_SEARCH_INDICES=no + +# When verbose output is enabled with `-v`, it's written to stdout by default. If +# you'd prefer it to be written to a separate file, set this option. +# +#GHE_VERBOSE_LOG="/var/log/backup-verbose.log" + +# Any extra options passed to the SSH command. +# In a single instance environment, nothing is required by default. +# In a clustering environment, "-i abs-path-to-ssh-private-key" is required. +# +#GHE_EXTRA_SSH_OPTS="" +# +# All backup processes are ran with the lowest priority for scheduling by default. +# To change throttling behaviour/allow higher priority for backup processes, set higher values for following variables. +# default value for GHENICE=nice -n 19 +# default value for GHE_IONICE=ionice -c 3 +#GHE_NICE="" +#GHE_IONICE="" + +# Any extra options passed to the rsync command. Nothing required by default. +# +#GHE_EXTRA_RSYNC_OPTS="" + +# If set to 'yes', rsync will be set to use compression during backups and restores transfers. Defaults to 'no'. +# +#GHE_RSYNC_COMPRESSION_ENABLED=yes + +# If enabled and set to 'no', rsync warning message during backups will be suppressed. +#RSYNC_WARNING=no + + +# If set to 'yes', logging output will be colorized. +# +#OUTPUT_COLOR=no + +# If set to 'no', GHE_DATA_DIR will not be created automatically +# and restore/backup will exit 8 +# +#GHE_CREATE_DATA_DIR=yes + +# If set to 'yes', git fsck will run on the repositories +# and print some additional info. +# +# WARNING: do not enable this, only useful for debugging/development +#GHE_BACKUP_FSCK=no + +# Cadence of MSSQL backups +# ,, all in minutes +# e.g. +# - Full backup every week (10080 minutes) +# - Differential backup every day (1440 minutes) +# - Transactionlog backup every 15 minutes +# +#GHE_MSSQL_BACKUP_CADENCE=10080,1440,15 + +# If set to 'yes', ghe-backup jobs will run in parallel. Defaults to 'no'. +# +#GHE_PARALLEL_ENABLED=yes + +# Sets the maximum number of jobs to run in parallel. Defaults to the number +# of available processing units on the machine. +# +#GHE_PARALLEL_MAX_JOBS=2 + +# Sets the maximum number of rsync jobs to run in parallel. Defaults to the +# configured GHE_PARALLEL_MAX_JOBS, or the number of available processing +# units on the machine. +# +# GHE_PARALLEL_RSYNC_MAX_JOBS=3 + +# When jobs are running in parallel wait as needed to avoid starting new jobs +# when the system's load average is not below the specified percentage. Defaults to +# unrestricted. +# +#GHE_PARALLEL_MAX_LOAD=50 + +# When running an external mysql database, run this script to trigger a MySQL backup +# rather than attempting to backup via backup-utils directly. +#EXTERNAL_DATABASE_BACKUP_SCRIPT="/bin/false" + +# When running an external mysql database, run this script to trigger a MySQL restore +# rather than attempting to backup via backup-utils directly. +#EXTERNAL_DATABASE_RESTORE_SCRIPT="/bin/false" + +# If set to 'yes', Pages data will be included in backup and restore. Defaults to 'yes' +#GHE_BACKUP_PAGES=no diff --git a/docs/getting-started.md b/docs/getting-started.md index ac85e5073..a272c796e 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -37,5 +37,5 @@ [1]: https://github.com/github/backup-utils/releases [2]: https://github.com/github/backup-utils/releases/tag/v2.11.4 -[3]: https://github.com/github/enterprise-backup-site/blob/master/backup.config-example +[3]: https://github.com/github/enterprise-backup-site/blob/master/docs/backup.config-example [4]: https://docs.github.com/enterprise-server/admin/configuration/configuring-your-enterprise/accessing-the-administrative-shell-ssh diff --git a/docs/usage.md b/docs/usage.md index 4324b58cc..33960eca7 100644 --- a/docs/usage.md +++ b/docs/usage.md @@ -13,7 +13,7 @@ These commands are run on the host you [installed][1] Backup Utilities on. You can supply your own configuration file or use the example configuration file as a template where you can set up your environment for backing up and restoring. -An example configuration file with documentation on possible settings can found in [backup.config-example](../backup.config-example). +An example configuration file with documentation on possible settings can found in [backup.config-example](backup.config-example). There are a number of command-line options that can also be passed to the `ghe-restore` command. Of particular note, if you use an external MySQL service but are restoring from a snapshot prior to enabling this, or vice versa, you must migrate the MySQL data outside of the context of backup-utils first, then pass the `--skip-mysql` flag to `ghe-restore`. From 63ab6d8b8192cd4d832dcc121557021b48470a82 Mon Sep 17 00:00:00 2001 From: "release-controller[bot]" <110195724+release-controller[bot]@users.noreply.github.com> Date: Tue, 22 Apr 2025 20:05:40 +0000 Subject: [PATCH 28/47] 3.13.4 release --- docs/backup.config-example | 147 ------------------------------------- docs/getting-started.md | 2 +- docs/usage.md | 2 +- 3 files changed, 2 insertions(+), 149 deletions(-) delete mode 100644 docs/backup.config-example diff --git a/docs/backup.config-example b/docs/backup.config-example deleted file mode 100644 index 552d04b01..000000000 --- a/docs/backup.config-example +++ /dev/null @@ -1,147 +0,0 @@ -# GitHub Enterprise Server backup configuration file - -# The hostname of the GitHub Enterprise Server appliance to back up. The host -# must be reachable via SSH from the backup host. -GHE_HOSTNAME="github.example.com" - -# Path to where backup data is stored. By default this is the "data" -# directory next to this file but can be set to an absolute path -# elsewhere for backing up to a separate partition / mount point. -GHE_DATA_DIR="data" - -# The number of backup snapshots to retain. Old snapshots are pruned after each -# successful ghe-backup run. This option should be tuned based on the frequency -# of scheduled backup runs. If backups are scheduled hourly, snapshots will be -# available for the past N hours; if backups are scheduled daily, snapshots will -# be available for the past N days ... -GHE_NUM_SNAPSHOTS=10 - -# Pruning snapshots can be scheduled outside of the backup process. -# If set to 'yes', snapshots will not be pruned by ghe-backup. -# Instead, ghe-prune-snapshots will need to be invoked separately via cron -#GHE_PRUNING_SCHEDULED=yes - -# If --incremental is used to generate incremental MySQL backups with ghe-backup, -# then you need to specify how many cycles of full and incremental backups will be -# performed before the next full backup is created. -# For example, if `GHE_INCREMENTAL_BACKUP_MAX` is set to 14, backup-utils will -# run 1 full backup and then 13 incremental backups before performing another full backup on the next cycle. -#GHE_INCREMENTAL_MAX_BACKUPS=14 - -# If GHE_ROUTE_VERIFICATION is set to true then ghe-repository-backup and -# ghe-storage-backup will issue a warning if the repositories and objects in -# the backup do not match the pre-backup inventory of routes. -#GHE_ROUTE_VERIFICATION=false - -# If GHE_MANAGE_CONSOLE_PW_RESTORE is set to false then management-console password -# will not be restored from backed-up snapshot data, it is restored by default -#GHE_MANAGE_CONSOLE_PW_RESTORE=true - -# If GHE_SKIP_CHECKS is set to true (or if --skip-checks is used with ghe-backup) then ghe-host-check -# disk space validation and software version checks on the backup-host will be disabled. -#GHE_SKIP_CHECKS=false - -# Cluster filesystem to check if it's writable as part of ghe-host-check -# By default it is /data/user/tmp but can be updated if needed -#GHE_FILE_SYSTEM_WRITE_CHECK="/data/user/tmp" - -# The hostname of the GitHub appliance to restore. If you've set up a separate -# GitHub appliance to act as a standby for recovery, specify its IP or hostname -# here. The host to restore to may also be specified directly when running -# ghe-restore so use of this variable isn't strictly required. -# -#GHE_RESTORE_HOST="github-standby.example.com" - -# If set to 'yes', ghe-restore will omit the restore of audit logs. -# -#GHE_RESTORE_SKIP_AUDIT_LOGS=no - -# If set to 'yes', backup and restore of Elasticsearch indices will be skipped -# -#GHE_SKIP_SEARCH_INDICES=no - -# When verbose output is enabled with `-v`, it's written to stdout by default. If -# you'd prefer it to be written to a separate file, set this option. -# -#GHE_VERBOSE_LOG="/var/log/backup-verbose.log" - -# Any extra options passed to the SSH command. -# In a single instance environment, nothing is required by default. -# In a clustering environment, "-i abs-path-to-ssh-private-key" is required. -# -#GHE_EXTRA_SSH_OPTS="" -# -# All backup processes are ran with the lowest priority for scheduling by default. -# To change throttling behaviour/allow higher priority for backup processes, set higher values for following variables. -# default value for GHENICE=nice -n 19 -# default value for GHE_IONICE=ionice -c 3 -#GHE_NICE="" -#GHE_IONICE="" - -# Any extra options passed to the rsync command. Nothing required by default. -# -#GHE_EXTRA_RSYNC_OPTS="" - -# If set to 'yes', rsync will be set to use compression during backups and restores transfers. Defaults to 'no'. -# -#GHE_RSYNC_COMPRESSION_ENABLED=yes - -# If enabled and set to 'no', rsync warning message during backups will be suppressed. -#RSYNC_WARNING=no - - -# If set to 'yes', logging output will be colorized. -# -#OUTPUT_COLOR=no - -# If set to 'no', GHE_DATA_DIR will not be created automatically -# and restore/backup will exit 8 -# -#GHE_CREATE_DATA_DIR=yes - -# If set to 'yes', git fsck will run on the repositories -# and print some additional info. -# -# WARNING: do not enable this, only useful for debugging/development -#GHE_BACKUP_FSCK=no - -# Cadence of MSSQL backups -# ,, all in minutes -# e.g. -# - Full backup every week (10080 minutes) -# - Differential backup every day (1440 minutes) -# - Transactionlog backup every 15 minutes -# -#GHE_MSSQL_BACKUP_CADENCE=10080,1440,15 - -# If set to 'yes', ghe-backup jobs will run in parallel. Defaults to 'no'. -# -#GHE_PARALLEL_ENABLED=yes - -# Sets the maximum number of jobs to run in parallel. Defaults to the number -# of available processing units on the machine. -# -#GHE_PARALLEL_MAX_JOBS=2 - -# Sets the maximum number of rsync jobs to run in parallel. Defaults to the -# configured GHE_PARALLEL_MAX_JOBS, or the number of available processing -# units on the machine. -# -# GHE_PARALLEL_RSYNC_MAX_JOBS=3 - -# When jobs are running in parallel wait as needed to avoid starting new jobs -# when the system's load average is not below the specified percentage. Defaults to -# unrestricted. -# -#GHE_PARALLEL_MAX_LOAD=50 - -# When running an external mysql database, run this script to trigger a MySQL backup -# rather than attempting to backup via backup-utils directly. -#EXTERNAL_DATABASE_BACKUP_SCRIPT="/bin/false" - -# When running an external mysql database, run this script to trigger a MySQL restore -# rather than attempting to backup via backup-utils directly. -#EXTERNAL_DATABASE_RESTORE_SCRIPT="/bin/false" - -# If set to 'yes', Pages data will be included in backup and restore. Defaults to 'yes' -#GHE_BACKUP_PAGES=no diff --git a/docs/getting-started.md b/docs/getting-started.md index a272c796e..ac85e5073 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -37,5 +37,5 @@ [1]: https://github.com/github/backup-utils/releases [2]: https://github.com/github/backup-utils/releases/tag/v2.11.4 -[3]: https://github.com/github/enterprise-backup-site/blob/master/docs/backup.config-example +[3]: https://github.com/github/enterprise-backup-site/blob/master/backup.config-example [4]: https://docs.github.com/enterprise-server/admin/configuration/configuring-your-enterprise/accessing-the-administrative-shell-ssh diff --git a/docs/usage.md b/docs/usage.md index 33960eca7..4324b58cc 100644 --- a/docs/usage.md +++ b/docs/usage.md @@ -13,7 +13,7 @@ These commands are run on the host you [installed][1] Backup Utilities on. You can supply your own configuration file or use the example configuration file as a template where you can set up your environment for backing up and restoring. -An example configuration file with documentation on possible settings can found in [backup.config-example](backup.config-example). +An example configuration file with documentation on possible settings can found in [backup.config-example](../backup.config-example). There are a number of command-line options that can also be passed to the `ghe-restore` command. Of particular note, if you use an external MySQL service but are restoring from a snapshot prior to enabling this, or vice versa, you must migrate the MySQL data outside of the context of backup-utils first, then pass the `--skip-mysql` flag to `ghe-restore`. From 321f4800d1ac21ada10b7dc4ec58c32b8bcaf6d2 Mon Sep 17 00:00:00 2001 From: "release-controller[bot]" <110195724+release-controller[bot]@users.noreply.github.com> Date: Tue, 22 Apr 2025 20:06:32 +0000 Subject: [PATCH 29/47] 3.16.2 release --- docs/backup.config-example | 147 +++++++++++++++++++++++++++++++++++++ docs/getting-started.md | 2 +- docs/usage.md | 2 +- 3 files changed, 149 insertions(+), 2 deletions(-) create mode 100644 docs/backup.config-example diff --git a/docs/backup.config-example b/docs/backup.config-example new file mode 100644 index 000000000..552d04b01 --- /dev/null +++ b/docs/backup.config-example @@ -0,0 +1,147 @@ +# GitHub Enterprise Server backup configuration file + +# The hostname of the GitHub Enterprise Server appliance to back up. The host +# must be reachable via SSH from the backup host. +GHE_HOSTNAME="github.example.com" + +# Path to where backup data is stored. By default this is the "data" +# directory next to this file but can be set to an absolute path +# elsewhere for backing up to a separate partition / mount point. +GHE_DATA_DIR="data" + +# The number of backup snapshots to retain. Old snapshots are pruned after each +# successful ghe-backup run. This option should be tuned based on the frequency +# of scheduled backup runs. If backups are scheduled hourly, snapshots will be +# available for the past N hours; if backups are scheduled daily, snapshots will +# be available for the past N days ... +GHE_NUM_SNAPSHOTS=10 + +# Pruning snapshots can be scheduled outside of the backup process. +# If set to 'yes', snapshots will not be pruned by ghe-backup. +# Instead, ghe-prune-snapshots will need to be invoked separately via cron +#GHE_PRUNING_SCHEDULED=yes + +# If --incremental is used to generate incremental MySQL backups with ghe-backup, +# then you need to specify how many cycles of full and incremental backups will be +# performed before the next full backup is created. +# For example, if `GHE_INCREMENTAL_BACKUP_MAX` is set to 14, backup-utils will +# run 1 full backup and then 13 incremental backups before performing another full backup on the next cycle. +#GHE_INCREMENTAL_MAX_BACKUPS=14 + +# If GHE_ROUTE_VERIFICATION is set to true then ghe-repository-backup and +# ghe-storage-backup will issue a warning if the repositories and objects in +# the backup do not match the pre-backup inventory of routes. +#GHE_ROUTE_VERIFICATION=false + +# If GHE_MANAGE_CONSOLE_PW_RESTORE is set to false then management-console password +# will not be restored from backed-up snapshot data, it is restored by default +#GHE_MANAGE_CONSOLE_PW_RESTORE=true + +# If GHE_SKIP_CHECKS is set to true (or if --skip-checks is used with ghe-backup) then ghe-host-check +# disk space validation and software version checks on the backup-host will be disabled. +#GHE_SKIP_CHECKS=false + +# Cluster filesystem to check if it's writable as part of ghe-host-check +# By default it is /data/user/tmp but can be updated if needed +#GHE_FILE_SYSTEM_WRITE_CHECK="/data/user/tmp" + +# The hostname of the GitHub appliance to restore. If you've set up a separate +# GitHub appliance to act as a standby for recovery, specify its IP or hostname +# here. The host to restore to may also be specified directly when running +# ghe-restore so use of this variable isn't strictly required. +# +#GHE_RESTORE_HOST="github-standby.example.com" + +# If set to 'yes', ghe-restore will omit the restore of audit logs. +# +#GHE_RESTORE_SKIP_AUDIT_LOGS=no + +# If set to 'yes', backup and restore of Elasticsearch indices will be skipped +# +#GHE_SKIP_SEARCH_INDICES=no + +# When verbose output is enabled with `-v`, it's written to stdout by default. If +# you'd prefer it to be written to a separate file, set this option. +# +#GHE_VERBOSE_LOG="/var/log/backup-verbose.log" + +# Any extra options passed to the SSH command. +# In a single instance environment, nothing is required by default. +# In a clustering environment, "-i abs-path-to-ssh-private-key" is required. +# +#GHE_EXTRA_SSH_OPTS="" +# +# All backup processes are ran with the lowest priority for scheduling by default. +# To change throttling behaviour/allow higher priority for backup processes, set higher values for following variables. +# default value for GHENICE=nice -n 19 +# default value for GHE_IONICE=ionice -c 3 +#GHE_NICE="" +#GHE_IONICE="" + +# Any extra options passed to the rsync command. Nothing required by default. +# +#GHE_EXTRA_RSYNC_OPTS="" + +# If set to 'yes', rsync will be set to use compression during backups and restores transfers. Defaults to 'no'. +# +#GHE_RSYNC_COMPRESSION_ENABLED=yes + +# If enabled and set to 'no', rsync warning message during backups will be suppressed. +#RSYNC_WARNING=no + + +# If set to 'yes', logging output will be colorized. +# +#OUTPUT_COLOR=no + +# If set to 'no', GHE_DATA_DIR will not be created automatically +# and restore/backup will exit 8 +# +#GHE_CREATE_DATA_DIR=yes + +# If set to 'yes', git fsck will run on the repositories +# and print some additional info. +# +# WARNING: do not enable this, only useful for debugging/development +#GHE_BACKUP_FSCK=no + +# Cadence of MSSQL backups +# ,, all in minutes +# e.g. +# - Full backup every week (10080 minutes) +# - Differential backup every day (1440 minutes) +# - Transactionlog backup every 15 minutes +# +#GHE_MSSQL_BACKUP_CADENCE=10080,1440,15 + +# If set to 'yes', ghe-backup jobs will run in parallel. Defaults to 'no'. +# +#GHE_PARALLEL_ENABLED=yes + +# Sets the maximum number of jobs to run in parallel. Defaults to the number +# of available processing units on the machine. +# +#GHE_PARALLEL_MAX_JOBS=2 + +# Sets the maximum number of rsync jobs to run in parallel. Defaults to the +# configured GHE_PARALLEL_MAX_JOBS, or the number of available processing +# units on the machine. +# +# GHE_PARALLEL_RSYNC_MAX_JOBS=3 + +# When jobs are running in parallel wait as needed to avoid starting new jobs +# when the system's load average is not below the specified percentage. Defaults to +# unrestricted. +# +#GHE_PARALLEL_MAX_LOAD=50 + +# When running an external mysql database, run this script to trigger a MySQL backup +# rather than attempting to backup via backup-utils directly. +#EXTERNAL_DATABASE_BACKUP_SCRIPT="/bin/false" + +# When running an external mysql database, run this script to trigger a MySQL restore +# rather than attempting to backup via backup-utils directly. +#EXTERNAL_DATABASE_RESTORE_SCRIPT="/bin/false" + +# If set to 'yes', Pages data will be included in backup and restore. Defaults to 'yes' +#GHE_BACKUP_PAGES=no diff --git a/docs/getting-started.md b/docs/getting-started.md index ac85e5073..a272c796e 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -37,5 +37,5 @@ [1]: https://github.com/github/backup-utils/releases [2]: https://github.com/github/backup-utils/releases/tag/v2.11.4 -[3]: https://github.com/github/enterprise-backup-site/blob/master/backup.config-example +[3]: https://github.com/github/enterprise-backup-site/blob/master/docs/backup.config-example [4]: https://docs.github.com/enterprise-server/admin/configuration/configuring-your-enterprise/accessing-the-administrative-shell-ssh diff --git a/docs/usage.md b/docs/usage.md index 4324b58cc..33960eca7 100644 --- a/docs/usage.md +++ b/docs/usage.md @@ -13,7 +13,7 @@ These commands are run on the host you [installed][1] Backup Utilities on. You can supply your own configuration file or use the example configuration file as a template where you can set up your environment for backing up and restoring. -An example configuration file with documentation on possible settings can found in [backup.config-example](../backup.config-example). +An example configuration file with documentation on possible settings can found in [backup.config-example](backup.config-example). There are a number of command-line options that can also be passed to the `ghe-restore` command. Of particular note, if you use an external MySQL service but are restoring from a snapshot prior to enabling this, or vice versa, you must migrate the MySQL data outside of the context of backup-utils first, then pass the `--skip-mysql` flag to `ghe-restore`. From 8a6db2aaf523d992ca1960e538a8b0b340ba4e7c Mon Sep 17 00:00:00 2001 From: "release-controller[bot]" <110195724+release-controller[bot]@users.noreply.github.com> Date: Tue, 22 Apr 2025 20:07:08 +0000 Subject: [PATCH 30/47] 3.15.3 release --- docs/backup.config-example | 147 ------------------------------------- docs/getting-started.md | 2 +- docs/usage.md | 2 +- 3 files changed, 2 insertions(+), 149 deletions(-) delete mode 100644 docs/backup.config-example diff --git a/docs/backup.config-example b/docs/backup.config-example deleted file mode 100644 index 552d04b01..000000000 --- a/docs/backup.config-example +++ /dev/null @@ -1,147 +0,0 @@ -# GitHub Enterprise Server backup configuration file - -# The hostname of the GitHub Enterprise Server appliance to back up. The host -# must be reachable via SSH from the backup host. -GHE_HOSTNAME="github.example.com" - -# Path to where backup data is stored. By default this is the "data" -# directory next to this file but can be set to an absolute path -# elsewhere for backing up to a separate partition / mount point. -GHE_DATA_DIR="data" - -# The number of backup snapshots to retain. Old snapshots are pruned after each -# successful ghe-backup run. This option should be tuned based on the frequency -# of scheduled backup runs. If backups are scheduled hourly, snapshots will be -# available for the past N hours; if backups are scheduled daily, snapshots will -# be available for the past N days ... -GHE_NUM_SNAPSHOTS=10 - -# Pruning snapshots can be scheduled outside of the backup process. -# If set to 'yes', snapshots will not be pruned by ghe-backup. -# Instead, ghe-prune-snapshots will need to be invoked separately via cron -#GHE_PRUNING_SCHEDULED=yes - -# If --incremental is used to generate incremental MySQL backups with ghe-backup, -# then you need to specify how many cycles of full and incremental backups will be -# performed before the next full backup is created. -# For example, if `GHE_INCREMENTAL_BACKUP_MAX` is set to 14, backup-utils will -# run 1 full backup and then 13 incremental backups before performing another full backup on the next cycle. -#GHE_INCREMENTAL_MAX_BACKUPS=14 - -# If GHE_ROUTE_VERIFICATION is set to true then ghe-repository-backup and -# ghe-storage-backup will issue a warning if the repositories and objects in -# the backup do not match the pre-backup inventory of routes. -#GHE_ROUTE_VERIFICATION=false - -# If GHE_MANAGE_CONSOLE_PW_RESTORE is set to false then management-console password -# will not be restored from backed-up snapshot data, it is restored by default -#GHE_MANAGE_CONSOLE_PW_RESTORE=true - -# If GHE_SKIP_CHECKS is set to true (or if --skip-checks is used with ghe-backup) then ghe-host-check -# disk space validation and software version checks on the backup-host will be disabled. -#GHE_SKIP_CHECKS=false - -# Cluster filesystem to check if it's writable as part of ghe-host-check -# By default it is /data/user/tmp but can be updated if needed -#GHE_FILE_SYSTEM_WRITE_CHECK="/data/user/tmp" - -# The hostname of the GitHub appliance to restore. If you've set up a separate -# GitHub appliance to act as a standby for recovery, specify its IP or hostname -# here. The host to restore to may also be specified directly when running -# ghe-restore so use of this variable isn't strictly required. -# -#GHE_RESTORE_HOST="github-standby.example.com" - -# If set to 'yes', ghe-restore will omit the restore of audit logs. -# -#GHE_RESTORE_SKIP_AUDIT_LOGS=no - -# If set to 'yes', backup and restore of Elasticsearch indices will be skipped -# -#GHE_SKIP_SEARCH_INDICES=no - -# When verbose output is enabled with `-v`, it's written to stdout by default. If -# you'd prefer it to be written to a separate file, set this option. -# -#GHE_VERBOSE_LOG="/var/log/backup-verbose.log" - -# Any extra options passed to the SSH command. -# In a single instance environment, nothing is required by default. -# In a clustering environment, "-i abs-path-to-ssh-private-key" is required. -# -#GHE_EXTRA_SSH_OPTS="" -# -# All backup processes are ran with the lowest priority for scheduling by default. -# To change throttling behaviour/allow higher priority for backup processes, set higher values for following variables. -# default value for GHENICE=nice -n 19 -# default value for GHE_IONICE=ionice -c 3 -#GHE_NICE="" -#GHE_IONICE="" - -# Any extra options passed to the rsync command. Nothing required by default. -# -#GHE_EXTRA_RSYNC_OPTS="" - -# If set to 'yes', rsync will be set to use compression during backups and restores transfers. Defaults to 'no'. -# -#GHE_RSYNC_COMPRESSION_ENABLED=yes - -# If enabled and set to 'no', rsync warning message during backups will be suppressed. -#RSYNC_WARNING=no - - -# If set to 'yes', logging output will be colorized. -# -#OUTPUT_COLOR=no - -# If set to 'no', GHE_DATA_DIR will not be created automatically -# and restore/backup will exit 8 -# -#GHE_CREATE_DATA_DIR=yes - -# If set to 'yes', git fsck will run on the repositories -# and print some additional info. -# -# WARNING: do not enable this, only useful for debugging/development -#GHE_BACKUP_FSCK=no - -# Cadence of MSSQL backups -# ,, all in minutes -# e.g. -# - Full backup every week (10080 minutes) -# - Differential backup every day (1440 minutes) -# - Transactionlog backup every 15 minutes -# -#GHE_MSSQL_BACKUP_CADENCE=10080,1440,15 - -# If set to 'yes', ghe-backup jobs will run in parallel. Defaults to 'no'. -# -#GHE_PARALLEL_ENABLED=yes - -# Sets the maximum number of jobs to run in parallel. Defaults to the number -# of available processing units on the machine. -# -#GHE_PARALLEL_MAX_JOBS=2 - -# Sets the maximum number of rsync jobs to run in parallel. Defaults to the -# configured GHE_PARALLEL_MAX_JOBS, or the number of available processing -# units on the machine. -# -# GHE_PARALLEL_RSYNC_MAX_JOBS=3 - -# When jobs are running in parallel wait as needed to avoid starting new jobs -# when the system's load average is not below the specified percentage. Defaults to -# unrestricted. -# -#GHE_PARALLEL_MAX_LOAD=50 - -# When running an external mysql database, run this script to trigger a MySQL backup -# rather than attempting to backup via backup-utils directly. -#EXTERNAL_DATABASE_BACKUP_SCRIPT="/bin/false" - -# When running an external mysql database, run this script to trigger a MySQL restore -# rather than attempting to backup via backup-utils directly. -#EXTERNAL_DATABASE_RESTORE_SCRIPT="/bin/false" - -# If set to 'yes', Pages data will be included in backup and restore. Defaults to 'yes' -#GHE_BACKUP_PAGES=no diff --git a/docs/getting-started.md b/docs/getting-started.md index a272c796e..ac85e5073 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -37,5 +37,5 @@ [1]: https://github.com/github/backup-utils/releases [2]: https://github.com/github/backup-utils/releases/tag/v2.11.4 -[3]: https://github.com/github/enterprise-backup-site/blob/master/docs/backup.config-example +[3]: https://github.com/github/enterprise-backup-site/blob/master/backup.config-example [4]: https://docs.github.com/enterprise-server/admin/configuration/configuring-your-enterprise/accessing-the-administrative-shell-ssh diff --git a/docs/usage.md b/docs/usage.md index 33960eca7..4324b58cc 100644 --- a/docs/usage.md +++ b/docs/usage.md @@ -13,7 +13,7 @@ These commands are run on the host you [installed][1] Backup Utilities on. You can supply your own configuration file or use the example configuration file as a template where you can set up your environment for backing up and restoring. -An example configuration file with documentation on possible settings can found in [backup.config-example](backup.config-example). +An example configuration file with documentation on possible settings can found in [backup.config-example](../backup.config-example). There are a number of command-line options that can also be passed to the `ghe-restore` command. Of particular note, if you use an external MySQL service but are restoring from a snapshot prior to enabling this, or vice versa, you must migrate the MySQL data outside of the context of backup-utils first, then pass the `--skip-mysql` flag to `ghe-restore`. From a0d3fe624c2ad695c2de6aeed31e645f031e0647 Mon Sep 17 00:00:00 2001 From: "release-controller[bot]" <110195724+release-controller[bot]@users.noreply.github.com> Date: Tue, 22 Apr 2025 20:42:57 +0000 Subject: [PATCH 31/47] 3.14.4 release From 9e26501c41ab8b97a652bd7c7aa07ef8645ecec2 Mon Sep 17 00:00:00 2001 From: "release-controller[bot]" <110195724+release-controller[bot]@users.noreply.github.com> Date: Thu, 15 May 2025 22:37:16 +0000 Subject: [PATCH 32/47] 3.17.0 release --- docs/backup.config-example | 140 ++++++++++++++++++ docs/getting-started.md | 2 +- .../incremental-mysql-backups-and-restores.md | 38 ----- docs/usage.md | 5 +- 4 files changed, 143 insertions(+), 42 deletions(-) create mode 100644 docs/backup.config-example delete mode 100644 docs/incremental-mysql-backups-and-restores.md diff --git a/docs/backup.config-example b/docs/backup.config-example new file mode 100644 index 000000000..39a4d417f --- /dev/null +++ b/docs/backup.config-example @@ -0,0 +1,140 @@ +# GitHub Enterprise Server backup configuration file + +# The hostname of the GitHub Enterprise Server appliance to back up. The host +# must be reachable via SSH from the backup host. +GHE_HOSTNAME="github.example.com" + +# Path to where backup data is stored. By default this is the "data" +# directory next to this file but can be set to an absolute path +# elsewhere for backing up to a separate partition / mount point. +GHE_DATA_DIR="data" + +# The number of backup snapshots to retain. Old snapshots are pruned after each +# successful ghe-backup run. This option should be tuned based on the frequency +# of scheduled backup runs. If backups are scheduled hourly, snapshots will be +# available for the past N hours; if backups are scheduled daily, snapshots will +# be available for the past N days ... +GHE_NUM_SNAPSHOTS=10 + +# Pruning snapshots can be scheduled outside of the backup process. +# If set to 'yes', snapshots will not be pruned by ghe-backup. +# Instead, ghe-prune-snapshots will need to be invoked separately via cron +#GHE_PRUNING_SCHEDULED=yes + +# If GHE_ROUTE_VERIFICATION is set to true then ghe-repository-backup and +# ghe-storage-backup will issue a warning if the repositories and objects in +# the backup do not match the pre-backup inventory of routes. +#GHE_ROUTE_VERIFICATION=false + +# If GHE_MANAGE_CONSOLE_PW_RESTORE is set to false then management-console password +# will not be restored from backed-up snapshot data, it is restored by default +#GHE_MANAGE_CONSOLE_PW_RESTORE=true + +# If GHE_SKIP_CHECKS is set to true (or if --skip-checks is used with ghe-backup) then ghe-host-check +# disk space validation and software version checks on the backup-host will be disabled. +#GHE_SKIP_CHECKS=false + +# Cluster filesystem to check if it's writable as part of ghe-host-check +# By default it is /data/user/tmp but can be updated if needed +#GHE_FILE_SYSTEM_WRITE_CHECK="/data/user/tmp" + +# The hostname of the GitHub appliance to restore. If you've set up a separate +# GitHub appliance to act as a standby for recovery, specify its IP or hostname +# here. The host to restore to may also be specified directly when running +# ghe-restore so use of this variable isn't strictly required. +# +#GHE_RESTORE_HOST="github-standby.example.com" + +# If set to 'yes', ghe-restore will omit the restore of audit logs. +# +#GHE_RESTORE_SKIP_AUDIT_LOGS=no + +# If set to 'yes', backup and restore of Elasticsearch indices will be skipped +# +#GHE_SKIP_SEARCH_INDICES=no + +# When verbose output is enabled with `-v`, it's written to stdout by default. If +# you'd prefer it to be written to a separate file, set this option. +# +#GHE_VERBOSE_LOG="/var/log/backup-verbose.log" + +# Any extra options passed to the SSH command. +# In a single instance environment, nothing is required by default. +# In a clustering environment, "-i abs-path-to-ssh-private-key" is required. +# +#GHE_EXTRA_SSH_OPTS="" +# +# All backup processes are ran with the lowest priority for scheduling by default. +# To change throttling behaviour/allow higher priority for backup processes, set higher values for following variables. +# default value for GHENICE=nice -n 19 +# default value for GHE_IONICE=ionice -c 3 +#GHE_NICE="" +#GHE_IONICE="" + +# Any extra options passed to the rsync command. Nothing required by default. +# +#GHE_EXTRA_RSYNC_OPTS="" + +# If set to 'yes', rsync will be set to use compression during backups and restores transfers. Defaults to 'no'. +# +#GHE_RSYNC_COMPRESSION_ENABLED=yes + +# If enabled and set to 'no', rsync warning message during backups will be suppressed. +#RSYNC_WARNING=no + + +# If set to 'yes', logging output will be colorized. +# +#OUTPUT_COLOR=no + +# If set to 'no', GHE_DATA_DIR will not be created automatically +# and restore/backup will exit 8 +# +#GHE_CREATE_DATA_DIR=yes + +# If set to 'yes', git fsck will run on the repositories +# and print some additional info. +# +# WARNING: do not enable this, only useful for debugging/development +#GHE_BACKUP_FSCK=no + +# Cadence of MSSQL backups +# ,, all in minutes +# e.g. +# - Full backup every week (10080 minutes) +# - Differential backup every day (1440 minutes) +# - Transactionlog backup every 15 minutes +# +#GHE_MSSQL_BACKUP_CADENCE=10080,1440,15 + +# If set to 'yes', ghe-backup jobs will run in parallel. Defaults to 'no'. +# +#GHE_PARALLEL_ENABLED=yes + +# Sets the maximum number of jobs to run in parallel. Defaults to the number +# of available processing units on the machine. +# +#GHE_PARALLEL_MAX_JOBS=2 + +# Sets the maximum number of rsync jobs to run in parallel. Defaults to the +# configured GHE_PARALLEL_MAX_JOBS, or the number of available processing +# units on the machine. +# +# GHE_PARALLEL_RSYNC_MAX_JOBS=3 + +# When jobs are running in parallel wait as needed to avoid starting new jobs +# when the system's load average is not below the specified percentage. Defaults to +# unrestricted. +# +#GHE_PARALLEL_MAX_LOAD=50 + +# When running an external mysql database, run this script to trigger a MySQL backup +# rather than attempting to backup via backup-utils directly. +#EXTERNAL_DATABASE_BACKUP_SCRIPT="/bin/false" + +# When running an external mysql database, run this script to trigger a MySQL restore +# rather than attempting to backup via backup-utils directly. +#EXTERNAL_DATABASE_RESTORE_SCRIPT="/bin/false" + +# If set to 'yes', Pages data will be included in backup and restore. Defaults to 'yes' +#GHE_BACKUP_PAGES=no diff --git a/docs/getting-started.md b/docs/getting-started.md index ac85e5073..f8806e616 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -37,5 +37,5 @@ [1]: https://github.com/github/backup-utils/releases [2]: https://github.com/github/backup-utils/releases/tag/v2.11.4 -[3]: https://github.com/github/enterprise-backup-site/blob/master/backup.config-example +[3]: https://github.com/github/backup-utils/blob/master/docs/backup.config-example [4]: https://docs.github.com/enterprise-server/admin/configuration/configuring-your-enterprise/accessing-the-administrative-shell-ssh diff --git a/docs/incremental-mysql-backups-and-restores.md b/docs/incremental-mysql-backups-and-restores.md deleted file mode 100644 index 53543c927..000000000 --- a/docs/incremental-mysql-backups-and-restores.md +++ /dev/null @@ -1,38 +0,0 @@ -# Incremental MySQL Backups and Restores - -Customers who have large MySQL databases who wish to save storage space can use the `--incremental` flag with `ghe-backup` and `ghe-restore`. -Using this flag performs backups for other parts of GHES as normal, but only performs a MySQL backup of the changes to the database from the previous snapshot. -For larger databases this can conserve a lot of storage space for backups. - -## Configuring number of backups - -In your backup.config file you will need to set the variable `GHE_INCREMENTAL_MAX_BACKUPS`. -This variable determines how many cycles of full and incremental backups will be performed before the next full backup is created. -For example, if `GHE_INCREMENTAL_MAX_BACKUPS` is set to 14, backup-utils will run 1 full backup and then 13 incremental backups before performing another full backup on the next cycle. - -Incremental backups require the previous snapshot backups before them to work. -This means they do not follow the pruning strategy based on `GHE_NUM_SNAPSHOTS`. - -## Performing incremental backups - -To perform incremental backups: - -`bin/ghe-backup --incremental` - -the program will detect whether it needs to performa full or incremental snapshot based on what is currently in `GHE_DATA_DIR`. - -To see what snapshots are part of your full and incremental backups, you can reference `GHE_DATA_DIR/inc_full_backup` and `GHE_DATA_DIR/inc_snapshot_data`, respectively. - -## Performing incremental restores - -To perform incremental restores: - -`bin/ghe-restore --incremental -s ` - -The program will use the MySQL folders from each previous incremental backup and the full backup to restore the database. - -:warning: Incremental restores require the other snapshots in the cycle to complete a restore. Erasing snapshot directories that are part of a cycle corrupts the restore and makes it impossible to restore for the MySQL database. - -### Previous cycles - -To ensure there is a rolling window of mySQL backups, incremental MySQL backups from the cycle before the current one are kept. Those snapshots are pre-pended with `inc_previous`. To perform a restore from there, just use the full directory name for the snapshot ID. diff --git a/docs/usage.md b/docs/usage.md index 4324b58cc..58a816bc8 100644 --- a/docs/usage.md +++ b/docs/usage.md @@ -13,7 +13,7 @@ These commands are run on the host you [installed][1] Backup Utilities on. You can supply your own configuration file or use the example configuration file as a template where you can set up your environment for backing up and restoring. -An example configuration file with documentation on possible settings can found in [backup.config-example](../backup.config-example). +An example configuration file with documentation on possible settings can found in [backup.config-example](backup.config-example). There are a number of command-line options that can also be passed to the `ghe-restore` command. Of particular note, if you use an external MySQL service but are restoring from a snapshot prior to enabling this, or vice versa, you must migrate the MySQL data outside of the context of backup-utils first, then pass the `--skip-mysql` flag to `ghe-restore`. @@ -112,8 +112,7 @@ Please refer to [GHES Documentation](https://docs.github.com/en/enterprise-serve ## Incremental MySQL Backups and Restores -If you are interested in performing incremental backups of the MySQL data in your GitHub Enterprise Server instance, see [Incremental MySQL Backups and Restores](incremental-mysql-backups-and-restores.md) for details. - +Incremental MySQL backup has been deprecated since 3.17 due to data integrity concerns. Restoring backups created with incremental backups remains supported for compatibility reasons. ## Rsync compression From backup-utils v3.11.0 onwards, we have disabled rsync compression by default to improve transfer speed and reduce CPU usage during the transfer process. From b2102577a1c4bab1058e470b751f2d7447427734 Mon Sep 17 00:00:00 2001 From: "release-controller[bot]" <110195724+release-controller[bot]@users.noreply.github.com> Date: Fri, 20 Jun 2025 19:43:33 +0000 Subject: [PATCH 33/47] 3.17.1 release From c1e16868a6196254a135c2ff82cd2bc5a7481e6d Mon Sep 17 00:00:00 2001 From: "release-controller[bot]" <110195724+release-controller[bot]@users.noreply.github.com> Date: Fri, 20 Jun 2025 19:49:25 +0000 Subject: [PATCH 34/47] 3.13.5 release --- docs/backup.config-example | 140 ------------------ docs/getting-started.md | 2 +- .../incremental-mysql-backups-and-restores.md | 38 +++++ docs/usage.md | 5 +- 4 files changed, 42 insertions(+), 143 deletions(-) delete mode 100644 docs/backup.config-example create mode 100644 docs/incremental-mysql-backups-and-restores.md diff --git a/docs/backup.config-example b/docs/backup.config-example deleted file mode 100644 index 39a4d417f..000000000 --- a/docs/backup.config-example +++ /dev/null @@ -1,140 +0,0 @@ -# GitHub Enterprise Server backup configuration file - -# The hostname of the GitHub Enterprise Server appliance to back up. The host -# must be reachable via SSH from the backup host. -GHE_HOSTNAME="github.example.com" - -# Path to where backup data is stored. By default this is the "data" -# directory next to this file but can be set to an absolute path -# elsewhere for backing up to a separate partition / mount point. -GHE_DATA_DIR="data" - -# The number of backup snapshots to retain. Old snapshots are pruned after each -# successful ghe-backup run. This option should be tuned based on the frequency -# of scheduled backup runs. If backups are scheduled hourly, snapshots will be -# available for the past N hours; if backups are scheduled daily, snapshots will -# be available for the past N days ... -GHE_NUM_SNAPSHOTS=10 - -# Pruning snapshots can be scheduled outside of the backup process. -# If set to 'yes', snapshots will not be pruned by ghe-backup. -# Instead, ghe-prune-snapshots will need to be invoked separately via cron -#GHE_PRUNING_SCHEDULED=yes - -# If GHE_ROUTE_VERIFICATION is set to true then ghe-repository-backup and -# ghe-storage-backup will issue a warning if the repositories and objects in -# the backup do not match the pre-backup inventory of routes. -#GHE_ROUTE_VERIFICATION=false - -# If GHE_MANAGE_CONSOLE_PW_RESTORE is set to false then management-console password -# will not be restored from backed-up snapshot data, it is restored by default -#GHE_MANAGE_CONSOLE_PW_RESTORE=true - -# If GHE_SKIP_CHECKS is set to true (or if --skip-checks is used with ghe-backup) then ghe-host-check -# disk space validation and software version checks on the backup-host will be disabled. -#GHE_SKIP_CHECKS=false - -# Cluster filesystem to check if it's writable as part of ghe-host-check -# By default it is /data/user/tmp but can be updated if needed -#GHE_FILE_SYSTEM_WRITE_CHECK="/data/user/tmp" - -# The hostname of the GitHub appliance to restore. If you've set up a separate -# GitHub appliance to act as a standby for recovery, specify its IP or hostname -# here. The host to restore to may also be specified directly when running -# ghe-restore so use of this variable isn't strictly required. -# -#GHE_RESTORE_HOST="github-standby.example.com" - -# If set to 'yes', ghe-restore will omit the restore of audit logs. -# -#GHE_RESTORE_SKIP_AUDIT_LOGS=no - -# If set to 'yes', backup and restore of Elasticsearch indices will be skipped -# -#GHE_SKIP_SEARCH_INDICES=no - -# When verbose output is enabled with `-v`, it's written to stdout by default. If -# you'd prefer it to be written to a separate file, set this option. -# -#GHE_VERBOSE_LOG="/var/log/backup-verbose.log" - -# Any extra options passed to the SSH command. -# In a single instance environment, nothing is required by default. -# In a clustering environment, "-i abs-path-to-ssh-private-key" is required. -# -#GHE_EXTRA_SSH_OPTS="" -# -# All backup processes are ran with the lowest priority for scheduling by default. -# To change throttling behaviour/allow higher priority for backup processes, set higher values for following variables. -# default value for GHENICE=nice -n 19 -# default value for GHE_IONICE=ionice -c 3 -#GHE_NICE="" -#GHE_IONICE="" - -# Any extra options passed to the rsync command. Nothing required by default. -# -#GHE_EXTRA_RSYNC_OPTS="" - -# If set to 'yes', rsync will be set to use compression during backups and restores transfers. Defaults to 'no'. -# -#GHE_RSYNC_COMPRESSION_ENABLED=yes - -# If enabled and set to 'no', rsync warning message during backups will be suppressed. -#RSYNC_WARNING=no - - -# If set to 'yes', logging output will be colorized. -# -#OUTPUT_COLOR=no - -# If set to 'no', GHE_DATA_DIR will not be created automatically -# and restore/backup will exit 8 -# -#GHE_CREATE_DATA_DIR=yes - -# If set to 'yes', git fsck will run on the repositories -# and print some additional info. -# -# WARNING: do not enable this, only useful for debugging/development -#GHE_BACKUP_FSCK=no - -# Cadence of MSSQL backups -# ,, all in minutes -# e.g. -# - Full backup every week (10080 minutes) -# - Differential backup every day (1440 minutes) -# - Transactionlog backup every 15 minutes -# -#GHE_MSSQL_BACKUP_CADENCE=10080,1440,15 - -# If set to 'yes', ghe-backup jobs will run in parallel. Defaults to 'no'. -# -#GHE_PARALLEL_ENABLED=yes - -# Sets the maximum number of jobs to run in parallel. Defaults to the number -# of available processing units on the machine. -# -#GHE_PARALLEL_MAX_JOBS=2 - -# Sets the maximum number of rsync jobs to run in parallel. Defaults to the -# configured GHE_PARALLEL_MAX_JOBS, or the number of available processing -# units on the machine. -# -# GHE_PARALLEL_RSYNC_MAX_JOBS=3 - -# When jobs are running in parallel wait as needed to avoid starting new jobs -# when the system's load average is not below the specified percentage. Defaults to -# unrestricted. -# -#GHE_PARALLEL_MAX_LOAD=50 - -# When running an external mysql database, run this script to trigger a MySQL backup -# rather than attempting to backup via backup-utils directly. -#EXTERNAL_DATABASE_BACKUP_SCRIPT="/bin/false" - -# When running an external mysql database, run this script to trigger a MySQL restore -# rather than attempting to backup via backup-utils directly. -#EXTERNAL_DATABASE_RESTORE_SCRIPT="/bin/false" - -# If set to 'yes', Pages data will be included in backup and restore. Defaults to 'yes' -#GHE_BACKUP_PAGES=no diff --git a/docs/getting-started.md b/docs/getting-started.md index f8806e616..ac85e5073 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -37,5 +37,5 @@ [1]: https://github.com/github/backup-utils/releases [2]: https://github.com/github/backup-utils/releases/tag/v2.11.4 -[3]: https://github.com/github/backup-utils/blob/master/docs/backup.config-example +[3]: https://github.com/github/enterprise-backup-site/blob/master/backup.config-example [4]: https://docs.github.com/enterprise-server/admin/configuration/configuring-your-enterprise/accessing-the-administrative-shell-ssh diff --git a/docs/incremental-mysql-backups-and-restores.md b/docs/incremental-mysql-backups-and-restores.md new file mode 100644 index 000000000..53543c927 --- /dev/null +++ b/docs/incremental-mysql-backups-and-restores.md @@ -0,0 +1,38 @@ +# Incremental MySQL Backups and Restores + +Customers who have large MySQL databases who wish to save storage space can use the `--incremental` flag with `ghe-backup` and `ghe-restore`. +Using this flag performs backups for other parts of GHES as normal, but only performs a MySQL backup of the changes to the database from the previous snapshot. +For larger databases this can conserve a lot of storage space for backups. + +## Configuring number of backups + +In your backup.config file you will need to set the variable `GHE_INCREMENTAL_MAX_BACKUPS`. +This variable determines how many cycles of full and incremental backups will be performed before the next full backup is created. +For example, if `GHE_INCREMENTAL_MAX_BACKUPS` is set to 14, backup-utils will run 1 full backup and then 13 incremental backups before performing another full backup on the next cycle. + +Incremental backups require the previous snapshot backups before them to work. +This means they do not follow the pruning strategy based on `GHE_NUM_SNAPSHOTS`. + +## Performing incremental backups + +To perform incremental backups: + +`bin/ghe-backup --incremental` + +the program will detect whether it needs to performa full or incremental snapshot based on what is currently in `GHE_DATA_DIR`. + +To see what snapshots are part of your full and incremental backups, you can reference `GHE_DATA_DIR/inc_full_backup` and `GHE_DATA_DIR/inc_snapshot_data`, respectively. + +## Performing incremental restores + +To perform incremental restores: + +`bin/ghe-restore --incremental -s ` + +The program will use the MySQL folders from each previous incremental backup and the full backup to restore the database. + +:warning: Incremental restores require the other snapshots in the cycle to complete a restore. Erasing snapshot directories that are part of a cycle corrupts the restore and makes it impossible to restore for the MySQL database. + +### Previous cycles + +To ensure there is a rolling window of mySQL backups, incremental MySQL backups from the cycle before the current one are kept. Those snapshots are pre-pended with `inc_previous`. To perform a restore from there, just use the full directory name for the snapshot ID. diff --git a/docs/usage.md b/docs/usage.md index 58a816bc8..4324b58cc 100644 --- a/docs/usage.md +++ b/docs/usage.md @@ -13,7 +13,7 @@ These commands are run on the host you [installed][1] Backup Utilities on. You can supply your own configuration file or use the example configuration file as a template where you can set up your environment for backing up and restoring. -An example configuration file with documentation on possible settings can found in [backup.config-example](backup.config-example). +An example configuration file with documentation on possible settings can found in [backup.config-example](../backup.config-example). There are a number of command-line options that can also be passed to the `ghe-restore` command. Of particular note, if you use an external MySQL service but are restoring from a snapshot prior to enabling this, or vice versa, you must migrate the MySQL data outside of the context of backup-utils first, then pass the `--skip-mysql` flag to `ghe-restore`. @@ -112,7 +112,8 @@ Please refer to [GHES Documentation](https://docs.github.com/en/enterprise-serve ## Incremental MySQL Backups and Restores -Incremental MySQL backup has been deprecated since 3.17 due to data integrity concerns. Restoring backups created with incremental backups remains supported for compatibility reasons. +If you are interested in performing incremental backups of the MySQL data in your GitHub Enterprise Server instance, see [Incremental MySQL Backups and Restores](incremental-mysql-backups-and-restores.md) for details. + ## Rsync compression From backup-utils v3.11.0 onwards, we have disabled rsync compression by default to improve transfer speed and reduce CPU usage during the transfer process. From 1b1660d2d685577e408e22ddb32b105ad8a818d2 Mon Sep 17 00:00:00 2001 From: "release-controller[bot]" <110195724+release-controller[bot]@users.noreply.github.com> Date: Fri, 20 Jun 2025 19:50:28 +0000 Subject: [PATCH 35/47] 3.15.4 release From 7db4882e4eff398fa779b68f0cda74d1ebf86086 Mon Sep 17 00:00:00 2001 From: "release-controller[bot]" <110195724+release-controller[bot]@users.noreply.github.com> Date: Fri, 20 Jun 2025 19:50:39 +0000 Subject: [PATCH 36/47] 3.16.3 release --- docs/backup.config-example | 147 +++++++++++++++++++++++++++++++++++++ docs/getting-started.md | 2 +- docs/usage.md | 2 +- 3 files changed, 149 insertions(+), 2 deletions(-) create mode 100644 docs/backup.config-example diff --git a/docs/backup.config-example b/docs/backup.config-example new file mode 100644 index 000000000..552d04b01 --- /dev/null +++ b/docs/backup.config-example @@ -0,0 +1,147 @@ +# GitHub Enterprise Server backup configuration file + +# The hostname of the GitHub Enterprise Server appliance to back up. The host +# must be reachable via SSH from the backup host. +GHE_HOSTNAME="github.example.com" + +# Path to where backup data is stored. By default this is the "data" +# directory next to this file but can be set to an absolute path +# elsewhere for backing up to a separate partition / mount point. +GHE_DATA_DIR="data" + +# The number of backup snapshots to retain. Old snapshots are pruned after each +# successful ghe-backup run. This option should be tuned based on the frequency +# of scheduled backup runs. If backups are scheduled hourly, snapshots will be +# available for the past N hours; if backups are scheduled daily, snapshots will +# be available for the past N days ... +GHE_NUM_SNAPSHOTS=10 + +# Pruning snapshots can be scheduled outside of the backup process. +# If set to 'yes', snapshots will not be pruned by ghe-backup. +# Instead, ghe-prune-snapshots will need to be invoked separately via cron +#GHE_PRUNING_SCHEDULED=yes + +# If --incremental is used to generate incremental MySQL backups with ghe-backup, +# then you need to specify how many cycles of full and incremental backups will be +# performed before the next full backup is created. +# For example, if `GHE_INCREMENTAL_BACKUP_MAX` is set to 14, backup-utils will +# run 1 full backup and then 13 incremental backups before performing another full backup on the next cycle. +#GHE_INCREMENTAL_MAX_BACKUPS=14 + +# If GHE_ROUTE_VERIFICATION is set to true then ghe-repository-backup and +# ghe-storage-backup will issue a warning if the repositories and objects in +# the backup do not match the pre-backup inventory of routes. +#GHE_ROUTE_VERIFICATION=false + +# If GHE_MANAGE_CONSOLE_PW_RESTORE is set to false then management-console password +# will not be restored from backed-up snapshot data, it is restored by default +#GHE_MANAGE_CONSOLE_PW_RESTORE=true + +# If GHE_SKIP_CHECKS is set to true (or if --skip-checks is used with ghe-backup) then ghe-host-check +# disk space validation and software version checks on the backup-host will be disabled. +#GHE_SKIP_CHECKS=false + +# Cluster filesystem to check if it's writable as part of ghe-host-check +# By default it is /data/user/tmp but can be updated if needed +#GHE_FILE_SYSTEM_WRITE_CHECK="/data/user/tmp" + +# The hostname of the GitHub appliance to restore. If you've set up a separate +# GitHub appliance to act as a standby for recovery, specify its IP or hostname +# here. The host to restore to may also be specified directly when running +# ghe-restore so use of this variable isn't strictly required. +# +#GHE_RESTORE_HOST="github-standby.example.com" + +# If set to 'yes', ghe-restore will omit the restore of audit logs. +# +#GHE_RESTORE_SKIP_AUDIT_LOGS=no + +# If set to 'yes', backup and restore of Elasticsearch indices will be skipped +# +#GHE_SKIP_SEARCH_INDICES=no + +# When verbose output is enabled with `-v`, it's written to stdout by default. If +# you'd prefer it to be written to a separate file, set this option. +# +#GHE_VERBOSE_LOG="/var/log/backup-verbose.log" + +# Any extra options passed to the SSH command. +# In a single instance environment, nothing is required by default. +# In a clustering environment, "-i abs-path-to-ssh-private-key" is required. +# +#GHE_EXTRA_SSH_OPTS="" +# +# All backup processes are ran with the lowest priority for scheduling by default. +# To change throttling behaviour/allow higher priority for backup processes, set higher values for following variables. +# default value for GHENICE=nice -n 19 +# default value for GHE_IONICE=ionice -c 3 +#GHE_NICE="" +#GHE_IONICE="" + +# Any extra options passed to the rsync command. Nothing required by default. +# +#GHE_EXTRA_RSYNC_OPTS="" + +# If set to 'yes', rsync will be set to use compression during backups and restores transfers. Defaults to 'no'. +# +#GHE_RSYNC_COMPRESSION_ENABLED=yes + +# If enabled and set to 'no', rsync warning message during backups will be suppressed. +#RSYNC_WARNING=no + + +# If set to 'yes', logging output will be colorized. +# +#OUTPUT_COLOR=no + +# If set to 'no', GHE_DATA_DIR will not be created automatically +# and restore/backup will exit 8 +# +#GHE_CREATE_DATA_DIR=yes + +# If set to 'yes', git fsck will run on the repositories +# and print some additional info. +# +# WARNING: do not enable this, only useful for debugging/development +#GHE_BACKUP_FSCK=no + +# Cadence of MSSQL backups +# ,, all in minutes +# e.g. +# - Full backup every week (10080 minutes) +# - Differential backup every day (1440 minutes) +# - Transactionlog backup every 15 minutes +# +#GHE_MSSQL_BACKUP_CADENCE=10080,1440,15 + +# If set to 'yes', ghe-backup jobs will run in parallel. Defaults to 'no'. +# +#GHE_PARALLEL_ENABLED=yes + +# Sets the maximum number of jobs to run in parallel. Defaults to the number +# of available processing units on the machine. +# +#GHE_PARALLEL_MAX_JOBS=2 + +# Sets the maximum number of rsync jobs to run in parallel. Defaults to the +# configured GHE_PARALLEL_MAX_JOBS, or the number of available processing +# units on the machine. +# +# GHE_PARALLEL_RSYNC_MAX_JOBS=3 + +# When jobs are running in parallel wait as needed to avoid starting new jobs +# when the system's load average is not below the specified percentage. Defaults to +# unrestricted. +# +#GHE_PARALLEL_MAX_LOAD=50 + +# When running an external mysql database, run this script to trigger a MySQL backup +# rather than attempting to backup via backup-utils directly. +#EXTERNAL_DATABASE_BACKUP_SCRIPT="/bin/false" + +# When running an external mysql database, run this script to trigger a MySQL restore +# rather than attempting to backup via backup-utils directly. +#EXTERNAL_DATABASE_RESTORE_SCRIPT="/bin/false" + +# If set to 'yes', Pages data will be included in backup and restore. Defaults to 'yes' +#GHE_BACKUP_PAGES=no diff --git a/docs/getting-started.md b/docs/getting-started.md index ac85e5073..a272c796e 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -37,5 +37,5 @@ [1]: https://github.com/github/backup-utils/releases [2]: https://github.com/github/backup-utils/releases/tag/v2.11.4 -[3]: https://github.com/github/enterprise-backup-site/blob/master/backup.config-example +[3]: https://github.com/github/enterprise-backup-site/blob/master/docs/backup.config-example [4]: https://docs.github.com/enterprise-server/admin/configuration/configuring-your-enterprise/accessing-the-administrative-shell-ssh diff --git a/docs/usage.md b/docs/usage.md index 4324b58cc..33960eca7 100644 --- a/docs/usage.md +++ b/docs/usage.md @@ -13,7 +13,7 @@ These commands are run on the host you [installed][1] Backup Utilities on. You can supply your own configuration file or use the example configuration file as a template where you can set up your environment for backing up and restoring. -An example configuration file with documentation on possible settings can found in [backup.config-example](../backup.config-example). +An example configuration file with documentation on possible settings can found in [backup.config-example](backup.config-example). There are a number of command-line options that can also be passed to the `ghe-restore` command. Of particular note, if you use an external MySQL service but are restoring from a snapshot prior to enabling this, or vice versa, you must migrate the MySQL data outside of the context of backup-utils first, then pass the `--skip-mysql` flag to `ghe-restore`. From f4b19f82a3b81f7ef17aecc4375da214cee43517 Mon Sep 17 00:00:00 2001 From: "release-controller[bot]" <110195724+release-controller[bot]@users.noreply.github.com> Date: Fri, 20 Jun 2025 19:51:42 +0000 Subject: [PATCH 37/47] 3.14.5 release --- docs/backup.config-example | 147 ------------------------------------- docs/getting-started.md | 2 +- docs/usage.md | 2 +- 3 files changed, 2 insertions(+), 149 deletions(-) delete mode 100644 docs/backup.config-example diff --git a/docs/backup.config-example b/docs/backup.config-example deleted file mode 100644 index 552d04b01..000000000 --- a/docs/backup.config-example +++ /dev/null @@ -1,147 +0,0 @@ -# GitHub Enterprise Server backup configuration file - -# The hostname of the GitHub Enterprise Server appliance to back up. The host -# must be reachable via SSH from the backup host. -GHE_HOSTNAME="github.example.com" - -# Path to where backup data is stored. By default this is the "data" -# directory next to this file but can be set to an absolute path -# elsewhere for backing up to a separate partition / mount point. -GHE_DATA_DIR="data" - -# The number of backup snapshots to retain. Old snapshots are pruned after each -# successful ghe-backup run. This option should be tuned based on the frequency -# of scheduled backup runs. If backups are scheduled hourly, snapshots will be -# available for the past N hours; if backups are scheduled daily, snapshots will -# be available for the past N days ... -GHE_NUM_SNAPSHOTS=10 - -# Pruning snapshots can be scheduled outside of the backup process. -# If set to 'yes', snapshots will not be pruned by ghe-backup. -# Instead, ghe-prune-snapshots will need to be invoked separately via cron -#GHE_PRUNING_SCHEDULED=yes - -# If --incremental is used to generate incremental MySQL backups with ghe-backup, -# then you need to specify how many cycles of full and incremental backups will be -# performed before the next full backup is created. -# For example, if `GHE_INCREMENTAL_BACKUP_MAX` is set to 14, backup-utils will -# run 1 full backup and then 13 incremental backups before performing another full backup on the next cycle. -#GHE_INCREMENTAL_MAX_BACKUPS=14 - -# If GHE_ROUTE_VERIFICATION is set to true then ghe-repository-backup and -# ghe-storage-backup will issue a warning if the repositories and objects in -# the backup do not match the pre-backup inventory of routes. -#GHE_ROUTE_VERIFICATION=false - -# If GHE_MANAGE_CONSOLE_PW_RESTORE is set to false then management-console password -# will not be restored from backed-up snapshot data, it is restored by default -#GHE_MANAGE_CONSOLE_PW_RESTORE=true - -# If GHE_SKIP_CHECKS is set to true (or if --skip-checks is used with ghe-backup) then ghe-host-check -# disk space validation and software version checks on the backup-host will be disabled. -#GHE_SKIP_CHECKS=false - -# Cluster filesystem to check if it's writable as part of ghe-host-check -# By default it is /data/user/tmp but can be updated if needed -#GHE_FILE_SYSTEM_WRITE_CHECK="/data/user/tmp" - -# The hostname of the GitHub appliance to restore. If you've set up a separate -# GitHub appliance to act as a standby for recovery, specify its IP or hostname -# here. The host to restore to may also be specified directly when running -# ghe-restore so use of this variable isn't strictly required. -# -#GHE_RESTORE_HOST="github-standby.example.com" - -# If set to 'yes', ghe-restore will omit the restore of audit logs. -# -#GHE_RESTORE_SKIP_AUDIT_LOGS=no - -# If set to 'yes', backup and restore of Elasticsearch indices will be skipped -# -#GHE_SKIP_SEARCH_INDICES=no - -# When verbose output is enabled with `-v`, it's written to stdout by default. If -# you'd prefer it to be written to a separate file, set this option. -# -#GHE_VERBOSE_LOG="/var/log/backup-verbose.log" - -# Any extra options passed to the SSH command. -# In a single instance environment, nothing is required by default. -# In a clustering environment, "-i abs-path-to-ssh-private-key" is required. -# -#GHE_EXTRA_SSH_OPTS="" -# -# All backup processes are ran with the lowest priority for scheduling by default. -# To change throttling behaviour/allow higher priority for backup processes, set higher values for following variables. -# default value for GHENICE=nice -n 19 -# default value for GHE_IONICE=ionice -c 3 -#GHE_NICE="" -#GHE_IONICE="" - -# Any extra options passed to the rsync command. Nothing required by default. -# -#GHE_EXTRA_RSYNC_OPTS="" - -# If set to 'yes', rsync will be set to use compression during backups and restores transfers. Defaults to 'no'. -# -#GHE_RSYNC_COMPRESSION_ENABLED=yes - -# If enabled and set to 'no', rsync warning message during backups will be suppressed. -#RSYNC_WARNING=no - - -# If set to 'yes', logging output will be colorized. -# -#OUTPUT_COLOR=no - -# If set to 'no', GHE_DATA_DIR will not be created automatically -# and restore/backup will exit 8 -# -#GHE_CREATE_DATA_DIR=yes - -# If set to 'yes', git fsck will run on the repositories -# and print some additional info. -# -# WARNING: do not enable this, only useful for debugging/development -#GHE_BACKUP_FSCK=no - -# Cadence of MSSQL backups -# ,, all in minutes -# e.g. -# - Full backup every week (10080 minutes) -# - Differential backup every day (1440 minutes) -# - Transactionlog backup every 15 minutes -# -#GHE_MSSQL_BACKUP_CADENCE=10080,1440,15 - -# If set to 'yes', ghe-backup jobs will run in parallel. Defaults to 'no'. -# -#GHE_PARALLEL_ENABLED=yes - -# Sets the maximum number of jobs to run in parallel. Defaults to the number -# of available processing units on the machine. -# -#GHE_PARALLEL_MAX_JOBS=2 - -# Sets the maximum number of rsync jobs to run in parallel. Defaults to the -# configured GHE_PARALLEL_MAX_JOBS, or the number of available processing -# units on the machine. -# -# GHE_PARALLEL_RSYNC_MAX_JOBS=3 - -# When jobs are running in parallel wait as needed to avoid starting new jobs -# when the system's load average is not below the specified percentage. Defaults to -# unrestricted. -# -#GHE_PARALLEL_MAX_LOAD=50 - -# When running an external mysql database, run this script to trigger a MySQL backup -# rather than attempting to backup via backup-utils directly. -#EXTERNAL_DATABASE_BACKUP_SCRIPT="/bin/false" - -# When running an external mysql database, run this script to trigger a MySQL restore -# rather than attempting to backup via backup-utils directly. -#EXTERNAL_DATABASE_RESTORE_SCRIPT="/bin/false" - -# If set to 'yes', Pages data will be included in backup and restore. Defaults to 'yes' -#GHE_BACKUP_PAGES=no diff --git a/docs/getting-started.md b/docs/getting-started.md index a272c796e..ac85e5073 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -37,5 +37,5 @@ [1]: https://github.com/github/backup-utils/releases [2]: https://github.com/github/backup-utils/releases/tag/v2.11.4 -[3]: https://github.com/github/enterprise-backup-site/blob/master/docs/backup.config-example +[3]: https://github.com/github/enterprise-backup-site/blob/master/backup.config-example [4]: https://docs.github.com/enterprise-server/admin/configuration/configuring-your-enterprise/accessing-the-administrative-shell-ssh diff --git a/docs/usage.md b/docs/usage.md index 33960eca7..4324b58cc 100644 --- a/docs/usage.md +++ b/docs/usage.md @@ -13,7 +13,7 @@ These commands are run on the host you [installed][1] Backup Utilities on. You can supply your own configuration file or use the example configuration file as a template where you can set up your environment for backing up and restoring. -An example configuration file with documentation on possible settings can found in [backup.config-example](backup.config-example). +An example configuration file with documentation on possible settings can found in [backup.config-example](../backup.config-example). There are a number of command-line options that can also be passed to the `ghe-restore` command. Of particular note, if you use an external MySQL service but are restoring from a snapshot prior to enabling this, or vice versa, you must migrate the MySQL data outside of the context of backup-utils first, then pass the `--skip-mysql` flag to `ghe-restore`. From b71896fe873d46e28cafb9c39dba7283ba07c77f Mon Sep 17 00:00:00 2001 From: "release-controller[bot]" <110195724+release-controller[bot]@users.noreply.github.com> Date: Sat, 21 Jun 2025 02:11:42 +0000 Subject: [PATCH 38/47] 3.16.3 release --- docs/backup.config-example | 147 +++++++++++++++++++++++++++++++++++++ docs/getting-started.md | 2 +- docs/usage.md | 2 +- 3 files changed, 149 insertions(+), 2 deletions(-) create mode 100644 docs/backup.config-example diff --git a/docs/backup.config-example b/docs/backup.config-example new file mode 100644 index 000000000..552d04b01 --- /dev/null +++ b/docs/backup.config-example @@ -0,0 +1,147 @@ +# GitHub Enterprise Server backup configuration file + +# The hostname of the GitHub Enterprise Server appliance to back up. The host +# must be reachable via SSH from the backup host. +GHE_HOSTNAME="github.example.com" + +# Path to where backup data is stored. By default this is the "data" +# directory next to this file but can be set to an absolute path +# elsewhere for backing up to a separate partition / mount point. +GHE_DATA_DIR="data" + +# The number of backup snapshots to retain. Old snapshots are pruned after each +# successful ghe-backup run. This option should be tuned based on the frequency +# of scheduled backup runs. If backups are scheduled hourly, snapshots will be +# available for the past N hours; if backups are scheduled daily, snapshots will +# be available for the past N days ... +GHE_NUM_SNAPSHOTS=10 + +# Pruning snapshots can be scheduled outside of the backup process. +# If set to 'yes', snapshots will not be pruned by ghe-backup. +# Instead, ghe-prune-snapshots will need to be invoked separately via cron +#GHE_PRUNING_SCHEDULED=yes + +# If --incremental is used to generate incremental MySQL backups with ghe-backup, +# then you need to specify how many cycles of full and incremental backups will be +# performed before the next full backup is created. +# For example, if `GHE_INCREMENTAL_BACKUP_MAX` is set to 14, backup-utils will +# run 1 full backup and then 13 incremental backups before performing another full backup on the next cycle. +#GHE_INCREMENTAL_MAX_BACKUPS=14 + +# If GHE_ROUTE_VERIFICATION is set to true then ghe-repository-backup and +# ghe-storage-backup will issue a warning if the repositories and objects in +# the backup do not match the pre-backup inventory of routes. +#GHE_ROUTE_VERIFICATION=false + +# If GHE_MANAGE_CONSOLE_PW_RESTORE is set to false then management-console password +# will not be restored from backed-up snapshot data, it is restored by default +#GHE_MANAGE_CONSOLE_PW_RESTORE=true + +# If GHE_SKIP_CHECKS is set to true (or if --skip-checks is used with ghe-backup) then ghe-host-check +# disk space validation and software version checks on the backup-host will be disabled. +#GHE_SKIP_CHECKS=false + +# Cluster filesystem to check if it's writable as part of ghe-host-check +# By default it is /data/user/tmp but can be updated if needed +#GHE_FILE_SYSTEM_WRITE_CHECK="/data/user/tmp" + +# The hostname of the GitHub appliance to restore. If you've set up a separate +# GitHub appliance to act as a standby for recovery, specify its IP or hostname +# here. The host to restore to may also be specified directly when running +# ghe-restore so use of this variable isn't strictly required. +# +#GHE_RESTORE_HOST="github-standby.example.com" + +# If set to 'yes', ghe-restore will omit the restore of audit logs. +# +#GHE_RESTORE_SKIP_AUDIT_LOGS=no + +# If set to 'yes', backup and restore of Elasticsearch indices will be skipped +# +#GHE_SKIP_SEARCH_INDICES=no + +# When verbose output is enabled with `-v`, it's written to stdout by default. If +# you'd prefer it to be written to a separate file, set this option. +# +#GHE_VERBOSE_LOG="/var/log/backup-verbose.log" + +# Any extra options passed to the SSH command. +# In a single instance environment, nothing is required by default. +# In a clustering environment, "-i abs-path-to-ssh-private-key" is required. +# +#GHE_EXTRA_SSH_OPTS="" +# +# All backup processes are ran with the lowest priority for scheduling by default. +# To change throttling behaviour/allow higher priority for backup processes, set higher values for following variables. +# default value for GHENICE=nice -n 19 +# default value for GHE_IONICE=ionice -c 3 +#GHE_NICE="" +#GHE_IONICE="" + +# Any extra options passed to the rsync command. Nothing required by default. +# +#GHE_EXTRA_RSYNC_OPTS="" + +# If set to 'yes', rsync will be set to use compression during backups and restores transfers. Defaults to 'no'. +# +#GHE_RSYNC_COMPRESSION_ENABLED=yes + +# If enabled and set to 'no', rsync warning message during backups will be suppressed. +#RSYNC_WARNING=no + + +# If set to 'yes', logging output will be colorized. +# +#OUTPUT_COLOR=no + +# If set to 'no', GHE_DATA_DIR will not be created automatically +# and restore/backup will exit 8 +# +#GHE_CREATE_DATA_DIR=yes + +# If set to 'yes', git fsck will run on the repositories +# and print some additional info. +# +# WARNING: do not enable this, only useful for debugging/development +#GHE_BACKUP_FSCK=no + +# Cadence of MSSQL backups +# ,, all in minutes +# e.g. +# - Full backup every week (10080 minutes) +# - Differential backup every day (1440 minutes) +# - Transactionlog backup every 15 minutes +# +#GHE_MSSQL_BACKUP_CADENCE=10080,1440,15 + +# If set to 'yes', ghe-backup jobs will run in parallel. Defaults to 'no'. +# +#GHE_PARALLEL_ENABLED=yes + +# Sets the maximum number of jobs to run in parallel. Defaults to the number +# of available processing units on the machine. +# +#GHE_PARALLEL_MAX_JOBS=2 + +# Sets the maximum number of rsync jobs to run in parallel. Defaults to the +# configured GHE_PARALLEL_MAX_JOBS, or the number of available processing +# units on the machine. +# +# GHE_PARALLEL_RSYNC_MAX_JOBS=3 + +# When jobs are running in parallel wait as needed to avoid starting new jobs +# when the system's load average is not below the specified percentage. Defaults to +# unrestricted. +# +#GHE_PARALLEL_MAX_LOAD=50 + +# When running an external mysql database, run this script to trigger a MySQL backup +# rather than attempting to backup via backup-utils directly. +#EXTERNAL_DATABASE_BACKUP_SCRIPT="/bin/false" + +# When running an external mysql database, run this script to trigger a MySQL restore +# rather than attempting to backup via backup-utils directly. +#EXTERNAL_DATABASE_RESTORE_SCRIPT="/bin/false" + +# If set to 'yes', Pages data will be included in backup and restore. Defaults to 'yes' +#GHE_BACKUP_PAGES=no diff --git a/docs/getting-started.md b/docs/getting-started.md index ac85e5073..a272c796e 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -37,5 +37,5 @@ [1]: https://github.com/github/backup-utils/releases [2]: https://github.com/github/backup-utils/releases/tag/v2.11.4 -[3]: https://github.com/github/enterprise-backup-site/blob/master/backup.config-example +[3]: https://github.com/github/enterprise-backup-site/blob/master/docs/backup.config-example [4]: https://docs.github.com/enterprise-server/admin/configuration/configuring-your-enterprise/accessing-the-administrative-shell-ssh diff --git a/docs/usage.md b/docs/usage.md index 4324b58cc..33960eca7 100644 --- a/docs/usage.md +++ b/docs/usage.md @@ -13,7 +13,7 @@ These commands are run on the host you [installed][1] Backup Utilities on. You can supply your own configuration file or use the example configuration file as a template where you can set up your environment for backing up and restoring. -An example configuration file with documentation on possible settings can found in [backup.config-example](../backup.config-example). +An example configuration file with documentation on possible settings can found in [backup.config-example](backup.config-example). There are a number of command-line options that can also be passed to the `ghe-restore` command. Of particular note, if you use an external MySQL service but are restoring from a snapshot prior to enabling this, or vice versa, you must migrate the MySQL data outside of the context of backup-utils first, then pass the `--skip-mysql` flag to `ghe-restore`. From b3ed0204fe0df56279f09f43f2d9e2e5a58576f3 Mon Sep 17 00:00:00 2001 From: "release-controller[bot]" <110195724+release-controller[bot]@users.noreply.github.com> Date: Sat, 21 Jun 2025 02:11:48 +0000 Subject: [PATCH 39/47] 3.15.4 release --- docs/backup.config-example | 147 ------------------------------------- docs/getting-started.md | 2 +- docs/usage.md | 2 +- 3 files changed, 2 insertions(+), 149 deletions(-) delete mode 100644 docs/backup.config-example diff --git a/docs/backup.config-example b/docs/backup.config-example deleted file mode 100644 index 552d04b01..000000000 --- a/docs/backup.config-example +++ /dev/null @@ -1,147 +0,0 @@ -# GitHub Enterprise Server backup configuration file - -# The hostname of the GitHub Enterprise Server appliance to back up. The host -# must be reachable via SSH from the backup host. -GHE_HOSTNAME="github.example.com" - -# Path to where backup data is stored. By default this is the "data" -# directory next to this file but can be set to an absolute path -# elsewhere for backing up to a separate partition / mount point. -GHE_DATA_DIR="data" - -# The number of backup snapshots to retain. Old snapshots are pruned after each -# successful ghe-backup run. This option should be tuned based on the frequency -# of scheduled backup runs. If backups are scheduled hourly, snapshots will be -# available for the past N hours; if backups are scheduled daily, snapshots will -# be available for the past N days ... -GHE_NUM_SNAPSHOTS=10 - -# Pruning snapshots can be scheduled outside of the backup process. -# If set to 'yes', snapshots will not be pruned by ghe-backup. -# Instead, ghe-prune-snapshots will need to be invoked separately via cron -#GHE_PRUNING_SCHEDULED=yes - -# If --incremental is used to generate incremental MySQL backups with ghe-backup, -# then you need to specify how many cycles of full and incremental backups will be -# performed before the next full backup is created. -# For example, if `GHE_INCREMENTAL_BACKUP_MAX` is set to 14, backup-utils will -# run 1 full backup and then 13 incremental backups before performing another full backup on the next cycle. -#GHE_INCREMENTAL_MAX_BACKUPS=14 - -# If GHE_ROUTE_VERIFICATION is set to true then ghe-repository-backup and -# ghe-storage-backup will issue a warning if the repositories and objects in -# the backup do not match the pre-backup inventory of routes. -#GHE_ROUTE_VERIFICATION=false - -# If GHE_MANAGE_CONSOLE_PW_RESTORE is set to false then management-console password -# will not be restored from backed-up snapshot data, it is restored by default -#GHE_MANAGE_CONSOLE_PW_RESTORE=true - -# If GHE_SKIP_CHECKS is set to true (or if --skip-checks is used with ghe-backup) then ghe-host-check -# disk space validation and software version checks on the backup-host will be disabled. -#GHE_SKIP_CHECKS=false - -# Cluster filesystem to check if it's writable as part of ghe-host-check -# By default it is /data/user/tmp but can be updated if needed -#GHE_FILE_SYSTEM_WRITE_CHECK="/data/user/tmp" - -# The hostname of the GitHub appliance to restore. If you've set up a separate -# GitHub appliance to act as a standby for recovery, specify its IP or hostname -# here. The host to restore to may also be specified directly when running -# ghe-restore so use of this variable isn't strictly required. -# -#GHE_RESTORE_HOST="github-standby.example.com" - -# If set to 'yes', ghe-restore will omit the restore of audit logs. -# -#GHE_RESTORE_SKIP_AUDIT_LOGS=no - -# If set to 'yes', backup and restore of Elasticsearch indices will be skipped -# -#GHE_SKIP_SEARCH_INDICES=no - -# When verbose output is enabled with `-v`, it's written to stdout by default. If -# you'd prefer it to be written to a separate file, set this option. -# -#GHE_VERBOSE_LOG="/var/log/backup-verbose.log" - -# Any extra options passed to the SSH command. -# In a single instance environment, nothing is required by default. -# In a clustering environment, "-i abs-path-to-ssh-private-key" is required. -# -#GHE_EXTRA_SSH_OPTS="" -# -# All backup processes are ran with the lowest priority for scheduling by default. -# To change throttling behaviour/allow higher priority for backup processes, set higher values for following variables. -# default value for GHENICE=nice -n 19 -# default value for GHE_IONICE=ionice -c 3 -#GHE_NICE="" -#GHE_IONICE="" - -# Any extra options passed to the rsync command. Nothing required by default. -# -#GHE_EXTRA_RSYNC_OPTS="" - -# If set to 'yes', rsync will be set to use compression during backups and restores transfers. Defaults to 'no'. -# -#GHE_RSYNC_COMPRESSION_ENABLED=yes - -# If enabled and set to 'no', rsync warning message during backups will be suppressed. -#RSYNC_WARNING=no - - -# If set to 'yes', logging output will be colorized. -# -#OUTPUT_COLOR=no - -# If set to 'no', GHE_DATA_DIR will not be created automatically -# and restore/backup will exit 8 -# -#GHE_CREATE_DATA_DIR=yes - -# If set to 'yes', git fsck will run on the repositories -# and print some additional info. -# -# WARNING: do not enable this, only useful for debugging/development -#GHE_BACKUP_FSCK=no - -# Cadence of MSSQL backups -# ,, all in minutes -# e.g. -# - Full backup every week (10080 minutes) -# - Differential backup every day (1440 minutes) -# - Transactionlog backup every 15 minutes -# -#GHE_MSSQL_BACKUP_CADENCE=10080,1440,15 - -# If set to 'yes', ghe-backup jobs will run in parallel. Defaults to 'no'. -# -#GHE_PARALLEL_ENABLED=yes - -# Sets the maximum number of jobs to run in parallel. Defaults to the number -# of available processing units on the machine. -# -#GHE_PARALLEL_MAX_JOBS=2 - -# Sets the maximum number of rsync jobs to run in parallel. Defaults to the -# configured GHE_PARALLEL_MAX_JOBS, or the number of available processing -# units on the machine. -# -# GHE_PARALLEL_RSYNC_MAX_JOBS=3 - -# When jobs are running in parallel wait as needed to avoid starting new jobs -# when the system's load average is not below the specified percentage. Defaults to -# unrestricted. -# -#GHE_PARALLEL_MAX_LOAD=50 - -# When running an external mysql database, run this script to trigger a MySQL backup -# rather than attempting to backup via backup-utils directly. -#EXTERNAL_DATABASE_BACKUP_SCRIPT="/bin/false" - -# When running an external mysql database, run this script to trigger a MySQL restore -# rather than attempting to backup via backup-utils directly. -#EXTERNAL_DATABASE_RESTORE_SCRIPT="/bin/false" - -# If set to 'yes', Pages data will be included in backup and restore. Defaults to 'yes' -#GHE_BACKUP_PAGES=no diff --git a/docs/getting-started.md b/docs/getting-started.md index a272c796e..ac85e5073 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -37,5 +37,5 @@ [1]: https://github.com/github/backup-utils/releases [2]: https://github.com/github/backup-utils/releases/tag/v2.11.4 -[3]: https://github.com/github/enterprise-backup-site/blob/master/docs/backup.config-example +[3]: https://github.com/github/enterprise-backup-site/blob/master/backup.config-example [4]: https://docs.github.com/enterprise-server/admin/configuration/configuring-your-enterprise/accessing-the-administrative-shell-ssh diff --git a/docs/usage.md b/docs/usage.md index 33960eca7..4324b58cc 100644 --- a/docs/usage.md +++ b/docs/usage.md @@ -13,7 +13,7 @@ These commands are run on the host you [installed][1] Backup Utilities on. You can supply your own configuration file or use the example configuration file as a template where you can set up your environment for backing up and restoring. -An example configuration file with documentation on possible settings can found in [backup.config-example](backup.config-example). +An example configuration file with documentation on possible settings can found in [backup.config-example](../backup.config-example). There are a number of command-line options that can also be passed to the `ghe-restore` command. Of particular note, if you use an external MySQL service but are restoring from a snapshot prior to enabling this, or vice versa, you must migrate the MySQL data outside of the context of backup-utils first, then pass the `--skip-mysql` flag to `ghe-restore`. From bee107428b8dc4729b302a8fec32348ab3c12d58 Mon Sep 17 00:00:00 2001 From: "release-controller[bot]" <110195724+release-controller[bot]@users.noreply.github.com> Date: Sat, 21 Jun 2025 02:12:33 +0000 Subject: [PATCH 40/47] 3.13.5 release From c825304b85712256bcd1866998fa3161b7a3e901 Mon Sep 17 00:00:00 2001 From: "release-controller[bot]" <110195724+release-controller[bot]@users.noreply.github.com> Date: Sat, 21 Jun 2025 02:13:41 +0000 Subject: [PATCH 41/47] 3.14.5 release From 7f67a61ab35c586a6394dda49e026e4acbb33ea6 Mon Sep 17 00:00:00 2001 From: "release-controller[bot]" <110195724+release-controller[bot]@users.noreply.github.com> Date: Wed, 27 Aug 2025 19:15:47 +0000 Subject: [PATCH 42/47] 3.17.2 release --- docs/backup.config-example | 140 ++++++++++++++++++ docs/getting-started.md | 2 +- .../incremental-mysql-backups-and-restores.md | 38 ----- docs/usage.md | 5 +- 4 files changed, 143 insertions(+), 42 deletions(-) create mode 100644 docs/backup.config-example delete mode 100644 docs/incremental-mysql-backups-and-restores.md diff --git a/docs/backup.config-example b/docs/backup.config-example new file mode 100644 index 000000000..39a4d417f --- /dev/null +++ b/docs/backup.config-example @@ -0,0 +1,140 @@ +# GitHub Enterprise Server backup configuration file + +# The hostname of the GitHub Enterprise Server appliance to back up. The host +# must be reachable via SSH from the backup host. +GHE_HOSTNAME="github.example.com" + +# Path to where backup data is stored. By default this is the "data" +# directory next to this file but can be set to an absolute path +# elsewhere for backing up to a separate partition / mount point. +GHE_DATA_DIR="data" + +# The number of backup snapshots to retain. Old snapshots are pruned after each +# successful ghe-backup run. This option should be tuned based on the frequency +# of scheduled backup runs. If backups are scheduled hourly, snapshots will be +# available for the past N hours; if backups are scheduled daily, snapshots will +# be available for the past N days ... +GHE_NUM_SNAPSHOTS=10 + +# Pruning snapshots can be scheduled outside of the backup process. +# If set to 'yes', snapshots will not be pruned by ghe-backup. +# Instead, ghe-prune-snapshots will need to be invoked separately via cron +#GHE_PRUNING_SCHEDULED=yes + +# If GHE_ROUTE_VERIFICATION is set to true then ghe-repository-backup and +# ghe-storage-backup will issue a warning if the repositories and objects in +# the backup do not match the pre-backup inventory of routes. +#GHE_ROUTE_VERIFICATION=false + +# If GHE_MANAGE_CONSOLE_PW_RESTORE is set to false then management-console password +# will not be restored from backed-up snapshot data, it is restored by default +#GHE_MANAGE_CONSOLE_PW_RESTORE=true + +# If GHE_SKIP_CHECKS is set to true (or if --skip-checks is used with ghe-backup) then ghe-host-check +# disk space validation and software version checks on the backup-host will be disabled. +#GHE_SKIP_CHECKS=false + +# Cluster filesystem to check if it's writable as part of ghe-host-check +# By default it is /data/user/tmp but can be updated if needed +#GHE_FILE_SYSTEM_WRITE_CHECK="/data/user/tmp" + +# The hostname of the GitHub appliance to restore. If you've set up a separate +# GitHub appliance to act as a standby for recovery, specify its IP or hostname +# here. The host to restore to may also be specified directly when running +# ghe-restore so use of this variable isn't strictly required. +# +#GHE_RESTORE_HOST="github-standby.example.com" + +# If set to 'yes', ghe-restore will omit the restore of audit logs. +# +#GHE_RESTORE_SKIP_AUDIT_LOGS=no + +# If set to 'yes', backup and restore of Elasticsearch indices will be skipped +# +#GHE_SKIP_SEARCH_INDICES=no + +# When verbose output is enabled with `-v`, it's written to stdout by default. If +# you'd prefer it to be written to a separate file, set this option. +# +#GHE_VERBOSE_LOG="/var/log/backup-verbose.log" + +# Any extra options passed to the SSH command. +# In a single instance environment, nothing is required by default. +# In a clustering environment, "-i abs-path-to-ssh-private-key" is required. +# +#GHE_EXTRA_SSH_OPTS="" +# +# All backup processes are ran with the lowest priority for scheduling by default. +# To change throttling behaviour/allow higher priority for backup processes, set higher values for following variables. +# default value for GHENICE=nice -n 19 +# default value for GHE_IONICE=ionice -c 3 +#GHE_NICE="" +#GHE_IONICE="" + +# Any extra options passed to the rsync command. Nothing required by default. +# +#GHE_EXTRA_RSYNC_OPTS="" + +# If set to 'yes', rsync will be set to use compression during backups and restores transfers. Defaults to 'no'. +# +#GHE_RSYNC_COMPRESSION_ENABLED=yes + +# If enabled and set to 'no', rsync warning message during backups will be suppressed. +#RSYNC_WARNING=no + + +# If set to 'yes', logging output will be colorized. +# +#OUTPUT_COLOR=no + +# If set to 'no', GHE_DATA_DIR will not be created automatically +# and restore/backup will exit 8 +# +#GHE_CREATE_DATA_DIR=yes + +# If set to 'yes', git fsck will run on the repositories +# and print some additional info. +# +# WARNING: do not enable this, only useful for debugging/development +#GHE_BACKUP_FSCK=no + +# Cadence of MSSQL backups +# ,, all in minutes +# e.g. +# - Full backup every week (10080 minutes) +# - Differential backup every day (1440 minutes) +# - Transactionlog backup every 15 minutes +# +#GHE_MSSQL_BACKUP_CADENCE=10080,1440,15 + +# If set to 'yes', ghe-backup jobs will run in parallel. Defaults to 'no'. +# +#GHE_PARALLEL_ENABLED=yes + +# Sets the maximum number of jobs to run in parallel. Defaults to the number +# of available processing units on the machine. +# +#GHE_PARALLEL_MAX_JOBS=2 + +# Sets the maximum number of rsync jobs to run in parallel. Defaults to the +# configured GHE_PARALLEL_MAX_JOBS, or the number of available processing +# units on the machine. +# +# GHE_PARALLEL_RSYNC_MAX_JOBS=3 + +# When jobs are running in parallel wait as needed to avoid starting new jobs +# when the system's load average is not below the specified percentage. Defaults to +# unrestricted. +# +#GHE_PARALLEL_MAX_LOAD=50 + +# When running an external mysql database, run this script to trigger a MySQL backup +# rather than attempting to backup via backup-utils directly. +#EXTERNAL_DATABASE_BACKUP_SCRIPT="/bin/false" + +# When running an external mysql database, run this script to trigger a MySQL restore +# rather than attempting to backup via backup-utils directly. +#EXTERNAL_DATABASE_RESTORE_SCRIPT="/bin/false" + +# If set to 'yes', Pages data will be included in backup and restore. Defaults to 'yes' +#GHE_BACKUP_PAGES=no diff --git a/docs/getting-started.md b/docs/getting-started.md index ac85e5073..f8806e616 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -37,5 +37,5 @@ [1]: https://github.com/github/backup-utils/releases [2]: https://github.com/github/backup-utils/releases/tag/v2.11.4 -[3]: https://github.com/github/enterprise-backup-site/blob/master/backup.config-example +[3]: https://github.com/github/backup-utils/blob/master/docs/backup.config-example [4]: https://docs.github.com/enterprise-server/admin/configuration/configuring-your-enterprise/accessing-the-administrative-shell-ssh diff --git a/docs/incremental-mysql-backups-and-restores.md b/docs/incremental-mysql-backups-and-restores.md deleted file mode 100644 index 53543c927..000000000 --- a/docs/incremental-mysql-backups-and-restores.md +++ /dev/null @@ -1,38 +0,0 @@ -# Incremental MySQL Backups and Restores - -Customers who have large MySQL databases who wish to save storage space can use the `--incremental` flag with `ghe-backup` and `ghe-restore`. -Using this flag performs backups for other parts of GHES as normal, but only performs a MySQL backup of the changes to the database from the previous snapshot. -For larger databases this can conserve a lot of storage space for backups. - -## Configuring number of backups - -In your backup.config file you will need to set the variable `GHE_INCREMENTAL_MAX_BACKUPS`. -This variable determines how many cycles of full and incremental backups will be performed before the next full backup is created. -For example, if `GHE_INCREMENTAL_MAX_BACKUPS` is set to 14, backup-utils will run 1 full backup and then 13 incremental backups before performing another full backup on the next cycle. - -Incremental backups require the previous snapshot backups before them to work. -This means they do not follow the pruning strategy based on `GHE_NUM_SNAPSHOTS`. - -## Performing incremental backups - -To perform incremental backups: - -`bin/ghe-backup --incremental` - -the program will detect whether it needs to performa full or incremental snapshot based on what is currently in `GHE_DATA_DIR`. - -To see what snapshots are part of your full and incremental backups, you can reference `GHE_DATA_DIR/inc_full_backup` and `GHE_DATA_DIR/inc_snapshot_data`, respectively. - -## Performing incremental restores - -To perform incremental restores: - -`bin/ghe-restore --incremental -s ` - -The program will use the MySQL folders from each previous incremental backup and the full backup to restore the database. - -:warning: Incremental restores require the other snapshots in the cycle to complete a restore. Erasing snapshot directories that are part of a cycle corrupts the restore and makes it impossible to restore for the MySQL database. - -### Previous cycles - -To ensure there is a rolling window of mySQL backups, incremental MySQL backups from the cycle before the current one are kept. Those snapshots are pre-pended with `inc_previous`. To perform a restore from there, just use the full directory name for the snapshot ID. diff --git a/docs/usage.md b/docs/usage.md index 4324b58cc..58a816bc8 100644 --- a/docs/usage.md +++ b/docs/usage.md @@ -13,7 +13,7 @@ These commands are run on the host you [installed][1] Backup Utilities on. You can supply your own configuration file or use the example configuration file as a template where you can set up your environment for backing up and restoring. -An example configuration file with documentation on possible settings can found in [backup.config-example](../backup.config-example). +An example configuration file with documentation on possible settings can found in [backup.config-example](backup.config-example). There are a number of command-line options that can also be passed to the `ghe-restore` command. Of particular note, if you use an external MySQL service but are restoring from a snapshot prior to enabling this, or vice versa, you must migrate the MySQL data outside of the context of backup-utils first, then pass the `--skip-mysql` flag to `ghe-restore`. @@ -112,8 +112,7 @@ Please refer to [GHES Documentation](https://docs.github.com/en/enterprise-serve ## Incremental MySQL Backups and Restores -If you are interested in performing incremental backups of the MySQL data in your GitHub Enterprise Server instance, see [Incremental MySQL Backups and Restores](incremental-mysql-backups-and-restores.md) for details. - +Incremental MySQL backup has been deprecated since 3.17 due to data integrity concerns. Restoring backups created with incremental backups remains supported for compatibility reasons. ## Rsync compression From backup-utils v3.11.0 onwards, we have disabled rsync compression by default to improve transfer speed and reduce CPU usage during the transfer process. From e6c6101e6922f1b2c74328b83453fccc80d9f5c7 Mon Sep 17 00:00:00 2001 From: "release-controller[bot]" <110195724+release-controller[bot]@users.noreply.github.com> Date: Tue, 14 Oct 2025 21:53:32 +0000 Subject: [PATCH 43/47] 3.18.0 release From 6caf749e801cd3589f41fd6cd3da16e6a101f1e1 Mon Sep 17 00:00:00 2001 From: "release-controller[bot]" <110195724+release-controller[bot]@users.noreply.github.com> Date: Wed, 15 Oct 2025 01:23:33 +0000 Subject: [PATCH 44/47] 3.18.0 release From b39101e730cb8d3088b8cf77c309c8ec3a13ddc7 Mon Sep 17 00:00:00 2001 From: "release-controller[bot]" <110195724+release-controller[bot]@users.noreply.github.com> Date: Thu, 16 Oct 2025 00:07:23 +0000 Subject: [PATCH 45/47] 3.18.0 release From 85135ac32b9fe7c4e6ae30e4169207d3f1534974 Mon Sep 17 00:00:00 2001 From: "release-controller[bot]" <110195724+release-controller[bot]@users.noreply.github.com> Date: Thu, 16 Oct 2025 00:23:34 +0000 Subject: [PATCH 46/47] 3.18.0 release From f8458204912a0f2a56bbb72cf0b7eb6853da9139 Mon Sep 17 00:00:00 2001 From: "release-controller[bot]" <110195724+release-controller[bot]@users.noreply.github.com> Date: Thu, 16 Oct 2025 00:49:03 +0000 Subject: [PATCH 47/47] 3.18.0 release