diff --git a/.travis.yml b/.travis.yml new file mode 100644 index 000000000..adc20bd4c --- /dev/null +++ b/.travis.yml @@ -0,0 +1,13 @@ +--- +matrix: + include: + - os: osx + before_install: + - brew update + - brew install gnu-tar + script: make test + - os: linux + addons: + apt: + packages: devscripts + script: debuild -uc -us diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index fec2ed2e3..ec3a16976 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -8,3 +8,5 @@ Looking to contribute something to this project? That is great, we always apprec 4. Add and then commit your changes (`git commit -am "Add a new backup endpoint."`). 5. Push your feature branch to GitHub.com (`git push -u origin my-feature-branch`). 6. Open a [Pull Request](https://github.com/github/backup-utils/compare/) and wait for our feedback. + +Have a look at the styleguide [styleguide](https://github.com/github/backup-utils/tree/master/STYLEGUIDE.md) to make sure your code style is consistent with the code in this repository. diff --git a/Makefile b/Makefile index 30b73decf..136411a16 100644 --- a/Makefile +++ b/Makefile @@ -1,7 +1,7 @@ SHELL = /bin/sh test: info - @script/cibuild + @script/cibuild --no-package info: @echo This is github/backup-utils @@ -12,6 +12,9 @@ info: dist: @script/package-tarball +deb: + @script/package-deb + clean: rm -rf dist diff --git a/README.md b/README.md index 7c702f69d..6893d75b9 100644 --- a/README.md +++ b/README.md @@ -9,10 +9,11 @@ This repository includes backup and recovery utilities for [GitHub Enterprise][1 - **[Storage requirements](#storage-requirements)** - **[GitHub Enterprise version requirements](#github-enterprise-version-requirements)** - **[Getting started](#getting-started)** -- **[Migrating from GitHub Enterprise v11.10.34x to v2.0](#migrating-from-github-enterprise-v111034x-to-v20)** +- **[Migrating from GitHub Enterprise v11.10.34x to v2.0](#migrating-from-github-enterprise-v111034x-to-v20-or-v21)** - **[Using the backup and restore commands](#using-the-backup-and-restore-commands)** - **[Scheduling backups](#scheduling-backups)** - **[Backup snapshot file structure](#backup-snapshot-file-structure)** +- **[How does backup utilities differ from a High Availability replica?](#how-does-backup-utilities-differ-from-a-high-availability-replica)** - **[Support](#support)** ### Features @@ -44,10 +45,11 @@ storage and must have network connectivity with the GitHub Enterprise appliance. ##### Backup host requirements Backup host software requirements are modest: Linux or other modern Unix -operating system with [rsync][4] v2.6.4 or newer. +operating system with bash and [rsync][4] v2.6.4 or newer. The backup host must be able to establish network connections outbound to the -GitHub appliance over SSH (port 22). +GitHub appliance over SSH. TCP port 122 is used to backup GitHub Enterprise 2.0 +or newer instances, and TCP port 22 is used for older versions (11.10.34X). ##### Storage requirements @@ -56,6 +58,9 @@ patterns of the GitHub appliance. We recommend allocating at least 5x the amount of storage allocated to the primary GitHub appliance for historical snapshots and growth over time. +The backup utilities use [hard links][12] to store data efficiently, so the backup +snapshots must be written to a filesystem with support for hard links. + ##### GitHub Enterprise version requirements The backup utilities are fully supported under GitHub Enterprise 2.0 or @@ -82,6 +87,16 @@ download the most recent GitHub Enterprise version. host name. Additional options are available and documented in the configuration file but none are required for basic backup functionality. + * backup-utils will attempt to load the backup configuration from the following locations, in this order: + + ``` + $GHE_BACKUP_CONFIG (User configurable environment variable) + $GHE_BACKUP_ROOT/backup.config (Root directory of backup-utils install) + $HOME/.github-backup-utils/backup.config + /etc/github-backup-utils/backup.config + ``` + * In a clustering environment, the `GHE_EXTRA_SSH_OPTS` key must be configured with the `-i ` SSH option. + 3. Add the backup host's SSH key to the GitHub appliance as an *Authorized SSH key*. See [Adding an SSH key for shell access][3] for instructions. @@ -92,13 +107,14 @@ download the most recent GitHub Enterprise version. [release]: https://github.com/github/backup-utils/releases -### Migrating from GitHub Enterprise v11.10.34x to v2.0 +### Migrating from GitHub Enterprise v11.10.34x to v2.0, or v2.1 -If you are migrating from GitHub Enterprise version 11.10.34x to 2.0 or greater, +If you are migrating from GitHub Enterprise version 11.10.34x to 2.0 or 2.1 +(note, migrations to versions greater than 2.1 are not officially supported), please see the [Migrating from GitHub Enterprise v11.10.34x][10] documentation in the [GitHub Enterprise System Administrator's Guide][11]. It includes important information on using the backup utilities to migrate data from your -v11.10.34x instance to v2.0. +v11.10.34x instance to v2.0 or v2.1. ### Using the backup and restore commands @@ -107,7 +123,8 @@ After the initial backup, use the following commands: - The `ghe-backup` command creates incremental snapshots of repository data, along with full snapshots of all other pertinent data stores. - The `ghe-restore` command restores snapshots to the same or separate GitHub - Enterprise appliance. + Enterprise appliance. You must add the backup host's SSH key to the target + GitHub Enterprise appliance before using this command. ##### Example backup and restore usage @@ -153,6 +170,11 @@ The `ghe-backup` and `ghe-restore` commands also have a verbose output mode (`-v`) that lists files as they're being transferred. It's often useful to enable when output is logged to a file. +When restoring to an already configured GHE instance, settings, certificate, and license data +are *not* restored to prevent overwriting manual configuration on the restore +host. This behavior can be overriden by passing the `-c` argument to `ghe-restore`, +forcing settings, certificate, and license data to be overwritten with the backup copy's data. + ### Scheduling backups Regular backups should be scheduled using `cron(8)` or similar command @@ -200,6 +222,11 @@ date and time the snapshot was taken. Each snapshot directory contains a full backup snapshot of all relevant data stores. Repository, Search, and Pages data is stored efficiently via hard links. +*Please note* Symlinks must be maintained when archiving backup snapshots. +Dereferencing or excluding symlinks, or storing the snapshot contents on a +filesystem which does not support symlinks will result in operational +problems when the data is restored. + The following example shows a snapshot file hierarchy for hourly frequency. There are five snapshot directories, with the `current` symlink pointing to the most recent successful snapshot: @@ -221,11 +248,21 @@ most recent successful snapshot: |- ssh-host-keys.tar |- strategy |- version - |- current -> 20140727T010000 + |- current -> 20140728T010000 Note: the `GHE_DATA_DIR` variable set in `backup.config` can be used to change the disk location where snapshots are written. +### How does backup utilities differ from a High Availability replica? +It is recommended that both backup utilities and an [High Availability replica](https://help.github.com/enterprise/admin/guides/installation/high-availability-cluster-configuration/) are used as part of a GitHub Enterprise deployment but they serve different roles. + +##### The purpose of the High Availability replica +The High Availability replica is a fully redundant secondary GitHub Enterprise instance, kept in sync with the primary instance via replication of all major datastores. This active/passive cluster configuration is designed to minimize service disruption in the event of hardware failure or major network outage affecting the primary instance. Because some forms of data corruption or loss may be replicated immediately from primary to replica, it is not a replacement for the backup utilities as part of your disaster recovery plan. + +##### The purpose of the backup utilities +Backup utilities are a disaster recovery tool. This tool takes date-stamped snapshots of all major datastores. These snapshots are used to restore an instance to a prior state or set up a new instance without having another always-on GitHub Enterprise instance (like the High Availability replica). + + ### Support If you find a bug or would like to request a feature in backup-utils, please @@ -242,5 +279,6 @@ site setup or recovery, please contact our [Enterprise support team][7] instead. [7]: https://enterprise.github.com/support/ [8]: https://enterprise.github.com/help/articles/backing-up-enterprise-data [9]: https://enterprise.github.com/help/articles/restoring-enterprise-data -[10]: https://help.github.com/enterprise/2.0/admin-guide/migrating/ +[10]: https://help.github.com/enterprise/2.0/admin-guide/migrating-to-a-different-platform-or-from-github-enterprise-11-10-34x/ [11]: https://help.github.com/enterprise/2.0/admin-guide/ +[12]: https://en.wikipedia.org/wiki/Hard_link diff --git a/RELEASING.md b/RELEASING.md new file mode 100644 index 000000000..692c74bfc --- /dev/null +++ b/RELEASING.md @@ -0,0 +1,13 @@ +# Making a backup-utils release + + 1. Install the debian devscripts package: + `sudo apt-get install devscripts` + 2. Add a new version and release notes to the `debian/changelog` file: + `dch --newversion 2.6.0 --release-heuristic log` + 3. Rev the `share/github-backup-utils/version` file. + 4. Tag the release: `git tag v2.0.2` + 5. Build that tarball package: `make dist` + 6. Build the deb package: `make deb`. All the tests should pass. + 7. Draft a new release at https://github.com/github/backup-utils/releases, including the release notes and attaching the tarball and deb packages. + The dist tarball you should upload has the git revision in the file name, i.e. something like `github-backup-utils-v2.5.0-1-g23c41cc.tar.gz` + 8. Push the head of the release to the 'stable' branch. diff --git a/STYLEGUIDE.md b/STYLEGUIDE.md new file mode 100644 index 000000000..a0a25d994 --- /dev/null +++ b/STYLEGUIDE.md @@ -0,0 +1,233 @@ +## Bash Styleguide + +If you've not done much Bash development before you may find these debugging tips useful: http://wiki.bash-hackers.org/scripting/debuggingtips. + +-- +##### Scripts must start with `#!/usr/bin/env bash` + +-- +##### Scripts must use `set -e` + +If the return value of a command can be ignored, suffix it with `|| true`: + +```bash +set -e +command_that_might_fail || true +command_that_should_not_fail +``` + +Note that ignoring an exit status with `|| true` is not a good practice though. Generally speaking, it's better to handle the error. + +-- +##### Scripts should not check exit status via `$?` manually + +Rely on `set -e` instead: + +```bash +cmd +if [ $? -eq 0 ]; then + echo worked +fi +``` + +should be written as: + +```bash +set -e +if cmd; then + echo worked +fi +``` + +-- +##### Scripts must include a usage, description and optional examples + +Use this format: + +```bash +#!/usr/bin/env bash +#/ Usage: ghe-this-is-my-script [options] +#/ +#/ This is a brief description of the script's purpose. +#/ +#/ OPTIONS: +#/ -h | --help Show this message. +#/ -l | --longopt An option. +#/ -c Another option. +#/ +#/ EXAMPLES: (optional section but nice to have when not trivial) +#/ +#/ This will do foo and bar: +#/ $ ghe-this-is-my-script --longopt foobar -c 2 +#/ +set -e +``` + +If there are no options or required arguments, the `OPTIONS` section can be ignored. + +-- +##### Customer-facing scripts must accept both -h and --help arguments + +They should also print the usage information and exit 2. + +For example: + +```bash +#!/usr/bin/env bash +#/ Usage: ghe-this-is-my-script [options] +#/ +#/ This is a brief description of the script's purpose. +set -e + +if [ "$1" = "--help" -o "$1" = "-h" ]; then + grep '^#/' <"$0" | cut -c 4- + exit 2 +fi + +``` + +-- +##### Scripts should not use Bash arrays + +Main issues: + +* Portability +* Important bugs in Bash versions < 4.3 + +-- +##### Scripts should use `test` or `[` whenever possible + +```bash +test -f /etc/passwd +test -f /etc/passwd -a -f /etc/group +if [ "string" = "string" ]; then + true +fi +``` + +-- +##### Scripts may use `[[` for advanced bash features + +```bash +if [[ "$(hostname)" = *.iad.github.net ]]; then + true +fi +``` + +-- +##### Scripts may use Bash for loops + +Preferred: + +```bash +for i in $(seq 0 9); do +done +``` + +or: + +```bash +for ((n=0; n<10; n++)); do +done +``` + +-- +##### Scripts should use `$[x+y*z]` for mathematical expressions + +```bash +local n=1 +let n++ +n=$[n+1] # preferred +n=$[$n+1] +n=$((n+1)) +n=$(($n+1)) +``` + +-- +##### Scripts should use variables sparingly + +Short paths and other constants should be repeated liberally throughout code since they +can be search/replaced easily if they ever change. + +```bash +DATA_DB_PATH=/data/user/db +mkdir -p $DATA_DB_PATH +rsync $DATA_DB_PATH remote:$DATA_DB_PATH +``` + +versus the much more readable: + +```bash +mkdir -p /data/user/db +rsync /data/user/db remote:/data/user/db +``` + +-- +##### Scripts should use lowercase variables for locals, and uppercase for variables inherited or exported via the environment + +```bash +#!/usr/bin/env bash +#/ Usage: [DEBUG=0] process_repo +nwo=$1 +[ -n $DEBUG ] && echo "** processing $nwo" >&2 + +export GIT_DIR=/data/repos/$nwo.git +git rev-list +``` + +-- +##### Scripts should use `${var}` for interpolation only when required + +```bash +greeting=hello +echo $greeting +echo ${greeting}world +``` + +-- +##### Scripts should use functions sparingly, opting for small/simple/sequential scripts instead whenever possible + +When defining functions, use the following style: + +```bash +my_function() { + local arg1=$1 + [ -n $arg1 ] || return + ... +} +``` + +-- +##### Scripts should use `< /etc/foo # interpolated before ssh + chmod 0600 /etc/foo +eof +``` + +-- +##### Scripts should quote variables that could reasonably have a space now or in the future + +```bash +if [ ! -z "$packages" ]; then + true +fi +``` + +### Testing + +See [the style guide](../test/STYLEGUIDE.md) diff --git a/backup.config-example b/backup.config-example index 54b221187..d2fe3828a 100644 --- a/backup.config-example +++ b/backup.config-example @@ -23,6 +23,23 @@ GHE_NUM_SNAPSHOTS=10 # #GHE_RESTORE_HOST="github-standby.example.com" -# Any extra options passed to the SSH command. Nothing required by default +# Any extra options passed to the SSH command. +# In a single instance environment, nothing is required by default. +# In a clustering environment, "-i abs-path-to-ssh-private-key" is required. # #GHE_EXTRA_SSH_OPTS="" + +# Any extra options passed to the rsync command. Nothing required by default. +# +#GHE_EXTRA_RSYNC_OPTS="" + +# If set to 'no', GHE_DATA_DIR will not be created automatically +# and restore/backup will exit 8 +# +#GHE_CREATE_DATA_DIR=yes + +# If set to 'yes', git fsck will run on the repositories +# and print some additional info. +# +# WARNING: do not enable this, only useful for debugging/development +#GHE_BACKUP_FSCK=no diff --git a/bin/ghe-backup b/bin/ghe-backup index 9c42bc6eb..7002879a4 100755 --- a/bin/ghe-backup +++ b/bin/ghe-backup @@ -1,4 +1,4 @@ -#!/bin/sh +#!/usr/bin/env bash #/ Usage: ghe-backup [-v] #/ Take snapshots of all GitHub Enterprise data, including Git repository data, #/ the MySQL database, instance settings, GitHub Pages data, etc. @@ -8,8 +8,7 @@ set -e # Bring in the backup configuration -cd $(dirname "$0")/.. -. share/github-backup-utils/ghe-backup-config +. $( dirname "${BASH_SOURCE[0]}" )/../share/github-backup-utils/ghe-backup-config # Used to record failed backup steps failures= @@ -28,37 +27,56 @@ touch "incomplete" GHE_MAINTENANCE_MODE_ENABLED=false # To prevent multiple backup runs happening at the same time, we create a -# in-progress symlink pointing to the snapshot directory. This will fail if -# another backup is already in progress, giving us a form of locking. +# in-progress file with the timestamp and pid of the backup process, +# giving us a form of locking. # -# Set up a trap to remove the in-progress symlink if we exit for any reason but -# verify that we own the in-progress symlink before doing so. +# Set up a trap to remove the in-progress file if we exit for any reason but +# verify that we are the same process before doing so. # # The cleanup trap also handles disabling maintenance mode on the appliance if # it was automatically enabled. cleanup () { - if [ $(readlink ../in-progress) = "$GHE_SNAPSHOT_TIMESTAMP" ]; then - unlink ../in-progress + if [ -f ../in-progress ]; then + progress=$(cat ../in-progress) + snapshot=$(echo "$progress" | cut -d ' ' -f 1) + pid=$(echo "$progress" | cut -d ' ' -f 2) + if [ "$snapshot" = "$GHE_SNAPSHOT_TIMESTAMP" -a "$$" = $pid ]; then + unlink ../in-progress fi + fi - if $GHE_MAINTENANCE_MODE_ENABLED; then - ghe-maintenance-mode-disable "$GHE_HOSTNAME" - fi + if $GHE_MAINTENANCE_MODE_ENABLED; then + ghe-maintenance-mode-disable "$GHE_HOSTNAME" + fi } # Setup exit traps trap 'cleanup' EXIT trap 'exit $?' INT # ^C always terminate -# Mark the snapshot as in-progress by creating the symlink. If this fails, it -# means another ghe-backup run is already in progress and we should exit. -# NOTE: The -n argument to ln is non-POSIX but widely supported. -if ! ln -sn "$GHE_SNAPSHOT_TIMESTAMP" ../in-progress 2>/dev/null; then - snapshot="$(readlink ../in-progress)" - echo "Error: backup of $GHE_HOSTNAME already in progress in snapshot $snapshot. Aborting." 1>&2 +if [ -h ../in-progress ]; then + echo "Error: detected a backup already in progress from a previous version of ghe-backup." 1>&2 + echo "If there is no backup in progress anymore, please remove" 1>&2 + echo "the $GHE_DATA_DIR/in-progress symlink." 1>&2 + exit 1 +fi + +if [ -f ../in-progress ]; then + progress=$(cat ../in-progress) + snapshot=$(echo "$progress" | cut -d ' ' -f 1) + pid=$(echo "$progress" | cut -d ' ' -f 2) + if ! ps -p $pid -o command= | grep ghe-backup; then + # We can safely remove in-progress, ghe-prune-snapshots + # will clean up the failed backup. + unlink ../in-progress + else + echo "Error: backup process $pid of $GHE_HOSTNAME already in progress in snapshot $snapshot. Aborting." 1>&2 exit 1 + fi fi +echo "$GHE_SNAPSHOT_TIMESTAMP $$" > ../in-progress + echo "Starting backup of $GHE_HOSTNAME in snapshot $GHE_SNAPSHOT_TIMESTAMP" # Perform a host connection check and establish the remote appliance version. @@ -67,6 +85,16 @@ echo "Starting backup of $GHE_HOSTNAME in snapshot $GHE_SNAPSHOT_TIMESTAMP" ghe_remote_version_required echo "$GHE_REMOTE_VERSION" > version +# Figure out if we're restoring into cluster +cluster=false +if ghe-ssh "$GHE_HOSTNAME" -- \ + "[ -f '$GHE_REMOTE_ROOT_DIR/etc/github/cluster' ]"; then + cluster=true +fi + +# Log backup start message in /var/log/syslog on remote instance +ghe_remote_logger "Starting backup from $(hostname) in snapshot $GHE_SNAPSHOT_TIMESTAMP ..." + # Determine whether to use the rsync or tarball backup strategy based on the # remote appliance version. The tarball strategy must be used with GitHub # Enterprise versions prior to 11.10.340 since rsync is not available. @@ -77,8 +105,13 @@ if [ $GHE_VERSION_MAJOR -eq 1 -a $GHE_VERSION_PATCH -lt 340 ]; then GHE_BACKUP_STRATEGY="tarball" fi +if $cluster; then + GHE_BACKUP_STRATEGY="cluster" +fi + # Record the strategy with the snapshot so we will know how to restore. echo "$GHE_BACKUP_STRATEGY" > strategy +export GHE_BACKUP_STRATEGY # If we're using the tarball backup strategy, put the appliance in maintenance # mode and wait for all writing processes to bleed out. @@ -87,6 +120,9 @@ if [ "$GHE_BACKUP_STRATEGY" = "tarball" ]; then GHE_MAINTENANCE_MODE_ENABLED=true fi +ghe-backup-store-version || +echo "Warning: storing backup-utils version remotely failed." + echo "Backing up GitHub settings ..." ghe-backup-settings || failures="$failures settings" @@ -100,13 +136,28 @@ ghe-ssh "$GHE_HOSTNAME" -- 'ghe-export-ssh-host-keys' > ssh-host-keys.tar || failures="$failures ssh-host-keys" echo "Backing up MySQL database ..." -echo 'ghe-export-mysql | gzip' | -ghe-ssh "$GHE_HOSTNAME" -- /bin/sh > mysql.sql.gz || +echo 'set -o pipefail; ghe-export-mysql | gzip' | +ghe-ssh "$GHE_HOSTNAME" -- /bin/bash > mysql.sql.gz || failures="$failures mysql" echo "Backing up Redis database ..." -ghe-backup-redis > redis.rdb || -failures="$failures redis" +if $cluster; then + ghe-backup-redis-cluster > redis.rdb || + failures="$failures redis" +else + ghe-backup-redis > redis.rdb || + failures="$failures redis" +fi + +if [ $GHE_VERSION_MAJOR -ge 2 ]; then + echo "Backing up audit log ..." + ghe-backup-es-audit-log || + failures="$failures audit-log" +fi + +echo "Backing up hookshot logs ..." +ghe-backup-es-hookshot || +failures="$failures hookshot" echo "Backing up Git repositories ..." ghe-backup-repositories-${GHE_BACKUP_STRATEGY} || @@ -117,18 +168,40 @@ ghe-backup-pages-${GHE_BACKUP_STRATEGY} || failures="$failures pages" if [ "$GHE_VERSION_MAJOR" -ge 2 ]; then - echo "Backing up asset attachments ..." - ghe-backup-userdata alambic_assets || - failures="$failures alambic_assets" - - echo "Backing up hook deliveries ..." - ghe-backup-userdata hookshot || - failures="$failures hookshot" + if $cluster; then + echo "Backing up asset attachments ..." + ghe-backup-alambic-cluster || + failures="$failures alambic_assets" + + echo "Backing up custom Git hooks ..." + ghe-backup-git-hooks-cluster || + failures="$failures git-hooks" + else + echo "Backing up asset attachments ..." + ghe-backup-userdata alambic_assets || + failures="$failures alambic_assets" + + echo "Backing up storage data ..." + ghe-backup-userdata storage || + failures="$failures storage" + + echo "Backing up hook deliveries ..." + ghe-backup-userdata hookshot || + failures="$failures hookshot" + + echo "Backing up custom Git hooks ..." + ghe-backup-userdata git-hooks/environments/tarballs || + failures="$failures git-hooks-environments" + ghe-backup-userdata git-hooks/repos || + failures="$failures git-hooks-repos" + fi fi -echo "Backing up Elasticsearch indices ..." -ghe-backup-es-${GHE_BACKUP_STRATEGY} || -failures="$failures elasticsearch" +if ! $cluster; then + echo "Backing up Elasticsearch indices ..." + ghe-backup-es-${GHE_BACKUP_STRATEGY} || + failures="$failures elasticsearch" +fi # If we're using the tarball backup strategy, bring the appliance out of # maintenance mode now instead of waiting until after pruning stale snapshots. @@ -138,6 +211,12 @@ if $GHE_MAINTENANCE_MODE_ENABLED; then GHE_MAINTENANCE_MODE_ENABLED=false fi +# git fsck repositories after the backup +if [ "$GHE_BACKUP_FSCK" = "yes" ]; then + ghe-backup-fsck $GHE_SNAPSHOT_DIR || + failures="$failures fsck" +fi + # If everything was successful, mark the snapshot as complete, update the # current symlink to point to the new snapshot and prune expired and failed # snapshots. @@ -153,8 +232,11 @@ fi echo "Completed backup of $GHE_HOSTNAME in snapshot $GHE_SNAPSHOT_TIMESTAMP at $(date +"%H:%M:%S")" # Exit non-zero and list the steps that failed. -if [ -n "$failures" ]; then +if [ -z "$failures" ]; then + ghe_remote_logger "Completed backup from $(hostname) / snapshot $GHE_SNAPSHOT_TIMESTAMP successfully." +else steps="$(echo $failures | sed 's/ /, /g')" + ghe_remote_logger "Completed backup from $(hostname) / snapshot $GHE_SNAPSHOT_TIMESTAMP with failures: ${steps}." echo "Error: Snapshot incomplete. Some steps failed: ${steps}. " exit 1 fi diff --git a/bin/ghe-host-check b/bin/ghe-host-check index faf5317fa..992b28ac8 100755 --- a/bin/ghe-host-check +++ b/bin/ghe-host-check @@ -1,12 +1,11 @@ -#!/bin/sh +#!/usr/bin/env bash #/ Usage: ghe-host-check [] #/ Verify connectivity with the GitHub Enterprise host. When no is #/ provided, the $GHE_HOSTNAME configured in backup.config is assumed. set -e -# Bring in the backup configuration. -cd $(dirname "$0")/.. -. share/github-backup-utils/ghe-backup-config +# Bring in the backup configuration +. $( dirname "${BASH_SOURCE[0]}" )/../share/github-backup-utils/ghe-backup-config # Use the host provided on the command line if provided, or fallback on the # $GHE_HOSTNAME configured in backup.config when not present. @@ -24,15 +23,21 @@ port=$(ssh_port_part "$host") hostname=$(ssh_host_part "$host") set +e -output=$(echo "cat \"$GHE_REMOTE_METADATA_FILE\" 2>/dev/null || exit 101" | ghe-ssh $options $host -- /bin/sh 2>&1) +output=$(echo "ghe-negotiate-version backup-utils $BACKUP_UTILS_VERSION" | ghe-ssh -o BatchMode=no $options $host -- /bin/sh 2>&1) rc=$? +if [ $rc = 127 ]; then + # ghe-negotiate-version not found, fallback to reading version file + legacy_version_output="1" + output=$(echo "cat \"$GHE_REMOTE_METADATA_FILE\" 2>/dev/null || exit 101" | ghe-ssh -o BatchMode=no $options $host -- /bin/sh 2>&1) + rc=$? +fi set -e if [ $rc -ne 0 ]; then case $rc in 255) - if echo "$output" | grep -i -q "port 22: connection refused"; then - exec "$0" "$hostname:122" + if echo "$output" | grep -i "port 22: connection refused\|port 22: no route to host\|ssh_exchange_identification: Connection closed by remote host\|Connection timed out during banner exchange" >/dev/null; then + exec "$(basename $0)" "$hostname:122" fi echo "$output" 1>&2 @@ -44,8 +49,8 @@ if [ $rc -ne 0 ]; then echo "Error: couldn't read GitHub Enterprise fingerprint on '$host' or this isn't a GitHub appliance." 1>&2 ;; 1) - if [ "${port:-22}" -eq 22 ] && echo "$output" | grep -q "use port 122"; then - exec "$0" "$hostname:122" + if [ "${port:-22}" -eq 22 ] && echo "$output" | grep "use port 122" >/dev/null; then + exec "$(basename $0)" "$hostname:122" else echo "$output" 1>&2 fi @@ -55,9 +60,14 @@ if [ $rc -ne 0 ]; then exit $rc fi -version=$(echo "$output" | grep version | cut -d'"' -f4) +if [ -z "$legacy_version_output" ]; then + version=$(echo "$output" | sed -n 's/GitHub Enterprise version \(.*\)/\1/p') +else + version=$(echo "$output" | grep version | cut -d'"' -f4) +fi + if [ -z "$version" ]; then - echo "Error: failed to parse version from $GHE_REMOTE_METADATA_FILE on '$host' or this isn't a GitHub appliance." 1>&2 + echo "Error: failed to parse version on '$host' or this isn't a GitHub appliance." 1>&2 exit 2 fi diff --git a/bin/ghe-restore b/bin/ghe-restore index 48ab357e6..ac465916a 100755 --- a/bin/ghe-restore +++ b/bin/ghe-restore @@ -1,4 +1,4 @@ -#!/bin/sh +#!/usr/bin/env bash #/ Usage: ghe-restore [-v] [-s ] [] #/ Restores a GitHub instance from local backup snapshots. The is the #/ hostname or IP of the GitHub instance. The may be omitted when @@ -21,9 +21,8 @@ #/ set -e -# Bring in the backup configuration. -cd $(dirname "$0")/.. -. share/github-backup-utils/ghe-backup-config +# Bring in the backup configuration +. $( dirname "${BASH_SOURCE[0]}" )/../share/github-backup-utils/ghe-backup-config # Parse arguments restore_settings=false @@ -97,6 +96,34 @@ elif [ "$GHE_VERSION_MAJOR" -ge 2 ]; then restore_settings=true fi +# Figure out if we're restoring into cluster +cluster=false +if ghe-ssh "$GHE_HOSTNAME" -- \ + "[ -f '$GHE_REMOTE_ROOT_DIR/etc/github/cluster' ]"; then + cluster=true + instance_configured=true + restore_settings=false +fi + +# Figure out if we're restoring into a cluster with an unsupported snapshot +if $cluster; then + snapshot_instance_version=$(cat $GHE_RESTORE_SNAPSHOT_PATH/version) + if ! echo $snapshot_instance_version | \ + grep -Eq "v2\.[5-9]|v2\.[1-9][0-9]|v[3-9]|v[1-9][0-9]"; then + echo "Error: Snapshot must be from GitHub Enterprise v2.5.0 or above to be restored" + echo " into a cluster (detected $snapshot_instance_version). Aborting." >&2 + exit 1 + fi +fi + +# Figure out if this instance is in a replication pair +if ghe-ssh "$GHE_HOSTNAME" -- "ghe-repl-status -r 2>/dev/null" \ + | grep -Eq "replica|primary"; then + instance_configured=true + echo "WARNING: Restoring to a server with replication enabled interrupts replication." + echo " You will need to reconfigure replication after the restore completes." +fi + # Prompt to verify the restore host given is correct. Restoring overwrites # important data on the destination appliance that cannot be recovered. This is # mostly to prevent accidents where the backup host is given to restore instead @@ -125,7 +152,11 @@ if $instance_configured && ! $force; then echo fi +# Log restore start message locally and in /var/log/syslog on remote instance echo "Starting restore of $GHE_HOSTNAME from snapshot $GHE_RESTORE_SNAPSHOT" +ghe_remote_logger "Starting restore from $(hostname) / snapshot $GHE_RESTORE_SNAPSHOT ..." + +# Update remote restore state file and setup failure trap trap "update_restore_status failed" EXIT update_restore_status "restoring" @@ -139,7 +170,7 @@ fi # Restoring Elasticsearch to 11.10.3x via rsync requires GNU tar if [ "$GHE_VERSION_MAJOR" -le 1 ] && [ "$GHE_BACKUP_STRATEGY" = "rsync" ]; then - if ! tar --version | grep -q GNU; then + if ! tar --version | grep GNU >/dev/null; then if ! command -v gtar >/dev/null 2>&1; then echo "GNU tar is required. Aborting." >&2 exit 1 @@ -147,12 +178,17 @@ if [ "$GHE_VERSION_MAJOR" -le 1 ] && [ "$GHE_BACKUP_STRATEGY" = "rsync" ]; then fi fi -# Make sure the GitHub appliance is in maintenance mode and all writing -# processes have bled out. +# Make sure the GitHub appliance is in maintenance mode. if $instance_configured; then - ghe-maintenance-mode-enable "$GHE_HOSTNAME" + if ! ghe-maintenance-mode-status "$GHE_HOSTNAME"; then + echo "Error: $GHE_HOSTNAME must be put in maintenance mode before restoring. Aborting." 1>&2 + exit 1 + fi fi +ghe-backup-store-version || +echo "Warning: storing backup-utils version remotely failed." + # Restore settings and license if restoring to an unconfigured appliance or when # specified manually. if $restore_settings; then @@ -162,42 +198,115 @@ fi # Make sure mysql and elasticsearch are prep'd and running before restoring into # appliances v2.x or greater. These services will not have been started on appliances # that have not been configured yet. -if [ "$GHE_VERSION_MAJOR" -ge 2 ]; then +if ! $cluster; then + if [ "$GHE_VERSION_MAJOR" -ge 2 ]; then echo "sudo ghe-service-ensure-mysql && sudo ghe-service-ensure-elasticsearch" | ghe-ssh "$GHE_HOSTNAME" -- /bin/sh 1>&3 + fi fi -echo "Restoring Git repositories ..." -ghe-restore-repositories-${GHE_BACKUP_STRATEGY} "$GHE_HOSTNAME" 1>&3 +echo "Restoring MySQL database ..." +bm_start "ghe-import-mysql" +gzip -dc "$GHE_RESTORE_SNAPSHOT_PATH/mysql.sql.gz" | ghe-ssh "$GHE_HOSTNAME" -- 'ghe-import-mysql' +bm_end "ghe-import-mysql" -echo "Restoring GitHub Pages ..." -ghe-restore-pages-${GHE_BACKUP_STRATEGY} "$GHE_HOSTNAME" 1>&3 +echo "Restoring Redis database ..." +bm_start "ghe-import-redis" +ghe-ssh "$GHE_HOSTNAME" -- 'ghe-import-redis' < "$GHE_RESTORE_SNAPSHOT_PATH/redis.rdb" 1>&3 +bm_end "ghe-import-redis" -if [ "$GHE_VERSION_MAJOR" -ge 2 ]; then - echo "Restoring asset attachments ..." - ghe-restore-userdata alambic_assets "$GHE_HOSTNAME" 1>&3 +if $cluster; then + echo "Restoring Git repositories into cluster ..." + if ghe-ssh "$GHE_HOSTNAME" test -f /data/github/current/script/dgit-cluster-restore-routes; then + ghe_verbose "* Using ghe-restore-repositories-dgit-ng to restore" + ghe-restore-repositories-dgit-ng "$GHE_HOSTNAME" 1>&3 + else + ghe-restore-repositories-dgit "$GHE_HOSTNAME" 1>&3 + fi - echo "Restoring hook deliveries ..." - ghe-restore-userdata hookshot "$GHE_HOSTNAME" 1>&3 -fi + echo "Restoring Gists into cluster ..." + if ghe-ssh "$GHE_HOSTNAME" test -f /data/github/current/script/gist-cluster-restore-routes; then + ghe_verbose "* Using ghe-restore-repositories-gist-ng to restore" + ghe-restore-repositories-gist-ng "$GHE_HOSTNAME" 1>&3 + else + ghe-restore-repositories-gist "$GHE_HOSTNAME" 1>&3 + fi +else + # Remove temporary 2.2 storage migration directory if it exists + echo "if [ -d /data/user/repositories-nw-backup ]; then sudo rm -rf /data/user/repositories-nw-backup; fi" | + ghe-ssh "$GHE_HOSTNAME" -- /bin/sh 1>&3 -echo "Restoring MySQL database ..." -gzip -dc "$GHE_RESTORE_SNAPSHOT_PATH/mysql.sql.gz" | ghe-ssh "$GHE_HOSTNAME" -- 'ghe-import-mysql' 1>&3 + echo "Restoring Git repositories ..." + ghe-restore-repositories-${GHE_BACKUP_STRATEGY} "$GHE_HOSTNAME" 1>&3 +fi -echo "Restoring Redis database ..." -ghe-ssh "$GHE_HOSTNAME" -- 'ghe-import-redis' < "$GHE_RESTORE_SNAPSHOT_PATH/redis.rdb" 1>&3 +if $cluster; then + echo "Restoring GitHub Pages into DPages..." + ghe-restore-pages-dpages "$GHE_HOSTNAME" 1>&3 +else + echo "Restoring GitHub Pages ..." + ghe-restore-pages-${GHE_BACKUP_STRATEGY} "$GHE_HOSTNAME" 1>&3 +fi echo "Restoring SSH authorized keys ..." ghe-ssh "$GHE_HOSTNAME" -- 'ghe-import-authorized-keys' < "$GHE_RESTORE_SNAPSHOT_PATH/authorized-keys.json" 1>&3 -echo "Restoring Elasticsearch indices ..." -ghe-restore-es-${GHE_BACKUP_STRATEGY} "$GHE_HOSTNAME" 1>&3 +if $cluster; then + echo "Restoring storage data ..." + if ghe-ssh "$GHE_HOSTNAME" test -f /data/github/current/script/storage-cluster-restore-routes; then + ghe_verbose "* Using ghe-restore-alambic-cluster-ng to restore" + # This restore script is faster but only available in + # GHE >= 2.6.3 + ghe-restore-alambic-cluster-ng "$GHE_HOSTNAME" 1>&3 + else + ghe-restore-alambic-cluster "$GHE_HOSTNAME" 1>&3 + fi + echo "Restoring custom Git hooks ..." + ghe-restore-git-hooks-cluster "$GHE_HOSTNAME" 1>&3 +elif [ "$GHE_VERSION_MAJOR" -ge 2 ]; then + echo "Restoring asset attachments ..." + ghe-restore-userdata alambic_assets "$GHE_HOSTNAME" 1>&3 + + echo "Restoring storage data ..." + ghe-restore-userdata storage "$GHE_HOSTNAME" 1>&3 + + echo "Restoring hook deliveries ..." + ghe-restore-userdata hookshot "$GHE_HOSTNAME" 1>&3 + + echo "Restoring custom Git hooks ..." + ghe-restore-userdata git-hooks/environments/tarballs "$GHE_HOSTNAME" 1>&3 + ghe-restore-userdata git-hooks/repos "$GHE_HOSTNAME" 1>&3 + ghe-restore-git-hooks-extract "$GHE_HOSTNAME" 1>&3 +fi + +if $cluster; then + echo "Restoring ElasticSearch Audit logs" + ghe-restore-es-audit-log "$GHE_HOSTNAME" 1>&3 +else + echo "Restoring Elasticsearch indices ..." + ghe-restore-es-${GHE_BACKUP_STRATEGY} "$GHE_HOSTNAME" 1>&3 +fi + +# Restart an already running memcached to reset the cache after restore +if [ "$GHE_VERSION_MAJOR" -ge 2 ]; then + echo "Restarting memcached ..." 1>&3 + echo "sudo restart -q memcached 2>/dev/null || true" | + ghe-ssh "$GHE_HOSTNAME" -- /bin/sh +fi -# When restoring to a 2.x host that has already been configured, kick off a +# When restoring to a host that has already been configured, kick off a # config run to perform data migrations. -if [ "$GHE_VERSION_MAJOR" -ge 2 ] && $instance_configured; then - echo "Configuring storage ..." - ghe-ssh "$GHE_HOSTNAME" -- "sudo ghe-config-apply" 1>&3 +if $cluster; then + echo "Configuring cluster ..." + ghe-ssh "$GHE_HOSTNAME" -- "ghe-cluster-config-apply" 1>&3 2>&3 +elif $instance_configured; then + echo "Configuring storage ..." + if [ "$GHE_VERSION_MAJOR" -ge 2 ]; then + ghe-ssh "$GHE_HOSTNAME" -- "ghe-config-apply --full" 1>&3 2>&3 + else + echo " This will take several minutes to complete..." + ghe-ssh "$GHE_HOSTNAME" -- "sudo enterprise-configure" 1>&3 2>&3 + fi fi # Update the remote status to "complete". This has to happen before importing @@ -206,8 +315,23 @@ fi trap "" EXIT update_restore_status "complete" -echo "Restoring SSH host keys ..." -ghe-ssh "$GHE_HOSTNAME" -- 'ghe-import-ssh-host-keys' < "$GHE_RESTORE_SNAPSHOT_PATH/ssh-host-keys.tar" 1>&3 +# Log restore complete message in /var/log/syslog on remote instance +ghe_remote_logger "Completed restore from $(hostname) / snapshot ${GHE_RESTORE_SNAPSHOT}." + +if ! $cluster; then + echo "Restoring SSH host keys ..." + ghe-ssh "$GHE_HOSTNAME" -- 'ghe-import-ssh-host-keys' < "$GHE_RESTORE_SNAPSHOT_PATH/ssh-host-keys.tar" 1>&3 +else + # This will make sure that Git over SSH host keys (babeld) are + # copied to all the cluster nodes so babeld uses the same keys. + echo "Restoring Git over SSH host keys ..." + ghe-ssh "$GHE_HOSTNAME" -- "sudo tar -xpf - -C /data/user/common" < "$GHE_RESTORE_SNAPSHOT_PATH/ssh-host-keys.tar" 1>&3 + ghe-ssh "$GHE_HOSTNAME" -- "sudo chown babeld:babeld /data/user/common/ssh_host_*" 1>&3 + ghe-ssh "$GHE_HOSTNAME" -- "/usr/local/share/enterprise/ghe-cluster-config-update -s" 1>&3 +fi echo "Completed restore of $GHE_HOSTNAME from snapshot $GHE_RESTORE_SNAPSHOT" -echo "Visit https://$hostname/setup/settings to review appliance configuration." + +if ! $cluster; then + echo "Visit https://$hostname/setup/settings to review appliance configuration." +fi diff --git a/debian/changelog b/debian/changelog index 394ae6c49..8edfa30a0 100644 --- a/debian/changelog +++ b/debian/changelog @@ -1,3 +1,152 @@ +github-backup-utils (2.6.3) UNRELEASED; urgency=medium + + * Cluster: git-hooks backup fixes #235 + + -- Sergio Rubio Wed, 29 Jun 2016 21:05:21 +0200 + +github-backup-utils (2.6.2) UNRELEASED; urgency=medium + + * git-hooks fixes #231 + * Cluster: speedup repositories restore #232 (requires GitHub Enterprise + 2.6.4) + * Cluster: restore Git over SSH keys #230 + * Benchmark restores #219 + + -- Sergio Rubio Wed, 22 Jun 2016 19:36:06 +0200 + +github-backup-utils (2.6.1) UNRELEASED; urgency=medium + + * Cluster: faster gist restores #220 + * Cluster: faster storage restores #212 + * Cluster: fix git-hooks restores #204 + + -- Sergio Rubio Tue, 31 May 2016 20:54:11 +0200 + +github-backup-utils (2.6.0) UNRELEASED; urgency=medium + + * Adds support for GitHub Enterprise 2.6 + * Adds an extra supported location for the backup configuration #197 + * New config option to check for corrupted repositories after the backup #195 + * General improvements and bug fixes + + -- Sergio Rubio Tue, 26 Apr 2016 18:03:01 +0200 + +github-backup-utils (2.5.2) UNRELEASED; urgency=medium + + * New configuration variable: GHE_CREATE_DATA_DIR #186 + * Require that snapshots originated from an instance running GitHub + Enterprise 2.5.0 or above when restoring to a cluster #182 + * Prevent Git GC operations and some other maintenance jobs from running + while repositories are being restored #179 + * Fix Solaris and SmartOS support, using Bash everywhere #177 + + -- Sergio Rubio Wed, 30 Mar 2016 14:32:15 +0200 + +github-backup-utils (2.5.1) UNRELEASED; urgency=medium + + * Fixes for cluster restores #173 + * Fix Elasticsearch backups for GitHub Enterprise <= 2.2 #175 + * Removed experimental S3 support #167 + * Remote logging output fixes #170 + * Update ghe-host-check to detect extra port 22 error #162 + + -- Sergio Rubio Wed, 09 Mar 2016 14:44:05 +0100 + +github-backup-utils (2.5.0) UNRELEASED; urgency=medium + + * Adds GitHub Enterpise 2.5 support + * Adds GitHub Enterprise Clustering support + * Backups and restores SAML keypairs + + -- Sergio Rubio Tue, 9 Feb 2016 00:02:37 +0000 + +github-backup-utils (2.4.0) UNRELEASED; urgency=medium + + * Moves the in-progress detection to a separate file with PID which is + removed if the process is no longer running after the backup. #145, #99 + * Updates the README to explain why backup-utils is useful even if you have + the high availability replica running. #140 + * Changes the use of the --link-dest option to only occur when backing up + populated directories. #138 + * Adds logging to /var/log/syslog on the remote GitHub Enterprise appliance + to both ghe-backup and ghe-restore. #131 + * Restarts memcached after restoring to an already configured appliance to + ensure it doesn't contain out-of-sync information. #130 + * Removes the temporary /data/user/repositories-nw-backup directory that + remains after successfully migrating the repository storage layout to the + new format used on GitHub Enterprise 2.2.0 and later after restoring a + backup from an older release of GitHub Enterprise. #129 + * Add devscripts to Debian's build-depends for checkbashisms. #101 + * Documents the -c option which forces the restoration of the configuration + information to an already configured appliance. #96 + + -- Colin Seymour Tue, 20 Oct 2015 00:02:37 +0000 + +github-backup-utils (2.2.0) UNRELEASED; urgency=medium + + * Adds support for the new repositories filesystem layout include in + GitHub Enterprise v2.2. #122, #124 + * ghe-restore now performs a config run on the instance after an incremental + restore to 11.10.x and 2.x instances. #100 + * ghe-restore now fails fast when run against a GHE instance with replication + enabled. Replication should be disabled during a restore and then setup + after the restore completes. #121 + * Fixes an issue with special port 122 detection failing when port is + overridden in an ssh config file. #102 + * Removes a warning message when running ghe-backup against an instance with + GitHub Pages disabled. #117 + * backup-utils release version numbers now track GitHub Enterprise releases + to ease the process of determining which version of backup-utils is + required for a given GitHub Enterprise version. + + -- Ryan Tomayko Wed, 29 Apr 2015 07:29:04 +0000 + +github-backup-utils (2.0.2) UNRELEASED; urgency=medium + + * ghe-restore now requires that an already-configured appliance be put into + maintenance mode manually. This is a safeguard against accidentally + overwriting data on the wrong instance. #62, #84 + * ghe-backup and ghe-restore now run a ghe-negotiate-version program on the + appliance to determine whether the backup-utils and GHE versions are + compatible. #91 + * Various portability fixes for problems surfaced when running on Solaris + and FreeBSD. #86, #87 + * Fixes an issue in ghe-backup where mysqldump failures weren't being + reported properly. #90 + * Automated builds are now run on Travis CI. #77 + + -- Ryan Tomayko Tue, 20 Jan 2015 16:00:00 +0000 + +github-backup-utils (2.0.1) UNRELEASED; urgency=medium + + * Adds /etc/github-backup-utils/backup.config as a default config file search + location for deb / system installs. + * Enables SSH BatchMode for all remote command invocation except initial host + check / version identification. + * Fixes a bug in ghe-backup where Git GC process detection would misclassify + long-running server processes matching /git.*gc/, causing the backup operation + to timeout. + * Adds a note and link to the Migrating from GitHub Enterprise v11.10.34x to + v2.0 documentation in the README. + * Adds example / documentation for the GHE_EXTRA_SSH_OPTS config value to the + backup.config-example file. + + -- Ryan Tomayko Mon, 17 Nov 2014 12:47:22 +0000 + +github-backup-utils (2.0.0) UNRELEASED; urgency=medium + + * Support for GitHub Enterprise 2.0. + * Support for migrating from GitHub Enterprise 11.10.34x to 2.0 (including from + VMware to AWS). + * ghe-backup retains hardlinks present on VM in backup snapshots, saving space. + * ghe-restore retains hardlinks present in backup snapshot when restoring to VM. + * backup-utils now includes debian packaging support. + * Fixes an issue with ghe-restore -s not using the snapshot specified. + * Fixes an issue with ghe-backup not waiting for nw-repack processes to finish + in some instances. + + -- Ryan Tomayko Mon, 10 Nov 2014 10:48:36 +0000 + github-backup-utils (1.1.0) UNRELEASED; urgency=medium * Updated documentation on minimum GitHub Enterprise version requirements for diff --git a/debian/control b/debian/control index c7a42cc30..4c9bb61b9 100644 --- a/debian/control +++ b/debian/control @@ -3,7 +3,7 @@ Maintainer: Twan Wolthof Section: misc Priority: optional Standards-Version: 3.9.2 -Build-Depends: debhelper (>= 9), git +Build-Depends: debhelper (>= 9), git, devscripts Package: github-backup-utils Architecture: any diff --git a/debian/rules b/debian/rules index 4915d7afb..290fed548 100755 --- a/debian/rules +++ b/debian/rules @@ -1,5 +1,7 @@ #!/usr/bin/make -f build-indep: +override_dh_auto_build: + %: dh $@ diff --git a/script/cibuild b/script/cibuild index cb712180e..32d13f504 100755 --- a/script/cibuild +++ b/script/cibuild @@ -1,5 +1,5 @@ -#!/bin/sh -# Usage: script/cibuild +#!/usr/bin/env bash +# Usage: script/cibuild [--no-package] set -e # GHE appliance versions to run tests against. Remote metadata files are put in @@ -9,6 +9,8 @@ set -e REMOTE_VERSIONS=" 11.10.344 2.0.0 + 2.2.0 + 2.5.0 " # Enable verbose logging of ssh commands @@ -20,7 +22,7 @@ for version in $REMOTE_VERSIONS do echo "==> Running testsuite with GHE_TEST_REMOTE_VERSION=$version" export GHE_TEST_REMOTE_VERSION="$version" - if ! ls -1 test/test-*.sh | xargs -P 4 -n 1 /bin/sh; then + if ! ls -1 test/test-*.sh | xargs -P 4 -n 1 /bin/bash; then res=false fi echo @@ -28,3 +30,44 @@ done # If any of the version tests failed, exit non-zero $res + +# Bail out when --no-package given +[ "$1" = "--no-package" ] && exit 0 + +# files we'll md5sum at the end +pkg_files= + +# Build the tarball +echo "Building tar.gz package ..." +if script/package-tarball 1>package-tarball.txt 2>&1; then + pkg_files=$(grep '^Package ' < package-tarball.txt | cut -f 2 -d ' ') +else + echo "Packaging tar.gz failed:" + cat package-tarball.txt | sed 's/^/ /' 1>&2 + exit 1 +fi + +# Skip deb packaging if debuild not installed +if ! type debuild 1>/dev/null 2>&1; then + echo "debuild not installed, skipping deb packaging ..." + exit 0 +fi + +# Build the deb related packages +echo "Building deb package ..." +if script/package-deb 1>package-deb-out.txt 2>package-deb-err.txt; then + pkg_files="$pkg_files $(cat package-deb-out.txt)" +else + echo "Package build failed:" + cat package-tarball.txt | sed 's/^/ /' 1>&2 + exit 1 +fi + +# Publish package files on wcat.io +echo "Uploading packages ..." +for f in $pkg_files; do + printf "%-32s %-s\n" "$(curl -sT- https://wcat.io <"$f" || true)" "$f" +done + +# Generate md5sums +md5sum $pkg_files diff --git a/script/package-deb b/script/package-deb new file mode 100755 index 000000000..b42f0d5fd --- /dev/null +++ b/script/package-deb @@ -0,0 +1,31 @@ +#!/usr/bin/env bash +# Usage: script/package-deb +# Script to build a deb release package from the current HEAD version. +# The package version comes from the debian/changelog file so that should +# be edited before running this. +set -e + +# Change into project root +cd "$(dirname "$0")"/.. + +# Basic package name and version. +PKG_BASE="github-backup-utils" +PKG_VERS="$(git describe --tags)" +PKG_NAME="${PKG_BASE}-${PKG_VERS}" +PKG_HEAD="$(git rev-parse HEAD)" + +# Run git-archive to generate tarball +rm -rf dist/debuild +trap "rm -rf dist/debuild" EXIT +mkdir -p dist/debuild + +distdir="$(pwd)/dist/debuild/$PKG_NAME" +git clone -q . "$distdir" +cd "$distdir" +git checkout -q "$PKG_HEAD" + +debuild -uc -us 1>&2 +cd .. +files=$(ls -1 *.deb *.tar.gz *.dsc *.changes) +mv $files ../ +for f in $files; do echo "dist/$f"; done diff --git a/script/package-tarball b/script/package-tarball index 6229cc33d..65f4ed51d 100755 --- a/script/package-tarball +++ b/script/package-tarball @@ -1,4 +1,4 @@ -#!/bin/sh +#!/usr/bin/env bash # Usage: script/package-tarball # Script to build a tarball release package from the current HEAD version. # The package version comes from `git-describe --tags' so the release tag should diff --git a/share/github-backup-utils/bm.sh b/share/github-backup-utils/bm.sh new file mode 100644 index 000000000..56421a569 --- /dev/null +++ b/share/github-backup-utils/bm.sh @@ -0,0 +1,51 @@ +#!/bin/bash +# bm.sh: benchmarking Bash code blocks +# +# Example: +# bm_start "wget request" +# wget --quiet https://www.google.com +# bm_end "wget request" +# +# Sample output: +# $ bash test.sh +# wget request took 2s + +bm_desc_to_varname(){ + echo "__bm$(echo $@ | tr -cd '[[:alnum:]]')" +} + +bm_start() +{ + eval "$(bm_desc_to_varname $@)_start=$(date +%s)" + + bm_file_path > /dev/null +} + +bm_file_path() { + if [ -n "$BM_FILE_PATH" ]; then + echo $BM_FILE_PATH + return + fi + + local logfile="benchmark.${BM_TIMESTAMP:-$(date +"%Y%m%dT%H%M%S")}.log" + if [ -n "$GHE_RESTORE_SNAPSHOT_PATH" ]; then + export BM_FILE_PATH=$GHE_RESTORE_SNAPSHOT_PATH/benchmarks/$logfile + else + export BM_FILE_PATH=$GHE_SNAPSHOT_DIR/benchmarks/$logfile + fi + + mkdir -p $(dirname $BM_FILE_PATH) + echo $BM_FILE_PATH +} + +bm_end() { + if [ -z "$BM_FILE_PATH" ]; then + echo "Call bm_start first" >&2 + exit 1 + fi + + local tend=$(date +%s) + local tstart=$(eval "echo \$$(bm_desc_to_varname $@)_start") + + echo "$1 took $(($tend - $tstart))s" >> $BM_FILE_PATH +} diff --git a/share/github-backup-utils/ghe-backup-alambic-cluster b/share/github-backup-utils/ghe-backup-alambic-cluster new file mode 100755 index 000000000..88e11a1f0 --- /dev/null +++ b/share/github-backup-utils/ghe-backup-alambic-cluster @@ -0,0 +1,85 @@ +#!/usr/bin/env bash +#/ Usage: ghe-backup-alambic-cluster +#/ Take an online, incremental snapshot of all Alambic Storage data +#/ +#/ Note: This command typically isn't called directly. It's invoked by +#/ ghe-backup when the cluster strategy is used. +set -e + +# Bring in the backup configuration +. $( dirname "${BASH_SOURCE[0]}" )/ghe-backup-config + +# Set up remote host and root backup snapshot directory based on config +host="$GHE_HOSTNAME" +backup_dir="$GHE_SNAPSHOT_DIR/storage" + +# Verify rsync is available. +if ! rsync --version 1>/dev/null 2>&1; then + echo "Error: rsync not found." 1>&2 + exit 1 +fi + +# Perform a host-check and establish GHE_REMOTE_XXX variables. +ghe_remote_version_required "$host" + +# Generate SSH config for forwarding + +config="" + +# Split host:port into parts +port=$(ssh_port_part "$GHE_HOSTNAME") +host=$(ssh_host_part "$GHE_HOSTNAME") + +# Add user / -l option +user="${host%@*}" +[ "$user" = "$host" ] && user="admin" + +# git server hostnames +hostnames=$(ghe_cluster_online_nodes "storage-server") + +for hostname in $hostnames; do + config="$config +Host $hostname + ProxyCommand ssh -q $GHE_EXTRA_SSH_OPTS -p $port $user@$host nc.openbsd %h %p + StrictHostKeyChecking=no +" +done + +config_file=$(mktemp -t cluster-backup-restore-XXXXXX) +echo "$config" > "$config_file" + +opts="$GHE_EXTRA_SSH_OPTS -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o PasswordAuthentication=no" + +# Make sure root backup dir exists if this is the first run +mkdir -p "$backup_dir" + +# Removes the remote sync-in-progress file on exit, re-enabling GC operations +# on the remote instance. +cleanup() { + rm -f $config_file +} +trap 'cleanup' EXIT INT + +# If we have a previous increment and it is not empty, avoid transferring existing files via rsync's +# --link-dest support. This also decreases physical space usage considerably. +if [ -d "$GHE_DATA_DIR/current/storage" ] && [ "$(ls -A $GHE_DATA_DIR/current/storage)" ]; then + link_dest="--link-dest=../../current/storage" +fi + +for hostname in $hostnames; do + echo 1>&3 + echo "* Starting backup for host: $hostname" + # Sync all auxiliary repository data. This includes files and directories like + # HEAD, audit_log, config, description, info/, etc. No refs or object data + # should be transferred here. + echo 1>&3 + echo "* Transferring storage files ..." 1>&3 + + # Transfer all data from the user data directory using rsync. + ghe-rsync -avz \ + -e "ssh -q $opts -p 122 -F $config_file -l $user" \ + --rsync-path='sudo -u git rsync' \ + $link_dest \ + "$hostname:$GHE_REMOTE_DATA_USER_DIR/storage/" \ + "$GHE_SNAPSHOT_DIR/storage" 1>&3 +done diff --git a/share/github-backup-utils/ghe-backup-config b/share/github-backup-utils/ghe-backup-config index 0b78d3a8f..277558a06 100755 --- a/share/github-backup-utils/ghe-backup-config +++ b/share/github-backup-utils/ghe-backup-config @@ -1,25 +1,29 @@ -#!/bin/sh +#!/usr/bin/env bash # Usage: . ghe-backup-config # GitHub Enterprise backup shell configuration. # # This file is sourced by the various utilities under bin and share/github-backup-utils to # load in backup configuration and ensure things are configured properly. # -# All commands should start with the following: +# All commands in share/github-backup-utils/ should start with the following: # -# cd $(dirname "$0")/../.. -# . share/github-backup-utils/ghe-backup-config +# . $( dirname "${BASH_SOURCE[0]}" )/ghe-backup-config +# +# And all commands in bin/ should start with the following: +# +# . $( dirname "${BASH_SOURCE[0]}" )/../share/github-backup-utils/ghe-backup-config # -# Assume the current directory is the root. This should be fine so long as all -# scripts source us in according to the instructions above. -GHE_BACKUP_ROOT="$(pwd)" +# Assume this script lives in share/github-backup-utils/ when setting the root +GHE_BACKUP_ROOT="$( cd "$( dirname "${BASH_SOURCE[0]}" )/../.." && pwd )" + +# Get the version from the version file. +BACKUP_UTILS_VERSION="$(cat $GHE_BACKUP_ROOT/share/github-backup-utils/version)" # Add the bin and share/github-backup-utils dirs to PATH PATH="$GHE_BACKUP_ROOT/bin:$GHE_BACKUP_ROOT/share/github-backup-utils:$PATH" -# The backup config file. This may be set in the environment. -: ${GHE_BACKUP_CONFIG:="$GHE_BACKUP_ROOT/backup.config"} +. $GHE_BACKUP_ROOT/share/github-backup-utils/bm.sh # Parse out -v (verbose) argument if [ "$1" = "-v" ]; then @@ -36,20 +40,34 @@ else exec 3>/dev/null fi -# Check that the config file exists before we source it in. -if [ ! -f "$GHE_BACKUP_CONFIG" ]; then - echo "Error: The backup config file ('$GHE_BACKUP_CONFIG') doesn't exist." 1>&2 - exit 2 -fi - # Save off GHE_HOSTNAME from the environment since we want it to override the # backup.config value when set. GHE_HOSTNAME_PRESERVE="$GHE_HOSTNAME" -# Source in the backup config file. -. "$GHE_BACKUP_CONFIG" +# Source in the backup config file from the copy specified in the environment +# first and then fall back to the backup-utils root, home directory and system. +config_found=false +for f in "$GHE_BACKUP_CONFIG" "$GHE_BACKUP_ROOT/backup.config" \ + "$HOME/.github-backup-utils/backup.config" "/etc/github-backup-utils/backup.config"; do + if [ -f "$f" ]; then + GHE_BACKUP_CONFIG="$f" + . "$GHE_BACKUP_CONFIG" + config_found=true + break + fi +done -# Restore saved of hostname. +# Check that the config file exists before we source it in. +if ! $config_found; then + echo "Error: No backup configuration file found. Tried:" 1>&2 + [ -n "$GHE_BACKUP_CONFIG" ] && echo " - $GHE_BACKUP_CONFIG" 1>&2 + echo " - $GHE_BACKUP_ROOT/backup.config" 1>&2 + echo " - $HOME/.github-backup-utils/backup.config" 1>&2 + echo " - /etc/github-backup-utils/backup.config" 1>&2 + exit 2 +fi + +# Restore saved off hostname. [ -n "$GHE_HOSTNAME_PRESERVE" ] && GHE_HOSTNAME="$GHE_HOSTNAME_PRESERVE" # Check that the GHE hostname is set. @@ -58,12 +76,26 @@ if [ -z "$GHE_HOSTNAME" ]; then exit 2 fi +# Convert the data directory path to an absolute path, basing any relative +# paths on the backup-utils root, and using readlink, if available, to +# canonicalize the path. +if [ ${GHE_DATA_DIR:0:1} != "/" ]; then + GHE_DATA_DIR="$( cd "$GHE_BACKUP_ROOT" && readlink -m "$GHE_DATA_DIR" 2> /dev/null || echo "$GHE_BACKUP_ROOT/$GHE_DATA_DIR" )" +fi + +GHE_CREATE_DATA_DIR=${GHE_CREATE_DATA_DIR:-yes} + # Check that the data directory is set and create it if it doesn't exist. -if [ ! -d "$GHE_DATA_DIR" ]; then +if [ ! -d "$GHE_DATA_DIR" ] && [ "$GHE_CREATE_DATA_DIR" = "yes" ]; then echo "Creating the backup data directory ..." 1>&3 mkdir -p "$GHE_DATA_DIR" fi +if [ ! -d "$GHE_DATA_DIR" ]; then + echo "Error: GHE_DATA_DIR $GHE_DATA_DIR does not exist." >&2 + exit 8 +fi + # Set some defaults if needed. : ${GHE_NUM_SNAPSHOTS:=10} @@ -98,8 +130,9 @@ GHE_SNAPSHOT_DIR="$GHE_DATA_DIR"/"$GHE_SNAPSHOT_TIMESTAMP" # allows the location to be overridden in tests. : ${GHE_REMOTE_LICENSE_FILE:="$GHE_REMOTE_DATA_DIR/enterprise/enterprise.ghl"} -# The location of the metadata file on the remote side. This is always -# "/data/enterprise/chef_metadata.json" for GitHub instances. Use of this variable +# The legacy location of the metadata file on the remote side. Only used if +# the newer "ghe-negotiate-version" script cannot be found or fails. This was +# "/data/enterprise/metadata.json" for GitHub instances. Use of this variable # allows the location to be overridden in tests. : ${GHE_REMOTE_METADATA_FILE:="$GHE_REMOTE_DATA_DIR/enterprise/chef_metadata.json"} @@ -115,6 +148,13 @@ GHE_SNAPSHOT_DIR="$GHE_DATA_DIR"/"$GHE_SNAPSHOT_TIMESTAMP" # Set "true" to get verbose logging of all ssh commands on stderr : ${GHE_VERBOSE_SSH:=false} +# The location of the cluster configuration file file on the remote side. +# This is always "/data/user/common/cluster.conf" for GitHub Cluster instances. +# Use of this variable allows the location to be overridden in tests. +: ${GHE_REMOTE_CLUSTER_CONF_FILE:="$GHE_REMOTE_DATA_DIR/user/common/cluster.conf"} + +# The location of the file used to disable GC operations on the remote side. +: ${SYNC_IN_PROGRESS_FILE:="$GHE_REMOTE_DATA_USER_DIR/repositories/.sync_in_progress"} ############################################################################### ### Dynamic remote version config @@ -218,3 +258,26 @@ ssh_host_part () { ssh_port_part () { [ "${1##*:}" = "$1" ] && echo 22 || echo "${1##*:}" } + +# Usage: ghe_remote_logger ... +# Log a message to /var/log/syslog on the remote instance. +# Note: Use sparingly. Remote logging requires an ssh connection per invocation. +ghe_remote_logger () { + echo "$@" | + ghe-ssh "$GHE_HOSTNAME" -- logger -t backup-utils || true +} + +# Usage: ghe_cluster_online_nodes role +# Returns the online nodes with a certain role in cluster +ghe_cluster_online_nodes () { + role=$1 + echo "ghe-config --get-regexp cluster.*.$role | egrep 'true$' | awk '{ print \$1; }' | awk 'BEGIN { FS=\".\" }; { print \$2 };' | xargs -I{} -n1 bash -c 'if [ \"\$(ghe-config cluster.\$hostname.offline)\" != true ]; then ghe-config cluster.{}.hostname; fi'" | ghe-ssh "$GHE_HOSTNAME" /bin/bash +} + +# Usage: ghe_verbose +# Log if verbose mode is enabled (GHE_VERBOSE or `-v`). +ghe_verbose() { + if [ -n "$GHE_VERBOSE" ]; then + echo "$@" 1>&3 + fi +} diff --git a/share/github-backup-utils/ghe-backup-es-audit-log b/share/github-backup-utils/ghe-backup-es-audit-log new file mode 100755 index 000000000..719725ea8 --- /dev/null +++ b/share/github-backup-utils/ghe-backup-es-audit-log @@ -0,0 +1,41 @@ +#!/usr/bin/env bash +#/ Usage: ghe-backup-es-audit-log +#/ Take a backup of audit logs in ElasticSearch. +#/ +#/ Note: This command typically isn't called directly. It's invoked by +#/ ghe-backup. +set -e + +# Bring in the backup configuration +. $( dirname "${BASH_SOURCE[0]}" )/ghe-backup-config + +# Set up remote host and root elastic backup directory based on config +host="$GHE_HOSTNAME" + +# Perform a host-check and establish GHE_REMOTE_XXX variables. +ghe_remote_version_required "$host" + +# Make sure root backup dir exists if this is the first run +mkdir -p "$GHE_SNAPSHOT_DIR/audit-log" + +if [ $GHE_VERSION_MAJOR -ge 2 ] && [ $GHE_VERSION_MINOR -ge 2 ]; then + es_port=9201 +else + es_port=9200 +fi + +if ! indices=$(ghe-ssh "$host" "curl -s \"localhost:$es_port/_cat/indices/audit_log*?h=index"\"); then + echo "Error: failed to retrieve audit log indices." 1>&2 + exit 1 +fi + +current_index=audit_log-$(ghe-ssh "$host" 'date +"%Y-%m"') + +for index in $indices; do + if [ -f $GHE_DATA_DIR/current/audit-log/$index.gz -a $index \< $current_index ]; then + # Hard link any older indices since they are read only and won't change + ln $GHE_DATA_DIR/current/audit-log/$index.gz $GHE_SNAPSHOT_DIR/audit-log/$index.gz + else + ghe-ssh "$host" "/usr/local/share/enterprise/ghe-es-dump-json \"http://localhost:$es_port/$index\"" | gzip > $GHE_SNAPSHOT_DIR/audit-log/$index.gz + fi +done diff --git a/share/github-backup-utils/ghe-backup-es-hookshot b/share/github-backup-utils/ghe-backup-es-hookshot new file mode 100755 index 000000000..2d16736da --- /dev/null +++ b/share/github-backup-utils/ghe-backup-es-hookshot @@ -0,0 +1,31 @@ +#!/usr/bin/env bash +#/ Usage: ghe-backup-es-hookshot +#/ Take a backup of hookshot logs in ElasticSearch. +#/ +#/ Note: This command typically isn't called directly. It's invoked by +#/ ghe-backup. +set -e + +# Bring in the backup configuration +. $( dirname "${BASH_SOURCE[0]}" )/ghe-backup-config + +# Set up remote host and root elastic backup directory based on config +host="$GHE_HOSTNAME" + +# Perform a host-check and establish GHE_REMOTE_XXX variables. +ghe_remote_version_required "$host" + +# Make sure root backup dir exists if this is the first run +mkdir -p "$GHE_SNAPSHOT_DIR/hookshot" + +indices=$(ghe-ssh "$host" 'curl -s "localhost:9201/_cat/indices/hookshot-logs-*"' | cut -d ' ' -f 3) +current_index=hookshot-logs-$(ghe-ssh "$host" 'date +"%Y-%m-%d"') + +for index in $indices; do + if [ -f $GHE_DATA_DIR/current/hookshot/$index.gz -a $index \< $current_index ]; then + # Hard link any older indices since they are read only and won't change + ln $GHE_DATA_DIR/current/hookshot/$index.gz $GHE_SNAPSHOT_DIR/hookshot/$index.gz + else + ghe-ssh "$host" "/usr/local/share/enterprise/ghe-es-dump-json 'http://localhost:9201/$index'" | gzip > $GHE_SNAPSHOT_DIR/hookshot/$index.gz + fi +done diff --git a/share/github-backup-utils/ghe-backup-es-rsync b/share/github-backup-utils/ghe-backup-es-rsync index c3d31cca5..e3722fd18 100755 --- a/share/github-backup-utils/ghe-backup-es-rsync +++ b/share/github-backup-utils/ghe-backup-es-rsync @@ -1,4 +1,4 @@ -#!/bin/sh +#!/usr/bin/env bash #/ Usage: ghe-backup-es-rsync #/ Take an online, incremental snapshot of Elasticsearch indices. #/ @@ -7,8 +7,7 @@ set -e # Bring in the backup configuration -cd $(dirname "$0")/../.. -. share/github-backup-utils/ghe-backup-config +. $( dirname "${BASH_SOURCE[0]}" )/ghe-backup-config # Set up remote host and root elastic backup directory based on config host="$GHE_HOSTNAME" diff --git a/share/github-backup-utils/ghe-backup-es-tarball b/share/github-backup-utils/ghe-backup-es-tarball index ebce38c47..25d9222d7 100755 --- a/share/github-backup-utils/ghe-backup-es-tarball +++ b/share/github-backup-utils/ghe-backup-es-tarball @@ -1,4 +1,4 @@ -#!/bin/sh +#!/usr/bin/env bash #/ Usage: ghe-backup-es-tarball #/ Take a tarball snapshot of all Elasticsearch data. #/ @@ -7,8 +7,7 @@ set -e # Bring in the backup configuration -cd $(dirname "$0")/../.. -. share/github-backup-utils/ghe-backup-config +. $( dirname "${BASH_SOURCE[0]}" )/ghe-backup-config # Snapshot all Elasticsearch data or fake it when no /data/elasticsearch # directory exists. diff --git a/share/github-backup-utils/ghe-backup-fsck b/share/github-backup-utils/ghe-backup-fsck new file mode 100755 index 000000000..19fd68e55 --- /dev/null +++ b/share/github-backup-utils/ghe-backup-fsck @@ -0,0 +1,85 @@ +#!/usr/bin/env bash +#/ Usage: ghe-backup-fsck [--print-nwo] +#/ +#/ Run git fsck on backed up repositories. +set -e + +# Bring in the backup configuration +. $( dirname "${BASH_SOURCE[0]}" )/ghe-backup-config + +echo "Running git fsck on repos..." + +# Verify git is available. +if ! git --version 1>/dev/null 2>&1; then + echo "Error: git not found." 1>&2 + exit 1 +fi + +sdir=$1 +repos=0 +errors=0 +log=$(mktemp -t ghe-backup-fsck-XXXXXX) +t_start=$(date +%s) +if git fsck -h | grep -q '\-\-dangling'; then + git_cmd='git fsck --no-dangling' +else + echo "Warning: old git version, --no-dangling not available" + git_cmd='git fsck' +fi + +if [ -z "$sdir" ] || [ ! -d "$sdir" ]; then + print_usage +fi + +if [ ! -d "$sdir/repositories" ]; then + echo "Error: $sdir is not a valid snapshot." >&2 + exit 1 +fi + +for repo in $(find $sdir/repositories/ -type d -name \*.git); do + repos=$(($repos+1)) + before_time=$(date +%s) + + status=$( + set -e + + cd $repo + + nwo="-" + if [ "$2" = "--print-nwo" ] && [ -f info/nwo ]; then + nwo="$(cat info/nwo)" + fi + + if [ ! -f objects/info/alternates ] || grep -q '^\.\.' objects/info/alternates; then + $git_cmd >$log 2>&1 && { + echo "OK $repo $nwo"; exit + } + else + GIT_ALTERNATE_OBJECT_DIRECTORIES=../network.git/objects $git_cmd >$log 2>&1 && { + echo "WARN $repo $nwo (alternates absolute path)"; exit + } + fi + + echo "ERROR $repo $nwo" + ) + + elapsed_time=$(($(date +%s) - before_time)) + + if [[ ! "$status" =~ ^OK ]] || [ $elapsed_time -gt 5 ]; then + echo "$status ${elapsed_time}s" 1>&3 + [ -n "$GHE_VERBOSE" ] && cat $log + fi + + case "$status" in + OK*) + ;; + ERROR*) + errors=$(($errors+1)) + ;; + esac + +done + +echo "* Repos verified: $repos, Errors: $errors, Took: $(($(date +%s) - $t_start))s" + +rm -f $log diff --git a/share/github-backup-utils/ghe-backup-git-hooks-cluster b/share/github-backup-utils/ghe-backup-git-hooks-cluster new file mode 100755 index 000000000..295ce1c52 --- /dev/null +++ b/share/github-backup-utils/ghe-backup-git-hooks-cluster @@ -0,0 +1,117 @@ +#!/usr/bin/env bash +#/ Usage: ghe-backup-git-hooks-cluster +#/ Take an online, incremental snapshot of custom Git hooks configuration. +#/ +#/ Note: This command typically isn't called directly. It's invoked by +#/ ghe-backup when the cluster strategy is used. +set -e + +# Bring in the backup configuration +. $( dirname "${BASH_SOURCE[0]}" )/ghe-backup-config + +# Verify rsync is available. +if ! rsync --version 1>/dev/null 2>&1; then + echo "Error: rsync not found." 1>&2 + exit 1 +fi + +bm_start "$(basename $0)" + +backup_dir="$GHE_SNAPSHOT_DIR/git-hooks" +# Location of last good backup for rsync --link-dest +backup_current="$GHE_DATA_DIR/current/git-hooks" + +# Perform a host-check and establish GHE_REMOTE_XXX variables. +ghe_remote_version_required "$host" + +# Split host:port into parts +port=$(ssh_port_part "$GHE_HOSTNAME") +host=$(ssh_host_part "$GHE_HOSTNAME") + +# Add user / -l option +user="${host%@*}" +[ "$user" = "$host" ] && user="admin" + +# Generate SSH config for forwarding +config="" + +# git server hostnames +hostnames=$(ghe_cluster_online_nodes "git-server") +for hostname in $hostnames; do + config="$config +Host $hostname + ProxyCommand ssh -q $GHE_EXTRA_SSH_OPTS -p $port $user@$host nc.openbsd %h %p + StrictHostKeyChecking=no +" +done + +config_file=$(mktemp -t cluster-backup-restore-XXXXXX) +echo "$config" > "$config_file" + +opts="$GHE_EXTRA_SSH_OPTS -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o PasswordAuthentication=no" + +# Removes the remote sync-in-progress file on exit, re-enabling GC operations +# on the remote instance. +cleanup() { + rm -f $config_file +} +trap 'cleanup' EXIT +trap 'exit $?' INT # ^C always terminate + +# Transfer Git hooks data from a GitHub instance to the current snapshot +# directory, using a previous snapshot to avoid transferring files that have +# already been transferred. A set of rsync filter rules are provided on stdin +# for each invocation. +rsync_git_hooks_data () { + port=$(ssh_port_part "$1") + host=$(ssh_host_part "$1") + + subpath=$2 + shift 2 + + # If we have a previous increment and it is not empty, avoid transferring existing files via rsync's + # --link-dest support. This also decreases physical space usage considerably. + if [ -d "$backup_current/$subpath" ] && [ "$(ls -A $backup_current/$subpath)" ]; then + + subdir="git-hooks/$subpath" + link_path=".." + while true; do + if [ $(dirname $subdir) = "." ]; then + break + fi + + if [ $(dirname $subdir) = "/" ]; then + break + fi + + link_path="../$link_path" + subdir=$(dirname $subdir) + done + + local link_dest="--link-dest=../${link_path}/current/git-hooks/$subpath" + fi + + # Ensure target directory exists, is needed with subdirectories + mkdir -p "$backup_dir/$subpath" + + ghe-rsync -av \ + -e "ssh -q $opts -p $port -F $config_file -l $user" $link_dest \ + --rsync-path='sudo -u git rsync' \ + "$host:$GHE_REMOTE_DATA_USER_DIR/git-hooks/$subpath/" \ + "$backup_dir/$subpath" 1>&3 +} + +hostname=$(echo $hostnames | awk '{ print $1; }') +if ghe-ssh -F $config_file "$hostname:122" -- "sudo -u git [ -d '$GHE_REMOTE_DATA_USER_DIR/git-hooks/environments/tarballs' ]"; then + rsync_git_hooks_data $hostname:122 environments/tarballs +else + ghe_verbose "git-hooks environment tarballs not found. Skipping ..." +fi + +if ghe-ssh -F $config_file "$hostname:122" -- "sudo -u git [ -d '$GHE_REMOTE_DATA_USER_DIR/git-hooks/repos' ]"; then + rsync_git_hooks_data $hostname:122 repos +else + ghe_verbose "git-hooks repositories not found. Skipping ..." +fi + +bm_end "$(basename $0)" diff --git a/share/github-backup-utils/ghe-backup-pages-cluster b/share/github-backup-utils/ghe-backup-pages-cluster new file mode 100755 index 000000000..55c912407 --- /dev/null +++ b/share/github-backup-utils/ghe-backup-pages-cluster @@ -0,0 +1,85 @@ +#!/usr/bin/env bash +#/ Usage: ghe-backup-pages-cluster +#/ Take an online, incremental snapshot of all Pages data +#/ +#/ Note: This command typically isn't called directly. It's invoked by +#/ ghe-backup when the cluster strategy is used. +set -e + +# Bring in the backup configuration +. $( dirname "${BASH_SOURCE[0]}" )/ghe-backup-config + +# Set up remote host and root backup snapshot directory based on config +host="$GHE_HOSTNAME" +backup_dir="$GHE_SNAPSHOT_DIR/pages" + +# Verify rsync is available. +if ! rsync --version 1>/dev/null 2>&1; then + echo "Error: rsync not found." 1>&2 + exit 1 +fi + +# Perform a host-check and establish GHE_REMOTE_XXX variables. +ghe_remote_version_required "$host" + +# Generate SSH config for forwarding + +config="" + +# Split host:port into parts +port=$(ssh_port_part "$GHE_HOSTNAME") +host=$(ssh_host_part "$GHE_HOSTNAME") + +# Add user / -l option +user="${host%@*}" +[ "$user" = "$host" ] && user="admin" + +# git server hostnames +hostnames=$(ghe_cluster_online_nodes "pages-server") + +for hostname in $hostnames; do + config="$config +Host $hostname + ProxyCommand ssh -q $GHE_EXTRA_SSH_OPTS -p $port $user@$host nc.openbsd %h %p + StrictHostKeyChecking=no +" +done + +config_file=$(mktemp -t cluster-backup-restore-XXXXXX) +echo "$config" > "$config_file" + +opts="$GHE_EXTRA_SSH_OPTS -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o PasswordAuthentication=no" + +# Make sure root backup dir exists if this is the first run +mkdir -p "$backup_dir" + +# Removes the remote sync-in-progress file on exit, re-enabling GC operations +# on the remote instance. +cleanup() { + rm -f $config_file +} +trap 'cleanup' EXIT INT + +# If we have a previous increment and it is not empty, avoid transferring existing files via rsync's +# --link-dest support. This also decreases physical space usage considerably. +if [ -d "$GHE_DATA_DIR/current/pages" ] && [ "$(ls -A $GHE_DATA_DIR/current/pages)" ]; then + link_dest="--link-dest=../../current/pages" +fi + +for hostname in $hostnames; do + echo 1>&3 + echo "* Starting backup for host: $hostname" + # Sync all auxiliary repository data. This includes files and directories like + # HEAD, audit_log, config, description, info/, etc. No refs or object data + # should be transferred here. + echo 1>&3 + echo "* Transferring pages files ..." 1>&3 + + # Transfer all data from the user data directory using rsync. + ghe-rsync -avz \ + -e "ssh -q $opts -p 122 -F $config_file -l $user" \ + --rsync-path='sudo -u git rsync' \ + $link_dest \ + "$hostname:$GHE_REMOTE_DATA_USER_DIR/pages/" \ + "$GHE_SNAPSHOT_DIR/pages" 1>&3 +done diff --git a/share/github-backup-utils/ghe-backup-pages-rsync b/share/github-backup-utils/ghe-backup-pages-rsync index 8f470b157..e2d541319 100755 --- a/share/github-backup-utils/ghe-backup-pages-rsync +++ b/share/github-backup-utils/ghe-backup-pages-rsync @@ -1,4 +1,4 @@ -#!/bin/sh +#!/usr/bin/env bash #/ Usage: ghe-backup-pages-rsync #/ Take an online, incremental snapshot of all Pages data. #/ @@ -7,8 +7,10 @@ set -e # Bring in the backup configuration -cd $(dirname "$0")/../.. -. share/github-backup-utils/ghe-backup-config +. $( dirname "${BASH_SOURCE[0]}" )/ghe-backup-config + +# Make sure root backup dir exists if this is the first run +mkdir -p "$GHE_SNAPSHOT_DIR/pages" # Use the common user data rsync backup utility. ghe-backup-userdata pages diff --git a/share/github-backup-utils/ghe-backup-pages-tarball b/share/github-backup-utils/ghe-backup-pages-tarball index 7e8bda265..574ee7c93 100755 --- a/share/github-backup-utils/ghe-backup-pages-tarball +++ b/share/github-backup-utils/ghe-backup-pages-tarball @@ -1,4 +1,4 @@ -#!/bin/sh +#!/usr/bin/env bash #/ Usage: ghe-backup-pages-tarball #/ Take a tarball snapshot of all Pages data. #/ @@ -7,8 +7,7 @@ set -e # Bring in the backup configuration -cd $(dirname "$0")/../.. -. share/github-backup-utils/ghe-backup-config +. $( dirname "${BASH_SOURCE[0]}" )/ghe-backup-config # Snapshot all Pages data or fake it when no /data/pages directory exists. echo ' diff --git a/share/github-backup-utils/ghe-backup-redis b/share/github-backup-utils/ghe-backup-redis index 10ea8bde6..d21623315 100755 --- a/share/github-backup-utils/ghe-backup-redis +++ b/share/github-backup-utils/ghe-backup-redis @@ -1,4 +1,4 @@ -#!/bin/sh +#!/usr/bin/env bash #/ Usage: ghe-backup-redis #/ Take a snapshot of all Redis data. This is needed because older versions of #/ the remote side ghe-export-redis command use a blocking SAVE instead of a @@ -9,13 +9,14 @@ set -e # Bring in the backup configuration -cd $(dirname "$0")/../.. -. share/github-backup-utils/ghe-backup-config +. $( dirname "${BASH_SOURCE[0]}" )/ghe-backup-config # Perform a host-check and establish GHE_REMOTE_XXX variables. ghe_remote_version_required "$GHE_HOSTNAME" # Force a redis BGSAVE, and wait for it to complete. +sudo= +[ "$GHE_VERSION_MAJOR" -ge 2 ] && sudo="sudo" ghe-ssh "$GHE_HOSTNAME" /bin/sh </dev/null || echo "localhost") + timestamp=\$(redis-cli -h \$redis_host LASTSAVE) + redis-cli -h \$redis_host BGSAVE 1>/dev/null + + while [ \$(redis-cli -h \$redis_host LASTSAVE) -eq \$timestamp ]; do + sleep 1 + done + + if [ "\$redis_host" != "localhost" ]; then + ssh \$redis_host $sudo cat '$GHE_REMOTE_DATA_USER_DIR/redis/dump.rdb' + else + $sudo cat '$GHE_REMOTE_DATA_USER_DIR/redis/dump.rdb' + fi +EOF diff --git a/share/github-backup-utils/ghe-backup-repositories-cluster b/share/github-backup-utils/ghe-backup-repositories-cluster new file mode 100755 index 000000000..20dbffd7c --- /dev/null +++ b/share/github-backup-utils/ghe-backup-repositories-cluster @@ -0,0 +1,289 @@ +#!/usr/bin/env bash +#/ Usage: ghe-backup-repositories-cluster +#/ Take an online, incremental snapshot of all Git repository data. +#/ +#/ Note: This command typically isn't called directly. It's invoked by +#/ ghe-backup when the cluster strategy is used. +set -e + +# This command is designed to allow for transferring active Git repository data +# from a GitHub instance to a backup site in a way that ensures data is +# captured in a consistent state even when being written to. +# +# - All Git GC operations are disabled on the GitHub instance for the duration of +# the backup. This removes the possibly of objects or packs being removed +# while the backup is in progress. +# +# - In progress Git GC operations are given a cooldown window to complete. The +# script will sleep for up to 60 seconds waiting for GC operations to finish. +# +# - Git repository data is transferred in a specific order: auxiliary files, +# packed refs, loose refs, reflogs, and finally objects and pack files in that +# order. This ensures that all referenced objects are captured. +# +# - Git GC operations are re-enabled on the GitHub instance. +# +# The script uses multiple runs of rsync to transfer repository files. Each run +# includes a list of filter rules that ensure only specific types of files are +# transferred. +# +# See the "FILTER RULES" and "INCLUDE/EXCLUDE PATTERN RULES" sections of the +# rsync(1) manual for more information: +# + +# Bring in the backup configuration +. $( dirname "${BASH_SOURCE[0]}" )/ghe-backup-config + +# Split host:port into parts +port=$(ssh_port_part "$GHE_HOSTNAME") +host=$(ssh_host_part "$GHE_HOSTNAME") + +# Add user / -l option +user="${host%@*}" +[ "$user" = "$host" ] && user="admin" + +backup_dir="$GHE_SNAPSHOT_DIR/repositories" + +# Location of last good backup for rsync --link-dest +backup_current="$GHE_DATA_DIR/current/repositories" + +# Verify rsync is available. +if ! rsync --version 1>/dev/null 2>&1; then + echo "Error: rsync not found." 1>&2 + exit 1 +fi + +# Perform a host-check and establish GHE_REMOTE_XXX variables. +ghe_remote_version_required "$host" + +# Generate SSH config for forwarding + +config="" + +# Split host:port into parts +port=$(ssh_port_part "$GHE_HOSTNAME") +host=$(ssh_host_part "$GHE_HOSTNAME") + +# Add user / -l option +user="${host%@*}" +[ "$user" = "$host" ] && user="admin" + +# git server hostnames +hostnames=$(ghe_cluster_online_nodes "git-server") +for hostname in $hostnames; do + config="$config +Host $hostname + ProxyCommand ssh -q $GHE_EXTRA_SSH_OPTS -p $port $user@$host nc.openbsd %h %p + StrictHostKeyChecking=no +" +done + +config_file=$(mktemp -t cluster-backup-restore-XXXXXX) +echo "$config" > "$config_file" + +opts="$GHE_EXTRA_SSH_OPTS -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o PasswordAuthentication=no" + +# Make sure root backup dir exists if this is the first run +mkdir -p "$backup_dir" + +# Removes the remote sync-in-progress file on exit, re-enabling GC operations +# on the remote instance. +cleanup() { + # Enable remote GC operations + for hostname in $hostnames; do + ghe-gc-enable -F $config_file $hostname:$port + done + rm -f $config_file +} +trap 'cleanup' EXIT +trap 'exit $?' INT # ^C always terminate + +# Disable remote GC operations +for hostname in $hostnames; do + ghe-gc-disable -F $config_file $hostname:$port +done + +# If we have a previous increment, avoid transferring existing files via rsync's +# --link-dest support. This also decreases physical space usage considerably. +if [ -d "$backup_current" ]; then + link_dest="--link-dest=../../current/repositories" +fi + +# Transfer repository data from a GitHub instance to the current snapshot +# directory, using a previous snapshot to avoid transferring files that have +# already been transferred. A set of rsync filter rules are provided on stdin +# for each invocation. +rsync_repository_data () { + port=$(ssh_port_part "$1") + host=$(ssh_host_part "$1") + + shift + ghe-rsync -av \ + -e "ssh -q $opts -p $port -F $config_file -l $user" \ + $link_dest "$@" \ + --rsync-path='sudo -u git rsync' \ + --include-from=- --exclude=\* \ + "$host:$GHE_REMOTE_DATA_USER_DIR/repositories/" \ + "$backup_dir" 1>&3 +} + + +for hostname in $hostnames; do + echo 1>&3 + echo "* Starting backup for host: $hostname" + # Sync all auxiliary repository data. This includes files and directories like + # HEAD, audit_log, config, description, info/, etc. No refs or object data + # should be transferred here. + echo 1>&3 + echo "* Transferring auxiliary files ..." 1>&3 + rsync_repository_data $hostname:122 -z <&3 + echo "* Transferring packed-refs files ..." 1>&3 + rsync_repository_data $hostname:122 -z <&3 +echo "* Transferring refs and reflogs ..." 1>&3 +rsync_repository_data $hostname:122 -z <&3 + echo "* Transferring objects and packs ..." 1>&3 + rsync_repository_data $hostname:122 -H <&3 + echo "* Transferring special data directories ..." 1>&3 + rsync_repository_data $hostname:122 <&3 +done diff --git a/share/github-backup-utils/ghe-backup-repositories-rsync b/share/github-backup-utils/ghe-backup-repositories-rsync index 9c2a98310..3a431fda2 100755 --- a/share/github-backup-utils/ghe-backup-repositories-rsync +++ b/share/github-backup-utils/ghe-backup-repositories-rsync @@ -1,4 +1,4 @@ -#!/bin/sh +#!/usr/bin/env bash #/ Usage: ghe-backup-repositories-rsync #/ Take an online, incremental snapshot of all Git repository data. #/ @@ -32,8 +32,7 @@ set -e # # Bring in the backup configuration -cd $(dirname "$0")/../.. -. share/github-backup-utils/ghe-backup-config +. $( dirname "${BASH_SOURCE[0]}" )/ghe-backup-config # Set up remote host and root backup snapshot directory based on config host="$GHE_HOSTNAME" @@ -51,46 +50,20 @@ fi # Perform a host-check and establish GHE_REMOTE_XXX variables. ghe_remote_version_required "$host" -# Remote sync-in-progress file location. When this file exists, Git GC -# operations are disabled on the GitHub instance. -sync_in_progress_file="$GHE_REMOTE_DATA_USER_DIR/repositories/.sync_in_progress" - # Make sure root backup dir exists if this is the first run mkdir -p "$backup_dir" # Removes the remote sync-in-progress file on exit, re-enabling GC operations # on the remote instance. cleanup() { - ghe-ssh "$host" -- "sudo rm -f '$sync_in_progress_file'" + # Enable remote GC operations + ghe-gc-enable $host } trap 'cleanup' EXIT trap 'exit $?' INT # ^C always terminate -# Touch the sync-in-progress file, disabling GC operations, and wait for all -# active GC processes to finish on the remote side. -echo " - set -e - sudo -u git touch '$sync_in_progress_file' - - sanity=0 - while [ \$sanity -lt $GHE_GIT_COOLDOWN_PERIOD ]; do - # note: the bracket synta[x] below is to prevent matches against the - # grep process itself. - if ps axo pid,args | grep -q -e 'git.*nw-repac[k]' -e 'git.*g[c]'; then - sleep 1 - sanity=\$(( sanity + 1 )) - else - exit 0 - fi - done - exit 7 -" | ghe-ssh "$host" -- /bin/sh || { - res=$? - if [ $res = 7 ]; then - echo "Error: Git GC processes remain after $GHE_GIT_COOLDOWN_PERIOD seconds. Aborting..." 1>&2 - fi - exit $res -} +# Disable remote GC operations +ghe-gc-disable $host # Transfer repository data from a GitHub instance to the current snapshot # directory, using a previous snapshot to avoid transferring files that have @@ -119,6 +92,8 @@ echo 1>&3 echo "* Transferring auxiliary files ..." 1>&3 rsync_repository_data -z <&3 echo "* Transferring packed-refs files ..." 1>&3 rsync_repository_data -z <&3 echo "* Transferring refs and reflogs ..." 1>&3 rsync_repository_data -z <&3 echo "* Transferring objects and packs ..." 1>&3 rsync_repository_data -H <&3 diff --git a/share/github-backup-utils/ghe-backup-repositories-tarball b/share/github-backup-utils/ghe-backup-repositories-tarball index 09ee15ac4..2f083a974 100755 --- a/share/github-backup-utils/ghe-backup-repositories-tarball +++ b/share/github-backup-utils/ghe-backup-repositories-tarball @@ -1,4 +1,4 @@ -#!/bin/sh +#!/usr/bin/env bash #/ Usage: ghe-backup-repositories-tarball #/ Take a tarball snapshot of all Git repository data. #/ @@ -7,8 +7,7 @@ set -e # Bring in the backup configuration -cd $(dirname "$0")/../.. -. share/github-backup-utils/ghe-backup-config +. $( dirname "${BASH_SOURCE[0]}" )/ghe-backup-config # Snapshot all Git repository data ghe-ssh "$GHE_HOSTNAME" -- 'ghe-export-repositories' > "$GHE_SNAPSHOT_DIR"/repositories.tar diff --git a/share/github-backup-utils/ghe-backup-settings b/share/github-backup-utils/ghe-backup-settings index 87c9c1d76..9c1caf40b 100755 --- a/share/github-backup-utils/ghe-backup-settings +++ b/share/github-backup-utils/ghe-backup-settings @@ -1,11 +1,10 @@ -#!/bin/sh +#!/usr/bin/env bash #/ Usage: ghe-backup-settings #/ Restore settings from a snapshot to the given . set -e # Bring in the backup configuration -cd $(dirname "$0")/../.. -. share/github-backup-utils/ghe-backup-config +. $( dirname "${BASH_SOURCE[0]}" )/ghe-backup-config # Perform a host-check and establish GHE_REMOTE_XXX variables. ghe_remote_version_required "$host" @@ -21,7 +20,9 @@ echo "* Transferring settings data ..." 1>&3 ghe-ssh "$host" -- 'ghe-export-settings' > settings.json echo "* Transferring license data ..." 1>&3 -ghe-ssh "$host" -- "cat '$GHE_REMOTE_LICENSE_FILE'" > enterprise.ghl +comm="cat '$GHE_REMOTE_LICENSE_FILE'" +[ "$GHE_VERSION_MAJOR" -ge 2 ] && comm="sudo $comm" +ghe-ssh "$host" -- "$comm" > enterprise.ghl if [ "$GHE_VERSION_MAJOR" -ge 2 ]; then echo "* Transferring management console password ..." 1>&3 @@ -35,4 +36,17 @@ if [ "$GHE_VERSION_MAJOR" -ge 2 ]; then else unlink manage-password+ fi + + if ghe-ssh "$host" -- "test -f $GHE_REMOTE_DATA_USER_DIR/common/idp.crt"; then + echo "* Transferring SAML keys ..." 1>&3 + ghe-ssh $host -- sudo tar -C $GHE_REMOTE_DATA_USER_DIR/common/ -cf - "idp.crt saml-sp.p12" > saml-keys.tar + fi +fi + +if [ "$GHE_BACKUP_STRATEGY" = "cluster" ]; then + echo "* Transferring cluster configuration ..." 1>&3 + if ! ghe-ssh "$host" -- "sudo cat $GHE_REMOTE_CLUSTER_CONF_FILE 2>/dev/null" > cluster.conf; then + echo "Error: Enterprise Cluster is not configured yet, backup will fail" >&2 + exit 1 + fi fi diff --git a/share/github-backup-utils/ghe-backup-store-version b/share/github-backup-utils/ghe-backup-store-version new file mode 100755 index 000000000..5826af85a --- /dev/null +++ b/share/github-backup-utils/ghe-backup-store-version @@ -0,0 +1,17 @@ +#!/usr/bin/env bash +#/ Usage: ghe-backup-store-version +#/ Stores information about the used version of backup-utils on +set -e + +# Bring in the backup configuration +. $( dirname "${BASH_SOURCE[0]}" )/ghe-backup-config + +if [ -d $GHE_BACKUP_ROOT/.git ]; then + version_info="$BACKUP_UTILS_VERSION" + ref=$(git --git-dir=$GHE_BACKUP_ROOT/.git rev-parse HEAD || true) + if [ -n "$ref" ]; then + version_info="$version_info:$ref" + fi + echo "$version_info" | + ghe-ssh "$GHE_HOSTNAME" -- "sudo dd of=$GHE_REMOTE_DATA_USER_DIR/common/backup-utils-version >/dev/null 2>&1" +fi diff --git a/share/github-backup-utils/ghe-backup-userdata b/share/github-backup-utils/ghe-backup-userdata index 187982511..5cb68f8eb 100755 --- a/share/github-backup-utils/ghe-backup-userdata +++ b/share/github-backup-utils/ghe-backup-userdata @@ -1,4 +1,4 @@ -#!/bin/sh +#!/usr/bin/env bash #/ Usage: ghe-backup-userdata #/ Take an online, incremental snapshot of a user data directory. This is used #/ for a number of different simple datastores kept under /data/user on the @@ -6,8 +6,7 @@ set -e # Bring in the backup configuration -cd $(dirname "$0")/../.. -. share/github-backup-utils/ghe-backup-config +. $( dirname "${BASH_SOURCE[0]}" )/ghe-backup-config # Verify rsync is available. if ! rsync --version 1>/dev/null 2>&1; then @@ -24,14 +23,33 @@ ghe_remote_version_required "$host" # Verify that the user data directory exists. Bail out if not, which may be due # to an older version of GHE or no data has been added to this directory yet. -ghe-ssh "$host" -- "[ -d '$GHE_REMOTE_DATA_USER_DIR/$dirname' ]" || exit 0 +ghe-ssh "$host" -- "sudo -u git [ -d '$GHE_REMOTE_DATA_USER_DIR/$dirname' ]" || exit 0 -# If we have a previous increment, avoid transferring existing files via rsync's +# If we have a previous increment and it is not empty, avoid transferring existing files via rsync's # --link-dest support. This also decreases physical space usage considerably. -if [ -d "$GHE_DATA_DIR/current/$dirname" ]; then - link_dest="--link-dest=../../current/$dirname" +if [ -d "$GHE_DATA_DIR/current/$dirname" ] && [ "$(ls -A $GHE_DATA_DIR/current/$dirname)" ]; then + + subdir=$dirname + link_path=".." + while true; do + if [ $(dirname $subdir) = "." ]; then + break + fi + + if [ $(dirname $subdir) = "/" ]; then + break + fi + + link_path="../$link_path" + subdir=$(dirname $subdir) + done + + link_dest="--link-dest=../${link_path}/current/$dirname" fi +# Ensure target directory exists, is needed with subdirectories +mkdir -p "$GHE_SNAPSHOT_DIR/$dirname" + # Transfer all data from the user data directory using rsync. ghe-rsync -avz \ -e "ghe-ssh -p $(ssh_port_part "$host")" \ diff --git a/share/github-backup-utils/ghe-gc-disable b/share/github-backup-utils/ghe-gc-disable new file mode 100755 index 000000000..fd3065007 --- /dev/null +++ b/share/github-backup-utils/ghe-gc-disable @@ -0,0 +1,52 @@ +#!/usr/bin/env bash +#/ Usage: ghe-gc-disable [