|
| 1 | +--- |
| 2 | +layout: post |
| 3 | +title: Backing up Datadog and Performing Bulk Edits |
| 4 | +tags: |
| 5 | +- datadog |
| 6 | +- monitoring |
| 7 | +- featured |
| 8 | +team: Core Infrastructure |
| 9 | +authors: |
| 10 | +- jimp |
| 11 | +- kamranf |
| 12 | + |
| 13 | +--- |
| 14 | + |
| 15 | +# Datadog Backup |
| 16 | + |
| 17 | +What would happen if someone accidentally deleted a dashboard or important |
| 18 | +monitor in Datadog? How would we know that it had changed? All of our monitoring |
| 19 | +and metrics at Scribd are configured in [Datadog](https://www.datadoghq.com), so |
| 20 | +it was important to have a mechanism to **track changes** across all these |
| 21 | +resources and be able to **revert back** to a previous state in case of a bad |
| 22 | +change. We also wanted the ability to **search and edit** across all dashboards |
| 23 | +and monitors (to refactor a tag, for example). |
| 24 | + |
| 25 | +When composing dashboards and monitors, immediate feedback is critical. |
| 26 | +Therefore, as we evaluated existing tools, we looked for the ability to coexist |
| 27 | +with Datadog’s native user interface. |
| 28 | + |
| 29 | +Finding none that exactly fit our needs, we wrote [Datadog |
| 30 | +Backup](https://github.com/scribd/datadog_backup), an open source Ruby Gem. |
| 31 | +Datadog Backup focuses on the ability to perform global backups and bulk editing |
| 32 | +without impeding the free use of the Datadog User Interface. |
| 33 | + |
| 34 | +Aspects of this project were inspired by the likes of |
| 35 | +[Doggy](https://github.com/Shopify/doggy). As you consider your approach to |
| 36 | +Datadog management, we highly recommend you also take a look at it and other |
| 37 | +client libraries at |
| 38 | +[https://docs.datadoghq.com/developers/libraries/#datadog-client-community-libraries](https://docs.datadoghq.com/developers/libraries/#datadog-client-community-libraries). |
| 39 | + |
| 40 | + |
| 41 | +## How to use Datadog Backup |
| 42 | + |
| 43 | +The intended use case for Datadog Backup is as a component of a Github Action |
| 44 | +workflow, or similar CD pipeline, that takes regular backups of Datadog, then |
| 45 | +commits to a git repository. This enables such a repository to be used as an |
| 46 | +audit trail, capturing the state of Datadog configuration at regular intervals. |
| 47 | + |
| 48 | +A local backup will additionally enable performing a search against the YAML or |
| 49 | +JSON definitions and performing mass updates which can then be pushed back to |
| 50 | +Datadog. To use the tool to make a mass edit to Datadog configuration, one |
| 51 | +modifies the repository locally, then runs the command in “restore” mode. |
| 52 | + |
| 53 | + |
| 54 | +### Running the tool |
| 55 | + |
| 56 | +``` |
| 57 | +gem install datadog_backup |
| 58 | +export DATADOG_API_KEY=abc123 |
| 59 | +export DATADOG_APP_KEY=abc123 |
| 60 | +
|
| 61 | +# Perform backup to optional/path/to/backupdir using YAML encoding |
| 62 | +datadog_backup backup --backup-dir optional/path/to/backupdir |
| 63 | +
|
| 64 | +# Make some changes |
| 65 | +
|
| 66 | +# Just review the changes since last backup |
| 67 | +datadog_backup diffs --backup-dir optional/path/to/backupdir |
| 68 | +
|
| 69 | +# Review and apply local changes to datadog |
| 70 | +
|
| 71 | +datadog_backup restore --backup-dir optional/path/to/backupdir |
| 72 | +``` |
| 73 | + |
| 74 | +### Using the tool in Github Actions |
| 75 | + |
| 76 | +Included in the git repo for Datadog Backup is an [example Github Actions |
| 77 | +workflow](https://github.com/scribd/datadog_backup/tree/master/example) for |
| 78 | +periodically backing up your Datadog deployment. |
| 79 | + |
| 80 | +### Further development |
| 81 | + |
| 82 | +Some areas to further expand the gem are: |
| 83 | + |
| 84 | +- Backup of metadata |
| 85 | + |
| 86 | +If you find this Gem useful, and would like to expand on it, [contributions are |
| 87 | +welcome](https://github.com/scribd/datadog_backup)! |
| 88 | + |
0 commit comments