|
| 1 | +--- |
| 2 | +layout: post |
| 3 | +title: "Using Terraform to integrate Datadog and AWS" |
| 4 | +authors: |
| 5 | +- jimp |
| 6 | +- qphou |
| 7 | +tags: |
| 8 | +- featured |
| 9 | +- terraform |
| 10 | +- monitoring |
| 11 | +team: Core Infrastructure |
| 12 | +--- |
| 13 | + |
| 14 | +We love metrics but hate manual processes. When we adopted |
| 15 | +[Datadog](https://datadoghq.com)'s builtin AWS |
| 16 | +[integration](https://docs.datadoghq.com/integrations/amazon_web_services/?tab=allpermissions) |
| 17 | +we couldn't wait to get AWS CloudWatch metrics into Datadog, but first we needed to automate |
| 18 | +the [numerous manual steps |
| 19 | +required](https://docs.datadoghq.com/integrations/amazon_web_services/?tab=allpermissions) |
| 20 | +to set it up. Datadog's AWS integration is quite powerful, once |
| 21 | +enabled it will automatically synchronize specified CloudWatch metrics into a |
| 22 | +Datadog account. Basically, anything available within CloudWatch, can be easily |
| 23 | +made available in Datadog, alongside all of our other metrics and dashboards. |
| 24 | + |
| 25 | + |
| 26 | +Despite the integration's power and convenience, its setup process is actually |
| 27 | +quite involved. As outlined in [Datadog's |
| 28 | +documentation](https://docs.datadoghq.com/integrations/amazon_web_services/?tab=allpermissions), |
| 29 | +there are *18 manual steps** required, including: |
| 30 | + |
| 31 | +- finding the right AWS account ID |
| 32 | +- creating the right IAM policy |
| 33 | +- copy pasting the right AWS resource ID into Datadog UI |
| 34 | + |
| 35 | +If you have more than a few AWS accounts like we do, you may prefer to automate this! In our case, that means using [Terraform](https://terraform.io) |
| 36 | + |
| 37 | +In this blog post, we would like to share how Scribd uses Terraform to automate |
| 38 | +our Datadog and AWS integration across the organization. |
| 39 | + |
| 40 | +# Enable Datadog’s builtin AWS integration |
| 41 | + |
| 42 | +To address this problem, we built the [terraform-aws-datadog |
| 43 | +module](https://github.com/scribd/terraform-aws-datadog). With only couple |
| 44 | +lines of HCL code, Terraform will perform all the necessary steps to setup |
| 45 | +Datadog integration with a specific AWS account with Scribd’s best practices: |
| 46 | + |
| 47 | +```terraform |
| 48 | +module "datadog" { |
| 49 | + source = "git::https://github.com/scribd/terraform-aws-datadog.git?ref=master" |
| 50 | + aws_account_id = data.aws_caller_identity.current.account_id |
| 51 | + datadog_api_key = var.datadog_api_key |
| 52 | + env = "prod" |
| 53 | + namespace = "team_foo" |
| 54 | +} |
| 55 | +``` |
| 56 | + |
| 57 | +The benefit from an AWS Account maintainer point of view is that using the |
| 58 | +module is a convenient way to inherit centralized best practice. For module |
| 59 | +maintainers, any change to the Datadog integration module can be released using |
| 60 | +a [standard Terraform module release process](https://www.terraform.io/docs/registry/modules/publish.html). |
| 61 | + |
| 62 | + |
| 63 | +# CloudWatch log synchronization |
| 64 | + |
| 65 | +Initially, the module only sets up the base integration. As adoption increased, more |
| 66 | +features were added to the module by various teams. One of these features is |
| 67 | +automation for setting up log ingestion for CloudWatch. |
| 68 | + |
| 69 | +Like setting up the official AWS integration app, the [instructions for log |
| 70 | +synchronization](https://docs.datadoghq.com/integrations/amazon_web_services/?tab=allpermissions#log-collection) |
| 71 | +are a bit overwhelming. |
| 72 | + |
| 73 | +However, using the `terraform-aws-datadog` module, we can enable the feature with a single parameter: |
| 74 | + |
| 75 | +```terraform |
| 76 | +module "datadog" { |
| 77 | + source = "git::https://github.com/scribd/terraform-aws-datadog.git?ref=master" |
| 78 | + datadog_api_key = var.datadog_api_key |
| 79 | + env = "prod" |
| 80 | + namespace = "project_foo" |
| 81 | + cloudwatch_log_groups = ["cloudwatch_log_group_1", "cloudwatch_log_group_2"] |
| 82 | +} |
| 83 | +``` |
| 84 | + |
| 85 | +That’s it! Terraform will automatically create the Datadog serverless function |
| 86 | +and triggers for specified log groups to forward all CloudWatch logs into |
| 87 | +Datadog. After running `terraform apply`, you should be able to see logs showing |
| 88 | +up in Datadog within minutes. |
| 89 | + |
| 90 | + |
| 91 | +# Future work |
| 92 | + |
| 93 | +With both metrics and logs synchronized into Datadog, we are able to leverage |
| 94 | +Datadog as the central hub for all things monitoring. We are planning to bring |
| 95 | +more features to the module as we migrate Scribd’s infrastructure into AWS. |
| 96 | + |
| 97 | +Metrics ingested through the official AWS integration are delayed by couple |
| 98 | +minutes, which is not ideal to use as signals for monitoring critical systems. |
| 99 | +There are opportunities to enable real time metrics synchronization by |
| 100 | +automating Datadog agent setup. |
| 101 | + |
| 102 | +The [datadog-serverless-functions |
| 103 | +repo](https://github.com/DataDog/datadog-serverless-functions/tree/master/aws) |
| 104 | +contains two other lambda based AWS augmentations that we may add as available |
| 105 | +features of the module: `vpc_flow_log_monitoring` and `rds_enhanced_monitoring`. |
| 106 | + |
| 107 | +Stay apprised of future releases by watching our [release page](https://github.com/scribd/terraform-aws-datadog/releases). |
| 108 | + |
| 109 | +_Special shout out to Taylor McClure and Hamilton Hord for starting the project, as well |
| 110 | +as Sai Kiran Burle, Kamran Farhadi and Eugene Pimenov for improvements and bug |
| 111 | +fixes._ |
0 commit comments