1
1
---
2
2
layout : post
3
- title : " How Scribd manages Datadog’s AWS integration using Terraform "
3
+ title : " Using Terraform to integrate Datadog and AWS "
4
4
authors :
5
5
- jimp
6
6
- qphou
@@ -11,22 +11,28 @@ tags:
11
11
team : Core Infrastructure
12
12
---
13
13
14
- Datadog comes with a builtin AWS
14
+ We love metrics but hate manual processes. When we adopted
15
+ [ Datadog] ( https://datadoghq.com ) 's builtin AWS
15
16
[ integration] ( https://docs.datadoghq.com/integrations/amazon_web_services/?tab=allpermissions )
16
- to ship CloudWatch metrics to your Datadog account. Once enabled, the
17
- integration will automatically synchronize whitelisted CloudWatch metrics into
18
- your Datadog account.
17
+ we couldn't wait to get AWS CloudWatch metrics into Datadog, but first we needed to automate
18
+ the [ numerous manual steps
19
+ required] ( https://docs.datadoghq.com/integrations/amazon_web_services/?tab=allpermissions )
20
+ to set it up. Datadog's AWS integration is quite powerful, once
21
+ enabled it will automatically synchronize specified CloudWatch metrics into a
22
+ Datadog account. Basically, anything available within CloudWatch, can be easily
23
+ made available in Datadog, alongside all of our other metrics and dashboards.
19
24
20
- While this integration is powerful and convenient to use, it’s setup process is
21
- actually quite involved. As outlined in [ Datadog's documentation] ( https://docs.datadoghq.com/integrations/amazon_web_services/?tab=allpermissions ) , there are 18
22
- manual steps required, including:
25
+
26
+ Despite the integration's power and convenience, its setup process is actually
27
+ quite involved. As outlined in [ Datadog's
28
+ documentation] ( https://docs.datadoghq.com/integrations/amazon_web_services/?tab=allpermissions ) ,
29
+ there are * 18 manual steps** required, including:
23
30
24
31
- finding the right AWS account ID
25
32
- creating the right IAM policy
26
33
- copy pasting the right AWS resource ID into Datadog UI
27
- - etc.
28
34
29
- If you have more than a few AWS accounts, you may prefer to use Terraform.
35
+ If you have more than a few AWS accounts like we do , you may prefer to automate this! In our case, that means using [ Terraform] ( https://terraform.io )
30
36
31
37
In this blog post, we would like to share how Scribd uses Terraform to automate
32
38
our Datadog and AWS integration across the organization.
@@ -50,21 +56,21 @@ module "datadog" {
50
56
51
57
The benefit from an AWS Account maintainer point of view is that using the
52
58
module is a convenient way to inherit centralized best practice. For module
53
- maintainers, any change to the datadog integration module can be released using
59
+ maintainers, any change to the Datadog integration module can be released using
54
60
a [ standard Terraform module release process] ( https://www.terraform.io/docs/registry/modules/publish.html ) .
55
61
56
62
57
- # Cloudwatch log synchronization
63
+ # CloudWatch log synchronization
58
64
59
65
Initially, the module only sets up the base integration. As adoption increased, more
60
66
features were added to the module by various teams. One of these features is
61
- automation for setting up log ingestion for cloudwatch .
67
+ automation for setting up log ingestion for CloudWatch .
62
68
63
69
Like setting up the official AWS integration app, the [ instructions for log
64
70
synchronization] ( https://docs.datadoghq.com/integrations/amazon_web_services/?tab=allpermissions#log-collection )
65
71
are a bit overwhelming.
66
72
67
- However, using the terraform-aws-datadog module, we can enable the feature with a single parameter:
73
+ However, using the ` terraform-aws-datadog ` module, we can enable the feature with a single parameter:
68
74
69
75
``` terraform
70
76
module "datadog" {
@@ -76,9 +82,9 @@ module "datadog" {
76
82
}
77
83
```
78
84
79
- That’s it, Terraform will automatically create the datadog serverless function
80
- and triggers for specified log groups to forward all cloudwatch logs into
81
- Datadog. After running terraform apply, you should be able to see logs showing
85
+ That’s it! Terraform will automatically create the Datadog serverless function
86
+ and triggers for specified log groups to forward all CloudWatch logs into
87
+ Datadog. After running ` terraform apply ` , you should be able to see logs showing
82
88
up in Datadog within minutes.
83
89
84
90
@@ -91,7 +97,7 @@ more features to the module as we migrate Scribd’s infrastructure into AWS.
91
97
Metrics ingested through the official AWS integration are delayed by couple
92
98
minutes, which is not ideal to use as signals for monitoring critical systems.
93
99
There are opportunities to enable real time metrics synchronization by
94
- automating datadog agent setup.
100
+ automating Datadog agent setup.
95
101
96
102
The [ datadog-serverless-functions
97
103
repo] ( https://github.com/DataDog/datadog-serverless-functions/tree/master/aws )
0 commit comments