Loading

Add cloud metadata

Tip

Inputs that collect logs and metrics use this processor by default, so you do not need to configure it explicitly.

The add_cloud_metadata processor enriches each event with instance metadata from the machine’s hosting provider. At startup the processor queries a list of hosting providers and caches the instance metadata.

The following providers are supported:

  • Amazon Web Services (AWS)
  • Digital Ocean
  • Google Compute Engine (GCE)
  • Tencent Cloud (QCloud)
  • Alibaba Cloud (ECS)
  • Huawei Cloud (ECS)
  • Azure Virtual Machine
  • Openstack Nova

The Alibaba Cloud and Tencent providers are disabled by default, because they require to access a remote host. Use the providers setting to select a list of default providers to query.

Example

This configuration enables the processor:

- add_cloud_metadata: ~

The metadata that is added to events varies by hosting provider. For examples, refer to Provider-specific metadata examples.

Configuration settings

Note

Elastic Agent processors execute before ingest pipelines, which means that they process the raw event data rather than the final event sent to Elasticsearch. For related limitations, refer to What are some limitations of using processors?

Name Required Default Description
timeout No 3s Maximum amount of time to wait for a successful response when detecting the hosting provider. If a timeout occurs, no instance metadata is added to the events. This makes it possible to enable this processor for all your deployments (in the cloud or on-premise).
providers No List of provider names to use. If providers is not configured, all providers that do not access a remote endpoint are enabled by default. The list of providers may alternatively be configured with the environment variable BEATS_ADD_CLOUD_METADATA_PROVIDERS, by setting it to a comma-separated list of provider names.

The list of supported provider names includes:

* alibaba or ecs for the Alibaba Cloud provider (disabled by default).
* azure for Azure Virtual Machine (enabled by default).
* digitalocean for Digital Ocean (enabled by default).
* aws or ec2 for Amazon Web Services (enabled by default).
* gcp for Google Compute Engine (enabled by default).
* openstack or nova for Openstack Nova (enabled by default).
* openstack-ssl or nova-ssl for Openstack Nova when SSL metadata APIs are enabled (enabled by default).
* tencent or qcloud for Tencent Cloud (disabled by default).
* huawei for Huawei Cloud (enabled by default).
overwrite No false Whether to overwrite existing cloud fields. If true, the processoroverwrites existing cloud.* fields.

The add_cloud_metadata processor supports SSL options to configure the http client used to query cloud metadata.

For more information, refer to SSL/TLS, specifically the settings under Table 7, Common configuration options and Table 8, Client configuration options.

The following sections show examples for each of the supported providers.

AWS

{
  "cloud": {
    "account.id": "123456789012",
    "availability_zone": "us-east-1c",
    "instance.id": "i-4e123456",
    "machine.type": "t2.medium",
    "image.id": "ami-abcd1234",
    "provider": "aws",
    "region": "us-east-1"
  }
}

Digital Ocean

{
  "cloud": {
    "instance.id": "1234567",
    "provider": "digitalocean",
    "region": "nyc2"
  }
}

GCP

{
  "cloud": {
    "availability_zone": "us-east1-b",
    "instance.id": "1234556778987654321",
    "machine.type": "f1-micro",
    "project.id": "my-dev",
    "provider": "gcp"
  }
}

Tencent Cloud

{
  "cloud": {
    "availability_zone": "gz-azone2",
    "instance.id": "ins-qcloudv5",
    "provider": "qcloud",
    "region": "china-south-gz"
  }
}

Huawei Cloud

{
  "cloud": {
    "availability_zone": "cn-east-2b",
    "instance.id": "37da9890-8289-4c58-ba34-a8271c4a8216",
    "provider": "huawei",
    "region": "cn-east-2"
  }
}

Alibaba Cloud

This metadata is only available when VPC is selected as the network type of the ECS instance.

{
  "cloud": {
    "availability_zone": "cn-shenzhen",
    "instance.id": "i-wz9g2hqiikg0aliyun2b",
    "provider": "ecs",
    "region": "cn-shenzhen-a"
  }
}

Azure Virtual Machine

{
  "cloud": {
    "provider": "azure",
    "instance.id": "04ab04c3-63de-4709-a9f9-9ab8c0411d5e",
    "instance.name": "test-az-vm",
    "machine.type": "Standard_D3_v2",
    "region": "eastus2"
  }
}

Openstack Nova

{
  "cloud": {
    "instance.name": "test-998d932195.mycloud.tld",
    "instance.id": "i-00011a84",
    "availability_zone": "xxxx-az-c",
    "provider": "openstack",
    "machine.type": "m2.large"
  }
}