Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions packages/aws/changelog.yml
Original file line number Diff line number Diff line change
@@ -1,4 +1,9 @@
# newer versions go on top
- version: "2.25.1"
changes:
- description: Add aws.metrics_names_fingerprint field and mark it as a dimension.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kaiyan-sheng I would include in the description this part you added in the issue:

For metrics coming in from Firehose, there can be cases two documents have the same timestamp, dimension, namespace, accountID, exportARN and region BUT from two different requests.

Does it cause document loss because of TSDB? I would include it.

I think it would help to navigate better when looking at the changelog in case this appears in a SDH in the future.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@constanca-m Yes it does cause document loss because of TSDB. Let me add it in the aws firehose integration since that's where the actual fingerprint processor is added: https://github.com/elastic/integrations/blob/main/packages/awsfirehose/data_stream/metrics/elasticsearch/ingest_pipeline/default.yml#L29
changelog: https://github.com/elastic/integrations/blob/main/packages/awsfirehose/changelog.yml#L7

type: bugfix
link: https://github.com/elastic/integrations/pull/11368
- version: "2.25.0"
changes:
- description: "Allow @custom pipeline access to event.original without setting preserve_original_event."
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,3 +5,9 @@
type: flattened
description: |
Tag key-value pairs from AWS resources.
- name: metrics_names_fingerprint
type: keyword
description: |
Autogenerated ID representing the fingerprint of the list of metrics names.
Applicable only for [Amazon Data Firehose integration](https://www.elastic.co/docs/current/integrations/awsfirehose).
dimension: true
Original file line number Diff line number Diff line change
Expand Up @@ -155,4 +155,4 @@
},
"version": "78.0.3904.108"
}
}
}
6 changes: 6 additions & 0 deletions packages/aws/data_stream/dynamodb/fields/package-fields.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,3 +5,9 @@
type: flattened
description: |
Tag key value pairs from aws resources.
- name: metrics_names_fingerprint
type: keyword
description: |
Autogenerated ID representing the fingerprint of the list of metrics names.
Applicable only for [Amazon Data Firehose integration](https://www.elastic.co/docs/current/integrations/awsfirehose).
dimension: true
6 changes: 6 additions & 0 deletions packages/aws/data_stream/ebs/fields/package-fields.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,3 +9,9 @@
type: keyword
description: |
Name of a S3 bucket.
- name: metrics_names_fingerprint
type: keyword
description: |
Autogenerated ID representing the fingerprint of the list of metrics names.
Applicable only for [Amazon Data Firehose integration](https://www.elastic.co/docs/current/integrations/awsfirehose).
dimension: true
Original file line number Diff line number Diff line change
Expand Up @@ -5,3 +5,9 @@
type: flattened
description: |
Tag key value pairs from aws resources.
- name: metrics_names_fingerprint
type: keyword
description: |
Autogenerated ID representing the fingerprint of the list of metrics names.
Applicable only for [Amazon Data Firehose integration](https://www.elastic.co/docs/current/integrations/awsfirehose).
dimension: true
Original file line number Diff line number Diff line change
Expand Up @@ -5,3 +5,9 @@
type: flattened
description: |
Tag key value pairs from aws resources.
- name: metrics_names_fingerprint
type: keyword
description: |
Autogenerated ID representing the fingerprint of the list of metrics names.
Applicable only for [Amazon Data Firehose integration](https://www.elastic.co/docs/current/integrations/awsfirehose).
dimension: true
Original file line number Diff line number Diff line change
Expand Up @@ -5,3 +5,9 @@
type: flattened
description: |
Tag key value pairs from aws resources.
- name: metrics_names_fingerprint
type: keyword
description: |
Autogenerated ID representing the fingerprint of the list of metrics names.
Applicable only for [Amazon Data Firehose integration](https://www.elastic.co/docs/current/integrations/awsfirehose).
dimension: true
Original file line number Diff line number Diff line change
Expand Up @@ -5,3 +5,9 @@
type: flattened
description: |
Tag key value pairs from aws resources.
- name: metrics_names_fingerprint
type: keyword
description: |
Autogenerated ID representing the fingerprint of the list of metrics names.
Applicable only for [Amazon Data Firehose integration](https://www.elastic.co/docs/current/integrations/awsfirehose).
dimension: true
Original file line number Diff line number Diff line change
Expand Up @@ -5,3 +5,9 @@
type: flattened
description: |
Tag key value pairs from aws resources.
- name: metrics_names_fingerprint
type: keyword
description: |
Autogenerated ID representing the fingerprint of the list of metrics names.
Applicable only for [Amazon Data Firehose integration](https://www.elastic.co/docs/current/integrations/awsfirehose).
dimension: true
6 changes: 6 additions & 0 deletions packages/aws/data_stream/kinesis/fields/package-fields.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,3 +5,9 @@
type: flattened
description: |
Tag key value pairs from aws resources.
- name: metrics_names_fingerprint
type: keyword
description: |
Autogenerated ID representing the fingerprint of the list of metrics names.
Applicable only for [Amazon Data Firehose integration](https://www.elastic.co/docs/current/integrations/awsfirehose).
dimension: true
6 changes: 6 additions & 0 deletions packages/aws/data_stream/lambda/fields/package-fields.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,3 +5,9 @@
type: flattened
description: |
Tag key value pairs from aws resources.
- name: metrics_names_fingerprint
type: keyword
description: |
Autogenerated ID representing the fingerprint of the list of metrics names.
Applicable only for [Amazon Data Firehose integration](https://www.elastic.co/docs/current/integrations/awsfirehose).
dimension: true
Original file line number Diff line number Diff line change
Expand Up @@ -5,3 +5,9 @@
type: flattened
description: |
Tag key value pairs from aws resources.
- name: metrics_names_fingerprint
type: keyword
description: |
Autogenerated ID representing the fingerprint of the list of metrics names.
Applicable only for [Amazon Data Firehose integration](https://www.elastic.co/docs/current/integrations/awsfirehose).
dimension: true
6 changes: 6 additions & 0 deletions packages/aws/data_stream/rds/fields/package-fields.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,3 +5,9 @@
type: flattened
description: |
Tag key value pairs from aws resources.
- name: metrics_names_fingerprint
type: keyword
description: |
Autogenerated ID representing the fingerprint of the list of metrics names.
Applicable only for [Amazon Data Firehose integration](https://www.elastic.co/docs/current/integrations/awsfirehose).
dimension: true
Original file line number Diff line number Diff line change
Expand Up @@ -10,3 +10,9 @@
dimension: true
description: |
Name of a S3 bucket.
- name: metrics_names_fingerprint
type: keyword
description: |
Autogenerated ID representing the fingerprint of the list of metrics names.
Applicable only for [Amazon Data Firehose integration](https://www.elastic.co/docs/current/integrations/awsfirehose).
dimension: true
Original file line number Diff line number Diff line change
Expand Up @@ -10,3 +10,9 @@
dimension: true
description: |
Name of a S3 bucket.
- name: metrics_names_fingerprint
type: keyword
description: |
Autogenerated ID representing the fingerprint of the list of metrics names.
Applicable only for [Amazon Data Firehose integration](https://www.elastic.co/docs/current/integrations/awsfirehose).
dimension: true
Original file line number Diff line number Diff line change
Expand Up @@ -9,3 +9,9 @@
type: double
description: |
Metrics that returned from Cloudwatch API query.
- name: metrics_names_fingerprint
type: keyword
description: |
Autogenerated ID representing the fingerprint of the list of metrics names.
Applicable only for [Amazon Data Firehose integration](https://www.elastic.co/docs/current/integrations/awsfirehose).
dimension: true
6 changes: 6 additions & 0 deletions packages/aws/data_stream/sns/fields/package-fields.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,3 +5,9 @@
type: flattened
description: |
Tag key value pairs from aws resources.
- name: metrics_names_fingerprint
type: keyword
description: |
Autogenerated ID representing the fingerprint of the list of metrics names.
Applicable only for [Amazon Data Firehose integration](https://www.elastic.co/docs/current/integrations/awsfirehose).
dimension: true
6 changes: 6 additions & 0 deletions packages/aws/data_stream/sqs/fields/package-fields.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,3 +5,9 @@
type: flattened
description: |
Tag key value pairs from aws resources.
- name: metrics_names_fingerprint
type: keyword
description: |
Autogenerated ID representing the fingerprint of the list of metrics names.
Applicable only for [Amazon Data Firehose integration](https://www.elastic.co/docs/current/integrations/awsfirehose).
dimension: true
Original file line number Diff line number Diff line change
Expand Up @@ -5,3 +5,9 @@
type: flattened
description: |
Tag key value pairs from aws resources.
- name: metrics_names_fingerprint
type: keyword
description: |
Autogenerated ID representing the fingerprint of the list of metrics names.
Applicable only for [Amazon Data Firehose integration](https://www.elastic.co/docs/current/integrations/awsfirehose).
dimension: true
6 changes: 6 additions & 0 deletions packages/aws/data_stream/usage/fields/package-fields.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,3 +5,9 @@
type: flattened
description: |
Tag key value pairs from aws resources.
- name: metrics_names_fingerprint
type: keyword
description: |
Autogenerated ID representing the fingerprint of the list of metrics names.
Applicable only for [Amazon Data Firehose integration](https://www.elastic.co/docs/current/integrations/awsfirehose).
dimension: true
6 changes: 6 additions & 0 deletions packages/aws/data_stream/vpn/fields/package-fields.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,3 +5,9 @@
type: flattened
description: |
Tag key value pairs from aws resources.
- name: metrics_names_fingerprint
type: keyword
description: |
Autogenerated ID representing the fingerprint of the list of metrics names.
Applicable only for [Amazon Data Firehose integration](https://www.elastic.co/docs/current/integrations/awsfirehose).
dimension: true
1 change: 1 addition & 0 deletions packages/aws/docs/apigateway.md
Original file line number Diff line number Diff line change
Expand Up @@ -189,6 +189,7 @@ Please refer to the following [document](https://www.elastic.co/guide/en/ecs/cur
| aws.dimensions.Resource | It represents an endpoint within the API that corresponds to a specific functionality, typically associated with a URL path segment. | keyword | | |
| aws.dimensions.Route | Routes define the path and HTTP methods that clients can use to access different functionalities of the API. | keyword | | |
| aws.dimensions.Stage | It represents a specific version of the API that is accessible to clients. A stage allows you to manage different environments or versions of your API, such as development, testing, and production. | keyword | | |
| aws.metrics_names_fingerprint | Autogenerated ID representing the fingerprint of the list of metrics names. Applicable only for [Amazon Data Firehose integration](https://www.elastic.co/docs/current/integrations/awsfirehose). | keyword | | |
| aws.tags | Tag key-value pairs from AWS resources. | flattened | | |
| cloud.account.id | The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier. | keyword | | |
| cloud.image.id | Image ID for the cloud instance. | keyword | | |
Expand Down
1 change: 0 additions & 1 deletion packages/aws/docs/cloudfront.md
Original file line number Diff line number Diff line change
Expand Up @@ -240,5 +240,4 @@ An example event for `cloudfront` looks as following:
"version": "78.0.3904.108"
}
}

```
1 change: 1 addition & 0 deletions packages/aws/docs/dynamodb.md
Original file line number Diff line number Diff line change
Expand Up @@ -172,6 +172,7 @@ Please refer to the following [document](https://www.elastic.co/guide/en/ecs/cur
| aws.dynamodb.metrics.TransactionConflict.avg | | double | gauge |
| aws.dynamodb.metrics.TransactionConflict.sum | | long | gauge |
| aws.dynamodb.metrics.WriteThrottleEvents.sum | Requests to DynamoDB that exceed the provisioned write capacity units for a table or a global secondary index. | long | gauge |
| aws.metrics_names_fingerprint | Autogenerated ID representing the fingerprint of the list of metrics names. Applicable only for [Amazon Data Firehose integration](https://www.elastic.co/docs/current/integrations/awsfirehose). | keyword | |
| aws.tags | Tag key value pairs from aws resources. | flattened | |
| cloud.account.id | The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier. | keyword | |
| cloud.image.id | Image ID for the cloud instance. | keyword | |
Expand Down
1 change: 1 addition & 0 deletions packages/aws/docs/ebs.md
Original file line number Diff line number Diff line change
Expand Up @@ -166,6 +166,7 @@ Please refer to the following [document](https://www.elastic.co/guide/en/ecs/cur
| aws.ebs.metrics.VolumeTotalWriteTime.sum | The total number of seconds spent by all write operations that completed in a specified period of time. | double | gauge |
| aws.ebs.metrics.VolumeWriteBytes.avg | Average size of each write operation during the period, except on volumes attached to a Nitro-based instance, where the average represents the average over the specified period. | double | gauge |
| aws.ebs.metrics.VolumeWriteOps.avg | The total number of write operations in a specified period of time. | double | gauge |
| aws.metrics_names_fingerprint | Autogenerated ID representing the fingerprint of the list of metrics names. Applicable only for [Amazon Data Firehose integration](https://www.elastic.co/docs/current/integrations/awsfirehose). | keyword | |
| aws.s3.bucket.name | Name of a S3 bucket. | keyword | |
| aws.tags | Tag key value pairs from aws resources. | flattened | |
| cloud.account.id | The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier. | keyword | |
Expand Down
1 change: 1 addition & 0 deletions packages/aws/docs/ec2.md
Original file line number Diff line number Diff line change
Expand Up @@ -385,6 +385,7 @@ Please refer to the following [document](https://www.elastic.co/guide/en/ecs/cur
| aws.ec2.metrics.StatusCheckFailed.avg | Reports whether the instance has passed both the instance status check and the system status check in the last minute. | long | gauge |
| aws.ec2.metrics.StatusCheckFailed_Instance.avg | Reports whether the instance has passed the instance status check in the last minute. | long | gauge |
| aws.ec2.metrics.StatusCheckFailed_System.avg | Reports whether the instance has passed the system status check in the last minute. | long | gauge |
| aws.metrics_names_fingerprint | Autogenerated ID representing the fingerprint of the list of metrics names. Applicable only for [Amazon Data Firehose integration](https://www.elastic.co/docs/current/integrations/awsfirehose). | keyword | |
| aws.tags | Tag key value pairs from aws resources. | flattened | |
| cloud.account.id | The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier. | keyword | |
| cloud.image.id | Image ID for the cloud instance. | keyword | |
Expand Down
1 change: 1 addition & 0 deletions packages/aws/docs/elb.md
Original file line number Diff line number Diff line change
Expand Up @@ -420,6 +420,7 @@ Please refer to the following [document](https://www.elastic.co/guide/en/ecs/cur
| aws.elb.metrics.SpilloverCount.sum | The total number of requests that were rejected because the surge queue is full. | long | | gauge |
| aws.elb.metrics.SurgeQueueLength.max | The total number of requests (HTTP listener) or connections (TCP listener) that are pending routing to a healthy instance. | long | | gauge |
| aws.elb.metrics.UnHealthyHostCount.max | The number of unhealthy instances registered with your load balancer. | long | | gauge |
| aws.metrics_names_fingerprint | Autogenerated ID representing the fingerprint of the list of metrics names. Applicable only for [Amazon Data Firehose integration](https://www.elastic.co/docs/current/integrations/awsfirehose). | keyword | | |
| aws.networkelb.metrics.ActiveFlowCount.avg | The total number of concurrent flows (or connections) from clients to targets. | double | | gauge |
| aws.networkelb.metrics.ActiveFlowCount_TCP.avg | The total number of concurrent TCP flows (or connections) from clients to targets. | double | | gauge |
| aws.networkelb.metrics.ActiveFlowCount_TLS.avg | The total number of concurrent TLS flows (or connections) from clients to targets. | double | | gauge |
Expand Down
1 change: 1 addition & 0 deletions packages/aws/docs/emr.md
Original file line number Diff line number Diff line change
Expand Up @@ -178,6 +178,7 @@ Please refer to the following [document](https://www.elastic.co/guide/en/ecs/cur
| aws.elasticmapreduce.metrics.TotalVCPURunning.avg | The current total number of vCPUs available in a running cluster. | long | | gauge |
| aws.elasticmapreduce.metrics.UnderReplicatedBlocks.sum | The number of blocks that need to be replicated one or more times. | long | | gauge |
| aws.elasticmapreduce.metrics.YARNMemoryAvailablePercentage.avg | The percentage of remaining memory available to YARN | double | percent | gauge |
| aws.metrics_names_fingerprint | Autogenerated ID representing the fingerprint of the list of metrics names. Applicable only for [Amazon Data Firehose integration](https://www.elastic.co/docs/current/integrations/awsfirehose). | keyword | | |
| aws.tags | Tag key value pairs from aws resources. | flattened | | |
| cloud.account.id | The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier. | keyword | | |
| cloud.image.id | Image ID for the cloud instance. | keyword | | |
Expand Down
1 change: 1 addition & 0 deletions packages/aws/docs/firewall.md
Original file line number Diff line number Diff line change
Expand Up @@ -347,6 +347,7 @@ Please refer to the following [document](https://www.elastic.co/guide/en/ecs/cur
| aws.dimensions.CustomAction | Dimension for a publish metrics custom action that you defined. You can define this for a rule action in a stateless rule group or for a stateless default action in a firewall policy. | keyword | |
| aws.dimensions.Engine | Rules engine that processed the packet. The value for this is either Stateful or Stateless. | keyword | |
| aws.dimensions.FirewallName | Name that you specified for the Network Firewall firewall. | keyword | |
| aws.metrics_names_fingerprint | Autogenerated ID representing the fingerprint of the list of metrics names. Applicable only for [Amazon Data Firehose integration](https://www.elastic.co/docs/current/integrations/awsfirehose). | keyword | |
| aws.networkfirewall.metrics.DroppedPackets.sum | The number of packets dropped by the Network Firewall. | long | gauge |
| aws.networkfirewall.metrics.Packets.sum | Number of packets inspected for a firewall policy or stateless rulegroup for which a custom action is defined. This metric is only used for the dimension CustomAction. | long | gauge |
| aws.networkfirewall.metrics.PassedPackets.sum | The number of packets passed by the Network Firewall. | long | gauge |
Expand Down
1 change: 1 addition & 0 deletions packages/aws/docs/kafka.md
Original file line number Diff line number Diff line change
Expand Up @@ -199,6 +199,7 @@ Please refer to the following [document](https://www.elastic.co/guide/en/ecs/cur
| aws.kafka.metrics.UnderMinIsrPartitionCount.avg | The average number of under minIsr partitions for the broker. | long | gauge |
| aws.kafka.metrics.UnderReplicatedPartitions.avg | The average number of under-replicated partitions for the broker. | long | gauge |
| aws.kafka.metrics.ZooKeeperRequestLatencyMsMean.avg | The mean latency in milliseconds for Apache ZooKeeper requests from broker. | long | gauge |
| aws.metrics_names_fingerprint | Autogenerated ID representing the fingerprint of the list of metrics names. Applicable only for [Amazon Data Firehose integration](https://www.elastic.co/docs/current/integrations/awsfirehose). | keyword | |
| aws.tags | Tag key value pairs from aws resources. | flattened | |
| cloud.account.id | The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier. | keyword | |
| cloud.image.id | Image ID for the cloud instance. | keyword | |
Expand Down
1 change: 1 addition & 0 deletions packages/aws/docs/kinesis.md
Original file line number Diff line number Diff line change
Expand Up @@ -156,6 +156,7 @@ Please refer to the following [document](https://www.elastic.co/guide/en/ecs/cur
| aws.kinesis.metrics.SubscribeToShard_RateExceeded.avg | This metric is emitted when a new subscription attempt fails because there already is an active subscription by the same consumer or if you exceed the number of calls per second allowed for this operation. | long | gauge |
| aws.kinesis.metrics.SubscribeToShard_Success.avg | This metric records whether the SubscribeToShard subscription was successfully established. | long | gauge |
| aws.kinesis.metrics.WriteProvisionedThroughputExceeded.avg | The number of records rejected due to throttling for the stream over the specified time period. This metric includes throttling from PutRecord and PutRecords operations. | long | gauge |
| aws.metrics_names_fingerprint | Autogenerated ID representing the fingerprint of the list of metrics names. Applicable only for [Amazon Data Firehose integration](https://www.elastic.co/docs/current/integrations/awsfirehose). | keyword | |
| aws.tags | Tag key value pairs from aws resources. | flattened | |
| cloud.account.id | The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier. | keyword | |
| cloud.image.id | Image ID for the cloud instance. | keyword | |
Expand Down
Loading