Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 23 additions & 6 deletions packages/azure/_dev/build/docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -129,7 +129,6 @@ If you are familiar with Kafka, here's a conceptual mapping between the two:
| Consumer Group | Consumer Group |
| Offset | Offset |


#### How many partitions?

The number of partitions is essential to balance the event hub cost and performance.
Expand Down Expand Up @@ -195,7 +194,6 @@ The number of partitions must be at least the number of agents.
└ Event Hub ─ ─ ─ ─ ─ ─ ─ ┘ └ Agent ─ ─ ─ ─ ─ ─ ─ ─ ─ ┘
```


##### Recommendations

Create an event hub with at least two partitions. Two partitions allow low-volume deployment to support high availability with two agents. Consider creating four partitions or more to handle medium-volume deployments with availability.
Expand Down Expand Up @@ -305,7 +303,7 @@ Select the **subscription** and the **Event Hubs namespace** you previously crea

### Create a Storage account container

The Elastic Agent stores the consumer group information (state, position, or offset) in a storage account container. Making this information available to all agents allows them to share the logs processing and resume from the last processed logs after a restart.
The Elastic Agent stores the event hub checkpoint information in a storage account container. Storing checkpoint information in a container allows agents to share message processing and resume from the last processed message after a restart.

NOTE: Use the storage account as a checkpoint store only.

Expand Down Expand Up @@ -335,11 +333,30 @@ This is the final diagram of the a setup for collecting Activity logs from the A
└──────────────┘
```

#### How many Storage account containers?
#### How many Storage Accounts?

The Elastic Agent can create containers for all integrations on a single Storage Account.

The Elastic Agent can use one Storage account container for all integrations.
The Agent uses the **integration name** and the **event hub name** to uniquely identify the container that holds the blobs with the checkpoint information.

The Agent will use the integration name and the event hub name to identify the blob to store the consumer group information uniquely.
```text
┌─────────────────────────────────┐ ┌──────────────────────────────────────────┐
│ │ │ │
│ ┌─────────────────────┐ │ │ ┌───────────────────────────────────┐ │
│ │ azure-eventhub │ │ │ │ filebeat-activitylogs-evehub1 │ │
│ │ <<input>> │──────┼──────┼─▶│ <<container>> │ │
│ └─────────────────────┘ │ │ └───────────────────────────────────┘ │
│ ┌─────────────────────┐ │ │ ┌───────────────────────────────────┐ │
│ │ azure-eventhub │ │ │ │ filebeat-signinlogs-evehub1 │ │
│ │ <<input>> │──────┼──────┼─▶│ <<container>> │ │
│ └─────────────────────┘ │ │ └───────────────────────────────────┘ │
│ ┌─────────────────────┐ │ │ ┌───────────────────────────────────┐ │
│ │ azure-eventhub │ │ │ │ filebeat-auditlogs-evehub1 │ │
│ │ <<input>> │──────┼──────┼─▶│ <<container>> │ │
│ └─────────────────────┘ │ │ └───────────────────────────────────┘ │
│ │ │ │
└─Elastic Agent───────────────────┘ └─Storage Account──────────────────────────┘
```

### Running the integration behind a firewall

Expand Down
8 changes: 4 additions & 4 deletions packages/azure/changelog.yml
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
- version: "1.19.0"
- version: "1.19.1"
changes:
- description: Add entity identifiers to `related.entity` in activitylogs.
type: enhancement
link: https://github.com/elastic/integrations/pull/11233
- description: Fix an error and clarify the docs about the Storage Account container.
type: bugfix
link: https://github.com/elastic/integrations/pull/11660
- version: "1.18.0"
changes:
- description: Add entity identifiers to `related.entity`.
Expand Down
29 changes: 23 additions & 6 deletions packages/azure/docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -129,7 +129,6 @@ If you are familiar with Kafka, here's a conceptual mapping between the two:
| Consumer Group | Consumer Group |
| Offset | Offset |


#### How many partitions?

The number of partitions is essential to balance the event hub cost and performance.
Expand Down Expand Up @@ -195,7 +194,6 @@ The number of partitions must be at least the number of agents.
└ Event Hub ─ ─ ─ ─ ─ ─ ─ ┘ └ Agent ─ ─ ─ ─ ─ ─ ─ ─ ─ ┘
```


##### Recommendations

Create an event hub with at least two partitions. Two partitions allow low-volume deployment to support high availability with two agents. Consider creating four partitions or more to handle medium-volume deployments with availability.
Expand Down Expand Up @@ -305,7 +303,7 @@ Select the **subscription** and the **Event Hubs namespace** you previously crea

### Create a Storage account container

The Elastic Agent stores the consumer group information (state, position, or offset) in a storage account container. Making this information available to all agents allows them to share the logs processing and resume from the last processed logs after a restart.
The Elastic Agent stores the event hub checkpoint information in a storage account container. Storing checkpoint information in a container allows agents to share message processing and resume from the last processed message after a restart.

NOTE: Use the storage account as a checkpoint store only.

Expand Down Expand Up @@ -335,11 +333,30 @@ This is the final diagram of the a setup for collecting Activity logs from the A
└──────────────┘
```

#### How many Storage account containers?
#### How many Storage Accounts?

The Elastic Agent can create containers for all integrations on a single Storage Account.

The Elastic Agent can use one Storage account container for all integrations.
The Agent uses the **integration name** and the **event hub name** to uniquely identify the container that holds the blobs with the checkpoint information.

The Agent will use the integration name and the event hub name to identify the blob to store the consumer group information uniquely.
```text
┌─────────────────────────────────┐ ┌──────────────────────────────────────────┐
│ │ │ │
│ ┌─────────────────────┐ │ │ ┌───────────────────────────────────┐ │
│ │ azure-eventhub │ │ │ │ filebeat-activitylogs-evehub1 │ │
│ │ <<input>> │──────┼──────┼─▶│ <<container>> │ │
│ └─────────────────────┘ │ │ └───────────────────────────────────┘ │
│ ┌─────────────────────┐ │ │ ┌───────────────────────────────────┐ │
│ │ azure-eventhub │ │ │ │ filebeat-signinlogs-evehub1 │ │
│ │ <<input>> │──────┼──────┼─▶│ <<container>> │ │
│ └─────────────────────┘ │ │ └───────────────────────────────────┘ │
│ ┌─────────────────────┐ │ │ ┌───────────────────────────────────┐ │
│ │ azure-eventhub │ │ │ │ filebeat-auditlogs-evehub1 │ │
│ │ <<input>> │──────┼──────┼─▶│ <<container>> │ │
│ └─────────────────────┘ │ │ └───────────────────────────────────┘ │
│ │ │ │
└─Elastic Agent───────────────────┘ └─Storage Account──────────────────────────┘
```

### Running the integration behind a firewall

Expand Down
2 changes: 1 addition & 1 deletion packages/azure/manifest.yml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
name: azure
title: Azure Logs
version: 1.19.0
version: 1.19.1
description: This Elastic integration collects logs from Azure
type: integration
icons:
Expand Down