Add support for Azure Spring Cloud logs #1355

narph · 2021-07-26T13:52:08Z

Enhancement

PR description will be updated soon

What does this PR do?

Add support for Azure Spring Cloud logs

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
If I'm introducing a new feature, I have modified the Kibana version constraint in my package's manifest.yml file to point to the latest Elastic stack release (e.g. ^7.13.0).

Author's Checklist

[ ]

How to test this PR locally

Related issues

Example:

 {
        "_index" : ".ds-logs-azure.springcloudlogs-default-2021.08.04-000001",
        "_type" : "_doc",
        "_id" : "B6R6EXsBdXPyPPk9OOPJ",
        "_score" : null,
        "_source" : {
          "agent" : {
            "hostname" : "docker-fleet-agent",
            "name" : "docker-fleet-agent",
            "id" : "dcb938c2-639e-4251-b63e-4e87d8d02cb9",
            "ephemeral_id" : "cf5e0780-ffa3-41b1-afb3-77231370a3b1",
            "type" : "filebeat",
            "version" : "7.14.0"
          },
          "log" : {
            "level" : "INFO"
          },
          "elastic_agent" : {
            "id" : "dcb938c2-639e-4251-b63e-4e87d8d02cb9",
            "version" : "7.14.0",
            "snapshot" : true
          },
          "message" : "Get config server actuator health status UP.",
          "azure-eventhub" : {
            "sequence_number" : 771,
            "consumer_group" : "$Default",
            "offset" : 600952,
            "eventhub" : "insights-logs-systemlogs",
            "enqueued_time" : "2021-08-04T14:04:21.559Z"
          },
          "tags" : [
            "azure-springcloudlogs"
          ],
          "cloud" : {
            "provider" : "azure"
          },
          "input" : {
            "type" : "azure-eventhub"
          },
          "@timestamp" : "2021-08-04T14:03:33.185Z",
          "ecs" : {
            "version" : "1.10.0"
          },
          "data_stream" : {
            "namespace" : "default",
            "type" : "logs",
            "dataset" : "azure.springcloudlogs"
          },
          "host" : {
            "hostname" : "docker-fleet-agent",
            "os" : {
              "kernel" : "4.19.128-microsoft-standard",
              "codename" : "Core",
              "name" : "CentOS Linux",
              "type" : "linux",
              "family" : "redhat",
              "version" : "7 (Core)",
              "platform" : "centos"
            },
            "containerized" : true,
            "ip" : [
              "192.168.0.7"
            ],
            "name" : "docker-fleet-agent",
            "id" : "78315c3233258f2dcd540ed749ab1701",
            "mac" : [
              "02:42:c0:a8:00:07"
            ],
            "architecture" : "x86_64"
          },
          "event" : {
            "agent_id_status" : "verified",
            "ingested" : "2021-08-04T14:03:47.784957800Z",
            "kind" : "event",
            "action" : "Microsoft.AppPlatform/Spring/SystemLogs",
            "dataset" : "azure.springcloudlogs"
          },
          "azure" : {
            "subscription_id" : "....",
            "springcloudlogs" : {
              "operation_name" : "Microsoft.AppPlatform/Spring/SystemLogs",
              "category" : "SystemLogs",
              "event_category" : "Administrative",
              "logtag" : "F",
              "properties" : {
                "stack" : null,
                "service_name" : "obssprincloud",
                "logger" : "com.microsoft.azure.spring.service.controller.DeploymentHealthCheckController",
                "thread" : "XNIO-1 task-1",
                "type" : "ConfigServer"
              }
            },
            "resource" : {
              "provider" : "MICROSOFT.APPPLATFORM/SPRING",
              "name" : "OBSSPRINCLOUD",
              "id" : "/SUBSCRIPTIONS/.../RESOURCEGROUPS/TESTM/PROVIDERS/MICROSOFT.APPPLATFORM/SPRING/OBSSPRINCLOUD",
              "group" : "TESTM"
            }
          }
        },
        "sort" : [
          1628085813185
        ]
      }

{
        "_index" : ".ds-logs-azure.springcloudlogs-default-2021.08.04-000001",
        "_type" : "_doc",
        "_id" : "XqWZEXsBdXPyPPk9OgOY",
        "_score" : null,
        "_source" : {
          "agent" : {
            "hostname" : "docker-fleet-agent",
            "name" : "docker-fleet-agent",
            "id" : "dcb938c2-639e-4251-b63e-4e87d8d02cb9",
            "type" : "filebeat",
            "ephemeral_id" : "cf5e0780-ffa3-41b1-afb3-77231370a3b1",
            "version" : "7.14.0"
          },
          "log" : {
            "level" : "Informational"
          },
          "elastic_agent" : {
            "id" : "dcb938c2-639e-4251-b63e-4e87d8d02cb9",
            "version" : "7.14.0",
            "snapshot" : true
          },
          "message" : "2021-08-04 14:37:03.902  INFO [thisissecond,,,] 1 --- [trap-executor-0] c.n.d.s.r.aws.ConfigClusterResolver      : Resolving eureka endpoints via configuration",
          "azure-eventhub" : {
            "sequence_number" : 157,
            "consumer_group" : "$Default",
            "offset" : 140536,
            "eventhub" : "insights-logs-applicationconsole",
            "enqueued_time" : "2021-08-04T14:38:13.477Z"
          },
          "tags" : [
            "azure-springcloudlogs"
          ],
          "geo" : {
            "name" : "westeurope"
          },
          "cloud" : {
            "provider" : "azure"
          },
          "input" : {
            "type" : "azure-eventhub"
          },
          "@timestamp" : "2021-08-04T14:37:03.903Z",
          "ecs" : {
            "version" : "1.10.0"
          },
          "data_stream" : {
            "namespace" : "default",
            "type" : "logs",
            "dataset" : "azure.springcloudlogs"
          },
          "host" : {
            "hostname" : "docker-fleet-agent",
            "os" : {
              "kernel" : "4.19.128-microsoft-standard",
              "codename" : "Core",
              "name" : "CentOS Linux",
              "family" : "redhat",
              "type" : "linux",
              "version" : "7 (Core)",
              "platform" : "centos"
            },
            "ip" : [
              "192.168.0.7"
            ],
            "containerized" : true,
            "name" : "docker-fleet-agent",
            "id" : "78315c3233258f2dcd540ed749ab1701",
            "mac" : [
              "02:42:c0:a8:00:07"
            ],
            "architecture" : "x86_64"
          },
          "event" : {
            "agent_id_status" : "verified",
            "ingested" : "2021-08-04T14:37:39.863775400Z",
            "kind" : "event",
            "action" : "Microsoft.AppPlatform/Spring/logs",
            "dataset" : "azure.springcloudlogs"
          },
          "azure" : {
            "subscription_id" : "...",
            "springcloudlogs" : {
              "operation_name" : "Microsoft.AppPlatform/Spring/logs",
              "log_format" : "RAW",
              "category" : "ApplicationConsole",
              "event_category" : "Administrative",
              "logtag" : "F",
              "properties" : {
                "app_name" : "thisissecond",
                "instance_name" : "thisissecond-default-12-696f9485f8-7m6l2",
                "stream" : "stdout",
                "service_name" : "obssprincloud",
                "service_id" : "99070c7524f14eaf970bbdf35f357772"
              }
            },
            "resource" : {
              "provider" : "MICROSOFT.APPPLATFORM/SPRING",
              "name" : "OBSSPRINCLOUD",
              "id" : "/SUBSCRIPTIONS/.../RESOURCEGROUPS/TESTM/PROVIDERS/MICROSOFT.APPPLATFORM/SPRING/OBSSPRINCLOUD",
              "group" : "TESTM"
            }
          }
        },
        "sort" : [
          1628087823903
        ]
      }

Screenshots

elasticmachine · 2021-07-26T14:03:10Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2021-08-05T10:23:44.422+0000
Duration: 14 min 39 sec
Commit: aa8d637

Test stats 🧪

Test	Results
Failed	0
Passed	72
Skipped	0
Total	72

Trends 🧪

narph · 2021-08-04T15:14:10Z

@sorantis , @hemantmalik , I have added the logs format and dashboards in the description, let me know what you think

hemantmalik · 2021-08-04T17:10:32Z

Hey, @narph those are looking great. Thank you for sharing the update.

mtojek · 2021-08-05T08:47:16Z

packages/azure/data_stream/activitylogs/fields/ecs.yml

-  type: keyword
+- name: ecs.version
+  external: ecs
+  description: ECS version


If you load an external field, you don't need to add descriptions here (I'm not sure if we have overriding feature implemented).

mtojek · 2021-08-05T08:48:45Z

packages/azure/data_stream/activitylogs/fields/ecs.yml

+  external: ecs
+- name: event
+  external: ecs
+- name: host


You have to mention all leaf-fields not just groups. We don't want to import definitions for all fields behind host, but just ones that are actually used. Unfortunately the tool can't figure it out on it's own.

mtojek · 2021-08-05T08:51:06Z

packages/azure/data_stream/activitylogs/fields/ecs.yml

@@ -1,249 +1,28 @@
- description: IP address of the client.
-  name: client.ip


Fields defining client family is here: https://github.com/elastic/ecs/blob/master/generated/beats/fields.ecs.yml#L147

mtojek

Just one comment about the ecs.yml file. Otherwise it LGTM, please wait for the CI status.

mtojek · 2021-08-05T09:09:56Z

packages/azure/data_stream/springcloudlogs/fields/ecs.yml

@@ -0,0 +1,27 @@
+- name: ecs.version


I guess you don't need this file now.

starting work on spring cloud logs

9fa35c1

narph self-assigned this Jul 26, 2021

narph added the Integration:azure Azure Logs label Jul 26, 2021

narph added 10 commits August 2, 2021 13:44

Merge branch 'master' into azure-spring

587fee6

spring logs

9a30b0e

fix

8fd675f

update ecs

52d2e44

fix mapping

ad8e05a

remove doc

7540dbc

update pipeline

1687fc1

Merge branch 'master' into azure-spring

d84d47b

update ecs

cc717f6

fix manifest

7b9b1c1

dashboards

1ea01cb

narph added 3 commits August 5, 2021 09:50

build

825cc29

clean up mapping

329539a

update test

b07531e

mtojek reviewed Aug 5, 2021

View reviewed changes

revert ecs

447b27a

mtojek approved these changes Aug 5, 2021

View reviewed changes

narph added 3 commits August 5, 2021 11:24

update schema

e45fe44

add field

8265e50

fix name

aa8d637

narph merged commit fe65d14 into elastic:master Aug 5, 2021

narph deleted the azure-spring branch August 5, 2021 10:38

narph mentioned this pull request Aug 10, 2021

Azure Spring Cloud Observability with Elastic #1285

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add support for Azure Spring Cloud logs #1355

Add support for Azure Spring Cloud logs #1355

Uh oh!

narph commented Jul 26, 2021 •

edited

Loading

Uh oh!

elasticmachine commented Jul 26, 2021 •

edited

Loading

Build stats

Test stats 🧪

Trends 🧪

Uh oh!

narph commented Aug 4, 2021 •

edited

Loading

Uh oh!

hemantmalik commented Aug 4, 2021

Uh oh!

mtojek Aug 5, 2021

Uh oh!

mtojek Aug 5, 2021

Uh oh!

mtojek Aug 5, 2021

Uh oh!

mtojek left a comment

Uh oh!

mtojek Aug 5, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

		@@ -1,249 +1,28 @@
		- description: IP address of the client.
		name: client.ip

Add support for Azure Spring Cloud logs #1355

Add support for Azure Spring Cloud logs #1355

Uh oh!

Conversation

narph commented Jul 26, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Screenshots

Uh oh!

elasticmachine commented Jul 26, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

Test stats 🧪

Trends 🧪

Uh oh!

narph commented Aug 4, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hemantmalik commented Aug 4, 2021

Uh oh!

mtojek Aug 5, 2021

Choose a reason for hiding this comment

Uh oh!

mtojek Aug 5, 2021

Choose a reason for hiding this comment

Uh oh!

mtojek Aug 5, 2021

Choose a reason for hiding this comment

Uh oh!

mtojek left a comment

Choose a reason for hiding this comment

Uh oh!

mtojek Aug 5, 2021

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

narph commented Jul 26, 2021 •

edited

Loading

elasticmachine commented Jul 26, 2021 •

edited

Loading

narph commented Aug 4, 2021 •

edited

Loading