ZooKeeper Integration

Serverless Observability Serverless Security Stack 9.0.0


Version	1.14.0 (View all)
Subscription level What's this?	Basic
Developed by What's this?	Elastic
Ingestion method(s)	Database

This integration periodically fetches metrics from the ZooKeeper service.

Compatibility

The ZooKeeper integration is tested with ZooKeeper 3.4.8 and is expected to work with all version >= 3.4.0. Versions prior to 3.4 do not support the mntr command.

Metrics

connection

The connection dataset fetches the data returned by the cons admin keyword.

		{
    "@timestamp": "2020-07-06T16:12:07.612Z",
    "agent": {
        "ephemeral_id": "4d221f8f-7147-4855-8ea3-b4d2a5b80ae0",
        "id": "2ff8a09c-c7f0-42f2-9fe1-65f7fd460651",
        "name": "zookeeper-01",
        "type": "metricbeat",
        "version": "8.0.0"
    },
    "client": {
        "ip": "172.28.0.1",
        "port": 44338
    },
    "ecs": {
        "version": "8.11.0"
    },
    "event": {
        "dataset": "zookeeper.connection",
        "duration": 3093417,
        "module": "zookeeper"
    },
    "host": {
        "name": "zookeeper-01"
    },
    "metricset": {
        "name": "connection",
        "period": 10000
    },
    "service": {
        "address": "localhost:2181",
        "type": "zookeeper"
    },
    "zookeeper": {
        "connection": {
            "interest_ops": 0,
            "queued": 0,
            "received": 1,
            "sent": 0
        }
    }
}
		
	

ECS Field Reference

Please refer to the following document for detailed information on ECS fields.

Field	Description	Type	Metric Type
@timestamp	Event timestamp.	date
agent.id		keyword
client.ip	IP address of the client (IPv4 or IPv6).	ip
client.port	Port of the client.	long
cloud.account.id	The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier.	keyword
cloud.availability_zone	Availability zone in which this host is running.	keyword
cloud.image.id	Image ID for the cloud instance.	keyword
cloud.instance.id	Instance ID of the host machine.	keyword
cloud.provider	Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean.	keyword
cloud.region	Region in which this host is running.	keyword
container.id	Unique container id.	keyword
data_stream.dataset	Data stream dataset.	constant_keyword
data_stream.namespace	Data stream namespace.	constant_keyword
data_stream.type	Data stream type.	constant_keyword
event.dataset	Event dataset	constant_keyword
event.module	Event module	constant_keyword
host.containerized	If the host is a container.	boolean
host.name	Name of the host. It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use.	keyword
host.os.build	OS build information.	keyword
host.os.codename	OS codename, if any.	keyword
service.address	Address where data about this service was collected from. This should be a URI, network address (ipv4:port or [ipv6]:port) or a resource path (sockets).	keyword
zookeeper.connection.interest_ops	Interest ops	long
zookeeper.connection.queued	Queued connections	long	gauge
zookeeper.connection.received	Received connections	long	counter
zookeeper.connection.sent	Connections sent	long	counter

mntr

The mntr Metricset fetches the data returned by the mntr admin keyword.

		{
    "@timestamp": "2020-07-06T16:12:08.494Z",
    "agent": {
        "ephemeral_id": "4d221f8f-7147-4855-8ea3-b4d2a5b80ae0",
        "id": "2ff8a09c-c7f0-42f2-9fe1-65f7fd460651",
        "name": "zookeeper-01",
        "type": "metricbeat",
        "version": "8.0.0"
    },
    "ecs": {
        "version": "8.11.0"
    },
    "event": {
        "dataset": "zookeeper.mntr",
        "duration": 15795652,
        "module": "zookeeper"
    },
    "host": {
        "name": "zookeeper-01"
    },
    "metricset": {
        "name": "mntr",
        "period": 10000
    },
    "service": {
        "address": "localhost:2181",
        "type": "zookeeper",
        "version": "3.5.5-390fe37ea45dee01bf87dc1c042b5e3dcce88653, built on 05/03/2019 12:07 GMT"
    },
    "zookeeper": {
        "mntr": {
            "approximate_data_size": 44,
            "ephemerals_count": 0,
            "latency": {
                "avg": 0,
                "max": 0,
                "min": 0
            },
            "max_file_descriptor_count": 1048576,
            "num_alive_connections": 1,
            "open_file_descriptor_count": 49,
            "outstanding_requests": 0,
            "packets": {
                "received": 152,
                "sent": 151
            },
            "server_state": "standalone",
            "watch_count": 0,
            "znode_count": 5
        }
    }
}
		
	

ECS Field Reference

Please refer to the following document for detailed information on ECS fields.

Field	Description	Type	Metric Type
@timestamp	Event timestamp.	date
agent.id		keyword
cloud.account.id	The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier.	keyword
cloud.availability_zone	Availability zone in which this host is running.	keyword
cloud.image.id	Image ID for the cloud instance.	keyword
cloud.instance.id	Instance ID of the host machine.	keyword
cloud.provider	Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean.	keyword
cloud.region	Region in which this host is running.	keyword
container.id	Unique container id.	keyword
data_stream.dataset	Data stream dataset.	constant_keyword
data_stream.namespace	Data stream namespace.	constant_keyword
data_stream.type	Data stream type.	constant_keyword
event.dataset	Event dataset	constant_keyword
event.module	Event module	constant_keyword
host.containerized	If the host is a container.	boolean
host.name	Name of the host. It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use.	keyword
host.os.build	OS build information.	keyword
host.os.codename	OS codename, if any.	keyword
service.address	Address where data about this service was collected from. This should be a URI, network address (ipv4:port or [ipv6]:port) or a resource path (sockets).	keyword
zookeeper.mntr.approximate_data_size	Approximate size of ZooKeeper data.	long	gauge
zookeeper.mntr.ephemerals_count	Number of ephemeral znodes.	long	gauge
zookeeper.mntr.followers	Number of followers seen by the current host.	long	gauge
zookeeper.mntr.latency.avg	Average latency between ensemble hosts in milliseconds.	long	gauge
zookeeper.mntr.latency.max	Maximum latency in milliseconds.	long	gauge
zookeeper.mntr.latency.min	Minimum latency in milliseconds.	long	gauge
zookeeper.mntr.max_file_descriptor_count	Maximum number of file descriptors allowed for the ZooKeeper process.	long	gauge
zookeeper.mntr.num_alive_connections	Number of connections to ZooKeeper that are currently alive.	long	gauge
zookeeper.mntr.open_file_descriptor_count	Number of file descriptors open by the ZooKeeper process.	long	gauge
zookeeper.mntr.outstanding_requests	Number of outstanding requests that need to be processed by the cluster.	long	gauge
zookeeper.mntr.packets.received	Number of ZooKeeper network packets received.	long	gauge
zookeeper.mntr.packets.sent	Number of ZooKeeper network packets sent.	long	gauge
zookeeper.mntr.pending_syncs	Number of pending syncs to carry out to ZooKeeper ensemble followers.	long	gauge
zookeeper.mntr.server_state	Role in the ZooKeeper ensemble.	keyword
zookeeper.mntr.synced_followers	Number of synced followers reported when a node server_state is leader.	long	gauge
zookeeper.mntr.watch_count	Number of watches currently set on the local ZooKeeper process.	long	gauge
zookeeper.mntr.znode_count	Number of znodes reported by the local ZooKeeper process.	long	gauge

server

The server Metricset fetches the data returned by the srvr admin keyword.

		{
    "@timestamp": "2020-07-06T16:12:12.409Z",
    "agent": {
        "ephemeral_id": "4d221f8f-7147-4855-8ea3-b4d2a5b80ae0",
        "id": "2ff8a09c-c7f0-42f2-9fe1-65f7fd460651",
        "name": "zookeeper-01",
        "type": "metricbeat",
        "version": "8.0.0"
    },
    "ecs": {
        "version": "8.11.0"
    },
    "event": {
        "dataset": "zookeeper.server",
        "duration": 3001938,
        "module": "zookeeper"
    },
    "host": {
        "name": "zookeeper-01"
    },
    "metricset": {
        "name": "server",
        "period": 10000
    },
    "service": {
        "address": "localhost:2181",
        "type": "zookeeper",
        "version": "3.5.5-390fe37ea45dee01bf87dc1c042b5e3dcce88653"
    },
    "zookeeper": {
        "server": {
            "connections": 1,
            "count": 0,
            "epoch": 0,
            "latency": {
                "avg": 0,
                "max": 0,
                "min": 0
            },
            "mode": "standalone",
            "node_count": 5,
            "outstanding": 0,
            "received": 156,
            "sent": 155,
            "version_date": "2019-05-03T12:07:00Z",
            "zxid": "0x0"
        }
    }
}
		
	

ECS Field Reference

Please refer to the following document for detailed information on ECS fields.

Field	Description	Type	Metric Type
@timestamp	Event timestamp.	date
agent.id		keyword
cloud.account.id	The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier.	keyword
cloud.availability_zone	Availability zone in which this host is running.	keyword
cloud.image.id	Image ID for the cloud instance.	keyword
cloud.instance.id	Instance ID of the host machine.	keyword
cloud.provider	Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean.	keyword
cloud.region	Region in which this host is running.	keyword
container.id	Unique container id.	keyword
data_stream.dataset	Data stream dataset.	constant_keyword
data_stream.namespace	Data stream namespace.	constant_keyword
data_stream.type	Data stream type.	constant_keyword
event.dataset	Event dataset	constant_keyword
event.module	Event module	constant_keyword
host.containerized	If the host is a container.	boolean
host.name	Name of the host. It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use.	keyword
host.os.build	OS build information.	keyword
host.os.codename	OS codename, if any.	keyword
service.address	Address where data about this service was collected from. This should be a URI, network address (ipv4:port or [ipv6]:port) or a resource path (sockets).	keyword
zookeeper.server.connections	Number of clients currently connected to the server	long	gauge
zookeeper.server.count	Total transactions of the leader in epoch	long	counter
zookeeper.server.epoch	Epoch value of the Zookeeper transaction ID. An epoch signifies the period in which a server is a leader	long
zookeeper.server.latency.avg	Average amount of time taken for the server to respond to a client request	long	gauge
zookeeper.server.latency.max	Maximum amount of time taken for the server to respond to a client request	long	gauge
zookeeper.server.latency.min	Minimum amount of time taken for the server to respond to a client request	long	gauge
zookeeper.server.mode	Mode of the server. In an ensemble, this may either be leader or follower. Otherwise, it is standalone	keyword
zookeeper.server.node_count	Total number of nodes	long	gauge
zookeeper.server.outstanding	Number of requests queued at the server. This exceeds zero when the server receives more requests than it is able to process	long	gauge
zookeeper.server.received	Number of requests received by the server	long	counter
zookeeper.server.sent	Number of requests sent by the server	long	counter
zookeeper.server.version_date	Date of the Zookeeper release currently in use	date
zookeeper.server.zxid	Unique value of the Zookeeper transaction ID. The zxid consists of an epoch and a counter. It is established by the leader and is used to determine the temporal ordering of changes	keyword

Screenshots

This integration includes one or more Kibana dashboards that visualizes the data collected by the integration. The screenshots below illustrate how the ingested data is displayed.

Changelog

Version	Details	Kibana version(s)
1.14.0	Enhancement (View pull request) Add support for Kibana `9.0.0`.	8.13.0 or higher 9.0.0 or higher
1.13.1	Bug fix (View pull request) Remove `hostname` field from mntr data stream.	8.13.0 or higher
1.13.0	Enhancement (View pull request) Add processor support for connection, mntr and server data streams.	8.13.0 or higher
1.12.0	Enhancement (View pull request) ECS version updated to 8.11.0. Update the kibana constraint to ^8.13.0. Modified the field definitions to remove ECS fields made redundant by the ecs@mappings component template.	8.13.0 or higher
1.11.0	Enhancement (View pull request) Add global filter on data_stream.dataset to improve performance.	8.8.0 or higher
1.10.0	Enhancement (View pull request) Update the package format_version to 3.0.0.	8.8.0 or higher
1.9.0	Enhancement (View pull request) Enable time series data streams for the metrics datasets. This improves storage usage and query performance. For more details, see https://www.elastic.co/guide/en/elasticsearch/reference/current/tsds.html	8.8.0 or higher
1.8.5	Enhancement (View pull request) Added dimension mapping for server datastream.	8.2.0 or higher
1.8.4	Enhancement (View pull request) Added dimension mapping for mntr datastream.	8.2.0 or higher
1.8.3	Enhancement (View pull request) Added dimension mapping for connection datastream.	8.2.0 or higher
1.8.2	Enhancement (View pull request) Added metric_type mapping for server datastream.	8.2.0 or higher
1.8.1	Enhancement (View pull request) Added metric_type mapping for mntr datastream.	8.2.0 or higher
1.8.0	Enhancement (View pull request) Added metric_type mapping for connection datastream	8.2.0 or higher
1.7.0	Enhancement (View pull request) Rename ownership from obs-service-integrations to obs-infraobs-integrations	8.2.0 or higher
1.6.0	Enhancement (View pull request) Migrate visualizations to lens.	8.2.0 or higher
1.5.1	Enhancement (View pull request) Added categories and/or subcategories.	7.14.0 or higher 8.0.0 or higher
1.5.0	Enhancement (View pull request) Update ECS version to 8.5.1	7.14.0 or higher 8.0.0 or higher
1.4.0	Enhancement (View pull request) Added infrastructure category.	7.14.0 or higher 8.0.0 or higher
1.3.1	Enhancement (View pull request) Add documentation for multi-fields	7.14.0 or higher 8.0.0 or higher
1.3.0	Enhancement (View pull request) Update to ECS 8.0	—
1.2.0	Enhancement (View pull request) Support Kibana 8.0	7.14.0 or higher 8.0.0 or higher
1.1.1	Enhancement (View pull request) Uniform with guidelines	—
1.1.0	Enhancement (View pull request) Update to ECS 1.12.0	7.14.0 or higher
1.0.0	Enhancement (View pull request) Release Kafka as GA	—
0.4.0	Enhancement (View pull request) Update integration description	—
0.3.1	Bug fix (View pull request) Add missing data_stream fields	—
0.3.0	Enhancement (View pull request) Set "event.module" and "event.dataset"	—
0.2.7	Enhancement (View pull request) Updating package owner	—
0.2.6	Bug fix (View pull request) Correct sample event file.	—
0.1.0	Enhancement (View pull request) initial release	—