[8.18/9.0]Change the size of region_start_bytes + add the field to the alert data stream

[AsuNa-jp]

Change Summary

We changed the size (string size) of the following fields from 32 bytes to 64 bytes. Thus, we would like to change the discription and example written in the endpoint-package as well. Moreover, this PR add the fields to the alert data stream.
The field itself was added at this PR.

• process.Ext.memory_region.region_start_bytes

Sample values

Sample document:

    "message": "Memory Threat Detection Alert: Multi.EICAR.Not-a-virus",
    "process": {
        "Ext": {
            "ancestry": [
                "pPVZfPJXb3LwUZ6tFZ8cNA",
                "zkRe4BMHSS5W62hRN/nEhw",
                "oMfl23/bSjrkV8n+vJdWKQ",
                "/kph1ryYQOxMtjxrL99sxQ",
                "tfZhGzm7UQh2y/dLc5PM5g"
            ],
            "architecture": "x86_64",
            "code_signature": [
                {
                    "exists": false
                }
            ],
            "memory_region": {
                "allocation_base": 6442450944,
                "allocation_protection": "RW-",
                "allocation_size": 110592,
                "allocation_type": "PRIVATE",
                "bytes_address": 6442450944,
                "bytes_allocation_offset": 0,
                "bytes_compressed_present": false,
                "malware_signature": {
                    "all_names": "Multi.EICAR.Not-a-virus",
                    "identifier": "diagnostic-malware-signature-v1-windows",
                    "primary": {
                        "matches": [
                            "WDVPIVAlQEFQWzRcUFpYNTQoUF4pN0NDKTd9JEVJQ0FSLVNUQU5EQVJELUFOVElWSVJVUy1URVNULUZJTEUhJEgrSCo="
                        ],
                        "signature": {
                            "hash": {
                                "sha256": "bb0e0bdf70ec65d98f652e2428e3567013d5413f2725a2905b372fd18da8b9dd"
                            },
                            "id": "ac8f42d6-52da-46ec-8db1-5a5f69222a38",
                            "name": "Multi.EICAR.Not-a-virus"
                        }
                    },
                    "secondary": [],
                    "version": "1.0.70"
                },
                "region_base": 6442536960,
                "region_protection": "RW-",
                "region_size": 8192,
                "region_start_bytes": "58354f2150254041505b345c505a58353428505e2937434329377d2445494341522d5354414e444152442d414e544956495255532d544553542d46494c452124",
                "region_state": "COMMIT"
            },

Release Target

8.18/9.0

Q/A

For mapping changes:

• I ran make after making the schema changes, and committed all changes

[AsuNa-jp]

Hi @pzl
In the 9.0.0 beta BC1, the process.Ext.memory_region.region_start_bytes appeared as an Unmapped fields even though it was added in this PR (#567.).

I assume this is because we changed the size of region_start_bytes from 32 bytes long string to 64 bytes long string. Thus, I’m updating the description and example in the endpoint-package in this PR to reflect that change, but please let me know if anything else needs to be adjusted.

[pzl]

@AsuNa-jp

what is the version number of the defend integration you have installed?

On the fleet integrations page, is the "install beta integrations" button enabled?

[AsuNa-jp]

Hi @pzl
Thank you for the reply. I am currently testing endpoint (defend integration) 8.18 and 9.0.
After applying the install beta integrations, the process.Ext.memory_region.region_start_bytes moved from Unmapped fields, but it still appers as empty fields even though there is a event which contains a value in the process.Ext.memory_region.region_start_bytes. Do you have any thoughts on why this is happening?

[pzl]

@AsuNa-jp unclear, perhaps an issue where documents were indexed before the mapping was created. Old documents will not be reindexed automatically. And kibana's discover may have an outdated heuristic about that field now.

Try with a clean install, where the integration is up-to-date and the mapping exists before ingesting the documents

[AsuNa-jp]

@pzl (CC: @magermark )
Thanks for the reply.
Based on your feedback, I've created a new 8.18 instance on the stating environement, and first turned on the install beta integrations, and then installed the 8.18 agent/Defend Integration, but nothing has changed.

After comparing Kibana before and after emitting events that include fields process.Ext.memory_region.region_start_bytes (and process.Ext.memory_region.region_state), I found that process.Ext.memory_region.region_start_bytes was still as an empty fields, even though it contains data.

• before generating the event
• process.Ext.memory_region.region_start_bytes -> empty fields
• process.Ext.memory_region.region_state -> empty fields
• after generating the event
• process.Ext.memory_region.region_start_bytes -> empty fields
• process.Ext.memory_region.region_state -> Available fields

To be honest, I’m not sure about the cause and wonder if it might be a bug on Kibana’s side. If there’s someone more suitable to ask, I’d be happy to reach out to them. Do you have any thoughts or know someone who might?

[AsuNa-jp]

hi @pzl
Thank you for pointing out that region_start_bytes was not added to the alert data stream. I have updated the PR, so I would appreciate it if you could review the changes when you have time

[pzl]

@AsuNa-jp please update the file package/endpoint/data_stream/alerts/sample_event.json to place an example value of this field where it would appear. This will aid in some automated testing for this field addition within the repo here

otherwise, all good to go 👍

[AsuNa-jp]

@pzl
Thanks for the approval.
I've added the "memory_region" block including region_start_bytes filed to sample_event.json, since it was missing.

            "memory_region": {
                "region_size": 4096,
                "region_protection": "RWX",
                "allocation_base": 2401471234048,
                "allocation_type": "PRIVATE",
                "bytes_allocation_offset": 0,
                "region_state": "COMMIT",
                "bytes_compressed_present": false,
                "region_start_bytes": "58354f2150254041505b345c505a58353428505e2937434329377d2445494341522d5354414e444152442d414e544956495255532d544553542d46494c452124",
                "allocation_protection": "RW-",
                "region_base": 2401471238144,
                "allocation_size": 8192,
                "bytes_address": 2401471234048