Add memfd_create fields to process datastream
#564
Conversation
Is this PR meant to add or change any data stream mappings? Then you would also make changes in other folders.
If the field(s) we are looking at do need to be mapped, then they should be added above. If the field is part of ECS (~8.10) then just adding the field name to the above data stream should be enough. If the field is part of a newer ECS release, then we may need to do some bookkeeping to update our ECS reference. If the field is not part of ECS (a pretty common case, for our workflow here), then the field definition goes in So then in But again, if we don't need to search on it, you do not have to map. Depends on the end use case here. if this is just getting that EAF error message to go away, then you might be done already.
|
It is rendered in the doc directory right next to the src directory. It looks like the tool to do that rendering was run already for this PR. |
|
@pzl thanks for you help. So, as far as I'm aware these aren't ECS fields, so it sounds like the changes also need to go in Is the info in |
|
so if these do need mapping for filtering/search, then yes, put the details about the fields in Yes, |
|
@pzl Does that look right? |
|
@fearful-symmetry stellar. Looks perfect. If you could add your sample values for |
|
@pzl I think we need a code owner review? |
|
Needs a review from someone on |
The memfd object/flags aren't in ECS but this stage0 RFC proposal (so still early/might be revised when aligning with OTEL) was merged which adds |
|
@fearful-symmetry FYI this is likely going to have to be backed out. A prerelease package version just went out, and kibana CI is failing with:
The recent merge of #555 into Looking at the error, I think there is a simple fix here. It should be Absolutely fascinating that this wasn't caught in the |
|
@pzl ack! Sorry about that! Yeah, weird that CI didn't catch that... |
5 participants
Change Summary
This is part of https://github.com/elastic/endpoint-dev/pull/15318, as we need to update the fields here in order for the EAF tests to pass.
The documentation here is a tad difficult so this is probably incomplete; I'm not sure what the difference is between
custom_schemasandcustom_documentation, and if I should just copy-and-paste fields between them?Sample values
Sample document:
{ "agent": { "id": "aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa", "type": "endpoint", "version": "9.0.0-SNAPSHOT" }, "process": { "Ext": { "ancestry": [ "2FglUZ0Lr/dQXMFoO1Bvqg", "ujM4C6D44yhXfnENvuGZnw", "ReLpB+OskmxuUDOXsozgyA", "TryIwW2T5O7gn1VvKhZgCQ", "lGOe4YnopeNXWmCEnKJp3Q" ], "memfd": { "flag_hugetlb": false, "flag_allow_seal": true, "flags": 2, "name": "test_memfd", "flag_exec": false, "flag_cloexec": false, "flag_noexec_seal": false } }, "parent": { "real_user": { "name": "alexk", "id": 1000 }, "interactive": true, "start": "2024-11-21T16:53:03.18Z", "pid": 29994, "working_directory": "/home/alexk/endpoint-dev/Python", "entity_id": "2FglUZ0Lr/dQXMFoO1Bvqg", "executable": "/usr/bin/python3.12", "args": [ "/home/alexk/endpoint-dev/.venv/EAF/bin/python3", "/home/alexk/endpoint-dev/.venv/EAF/bin/pytest", "-v", "endpoint/test/test_events.py::ProcessEventTests_0_capture_mode__ebpf::test_events_memfd_create" ], "name": "python3.12", "tty": { "char_device": { "major": 136, "minor": 6 } }, "real_group": { "name": "alexk", "id": 1000 }, "args_count": 4, "user": { "name": "alexk", "id": 1000 }, "command_line": "/home/alexk/endpoint-dev/.venv/EAF/bin/python3 /home/alexk/endpoint-dev/.venv/EAF/bin/pytest -v endpoint/test/test_events.py::ProcessEventTests_0_capture_mode__ebpf::test_events_memfd_create", "group": { "name": "alexk", "id": 1000 }, "supplemental_groups": [ { "name": "wheel", "id": 10 }, { "name": "libvirt", "id": 985 }, { "name": "docker", "id": 988 } ] }, "group_leader": { "real_user": { "name": "alexk", "id": 1000 }, "interactive": true, "start": "2024-11-21T16:53:03.18Z", "pid": 29994, "working_directory": "/home/alexk/endpoint-dev/Python", "entity_id": "2FglUZ0Lr/dQXMFoO1Bvqg", "executable": "/usr/bin/python3.12", "args": [ "/home/alexk/endpoint-dev/.venv/EAF/bin/python3", "/home/alexk/endpoint-dev/.venv/EAF/bin/pytest", "-v", "endpoint/test/test_events.py::ProcessEventTests_0_capture_mode__ebpf::test_events_memfd_create" ], "name": "python3.12", "tty": { "char_device": { "major": 136, "minor": 6 } }, "real_group": { "name": "alexk", "id": 1000 }, "args_count": 4, "same_as_process": false, "user": { "name": "alexk", "id": 1000 }, "group": { "name": "alexk", "id": 1000 }, "supplemental_groups": [ { "name": "wheel", "id": 10 }, { "name": "libvirt", "id": 985 }, { "name": "docker", "id": 988 } ] }, "previous": [ { "args": [ "/home/alexk/endpoint-dev/.venv/EAF/bin/python3", "/home/alexk/endpoint-dev/.venv/EAF/bin/pytest", "-v", "endpoint/test/test_events.py::ProcessEventTests_0_capture_mode__ebpf::test_events_memfd_create" ], "args_count": 4, "executable": "/usr/bin/python3.12" } ], "real_user": { "name": "alexk", "id": 1000 }, "interactive": true, "start": "2024-11-21T16:54:21.88Z", "pid": 30545, "working_directory": "/home/alexk/endpoint-dev/Python", "entity_id": "5wL0HgFagbHMKoIHNuBegA", "executable": "/home/alexk/endpoint-dev/.venv/EAF/bin/python", "args": [ "python", "-c", "import os; os.memfd_create('test_memfd', os.MFD_ALLOW_SEALING)" ], "session_leader": { "real_user": { "name": "alexk", "id": 1000 }, "interactive": true, "start": "2024-11-21T15:26:51.11Z", "pid": 25614, "working_directory": "/home/alexk/endpoint-dev/Python", "entity_id": "ujM4C6D44yhXfnENvuGZnw", "executable": "/usr/bin/zsh", "args": [ "/usr/bin/zsh", "-i" ], "name": "zsh", "tty": { "char_device": { "major": 136, "minor": 6 } }, "real_group": { "name": "alexk", "id": 1000 }, "args_count": 2, "same_as_process": false, "user": { "name": "alexk", "id": 1000 }, "group": { "name": "alexk", "id": 1000 }, "supplemental_groups": [ { "name": "wheel", "id": 10 }, { "name": "libvirt", "id": 985 }, { "name": "docker", "id": 988 } ] }, "entry_leader": { "parent": { "start": "2024-11-20T15:01:44.02Z", "pid": 1, "entity_id": "AdtuAiv+v3CdZ92vB6q0RQ" }, "real_user": { "name": "alexk", "id": 1000 }, "interactive": false, "start": "2024-11-21T15:05:51.8Z", "entry_meta": { "type": "unknown" }, "pid": 23494, "working_directory": "/home/alexk", "entity_id": "lGOe4YnopeNXWmCEnKJp3Q", "executable": "/usr/bin/bash", "args": [ "sh", "/home/alexk/.vscode-server/cli/servers/Stable-e8653663e8840adaf45af01eab5c627a5af81807/server/bin/code-server", "--connection-token=remotessh", "--accept-server-license-terms", "--start-server", "--enable-remote-auto-shutdown", "--socket-path=/tmp/code-25b9efc4-a1ec-4e3c-af48-8666aa4b847c" ], "name": "bash", "real_group": { "name": "alexk", "id": 1000 }, "args_count": 7, "same_as_process": false, "user": { "name": "alexk", "id": 1000 }, "group": { "name": "alexk", "id": 1000 }, "supplemental_groups": [ { "name": "wheel", "id": 10 }, { "name": "libvirt", "id": 985 }, { "name": "docker", "id": 988 } ] }, "name": "python", "tty": { "char_device": { "major": 136, "minor": 6 } }, "real_group": { "name": "alexk", "id": 1000 }, "args_count": 3, "user": { "name": "alexk", "id": 1000 }, "command_line": "python -c import os; os.memfd_create('test_memfd', os.MFD_ALLOW_SEALING)", "hash": { "sha256": "79d8d6661062c882888095db44299a4f3c58a45819eb71b5823733b41d7c7d01" }, "group": { "name": "alexk", "id": 1000 }, "supplemental_groups": [ { "name": "wheel", "id": 10 }, { "name": "libvirt", "id": 985 }, { "name": "docker", "id": 988 } ] }, "@timestamp": "2024-11-21T16:54:21.8897838Z", "ecs": { "version": "8.10.0" }, "data_stream": { "namespace": "default", "type": "logs", "dataset": "endpoint.events.process" }, "elastic": { "agent": { "id": "aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa" } }, "host": { "os": { "type": "linux" }, "name": "motmot", "id": "dabadaba-0000-0000-0000-000000000000" }, "event": { "sequence": 1639, "created": "2024-11-21T16:54:21.8897838Z", "kind": "event", "module": "endpoint", "action": [ "memfd_create" ], "id": "NnxZeOc0tCK3sTxG++++++TU", "category": [ "process" ], "type": [ "start" ], "dataset": "endpoint.events.process", "outcome": "unknown" }, "message": "Endpoint process event", "user": { "Ext": { "real": { "name": "alexk", "id": 1000 } }, "name": "alexk", "id": 1000 }, "group": { "Ext": { "real": { "name": "alexk", "id": 1000 } }, "name": "alexk", "id": 1000 } }Release Target
Q/A
For mapping changes:
makeafter making the schema changes, and committed all changesmetadatachange, I also updated both transform destination schemas to match