zeronetworks v0.1.0 #5736

dicolanl · 2023-03-29T13:21:32Z

Enhancement

What does this PR do?

New integration for Zero Networks to pull audit logs into Elastic.

Checklist

[X ] I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

[ ]

How to test this PR locally

cd pacakages/zeronetworks
elastic-package test

Related issues

None

Screenshots

cla-checker-service · 2023-03-29T13:21:37Z

💚 CLA has been signed

elasticmachine · 2023-03-29T13:26:15Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-04-28T20:07:37.613+0000
Duration: 13 min 28 sec

Test stats 🧪

Test	Results
Failed	0
Passed	5
Skipped	0
Total	5

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

dicolanl · 2023-03-29T14:49:38Z

/test

dicolanl · 2023-04-14T18:48:57Z

/test

kcreddy · 2023-04-27T10:00:04Z

/test

elasticmachine · 2023-04-27T10:02:39Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

elasticmachine · 2023-04-27T10:14:54Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`1/1`)	💚
Files	100.0% (`1/1`)	💚
Classes	100.0% (`1/1`)	💚
Methods	100.0% (`13/13`)	💚 1.613
Lines	98.045% (`702/716`)	👍 1.751
Conditionals	100.0% (`0/0`)	💚

andrewkroh · 2023-04-27T11:33:02Z

Can you please add pipeline and system tests. I saw you were having an issue with the pipeline tests. If you can push that commit up we can help troubleshoot the issue.

The "system test" is a end-to-end test where you mock the API response. It allows testing the httpjson input configuration. There are lots of examples of this in other packages that use the httpjson input.

dicolanl · 2023-04-27T13:35:45Z

Hey @andrewkroh
Thanks!

I will work on the system test, but i just committed the two files for pipeline test that seems to break. I hadnt tried system test cause I figured this had to be fixed first.

elastic-package test                     
2023/04/27 09:34:08  INFO New version is available - v0.79.0. Download from: https://github.com/elastic/elastic-package/releases/tag/v0.79.0
Run test suite for the package
Run system tests for the package
--- Test results for package: zeronetworks - START ---
No test results
--- Test results for package: zeronetworks - END   ---
Done
Run asset tests for the package
--- Test results for package: zeronetworks - START ---
╭──────────────┬─────────────┬───────────┬─────────────────────────────────────────────────────────┬────────┬──────────────╮
│ PACKAGE      │ DATA STREAM │ TEST TYPE │ TEST NAME                                               │ RESULT │ TIME ELAPSED │
├──────────────┼─────────────┼───────────┼─────────────────────────────────────────────────────────┼────────┼──────────────┤
│ zeronetworks │ audit       │ asset     │ index_template logs-zeronetworks.audit is loaded        │ PASS   │        875ns │
│ zeronetworks │ audit       │ asset     │ ingest_pipeline logs-zeronetworks.audit-0.1.0 is loaded │ PASS   │        292ns │
╰──────────────┴─────────────┴───────────┴─────────────────────────────────────────────────────────┴────────┴──────────────╯
--- Test results for package: zeronetworks - END   ---
Done
Run pipeline tests for the package
Error: error running package pipeline tests: could not complete test run: installing ingest pipelines failed: marshalling pipeline content failed (pipeline: default-1682602450006393000): json: unsupported type: map[interface {}]interface {}

kcreddy · 2023-04-27T14:42:30Z

/test

kcreddy · 2023-04-27T15:19:00Z

@dicolanl In your script processor, you need to define params with keys as strings instead of integers.

Example:

This does not work:

    params:
      1:
        action: 'Asset is being added to protection'
        outcome: success
      2:
        action: 'Asset added to protection'
        outcome: success

This works

    params:
      "1":
        action: 'Asset is being added to protection'
        outcome: success
      "2":
        action: 'Asset added to protection'
        outcome: success

You can take a look at other script processors for example: https://github.com/elastic/integrations/blob/main/packages/windows/data_stream/sysmon_operational/elasticsearch/ingest_pipeline/default.yml#L50

dicolanl · 2023-04-27T18:00:53Z

@kcreddy Amazing! thank you. that fixed pipeline testing!

@andrewkroh added system testing and tested locally. both pipeline and system testing and pass now.

kcreddy · 2023-04-27T18:12:34Z

/test

kcreddy · 2023-04-27T18:32:45Z

CI is failing with error _dev/deploy folder doesn't exist.

Error: error running package system tests: could not complete test run: _dev/deploy directory not found: "_dev/deploy" directory doesn't exist

Can you check other integrations about how the folder structure needs to be for system tests?
Example: https://github.com/elastic/integrations/blob/main/packages/windows/

dicolanl · 2023-04-27T20:54:08Z

@kcreddy apologies. just fixed it. I had them when i test locally, must have discarded them by accident.

kcreddy · 2023-04-28T06:11:05Z

/test

kcreddy

System tests seem to fail with error:

one or more errors found in document: [0] field "event.type" is not normalized as expected: expected array, found "info" (string)
[1] field "event.category" is not normalized as expected: expected array, found "configuration" (string)

Can you run system tests once again and push sample_event.json file that gets generated?

packages/zeronetworks/data_stream/audit/elasticsearch/ingest_pipeline/default.yml

dicolanl · 2023-04-28T11:54:42Z

Generated Sample event.

Tested locally

Run system tests for the package
2023/04/28 07:53:55  INFO Write container logs to file: /Volumes/Repos/integrations/build/container-logs/zeronetworks-1682682835246862000.log
--- Test results for package: zeronetworks - START ---
╭──────────────┬─────────────┬───────────┬───────────┬────────┬──────────────╮
│ PACKAGE      │ DATA STREAM │ TEST TYPE │ TEST NAME │ RESULT │ TIME ELAPSED │
├──────────────┼─────────────┼───────────┼───────────┼────────┼──────────────┤
│ zeronetworks │ audit       │ system    │ default   │ PASS   │   22.219748s │
╰──────────────┴─────────────┴───────────┴───────────┴────────┴──────────────╯
--- Test results for package: zeronetworks - END   ---
Done

kcreddy · 2023-04-28T14:59:54Z

/test

packages/zeronetworks/data_stream/audit/elasticsearch/ingest_pipeline/default.yml

dicolanl · 2023-04-28T16:01:01Z

@kcreddy
fixed per your comments.

Run pipeline tests for the package
--- Test results for package: zeronetworks - START ---
╭──────────────┬─────────────┬───────────┬────────────────┬────────┬──────────────╮
│ PACKAGE      │ DATA STREAM │ TEST TYPE │ TEST NAME      │ RESULT │ TIME ELAPSED │
├──────────────┼─────────────┼───────────┼────────────────┼────────┼──────────────┤
│ zeronetworks │ audit       │ pipeline  │ test-audit.log │ PASS   │   4.283042ms │
╰──────────────┴─────────────┴───────────┴────────────────┴────────┴──────────────╯
--- Test results for package: zeronetworks - END   --

kcreddy · 2023-04-28T20:06:29Z

/test

dicolanl · 2023-05-03T21:43:35Z

Hey @kcreddy let me know if you need anything else for this. thanks!

kcreddy · 2023-05-04T12:55:53Z

Hey @jamiehynds This is a new package from the community. Can I go ahead with the merge?

jamiehynds · 2023-05-05T08:47:26Z

Looks good @kcreddy

Thanks for the contribution @dicolanl! Do you work with Zero Networks, if so, it would be great to walk you through our partnership program which vendors generally sign up to when contributing integrations.

kcreddy

LGTM 👍🏼

elasticmachine · 2023-05-05T10:35:50Z

Package zeronetworks - 0.1.0 containing this change is available at https://epr.elastic.co/search?package=zeronetworks

jamiehynds · 2023-06-30T10:36:33Z

@dicolanl I just noticed the Zero Networks integration includes a screenshot of a dashboard, but a dashboard isn't included with the integration. Just wanted to flag, incase it's an oversight.

dicolanl · 2023-06-30T13:51:32Z

@jamiehynds
Thanks. Ill work on this for the next update.

zeronetworks v0.1.0

afa24ca

dicolanl added 2 commits March 29, 2023 09:21

Update changelog.yml

79b0f25

Merge branch 'main' into ZeroNetworks

d3ca528

dicolanl mentioned this pull request Apr 4, 2023

Error running pipeline test elastic/elastic-package#1202

Open

dicolanl marked this pull request as draft April 14, 2023 18:53

dicolanl marked this pull request as ready for review April 14, 2023 18:54

kcreddy mentioned this pull request Apr 27, 2023

How does one get a PR reviewed? #5977

Closed

kcreddy added enhancement New feature or request New Integration Issue or pull request for creating a new integration package. Team:Security-External Integrations labels Apr 27, 2023

Pipeline test

8cc09a4

Fixes for system and pipeline testing

cf2ccf9

Fix for system test.

72d5d5e

kcreddy reviewed Apr 28, 2023

View reviewed changes

generate sample event.

4f036f1

fix typos

c0fac94

kcreddy reviewed Apr 28, 2023

View reviewed changes

packages/zeronetworks/data_stream/audit/elasticsearch/ingest_pipeline/default.yml Outdated Show resolved Hide resolved

fixes per review

099b684

kcreddy closed this Apr 28, 2023

kcreddy reopened this Apr 28, 2023

kcreddy approved these changes May 5, 2023

View reviewed changes

kcreddy merged commit a297c20 into elastic:main May 5, 2023

zeronetworks v0.1.0 #5736

zeronetworks v0.1.0 #5736

Uh oh!

Conversation

dicolanl commented Mar 29, 2023

What does this PR do?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Screenshots

Uh oh!

cla-checker-service bot commented Mar 29, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticmachine commented Mar 29, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

Uh oh!

dicolanl commented Mar 29, 2023

Uh oh!

dicolanl commented Apr 14, 2023

Uh oh!

kcreddy commented Apr 27, 2023

Uh oh!

elasticmachine commented Apr 27, 2023

Uh oh!

elasticmachine commented Apr 27, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🌐 Coverage report

Uh oh!

andrewkroh commented Apr 27, 2023

Uh oh!

dicolanl commented Apr 27, 2023

Uh oh!

kcreddy commented Apr 27, 2023

Uh oh!

kcreddy commented Apr 27, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dicolanl commented Apr 27, 2023

Uh oh!

kcreddy commented Apr 27, 2023

Uh oh!

kcreddy commented Apr 27, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dicolanl commented Apr 27, 2023

Uh oh!

kcreddy commented Apr 28, 2023

Uh oh!

kcreddy left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dicolanl commented Apr 28, 2023

Uh oh!

kcreddy commented Apr 28, 2023

Uh oh!

Uh oh!

dicolanl commented Apr 28, 2023

Uh oh!

kcreddy commented Apr 28, 2023

Uh oh!

dicolanl commented May 3, 2023

Uh oh!

kcreddy commented May 4, 2023

Uh oh!

jamiehynds commented May 5, 2023

Uh oh!

kcreddy left a comment

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented May 5, 2023

Uh oh!

cla-checker-service bot commented Mar 29, 2023 •

edited

Loading

elasticmachine commented Mar 29, 2023 •

edited

Loading

elasticmachine commented Apr 27, 2023 •

edited

Loading

kcreddy commented Apr 27, 2023 •

edited

Loading

kcreddy commented Apr 27, 2023 •

edited

Loading