Add support for mobile UI filtering

[bryce-b]

Motivation/summary

The mobile ui wants to filter data based on host.os.version & service.version. Http spans are a key part of the mobile ui, and when filters are added for either of these attributes all http related charts will be empty.

Checklist

• Update CHANGELOG.asciidoc
• Update package changelog.yml (only if changes to apmpackage have been made)
• Documentation has been updated

For functional changes, consider:

• Is it observable through the addition of either logging or metrics?
• Is its use being published in telemetry to enable product improvement?
• Have system tests been added to avoid regression?

How to test these changes

Related issues

elastic/apm-agent-ios#145

[mergify]

This pull request does not have a backport label. Could you fix it @bryce-b? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-7.x is the label to automatically backport to the 7.x branch.
• backport-7./d is the label to automatically backport to the 7./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

[ghost]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2023-04-28T15:43:44.811+0000

• Duration: 18 min 48 sec

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate and publish the docker images.

• /test windows : Build & tests on Windows.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[ghost]

📚 Go benchmark report

Diff with the main branch

goos: linux
goarch: amd64
pkg: github.com/elastic/apm-server/internal/agentcfg
cpu: 12th Gen Intel(R) Core(TM) i5-12500
                                  │ build/main/bench.out │             bench.out             │
                                  │        sec/op        │   sec/op     vs base              │
geomean                                      68.79n        68.45n       -0.49%

                                  │ build/main/bench.out │             bench.out              │
                                  │         B/op         │    B/op     vs base                │
geomean                                                ²               +0.00%               ²
¹ all samples are equal
² summaries must be >0 to compute geomean

                                  │ build/main/bench.out │             bench.out              │
                                  │      allocs/op       │ allocs/op   vs base                │
geomean                                                ²               +0.00%               ²
¹ all samples are equal
² summaries must be >0 to compute geomean

pkg: github.com/elastic/apm-server/internal/beater/request
                                             │ build/main/bench.out │              bench.out              │
                                             │        sec/op        │    sec/op     vs base               │
ContextReset/Remote_Addr_ipv6-12                       606.5n ± 16%   717.7n ± 13%  +18.34% (p=0.015 n=6)
geomean                                                888.3n         914.5n         +2.94%

                                             │ build/main/bench.out │              bench.out               │
                                             │         B/op         │     B/op      vs base                │
geomean                                                           ²                 +0.00%               ²
¹ all samples are equal
² summaries must be >0 to compute geomean

                                             │ build/main/bench.out │             bench.out              │
                                             │      allocs/op       │ allocs/op   vs base                │
geomean                                                           ²               +0.00%               ²
¹ all samples are equal
² summaries must be >0 to compute geomean

pkg: github.com/elastic/apm-server/internal/publish
             │ build/main/bench.out │          bench.out          │
             │        sec/op        │   sec/op    vs base         │

             │ build/main/bench.out │           bench.out            │
             │         B/op         │     B/op       vs base         │

             │ build/main/bench.out │           bench.out           │
             │      allocs/op       │  allocs/op    vs base         │

pkg: github.com/elastic/apm-server/x-pack/apm-server/aggregation/spanmetrics
                 │ build/main/bench.out │           bench.out           │
                 │        sec/op        │    sec/op     vs base         │

                 │ build/main/bench.out │            bench.out            │
                 │         B/op         │     B/op      vs base           │
¹ all samples are equal

                 │ build/main/bench.out │           bench.out           │
                 │      allocs/op       │ allocs/op   vs base           │
¹ all samples are equal

pkg: github.com/elastic/apm-server/x-pack/apm-server/aggregation/txmetrics
                        │ build/main/bench.out │          bench.out           │
                        │        sec/op        │   sec/op     vs base         │

                        │ build/main/bench.out │           bench.out           │
                        │         B/op         │    B/op     vs base           │
¹ all samples are equal

                        │ build/main/bench.out │           bench.out           │
                        │      allocs/op       │ allocs/op   vs base           │
¹ all samples are equal

pkg: github.com/elastic/apm-server/x-pack/apm-server/sampling
               │ build/main/bench.out │             bench.out              │
               │        sec/op        │    sec/op     vs base              │
geomean                  594.3n         606.6n        +2.09%

               │ build/main/bench.out │              bench.out               │
               │         B/op         │     B/op      vs base                │
geomean                             ²                 +0.22%               ²
¹ all samples are equal
² summaries must be >0 to compute geomean

               │ build/main/bench.out │             bench.out              │
               │      allocs/op       │ allocs/op   vs base                │
geomean                             ²               +0.00%               ²
¹ all samples are equal
² summaries must be >0 to compute geomean

pkg: github.com/elastic/apm-server/x-pack/apm-server/sampling/eventstorage
                                            │ build/main/bench.out │             bench.out              │
                                            │        sec/op        │    sec/op     vs base              │
ReadEvents/nop_codec/1000_events-12                  904.0µ ±   5%   959.1µ ±  8%  +6.09% (p=0.009 n=6)
geomean                                              30.48µ          31.80µ        +4.32%

                                            │ build/main/bench.out │               bench.out                │
                                            │         B/op         │      B/op       vs base                │
WriteTransaction/json_codec_big_tx-12                3.687Ki ±  0%    3.688Ki ±  0%  +0.03% (p=0.045 n=6)
geomean                                              31.50Ki          31.44Ki        -0.19%
¹ all samples are equal

                                            │ build/main/bench.out │              bench.out              │
                                            │      allocs/op       │  allocs/op   vs base                │
geomean                                                 144.7         144.7       +0.00%
¹ all samples are equal

report generated with https://pkg.go.dev/golang.org/x/perf/cmd/benchstat

[bryce-b]

I'm not sure if this is the correct solution to this issue. Add these attributes on to spans may result in considerable change in storage size.

[axw]

@felixbarny do you think this is worth benchmarking, or would the design win anyway?

I wonder if we should drop that remove processor altogether. This has been around since forever, when we used to store a transaction document even for non-sampled transactions, and when we didn't have a default ILM delete phase . I feel like we have better options for storage management these days.

[felixbarny]

If we already had synthetic _source, it would probably be a no-brainer to drop the remove processor altogether. Instinctively, I don't expect this to have a huge impact even now but I'd prefer to do some testing for that. If that testing would be very difficult to do (which I hope it isn't, that would kinda be another issue in itself) I suppose we could skip it if we preserve just host and service.name.

[kruskall]

@bryce-b could you run make update check-approvals, confirm the output documents are correct and push the updated documents ? We have system-tests in place to compare the output documents to what we have in the repo and they need to be updated to contain the new fields.

[bryce-b]

@kruskall I ran that command but I'm not seeing any changes to push in the repo. Is that expected?

[kruskall]

Ah sorry, you might need to run the systemtests before that: make system-test

Running the systemtests might take a long time. To make sure the command is working you can run a failing test and update the approval file:

• GOTESTFLAGS="-run=TestCompressedSpans" make system-test
• make update check-approvals
• confirm the new output and you should see changes in your local repo

[mergify]

This pull request is now in conflicts. Could you fix it @bryce-b? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b mobile-filtering-support upstream/mobile-filtering-support
git merge upstream/main
git push upstream mobile-filtering-support

changes accepted

[axw]

Verified with BC2. Send testdata/intake-v2/spans.ndjson to the hosted APM Server, confirmed that host.* and service.version fields are present in span documents.