Integrate ANN into _search endpoint #88694
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This PR adds a new `knn` section to the search request. As explained in #87625, the search returns a global top k documents across all shards: * In the DFS phase, collect the top k vector matches per shard, combine them and keep the global top k. * Then move to the query phase. Convert the top k vector matches into a new `KnnScoreDocQueryBuilder` that matches only those top documents. For the final query, take a boolean disjunction between this `KnnScoreDocQueryBuilder` query and the search request `query`. Commits: * Rename KnnSearchRequestBuilder -> KnnSearchRequestParser to avoid confusion with new KnnSearchBuilder * Add `KnnScoreDocQueryBuilder` * Implement kNN search using DFS phase Addresses #87625.
This PR introduces a 'filter' subsection where filters can be provided to the kNN search. Like all other queries, the filters are rewritten on the coordinating nodes as well as data nodes.
When kNN search is enabled, it automatically uses the `dfs_query_then_fetch` search type. The user can no longer control the search type that's used. This commit throws an error if `search_type` is explicitly set when `knn` search is used.
This change adds docs for the new search `knn` option: * Add `knn` option to `_search` API docs. * Update the kNN search guide to use the `_search` endpoint instead of `_knn_search` endpoint. Move relevant information from the `_knn_search` API docs to this guide, like the filtering example. * Add an example to the kNN search guide of configuring `index_options` (just seemed nice to have, not core to this change). The PR doesn't deprecate the `_knn_search` endpoint yet.
jtibshirani
added
>feature
release highlight
v8.4.0
:Search Relevance/Vectors
|
Each commit was reviewed earlier:
There are some follow-ups tracked in #87625 that we'll tackle after this initial merge. |
|
Pinging @elastic/es-search (Team:Search) |
|
Hi @jtibshirani, I've created a changelog YAML for you. Note that since this PR is labelled |
mayya-sharipova
approved these changes
Jul 21, 2022
There was a problem hiding this comment.
@jtibshirani Thanks Julie, considering that everything was reviewed before, I only quickly browsed this PR and everything LGTM.
mayya-sharipova
added a commit
to mayya-sharipova/elasticsearch
that referenced
this pull request
May 18, 2023
Add a note that approximate knn search doesn't support the parameter ccs_minimize_roundtrips in CCS search. Relates to elastic#88694
mayya-sharipova
added a commit
that referenced
this pull request
May 23, 2023
Add a note that approximate knn search doesn't support the parameter ccs_minimize_roundtrips in CCS search. Relates to #88694
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds a new
knnoption to the_searchAPI to support ANN search.It's powered by the same Lucene ANN capabilities as the old
_knn_searchendpoint. The
knnoption can be combined with other search features likequeries and aggregations.
Addresses #87625