ES|QL FUSE command

Serverless Preview Stack Preview 9.2.0

The FUSE processing command merges rows from multiple result sets and assigns new relevance scores.

FUSE enables hybrid search to combine and score results from multiple queries, together with the FORK command.

FUSE works by:

1. Merging rows with matching <key_columns> values
2. Assigning new relevance scores using the specified <fuse_method> algorithm and the values from the <group_column> and <score_column>

Use default parameters:

FUSE
		

Specify custom parameters:

FUSE <fuse_method> SCORE BY <score_column> GROUP BY <group_column> KEY BY <key_columns> WITH <options>
		

fuse_method

Defaults to RRF. Can be one of RRF (for Reciprocal Rank Fusion) or LINEAR (for linear combination of scores). Designates which method to use to assign new relevance scores.

options

Options for the fuse_method.

score_column

Defaults to _score. Designates which column to use to retrieve the relevance scores of the input row and where to output the new relevance scores of the merged rows.

group_column

Defaults to _fork. Designates which column represents the result set.

key_columns

Defaults to _id, _index. Rows with matching key_columns values are merged.

In the following example, we use the FORK command to run two different queries: a lexical and a semantic query. We then use FUSE to merge the results (applies RRF by default):

FROM books METADATA _id, _index, _score
| FORK (WHERE title:"Shakespeare" | SORT _score DESC)
       (WHERE semantic_title:"Shakespeare" | SORT _score DESC)
| FUSE
		

1. Include document ID, index name, and relevance score
2. Fork 1: Lexical search on title field, sorted by relevance score
3. Fork 2: Semantic search on semantic_title field, sorted by relevance score
4. Merge results using RRF algorithm by default

FUSE can also use linear score combination:

FROM books METADATA _id, _index, _score
| FORK (WHERE title:"Shakespeare" | SORT _score DESC)
       (WHERE semantic_title:"Shakespeare" | SORT _score DESC)
| FUSE LINEAR
		

1. Fork 1: Lexical search on title
2. Fork 2: Semantic search on semantic_title
3. Merge results using linear combination of scores (equal weights by default)

When combining results from semantic and lexical queries through linear combination, we recommend first normalizing the scores from each result set.

The following example uses minmax score normalization. This means the scores normalize and assign values between 0 and 1, before combining the rows:

FROM books METADATA _id, _index, _score
| FORK (WHERE title:"Shakespeare" | SORT _score DESC)
       (WHERE semantic_title:"Shakespeare" | SORT _score DESC)
| FUSE LINEAR WITH { "normalizer": "minmax" }
		

1. Fork 1: Lexical search
2. Fork 2: Semantic search
3. Linear combination with min-max normalization (scales scores to 0-1 range)

FUSE allows you to specify different weights to scores, based on the _fork column values, enabling you to control the relative importance of each query branch in the final results.

FROM books METADATA _id, _index, _score
| FORK (WHERE title:"Shakespeare" | SORT _score DESC)
       (WHERE semantic_title:"Shakespeare" | SORT _score DESC)
| FUSE LINEAR WITH { "weights": { "fork1": 0.7, "fork2": 0.3 }, "normalizer": "minmax" }
		

1. Fork 1: Lexical search
2. Fork 2: Semantic search
3. Weighted linear combination: 70% lexical, 30% semantic, with min-max normalization

These limitations can be present either when:

• FUSE is not combined with FORK

• FUSE doesn't use the default metadata columns _id, _index, _score and _fork

  1. FUSE assumes that key_columns are single valued. When key_columns are multivalued, FUSE can produce unreliable relevance scores.
  2. FUSE automatically assigns a score value of NULL if the <score_column> or <group_column> are multivalued.
  3. FUSE assumes that the combination of key_columns and group_column is unique. If not, FUSE can produce unreliable relevance scores.