diff --git a/content/commands/vadd.md b/content/commands/vadd.md index 735077229d..a2418d0a48 100644 --- a/content/commands/vadd.md +++ b/content/commands/vadd.md @@ -18,7 +18,6 @@ since: 8.0.0 summary: Add a new element to a vector set, or update its vector if it already exists. syntax_fmt: "VADD key [REDUCE dim] (FP32 | VALUES num) vector element [CAS] [NOQUANT | Q8 | BIN]\n [EF build-exploration-factor] [SETATTR attributes] [M numlinks]" title: VADD -bannerText: Vector set is a new data type that is currently in preview and may be subject to change. --- Add a new element into the vector set specified by `key`. The vector can be provided as 32-bit floating point (`FP32`) blob of values, or as floating point numbers as strings, prefixed by the number of elements (3 in the example below): diff --git a/content/commands/vcard.md b/content/commands/vcard.md index c0c1159085..ab77d1f7e3 100644 --- a/content/commands/vcard.md +++ b/content/commands/vcard.md @@ -18,7 +18,6 @@ since: 8.0.0 summary: Return the number of elements in a vector set. syntax_fmt: "VCARD key" title: VCARD -bannerText: Vector set is a new data type that is currently in preview and may be subject to change. --- Return the number of elements in the specified vector set. diff --git a/content/commands/vdim.md b/content/commands/vdim.md index c133f1d5a6..1a889812dc 100644 --- a/content/commands/vdim.md +++ b/content/commands/vdim.md @@ -18,7 +18,6 @@ since: 8.0.0 summary: Return the dimension of vectors in the vector set. syntax_fmt: "VDIM key" title: VDIM -bannerText: Vector set is a new data type that is currently in preview and may be subject to change. --- Return the number of dimensions of the vectors in the specified vector set. diff --git a/content/commands/vemb.md b/content/commands/vemb.md index 40eefa0249..5f2f0ab8a8 100644 --- a/content/commands/vemb.md +++ b/content/commands/vemb.md @@ -18,7 +18,6 @@ since: 8.0.0 summary: Return the vector associated with an element. syntax_fmt: "VEMB key element [RAW]" title: VEMB -bannerText: Vector set is a new data type that is currently in preview and may be subject to change. --- Return the approximate vector associated with a given element in the vector set. diff --git a/content/commands/vgetattr.md b/content/commands/vgetattr.md index 938b9dc7e6..3cfe7b4c29 100644 --- a/content/commands/vgetattr.md +++ b/content/commands/vgetattr.md @@ -18,7 +18,6 @@ since: 8.0.0 summary: Retrieve the JSON attributes of elements. syntax_fmt: "VGETATTR key element" title: VGETATTR -bannerText: Vector set is a new data type that is currently in preview and may be subject to change. --- Return the JSON attributes associated with an element in a vector set. diff --git a/content/commands/vinfo.md b/content/commands/vinfo.md index a8a77485dc..9c29d75e9a 100644 --- a/content/commands/vinfo.md +++ b/content/commands/vinfo.md @@ -18,7 +18,6 @@ since: 8.0.0 summary: Return information about a vector set. syntax_fmt: "VINFO key" title: VINFO -bannerText: Vector set is a new data type that is currently in preview and may be subject to change. --- Return metadata and internal details about a vector set, including size, dimensions, quantization type, and graph structure. diff --git a/content/commands/vismember.md b/content/commands/vismember.md index 4093d1e32f..84e0cdd04b 100644 --- a/content/commands/vismember.md +++ b/content/commands/vismember.md @@ -28,7 +28,6 @@ summary: Check if an element exists in a vector set. syntax_fmt: VISMEMBER key element syntax_str: element title: VISMEMBER -bannerText: Vector set is a new data type that is currently in preview and may be subject to change. --- Check if an element exists in a vector set. diff --git a/content/commands/vlinks.md b/content/commands/vlinks.md index 4de1fd9091..cd1a53bf8b 100644 --- a/content/commands/vlinks.md +++ b/content/commands/vlinks.md @@ -18,7 +18,6 @@ since: 8.0.0 summary: Return the neighbors of an element at each layer in the HNSW graph. syntax_fmt: "VLINKS key element [WITHSCORES]" title: VLINKS -bannerText: Vector set is a new data type that is currently in preview and may be subject to change. --- Return the neighbors of a specified element in a vector set. The command shows the connections for each layer of the HNSW graph. diff --git a/content/commands/vrandmember.md b/content/commands/vrandmember.md index 26f751464b..ff08c68191 100644 --- a/content/commands/vrandmember.md +++ b/content/commands/vrandmember.md @@ -18,7 +18,6 @@ since: 8.0.0 summary: Return one or multiple random members from a vector set. syntax_fmt: "VRANDMEMBER key [count]" title: VRANDMEMBER -bannerText: Vector set is a new data type that is currently in preview and may be subject to change. --- Return one or more random elements from a vector set. diff --git a/content/commands/vrange.md b/content/commands/vrange.md new file mode 100644 index 0000000000..6261330803 --- /dev/null +++ b/content/commands/vrange.md @@ -0,0 +1,126 @@ +--- +arguments: +- name: key + type: key +- name: start + type: string +- name: end + type: string +- name: count + optional: true + type: integer +arity: -4 +categories: +- docs +- develop +- stack +- oss +- rs +- rc +- oss +- kubernetes +- clients +command_flags: +- READONLY +complexity: O(log(K)+M) where K is the number of elements in the start prefix, and + M is the number of elements returned. In practical terms, the command is just O(M) +description: Return elements in a lexicographical range +function: vrangeCommand +group: vector_set +hidden: false +linkTitle: VRANGE +since: 8.4.0 +summary: Return elements in a lexicographical range +syntax_fmt: VRANGE key start end [count] +syntax_str: start end [count] +title: VRANGE +--- +The `VRANGE` command provides a stateless iterator for the elements inside a vector set. It allows you to retrieve all the elements inside a vector set in small amounts for each call, without an explicit cursor, and with guarantees about what you will miss in case the vector set is changing (elements added and/or removed) during the iteration. + +The command returns elements in lexicographical order, using byte-by-byte comparison (like `memcmp()`) to establish a total order among elements. + +## Required arguments + +
key + +The name of the vector set key from which to retrieve elements. + +
+ +
start + +The starting point of the lexicographical range. Can be: +- A string prefixed with `[` for inclusive range (e.g., `[Redis`) +- A string prefixed with `(` for exclusive range (e.g., `(a7`) +- The special symbol `-` to indicate the minimum element + +
+ +
end + +The ending point of the lexicographical range. Can be: +- A string prefixed with `[` for inclusive range +- A string prefixed with `(` for exclusive range +- The special symbol `+` to indicate the maximum element + +
+ +## Optional arguments + +
count + +The maximum number of elements to return. If `count` is negative, the command returns all elements in the specified range (which may block the server for a long time with large sets). + +
+ +## Examples + +Retrieve the first 10 elements starting from the string "Redis" (inclusive): + +``` +VRANGE word_embeddings [Redis + 10 +``` + +Iterate through all elements, 10 at a time: + +``` +VRANGE mykey - + 10 +``` + +Continue iteration from the last element of the previous result (exclusive): + +``` +VRANGE mykey (a7 + 10 +``` + +Return all elements in the set (use with caution): + +``` +VRANGE mykey - + -1 +``` + +## Return information + +{{< multitabs id="return-info" + tab1="RESP2" + tab2="RESP3" >}} + +One of the following: + +- [Array reply]({{< relref "/develop/reference/protocol-spec#arrays" >}}) of elements in lexicographical order within the specified range. +- [Array reply]({{< relref "/develop/reference/protocol-spec#arrays" >}}) (empty array) if the key doesn't exist. + +-tab-sep- + +One of the following: + +- [Array reply]({{< relref "/develop/reference/protocol-spec#arrays" >}}) of elements in lexicographical order within the specified range. +- [Array reply]({{< relref "/develop/reference/protocol-spec#arrays" >}}) (empty array) if the key doesn't exist. + +{{< /multitabs >}} + +## Behavior + +- **Iteration guarantees**: Each range will produce exactly the elements that were present in the range at the moment the `VRANGE` command was executed. +- **Concurrent modifications**: Elements removed or added during iteration may or may not be returned, depending on when they were modified. +- **Empty key**: If the key doesn't exist, returns an empty array. diff --git a/content/commands/vrem.md b/content/commands/vrem.md index 227c1e65e2..ac5f01bed5 100644 --- a/content/commands/vrem.md +++ b/content/commands/vrem.md @@ -18,7 +18,6 @@ since: 8.0.0 summary: Remove an element from a vector set. syntax_fmt: "VREM key element" title: VREM -bannerText: Vector set is a new data type that is currently in preview and may be subject to change. --- Remove an element from a vector set. diff --git a/content/commands/vsetattr.md b/content/commands/vsetattr.md index 46688b309f..7362819f2c 100644 --- a/content/commands/vsetattr.md +++ b/content/commands/vsetattr.md @@ -18,7 +18,6 @@ since: 8.0.0 summary: Associate or remove the JSON attributes of elements. syntax_fmt: "VSETATTR key element \"{ JSON obj }\"" title: VSETATTR -bannerText: Vector set is a new data type that is currently in preview and may be subject to change. --- Associate a JSON object with an element in a vector set. Use this command to store attributes that can be used in filtered similarity searches with `VSIM`. diff --git a/content/commands/vsim.md b/content/commands/vsim.md index f7bb587ef8..0f8d5564d8 100644 --- a/content/commands/vsim.md +++ b/content/commands/vsim.md @@ -21,7 +21,6 @@ since: 8.0.0 summary: Return elements by vector similarity. syntax_fmt: "VSIM key (ELE | FP32 | VALUES num) (vector | element) [WITHSCORES] [WITHATTRIBS] [COUNT num]\n [EPSILON delta] [EF search-exploration-factor] [FILTER expression] [FILTER-EF max-filtering-effort]\n [TRUTH] [NOTHREAD]" title: VSIM -bannerText: Vector set is a new data type that is currently in preview and may be subject to change. --- Return elements similar to a given vector or element. Use this command to perform approximate or exact similarity searches within a vector set. diff --git a/content/develop/ai/search-and-query/administration/overview.md b/content/develop/ai/search-and-query/administration/overview.md index d59e5ad0d6..c9ace1efb2 100644 --- a/content/develop/ai/search-and-query/administration/overview.md +++ b/content/develop/ai/search-and-query/administration/overview.md @@ -239,9 +239,23 @@ These are the pre-bundled scoring functions available in Redis: * Identical to the default TFIDF scorer, with one important distinction: -* **BM25** +* **BM25STD (default)** - A variation on the basic TF-IDF scorer. See [this Wikipedia article for more information](https://en.wikipedia.org/wiki/Okapi_BM25). + A variation on the basic `TFIDF` scorer, see [this Wikipedia article for more info](https://en.wikipedia.org/wiki/Okapi_BM25). + + The relevance score for each document is multiplied by the presumptive document score and a penalty is applied based on slop as in `TFIDF`. + + {{< note >}} + The `BM25` scorer was renamed `BM25STD` in Redis Open Source 8.4. `BM25` is deprecated. + {{< /note >}} + +* **BM25STD.NORM** + + A variation of `BM25STD`, where the scores are normalized by the minimum and maximum score. + +* **BM25STD.TANH** + + A variation of `BM25STD.NORM`, where the scores are normalised by linear function `tanh(x)`. `BMSTDSTD.TANH` can take an optional argument, `BM25STD_TANH_FACTOR Y`, which is used to smooth the function and the score values. The default value for `Y` is 4. * **DISMAX** diff --git a/content/develop/ai/search-and-query/advanced-concepts/scoring.md b/content/develop/ai/search-and-query/advanced-concepts/scoring.md index c09bb64e80..0a8e2e410d 100644 --- a/content/develop/ai/search-and-query/advanced-concepts/scoring.md +++ b/content/develop/ai/search-and-query/advanced-concepts/scoring.md @@ -19,7 +19,7 @@ weight: 8 When searching, documents are scored based on their relevance to the query. The score is a floating point number between 0.0 and 1.0, where 1.0 is the highest score. The score is returned as part of the search results and can be used to sort the results. -Redis Open Source comes with a few very basic scoring functions to evaluate document relevance. They are all based on document scores and term frequency. This is regardless of the ability to use [sortable fields]({{< relref "/develop/ai/search-and-query/advanced-concepts/sorting" >}}). Scoring functions are specified by adding the `SCORER {scorer_name}` argument to a search query. +Redis Open Source comes with a few scoring functions to evaluate document relevance. They are all based on document scores and term frequency. This is regardless of the ability to use [sortable fields]({{< relref "/develop/ai/search-and-query/advanced-concepts/sorting" >}}). Scoring functions are specified by adding the `SCORER {scorer_name}` argument to a search query. If you prefer a custom scoring function, it is possible to add more functions using the [extension API]({{< relref "/develop/ai/search-and-query/administration/extensions" >}}). @@ -78,14 +78,36 @@ Term frequencies are normalized by the length of the document, expressed as the FT.SEARCH myIndex "foo" SCORER TFIDF.DOCNORM ``` -## BM25 (default) +## BM25STD (default) A variation on the basic `TFIDF` scorer, see [this Wikipedia article for more info](https://en.wikipedia.org/wiki/Okapi_BM25). The relevance score for each document is multiplied by the presumptive document score and a penalty is applied based on slop as in `TFIDF`. +{{< note >}} +The `BM25` scorer was renamed `BM25STD` in Redis Open Source 8.4. `BM25` is deprecated. +{{< /note >}} + +``` +FT.SEARCH myIndex "foo" SCORER BM25STD +``` + +## BM25STD.NORM + +A variation of `BM25STD`, where the scores are normalized by the minimum and maximum scores. + +`BM25STD.NORM` uses min–max normalization across the collection, making it more accurate in distinguishing documents when term frequency distributions vary significantly. Because it depends on global statistics, results adapt better to collection-specific characteristics, but this comes at a performance cost: min and max values must be computed and updated whenever the collection changes. This method is recommended when ranking precision is critical and the dataset is relatively stable. + +## BM25STD.TANH + +A variation of `BM25STD.NORM`, where the scores are normalised by linear function `tanh(x)`. `BMSTDSTD.TANH` can take an optional argument, `BM25STD_TANH_FACTOR Y`, which is used to smooth the function and the score values. The default value for `Y` is 4. + +`BM25STD.TANH` applies a smooth transformation using the `tanh(x/factor)` function, which avoids collection-dependent statistics and yields faster, more efficient scoring. While this makes it more scalable and consistent across different datasets, the trade-off is reduced accuracy in cases where min–max normalization provides sharper separation. This method is recommended when performance and throughput are prioritized over fine-grained ranking sensitivity. + +Following is an example of how to use `BM25STD_TANH_FACTOR Y` in a query. + ``` -FT.SEARCH myIndex "foo" SCORER BM25 +FT.SEARCH idx "term" SCORER BM25STD.TANH BM25STD_TANH_FACTOR 12 WITHSCORES ``` ## DISMAX diff --git a/content/develop/ai/search-and-query/vectors/svs-compression.md b/content/develop/ai/search-and-query/vectors/svs-compression.md index aa2c581447..735fc435c6 100644 --- a/content/develop/ai/search-and-query/vectors/svs-compression.md +++ b/content/develop/ai/search-and-query/vectors/svs-compression.md @@ -102,3 +102,27 @@ The strong performance of LVQ and LeanVec stems from their ability to adapt to t A minimum number of representative vectors is required during index initialization to train the compression parameters (see the [TRAINING_THRESHOLD]({{< relref "/develop/ai/search-and-query/vectors/#svs-vamana-index" >}}) parameter). A random sample from the dataset typically works well. * **Handling data drift:** If the characteristics of incoming vectors change significantly over time (that is, a data distribution shift), compression quality may degrade. This is a general limitation of all data-dependent compression methods,not just LVQ and LeanVec. When the data no longer resembles the original training sample, the learned representation becomes less effective. + +## Build Redis Open Source with Intel SVS support + +By default, Redis Open Source with the Redis Query Engine supports SVS-VAMANA indexing with the global 8-bit quantisation. To compile Redis with the Intel SVS-VAMANA optimisations, LeanVec and LVQ, for Intel platforms, follow the instructions below. + +{{< warning >}} +If you are using Redis Open Source under the AGPLv3 or SSPLv1 licenses, you cannot use it together with the Intel Optimization binaries (LeanVec and LVQ). The reason is that the Intel SVS license is not compatible with those licenses. +The LeanVec and LVQ techniques are closed source and are only available for use with Redis Open Source when distributed under the RSALv2 license. +For more details, please refer to the [information provided by Intel](https://github.com/intel/ScalableVectorSearch). +{{< /warning >}} + +### Build Redis Open Source + +Follow the [Redis Open Source build instructions]({{< relref "/operate/oss_and_stack/install/build-stack" >}}). Before executing `make`, define the following environment variable. + +```sh +export BUILD_INTEL_SVS_OPT=yes +``` + +Alternatively, you can define the `BUILD_INTEL_SVS_OPT` variable as part of the `make` command: + +```sh +make BUILD_INTEL_SVS_OPT=yes +``` diff --git a/content/develop/data-types/vector-sets/_index.md b/content/develop/data-types/vector-sets/_index.md index b9046b705f..2baff73bf1 100644 --- a/content/develop/data-types/vector-sets/_index.md +++ b/content/develop/data-types/vector-sets/_index.md @@ -13,8 +13,6 @@ description: Introduction to Redis vector sets linkTitle: Vector sets title: Redis vector sets weight: 55 -bannerText: Vector set is a new data type that is currently in preview and may be subject to change. -bannerChildren: true --- Vector sets are a data type similar to sorted sets, but instead of a score, vector set elements have a string representation of a vector.