<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[GSI Technology - Medium]]></title>
        <description><![CDATA[GSI Technology’s mission is to create world-class development and production partnerships using current and emerging technologies to help our customers, suppliers, and employees grow. Here on Medium, we discuss the applications of this tech through our blogs. - Medium]]></description>
        <link>https://medium.com/gsi-technology?source=rss----12ba45b97995---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>GSI Technology - Medium</title>
            <link>https://medium.com/gsi-technology?source=rss----12ba45b97995---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 07:08:21 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/gsi-technology" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Efficient HNSW Indexing: Reducing Index Build Time Through Massive Parallelism]]></title>
            <link>https://medium.com/gsi-technology/efficient-hnsw-indexing-reducing-index-build-time-through-massive-parallelism-0fc848f68a17?source=rss----12ba45b97995---4</link>
            <guid isPermaLink="false">https://medium.com/p/0fc848f68a17</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[vector-database]]></category>
            <category><![CDATA[vector-search]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[ecommerce]]></category>
            <dc:creator><![CDATA[Pat Lasserre]]></dc:creator>
            <pubDate>Fri, 21 Jun 2024 21:19:27 GMT</pubDate>
            <atom:updated>2024-06-21T21:56:52.181Z</atom:updated>
            <content:encoded><![CDATA[<h4><strong>Introduction</strong></h4><p>In the growing field of vector databases, driven by applications like retrieval-augmented generation (RAG) for Generative AI (GenAI) and recommendations for ecommerce, the HNSW (Hierarchical Navigable Small Worlds) algorithm is used by a lot of the top vector database providers for approximate nearest neighbor (ANN) search. It provides fast search with good recall but comes with the tradeoff of long index build time.</p><p>Long index build time creates several challenges, including scalability limitations, increased operational costs, reduced experimentation for developers, and slow updates for dynamic datasets.</p><p>This blog post explains why building an HNSW index takes a long time, and it presents a solution that leverages a high degree of parallelism to reduce index build time by roughly 85% compared to traditional solutions based on CPU.</p><h4><strong>HNSW Overview</strong></h4><p>HNSW is a graph-based algorithm that efficiently searches for the approximate nearest neighbors of a query. The graph has hierarchical layers of nodes (vector embeddings), where each layer contains a subset of the previous layer. Nodes are connected by edges, which represent the distance between them, and distance is measured by a metric such as cosine similarity.</p><p>The higher layers have fewer nodes and longer edges between them, enabling the bridging of distant regions of the graph and fast exploration of the vector space The bottom layer includes all the vectors and has short-range edges (connecting to nearby nodes), which allows for more refined searches.</p><p>Figure 1 provides a simple example of an HNSW graph structure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/818/1*oNlFxfopuKBH4HRPPycnrg.png" /><figcaption>Figure 1 — Example HNSW graph showing the hierarchical layers structure. The top layer has the fewest nodes and longer connections, while the bottom layer contains all the nodes and has short-range edges.</figcaption></figure><p>During index building or updating, new nodes are inserted based on their distance from existing nodes. At each layer, a dynamic list of the nearest neighbors for each vertex is maintained, with its size determined by the parameter ef_construction.</p><p>The algorithm iterates over the nodes in this list, performing nearest neighbor distance calculations to see if a node’s neighbors are closer to the query. If so, the neighbors are considered for addition to the list.</p><p>A larger ef_construction tracks and evaluates more candidates, increasing the likelihood of finding the true nearest neighbors, which improves accuracy but also increases index build time because more distance calculations are needed.</p><h4><strong>Consequences of Slow Index Build Times</strong></h4><p>As seen in this previous section, building an HNSW index requires a lot of distance computations to find the nearest neighbors in a hierarchical graph. While this results in low search latency and good recall, it comes with the tradeoff of a graph that is slow to build and update.</p><p>For example, as seen in Figure 2, <a href="https://innovation.ebayinc.com/tech/engineering/ebays-blazingly-fast-billion-scale-vector-similarity-engine/">eBay’s experience</a> shows that building an HNSW index for 160 million vectors can take between three to six hours, and that build time increases quickly as the dataset grows.</p><p>This can be problematic for a company like <a href="https://cloud.google.com/blog/products/ai-machine-learning/ebay-uses-vertex-ai-vector-search-for-recommendations">eBay</a> that has billions of live listings.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/876/1*ukAxgjLDgpzhByA1ZFFjtg.png" /><figcaption><a href="https://innovation.ebayinc.com/tech/engineering/ebays-blazingly-fast-billion-scale-vector-similarity-engine/">Figure 2</a>- Index build time vs. index size. Different quantization, M, and ef_construction values are used.</figcaption></figure><p>Slow index build times can lead to many challenges, such as: scalability issues, increased operational costs, limited developer application experimentation, and slow update of dynamic datasets.</p><p><strong>Scalability Issues</strong></p><p>Slow index build time restricts the scalability of applications like ecommerce and RAG. For ecommerce, growing customer bases and expanding product catalogs increase index build time, which delays the serving of relevant product recommendations. In RAG, larger datasets allow for higher-quality responses, but slow index build limit the amount of data that can be efficiently managed.</p><p><strong>Increased Operational Costs</strong></p><p>Slow index build times means computational resources like CPUs and GPUs are in use longer. This increases operational costs, especially in cloud computing environments where costs are based on resource usage</p><p><strong>Limited Developer Application Experimentation</strong></p><p>Developers need to experiment quickly to fine-tune models and improve application performance. Long index build time restricts the number of experiments they can run and increases the time needed to evaluate those experiments, slowing innovation and delaying improvements.</p><p><strong>Slow Update of Dynamic Datasets</strong></p><p>Applications like ecommerce and RAG have dynamic datasets that are frequently updated. Slow index build delays the incorporation of new data, leading to outdated recommendations and responses, which can negatively impact user satisfaction and trust.</p><h4><strong>Ways to Reduce Index Build Time</strong></h4><p>Two effective ways to reduce index build time are parallel processing and vector quantization.</p><p><strong>Parallel Processing</strong></p><p>The dataset is split into clusters of vectors and the nearest neighbor distance calculations within those clusters are performed in parallel. Parallel processing significantly reduces the time needed for the distance calculations, which is the longest part of the index build process.</p><p>Unfortunately, CPUs, which most HNSW index build solutions use, have limited parallel processing capabilities, so a solution with higher parallel processing capabilities is needed.</p><p><strong>Vector Quantization</strong></p><p>Vector quantization compresses vectors, enabling more vectors to be packed per data transfer and simplifies nearest neighbor distance calculations by performing them on fewer bits. This reduces the number of memory accesses from slower external memory and speeds up the distance calculations.</p><h4><strong>Compute-In-Memory (CiM) Associative Processing — Flexible Massive Parallel Processing</strong></h4><p>GSI Technology’s compute-in-memory Associative Processing Unit (APU) is a technology with millions of bit processors that perform computation at the bit level, which allows for flexible quantization. This enables massive parallel processing on any sized data element.</p><p>With the APU, computation is done in place, directly in memory, which avoids the traditional bottleneck between processors and memory.</p><p><strong>APU HNSW Index Build Process</strong></p><p>1. <strong>Quantization</strong>: Compress the dataset to a desired bit length (e.g., 4 bits per feature).</p><p>2. <strong>Clustering</strong>: Use K-means clustering to divide the dataset into clusters.</p><p>3. <strong>Assignment</strong>: Assign each vertex to its closest clusters based on distance to the cluster centroids.</p><p>4. <strong>Load Data</strong>: Load multiple clusters into the APU.</p><p>5. <strong>Nearest Neighbor Search</strong>: Perform nearest neighbor search for the vertices assigned to the clusters loaded in the APU.</p><p>6. <strong>Repeat Steps 4 and 5: </strong>Repeat steps 4 and 5 until all clusters have been processed and all vertices inserted into the graph</p><p>7. <strong>Union of Neighbors</strong>: For each vertex, merge nearest neighbors from multiple clusters.</p><p>8. <strong>Optimization</strong>: After the union, make the edges bidirectional and ensure each vertex has &lt;= K neighbors by pruning redundant edges.</p><p>The APU uses its millions of bit processors to perform the nearest neighbor distance calculations from step 5 in parallel. This massive parallel processing, along with flexible quantization, significantly reduces the time needed to compute the distance calculations<strong><em>. Performing nearest neighbor distance calculations is the most time-consuming part of building an index, so reducing it will have the greatest impact on speeding up the index build.</em></strong></p><h4><strong>Results</strong></h4><p>Figure 3, <a href="https://developer.nvidia.com/blog/accelerating-vector-search-using-gpu-powered-indexes-with-rapids-raft/">from Nvidia</a>, shows that an Intel Xeon Platinum 8480CL CPU takes 5,636 seconds (about 1.5 hours) to build an HNSW index for 100M vectors.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/793/1*d2TpOe0JQYBCHpeYWCrKWg.png" /><figcaption><a href="https://developer.nvidia.com/blog/accelerating-vector-search-using-gpu-powered-indexes-with-rapids-raft/">Figure 3</a> — CPU HNSW index build time for 100M vectors</figcaption></figure><p>Figure 4 shows that an APU system can build a 100 million vector HNSW index in 864 seconds (about 15 minutes). <strong>This is roughly an 85% reduction in build time compared to the 5,636 seconds (about 1.5 hours) for an Intel Xeon Platinum 8480CL CPU.</strong></p><p>Figure 4 also shows that an APU system can build a 1 billion vector HNSW index in roughly 2 hours, which is a dramatic improvement compared to the eBay performance from Figure 2, if it were to be extrapolated to 1 billion vectors.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*fwBOMKMHVrYGEQgpY6wnYA.png" /><figcaption><strong>Figure 4 — </strong>HNSW index build times using an APU system for 100 million — 1 billion vectors</figcaption></figure><h4><strong>Conclusion</strong></h4><p>HNSW is a leading algorithm used for vector similarity search in applications like GenAI and ecommerce. It provides high recall and fast search, but with the tradeoff of long index build time.</p><p>One of the main ways to reduce index build time is through parallelization. Unfortunately, most current solutions use CPUs, which have limited parallel processing capabilities.</p><p>Through massive parallel processing and flexible quantization, GSI Technology’s APU reduces index build time by approximately 85% compared to traditional CPU-based solutions, enhancing scalability, lowering operational costs, enabling faster developer application experimentation, and ensuring timely updates for dynamic datasets.</p><p>For more details about how the APU’s flexible bit-level processing provides massive parallel processing to significantly reduce index build time, read the white paper <a href="https://gsitechnology.com/enhancing-hnsw-efficiency-cutting-index-build-time-by-85-percent/"><strong>Enhancing HNSW Efficiency: Cutting Index Build Time by 85%</strong></a>.</p><p>To learn about GSI Technology’s on-prem and cloud-based HNSW index build offerings, contact us at associativecomputing@gsitechnology.com.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0fc848f68a17" width="1" height="1" alt=""><hr><p><a href="https://medium.com/gsi-technology/efficient-hnsw-indexing-reducing-index-build-time-through-massive-parallelism-0fc848f68a17">Efficient HNSW Indexing: Reducing Index Build Time Through Massive Parallelism</a> was originally published in <a href="https://medium.com/gsi-technology">GSI Technology</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Compute-in-Memory Computational Devices]]></title>
            <link>https://medium.com/gsi-technology/compute-in-memory-computational-devices-3a73bc722292?source=rss----12ba45b97995---4</link>
            <guid isPermaLink="false">https://medium.com/p/3a73bc722292</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[cnm]]></category>
            <category><![CDATA[gpu]]></category>
            <category><![CDATA[cim]]></category>
            <dc:creator><![CDATA[Sarah Hickey]]></dc:creator>
            <pubDate>Tue, 23 Apr 2024 23:16:27 GMT</pubDate>
            <atom:updated>2024-07-12T16:36:04.506Z</atom:updated>
            <content:encoded><![CDATA[<h3>Compute-In-Memory (CIM) Definition</h3><p>Compute-in-memory (CIM) conceptually means doing compute directly <strong>in</strong> the memory. As will be discussed in detail in subsequent sections, the memory itself is capable of performing efficient computing, which incurs much less data transfers resulting in higher performance and much lower power. These characteristics make it very beneficial for AI and high-performance computing workloads. Sometimes In-memory compute (IMC) is used for this definition also, but as will be explained in detail care must be observed to distinguish some of the uses of this term with a more appropriate definition of compute-near-memory which has different characteristics.</p><p>In contrast, near-memory computing architectures have the compute and memory blocks, even within the same chip or die, physically separate as standalone circuit blocks. CPUs, GPUs, or in general any modern AI ASICs (e.g., TPU, Inferentia, etc.) integrate various heterogeneous blocks and larger memory blocks — commonly referred to as caches and register files. These modern processors, SoCs, and AI accelerators are good examples of near-memory computing architectures. These follow what’s known as a von Neumann architecture, which requires the movement of data and instructions from memory to these heterogeneous computational units via direction from a sequencer and pipelining mechanism. The minimum block that computations can be measured from are a computation unit and the local register memory that provides the inputs for it to operate on.</p><p>The traditional von Neumann architecture extends to the near memory concept at a system level, with separate compute and memory chips in a system. In traditional von Neumann, including near-memory architectures, data movement between compute and memory at board level or chip level becomes costly in terms of time and energy. In recent data-centric applications, such as Artificial Intelligence/Machine Learning (AI/ML), this problem is aggravated with large amounts of data movement.</p><p><strong>Figure 1</strong> below is a side-by-side conceptual illustration of the three architecture types. The blocks show the minimum circuit blocks required for the different architectures to compute something. A primary difference that becomes visible is that CIM structures do not need additional memory to perform their computations. The basic structure can perform as memory and compute.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/972/1*xuyju5ayFDP7OhIINLEocQ.png" /><figcaption><strong>Figure 1: Computing Architectural Types</strong></figcaption></figure><h3>Example Compute in Memory Architecture</h3><p>To solve the computation within memory challenge that brings CIM the power and performance benefits, we must look at memory and build on its capabilities. Two structures provide a base to build from. First is a structure called content-addressable-memory or CAM. The second deals with loosening some operational rules that is inherent to traditional memory.</p><p>CAM is unique but a simple twist on standard memory. In standard memory a specific location is addressed for writing and then reading data. In CAM the read operation is not performed as with traditional memory. Data is presented to CAM, instead of an address, which the device will use to simultaneously compare all locations it has to that data and provide the result address if the data is stored in the part. This may sound weird, but if you think about it, that is similar to how our own brain compares things to memories: have I seen this before and if so, what is it, when did that occur, what should I remember about it. The CAM provides the first step of this flow. A specialized version of CAM adds a “don’t care” capability. That is, data that is sent for comparison can have certain portions or bits defined as “don’t care” if they match. This specialized operation again is similar to our brain operation where we may filter looking for our car in a parking lot and not caring about any vehicle that is not our make, model and color. This type of memory is called a ternary content addressable memory or TCAM. TCAM can be very useful for processing, but by itself it is just a memory with massive lookup capability. Providing the ability to loop back CAM data results for iterative searches or other processing starts to add discerning capability to this function. See<strong> Figure 2</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/840/1*zC3o7mKRKRG_NhuZL4fCIQ.png" /><figcaption><strong>Figure 2: CAM Pattern Processing within a Memory Construct (From Patent US 8238173B2)</strong></figcaption></figure><p>Traditional memory, not CAM, has an array of memory and a unique address for each location. Memories are organized into rows and columns that is a decoding of the address and operate by reading charge from a particular bit at that intersection at a time. These are organized such that a particular memory width (multiple bits) is supported for a particular memory. This is often 8 bits, 16 bits, 32 bits, or 64 bits. Let’s loosely define the width that the memory is configured as a “word”1. Only one bit is read per bit position in a word address. Only the specific result that is at the intersection of the row and column is read and latched into a result register for each bit location in the word. This is a standard rule in memory.</p><p>In order to read the charge from a memory during the read process, memory lines holding bits are pre-charged or have other mechanisms to maintain charge. Instead of reading only one bit from a memory row as is done in a standard memory, if we were to allow multiple bits on a bit line to be triggered an interesting function occurs. Normally, this is prohibited as the state of several bits would collide and we wouldn’t read what we wanted. But if we understood what could happen and controlled it, then we could get logic functions to occur from memory reads like what is shown in <strong>Figure 3</strong>. In that figure we see that if multiple bits can be enabled at the same time onto the same bit line, then just the operation of reading the data incurs a NOR gate compute function. Since memory reads occur at the fastest time in a processor such a construct could be created to create a very fast Boolean logic processor <em>with just the memory lines</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/610/1*zh1OZSZuiw6tLmFO9KE-Sg.png" /><figcaption><strong>Figure 3: Using Storage Cells to Perform Computation: Boolean (From Patent US 8238173B2)</strong></figcaption></figure><h3>Alternative Architectures</h3><h3>Near-Memory Compute Architectures</h3><p>Since compute blocks or Arithmetic Logic Units (ALUs) are well known and implement compute structures the use of these with memory have been the traditional method of creating processors. Putting memory very close to these or in-between stages has been one way of improving local performance.</p><p>GPUs and high performance CPUs with cache close to the processing elements are the best examples of near-memory compute architectures. Compute near memory can also be implemented as separate ALU structures with tightly coupled memory. In these structures the tensor core (in the GPU), or multiply/accumulate (MAC) or other ALU functions in special purpose parts are not inherent functions that occur in a special memory construct. The block diagram of the latest Nvidia GPU H100 is shown in <strong>Figure 4</strong> below to provide an illustration.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/405/1*hO2AC83HGGJ7PTy12_yWGw.png" /><figcaption><strong>Figure 4: Nvidia H100 GPU </strong><a href="https://developer.nvidia.com/blog/nvidia-hopper-architecture-in-depth/"><strong>Block diagram</strong></a></figcaption></figure><p>Three key points to note:</p><p>1) There are multiple types of cores surrounded by memory. Nvidia continues to evolve their cores to support different data formats independently from the memory. For example, the H100 adds FP8/Int8 precision to the Tensor cores. The register file sizes can and do vary separately from the cores from part to part. <br> <br> 2) The register files, L1 cache and shared caches are significantly farther from the tensor compute cores, which would result in higher energy spent in data transfers between compute and memory.<br> <br> 3) As we get farther from the compute cores, more levels of cache are required to keep the cores busy with larger cache sizes but lower bandwidth as we get farther away.</p><p>As one can see from the wide list of features and data format support, a general-purpose GPU or CPU may not deliver the best performance/watt/$ and overall TCO efficiency because not all resources would be used and memory transfers has to be routed to where processing is occurring.</p><p>Another example of a near-memory AI accelerator is Google’s TPU ASIC. The block diagram of TPU v1 (<strong>Figure 5</strong>) clearly illustrates the near-memory architecture where the compute and memory blocks are physically separate. The concept is extended to subsequent generations where external HBM memory is used for storing data with scaler and vector units having limited internal SRAM memory. See <strong>Figure 6</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/315/1*yox0lLtXm8Vhdu-hsAW_UA.png" /><figcaption><strong>Figure 5: TPU v1 </strong><a href="https://cloud.google.com/blog/products/ai-machine-learning/an-in-depth-look-at-googles-first-tensor-processing-unit-tpu"><strong>Block Diagram</strong></a></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/420/1*pIWQ80biy3Cf80DyuCBL0Q.png" /><figcaption><strong>Figure 6: TPU v5e </strong><a href="https://cloud.google.com/tpu/docs/system-architecture-tpu-vm"><strong>Block Diagram</strong></a></figcaption></figure><p>Similarly, many other AI accelerator ASICs developed by startups are mainly near-memory as illustrated in the generic block diagram of <strong>Figure 7</strong> below. The ALUs could be multipliers, multiplier/accumulators (MACs), digital signal processing units (DSP), other general mathematical arithmetic logic units, or vector processing units.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/367/1*eSk6FCHiVGDOAMW-7HTrLw.png" /><figcaption><strong>Figure 7: AI Accelerator ASIC Block Diagram</strong></figcaption></figure><p>Referring back to <strong>Figure 1</strong>, CNM structures are different from CIM in that the structure that does the compute is not a memory in itself. It requires memory to operate, and this memory is positioned closely. This memory is operating as register memory for the actual compute engine, so a CNM structure is a more optimized von Neumann structure with memory close by. The power savings in these structures diminishes as you scale up to more parts or connect to larger memory storage for bigger workloads. It is for this reason that GPUs and multi-core CPUs with large caches for each core and ALU end up consuming large amounts of data when scaled to large arrays. At the high system level, the benefits of the local cache are all but lost when there are so many transfers required to get the data to the actual compute structures.</p><h3>PIM (Processing-In-Memory)</h3><p>The Processing-in-Memory or PIM concept does attempt to reduce energy associated with data transfers between compute and memory, but in current form is merely a near-memory architecture and not true compute-in-memory, because computing is not performed on bit lines of the memory array. Instead, arithmetic logic units or ALUs are embedded close to the memory arrays as we have defined for CNM architectures. At the component level, the difference generally between CNM processors and PIM comes down to the device function — primarily being a processor (CNM) or memory (PIM).</p><p>With current implementation, the computation is limited to fixed data formats, such as FP16 multiply and add. As data formats continue to evolve, as highlighted with the formation of the recent <a href="https://www.opencompute.org/blog/amd-arm-intel-meta-microsoft-nvidia-and-qualcomm-standardize-next-generation-narrow-precision-data-formats-for-ai">Microscaling Formats (MX) Alliance</a>, there is a need to support flexibility in supporting data formats with lower or mixed precision. Efficient software implementations could also be delayed due to changes required in the hardware to address new frameworks when using parts with pre-defined data formats.</p><p>Samsung’s PIM integrates limited and non-programmable compute capability in their HBM2 stack, which could limit flexibility and applicability in future.</p><h3>Analog CIM (Compute-In-Memory)</h3><p>Analog compute-in-memory leverages designed memory elements by using tunable resistors, such as Memristors, to perform computations directly inside the memory array itself.</p><p>This architecture can be called true compute-in-memory (CIM) and can deliver computational efficiency, but there are many technical challenges with this approach as described below.</p><p>Analog CIM has been explored by many research labs for a long time, and startups have started creating test parts for the direction, but due to challenges described below, this approach has hurdles to becoming mainstream in near future. See <strong>Figure 8</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/847/1*t29ginkIQE2vovl_UxHJQw.png" /><figcaption><strong>Figure 8: Computing Architectures</strong></figcaption></figure><p>The limitations of analog compute-in-memory architecture include the following:</p><p><strong>Compute precision:</strong> There are concerns about whether this approach can provide sufficient compute precision to ensure that application accuracies are not compromised. Increasing precision can be achieved by increasing the dynamic range of analog components, but due to noise limitations, this can lead to higher voltage requirements. Analog resistive memory devices also suffer from device-to-device and cycle-to-cycle variances that can limit absolute precision (see Ref. 3, Le Gallo 2023).</p><p><strong>Integration with digital compute:</strong> The seamless integration of analog compute with digital compute is still a question, particularly in the context of achieving full end-to-end realization of deep neural networks on-chip. The ADC/DACs used for analog/digital conversion are susceptible to noise that limits the performance too.</p><p><strong>Latency and cost challenges:</strong> The architecture introduces challenges related to overall latency of computation. This occurs in charge and current domain architectures because of integration steps for results. For analog resistive memory devices, it results from the analog conversion steps. Cost too is a challenge, since most of the companies pioneering analog compute are small startups and the technology hasn’t been deployed in high volume to gain associated cost advantages.</p><p><strong>Design complexity:</strong> While analog compute-in-memory architecture offers some power advantages, its design and implementation can be more complex compared to traditional digital computing architectures.</p><p>In summary, while analog compute-in-memory architecture offers a few advantages, such as energy efficiency, it also presents challenges related to precision, integration, and design complexity that need to be carefully addressed before it is ready for high volume deployment in Data Centers or Edge.</p><h3>Technology Challenges and Market Needs</h3><p>Two key technology challenges are challenging the supremacy of the von Neumann CPU and by extension, the GPU architecture.</p><h3>General Purpose GPUs Are Not Keeping Up with Compute Demand</h3><p>As ChatGPT demonstrated the fastest adoption of any technology to-date, the underlying generative AI technology and large-language models (LLMs) have tremendous promise of increasing productivity of many applications. This has led to an exponential rise in compute demand that is currently met with general purpose GPUs, primarily from Nvidia and AMD. Unfortunately, traditional GPU architectures are not suitable for these emerging AI workloads, which need compute to scale 750 times or more every 2 years, as shown by the red line in chart below (see <strong>Figure 9</strong>). Furthermore, these general-purpose GPUs traditionally relied on Moore’s law scaling that delivered 2x transistor packing density and roughly the same 2x improvement in performance every 2 years, as illustrated by the gray line in the image below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/830/1*bsPmXjVYwTNbTAY3aKnx6Q.png" /><figcaption><strong>Figure 9: Training FLOPs Scaling</strong></figcaption></figure><p>Source: https://medium.com/riselab/ai-and-memory-wall-2cb4265cb0b8</p><h3>Memory and IO Bandwidth Are Not Keeping Up with AI Workload Demand</h3><p>Besides falling short of meeting the compute demands ~3x/2 years vs 750x/2 years required for training Transformer network, the external memory and IO are not keeping up with the generative AI workload demand either as illustrated by green and blue trajectories respectively (see <strong>Figure 10</strong>).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/817/1*esRSYM0R3sf_bX56lGF7Hw.png" /><figcaption><strong>Figure 10: Scaling of Peak Hardware FLOPs</strong></figcaption></figure><p>Source: <a href="https://medium.com/riselab/ai-and-memory-wall-2cb4265cb0b8">https://medium.com/riselab/ai-and-memory-wall-2cb4265cb0b8</a></p><h3>Advantages and Business Value</h3><p>Compute-in-memory represents a paradigm shift in computer architecture, offering a solution to the diminishing returns traditional processors have been facing of controlling power while increasing performance.</p><p><strong>1.</strong> <strong>High power efficiency and low power consumption:</strong></p><p>CIM architectures can compute on data that is in memory. Specifically, data computations within the storage elements can have compute operations applied in-place and looped if required for iterative processing to build up complex functions with reduced instructions and minimal data transfers to storage memory. Data transfer power associated with CIM architectures can also be very low compared to traditional von Neumann architectures by keeping additional memory very close. The combination of these characteristics can offer orders of magnitude less power than the comparable distances in traditional von Neumann architectures, which rely on data transfers from register files and various levels of cache memory to the processing elements for all compute operations.</p><p><strong>2.</strong> <strong>Sustainable Application Performance:</strong></p><p>Because compute-in-memory devices are inherently storage, immediate shaped inputs from sensors can be placed in them for processing. With appropriate register buffer memory and I/O port architecture devices can be achieved that offer “through-wire” processing or “bump-in-the-wire” processing for real-time data. This provides a paradigm change in low power and high throughput which can be taken advantage of for AI and HPC processing for real-time edge applications. These architectures are capable of eliminating application memory bottlenecks for models that can be housed in its array. This leads to 100% core utilization.</p><p><strong>3.</strong> <strong>Scalability:</strong></p><p>Systems utilizing compute-in-memory components can potentially scale as you would add memory to a system: by merely adding more components and without the need for complex connectivity. Using this methodology, it becomes quite normal to have a single low-cost server system with many cards in place. Massive datacenter magnitude scaling now becomes possible with commodity 10gE server interconnects.</p><h3>Conclusion</h3><p>Compute-in-memory represents a significant shift in computing paradigm with proven SRAM technology that is ready for high volume deployment. As the cost of compute and memory — in terms of energy, cooling, and diminishing returns on scale — continues to increase and the volume of data generated by businesses grows, compute-in-memory is becoming an increasingly vital component of modern IT infrastructure, enabling applications and actions that were previously economically unattainable. This whitepaper serves to clearly describe a true digital compute-in-memory architecture and contrast it with alternative architectures, particularly near-memory and PIM. We would like to leave you with a simple checklist or test to determine whether the architecture is true compute-in-memory. If the architecture that is evaluated does not meet the criteria below, they are not compute-in-memory.</p><p><strong>CIM — Simple Checklist/Test</strong></p><p>There are myriad implementations in industry referred to as compute-in-memory. Here is a simple list of checks that should define true compute-in-memory architecture.</p><p>1. Computation occurs where the data resides. For example, external sensor data could be stored in the compute structure, rather than storage memory. This data can then be computed in place, without moving from storage memory. This data transfer savings is significant to performance and low power performance of CIM.</p><p>2. Local or Register memory is not required for the compute operation but may be available to quickly transfer in new data or switch tasks.</p><p>3. Compute logic, if present, is embedded within the memory lines themselves, typically within a few microns. This characteristic is the most open to re-interpretation, as ALUs or computation blocks may be embedded with memory. Such structures will be greater than 100 microns from memory and will not conform to rule 1 above.</p><p>4. Offers the lowest data movement overhead (pJ/bit) and the best TCO measuring performance and power.</p><h3>References</h3><p>1. “Benchmarking and Modeling of Analog and Digital SRAM In-Memory Computing Architectures”. Houshmand, Pouya. Sun, Jiacong. Verchelst, Marian. arXiv:2305.18335[cs.AR], 2023</p><p>2. “AiDAC: A Low-Cost In-Memory Computing Architecture with All-Analog Multi-Bit Compute and Interconnect”. Xuan, Zihao. Chen, Song. Kang, Yi. arXiv:2312.11836[cs.AR], 2023.</p><p>3. “Using the IBM Analog In-Memory Hardware Acceleration Kit for Neural Network Training and Inference”. Le Gallo, Michael. Lammie, Corey. Buchel, Julien. Carta, Fabio. Fagbohungbe, Omobayode. Mackin, Charles. Tsai, Hsinyu. Narayanan, Vijay. Sebastian, Abu. El Maghraoui, Kaoutar. Rasch, Malte. APL Machine Learning. Volume 1, Issue 4. AIP Publishing, 2023.</p><p>4. Akerib, Avidan. Agam, Oren. Ehrman, Eli. Meyassed, Moshe. 2012. Using Storage Cells to Perform Computation (U.S. Patent №8238173B2). U.S. Patent and Trademark Office. <a href="https://patents.google.com/patent/US8238173B2/en">https://patents.google.com/patent/US8238173B2/en</a></p><p><strong>Endnote</strong></p><p>1. <strong>Word</strong> — While not technically precise, this term is used in this paper to reflect different widths that may be inherently stored in a memory. In computer architecture, various words, bytes, words, long words, etc. are used to reflect the different widths possible. For simplicity these are all attributed to the use of “word” in this description.</p><p>For more information, please contact <a href="/service/mailto:associativecomputing@gsitechnology.com"><strong>associativecomputing@gsitechnology.com</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3a73bc722292" width="1" height="1" alt=""><hr><p><a href="https://medium.com/gsi-technology/compute-in-memory-computational-devices-3a73bc722292">Compute-in-Memory Computational Devices</a> was originally published in <a href="https://medium.com/gsi-technology">GSI Technology</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Optimizing Genomics Research: Reducing the High Computational Cost of DNA Sequence Alignment]]></title>
            <link>https://medium.com/gsi-technology/optimizing-genomics-research-reducing-the-high-computational-cost-of-dna-sequence-alignment-f7f3581cbed7?source=rss----12ba45b97995---4</link>
            <guid isPermaLink="false">https://medium.com/p/f7f3581cbed7</guid>
            <category><![CDATA[healthtech]]></category>
            <category><![CDATA[genomics]]></category>
            <category><![CDATA[bioinformatics]]></category>
            <category><![CDATA[genetics]]></category>
            <category><![CDATA[in-memory-computing]]></category>
            <dc:creator><![CDATA[Pat Lasserre]]></dc:creator>
            <pubDate>Fri, 01 Mar 2024 16:45:54 GMT</pubDate>
            <atom:updated>2024-03-07T16:37:08.576Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Introduction:</strong></p><p>DNA sequence alignment is an important tool used in many genomic computation applications like understanding genetic diseases and evolutionary biology. The computational cost of aligning DNA sequences to reference genomes, however, is high. With the increasing volume of genetic data, it’s important to find solutions that minimize this computational cost.</p><p>This post explains why DNA sequence alignment is computationally expensive and shows how effective filtering of candidates can reduce this computational load. We will explore how accelerating the Myers’ bit-parallel edit-distance algorithm with GSI Technology’s Gemini-I APU, a compute-in-memory chip, can optimize the candidate filtering step.</p><p>By the end of this post, you’ll have a better understanding of the challenges in DNA sequence alignment, solutions to address them, and how these advancements will help biological and medical research.</p><p><strong>DNA Sequence Alignment</strong></p><p>DNA sequence alignment is a key tool used in computational genomics for biological and medical research and discovery.</p><p>Sequence alignment helps researchers with tasks such as better understanding diseases and studying evolution. By comparing the DNA sequences of different species, scientists can understand their evolutionary relationships to see how genes have evolved over time. Sequence alignment can also help identify genetic mutations that may cause diseases by comparing the DNA sequences of patients with certain diseases to those without.</p><p>Sequence alignment works by aligning DNA sequences of bases adenine (A), cytosine ©, guanine (G) and thymine (T) to reference genomes. This allows scientists to identify regions of similarity and difference. Figure 1 below shows a query sequence being compared to a target sequence.</p><figure><img alt="Source — Figure 1. Comparing a query and target sequence." src="https://cdn-images-1.medium.com/max/885/1*mhL-HredVLVpsb5WkN3sOg.png" /><figcaption><a href="https://bmcbioinformatics.biomedcentral.com/articles/10.1186/s12859-019-2827-0">Source</a> — Figure 1. Comparing a query and target sequence.</figcaption></figure><p><strong>The Computational Cost of DNA Sequence Alignment</strong></p><p>Reference-guided genome assembly requires multiple steps to align a collection of read sequences against a long reference sequence.</p><p>The first step is to find candidate locations for each read (query) in the reference genome. Then, each query is aligned to its set of candidates.</p><p>Since DNA sequences can be very long, aligning them can take a lot of computation. The alignment step is the most computationally expensive step in the reference-guided genome assembly pipeline.</p><p>To help reduce this computation, filtering can be used to reduce the number of candidates for alignment. It also allows the researcher to focus on candidates that are most likely to add meaningful information.</p><p>One approach to filtering is to calculate the edit distance between each query and candidate pair to estimate their similarity. The edit distance algorithm calculates the minimum number of edits needed to make a query and candidate align exactly, with an edit being an insertion, deletion, or substitution (Figure 2). Candidates can be filtered out based on a predefined edit distance threshold.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/928/1*dJIFu3B1aGt8aKVFZIKjvQ.png" /><figcaption><a href="https://www.lexogen.com/rna-lexicon-indexing-strategies-and-solutions/">Source</a> — Figure 2. Example of the different edit events used to measure edit distance.</figcaption></figure><p>Researchers from Cornell accelerated the filtering step of DNA sequence alignment by using the Myers’ bit-parallel algorithm to measure the edit distance between two DNA sequences.</p><p><strong>The Myer’s Bit-Parallel Edit Distance Algorithm</strong></p><p>The Myers’ algorithm aligns DNA sequences by calculating edit distances through bitwise operations such as AND, OR, XOR, and NOT. Bitwise operations work on the individual bits of a binary value as seen in Figure 3 below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/597/1*YCCYsP1c759keZwfrvUEhg.png" /><figcaption><a href="https://www.omnicalculator.com/math/bitwise">Source</a>- Figure 3. Example of bitwise AND, OR, XOR operations for two numbers</figcaption></figure><p><strong>Accelerating The Myers’ Algorithm</strong></p><p>To accelerate the Myers’ bit-parallel edit distance, Cornell used <a href="https://gsitechnology.com/">GSI Technology’s</a> Gemini-I, which is a compute-in-memory (CIM) chip that enables bit processing.</p><p>The Gemini-I uses bit-slicing, where each data bit is processed independently. For example, for a 16-bit data item, each of the 16 bits can be processed individually.</p><p>The Gemini-I’s bit-slicing architecture stores the corresponding bits of multiple data elements together. For example, the first bits of a set of 16-bit numbers are stored together, all the second bits together, etc. This can be seen in Figure 4, where Bit Slice 0 holds all the 0 bits for each 16-bit data item, Bit Slice 1 holds all the 1 bits, etc.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*MGPBIaoIl198VRKKiQxNnA.png" /><figcaption><a href="https://arxiv.org/pdf/2401.11685.pdf">Source</a> — Figure 4. This figure shows the Gemini-I APU’s bit-slice architecture that stores the corresponding bits of multiple data elements together. For example, Bit Slice 0 holds the 0 bits for each data item in a set of items.</figcaption></figure><p>As mentioned previously, the Myers’ bit-parallel algorithm uses bitwise operations such as AND, OR, XOR, and NOT to calculate edit distances. Bit-slicing is well-suited for these operations because each bit is processed independently, and it can execute them in parallel across all bits of the data elements. This makes Gemini-I’s bit-slicing architecture ideal for the Myers’ algorithm.</p><p>The Cornell researchers compared the performance of the APU to an Intel Xeon Gold 6230R CPU @ 2.1GHz with DDR4 when running the Myers’ bit-parallel edit distance algorithm. In their paper, they only used 1 of the 4 cores available on a Gemini-I APU chip for their performance comparison, but they “expect a roughly linear multicore speedup on the APU (similar to what was seen with the CPU), leading to an estimated total speedup of 5–6x for a 4-core APU over a 16-core CPU.”</p><p><strong>Conclusion</strong></p><p>DNA sequence alignment plays a key role in computational genomics applications such as disease research and understanding evolutionary relationships. It requires a lot of computation, but filtering candidates can greatly reduce that computational load.</p><p>Researchers from Cornell used the Myers’ bit-parallel edit-distance algorithm to filter candidates. They found that GSI Technology’s Gemini-I could accelerate the Myers’ algorithm 5–6x when compared to traditional CPUs.</p><p>The Gemini-I is a compute-in-memory chip that uses bit-slicing, where each data bit is processed independently — ideal for accelerating the bitwise operations of the Myers’ algorithm.</p><p>Click <a href="https://arxiv.org/pdf/2401.11685.pdf">here</a> to access the full Cornell paper, where you will find the details, including microcode, of how the Gemini-I can accelerate the filtering step in DNA sequence alignment.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f7f3581cbed7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/gsi-technology/optimizing-genomics-research-reducing-the-high-computational-cost-of-dna-sequence-alignment-f7f3581cbed7">Optimizing Genomics Research: Reducing the High Computational Cost of DNA Sequence Alignment</a> was originally published in <a href="https://medium.com/gsi-technology">GSI Technology</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Making Large Language Models More Explainable]]></title>
            <link>https://medium.com/gsi-technology/making-large-language-models-more-explainable-b8215696a659?source=rss----12ba45b97995---4</link>
            <guid isPermaLink="false">https://medium.com/p/b8215696a659</guid>
            <category><![CDATA[vector-search]]></category>
            <category><![CDATA[explainable-ai]]></category>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Pat Lasserre]]></dc:creator>
            <pubDate>Mon, 27 Mar 2023 21:18:16 GMT</pubDate>
            <atom:updated>2023-04-12T17:58:10.477Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*sGAgNloNJZ87zV13LVaAOg.png" /><figcaption>Photo by <a href="https://unsplash.com/@emilymorter?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Emily Morter</a> on <a href="https://unsplash.com/photos/8xAA0f9yQnE?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>I recently watched a <a href="https://www.youtube.com/watch?v=-lnHHWRCDGk">presentation</a> given by Professor <a href="https://medium.com/u/5fd85bd5c0c4">Christopher Potts</a> of Stanford University, where he mentioned that if we’re going to trust large language models (LLMs), we need to better understand how they arrive at their answers.</p><p>This touches on one of the key issues with LLMs — their lack of explainability. They’re so complex that they’re basically black boxes, and we’re not sure how they came up with their answers. The more sophisticated the model, the less transparent it becomes. This is the opposite of what is needed — explainable AI (XAI) where the responses can be explained.</p><p>This becomes problematic in areas such as healthcare, finance, and law, where users need to have a clear understanding of how the model arrived at its decision.</p><p>Professor Potts compared two different approaches to using LLMs, one of which we’ll show can be used to make LLMs more explainable.</p><p><strong>How Large Language Models Work</strong></p><p>LLMs learn statistical co-occurrence patterns in sequences of tokens (words) to predict the probability of a sequence of words. They use those statistics to assign the probability of the next word in a sequence.</p><p>The longer the sequence grows, the greater the chance that it can go off the rails and generates text that appears plausible but is inaccurate.</p><p>ChatGPT acknowledged that it can’t be fully trusted to produce facts when I asked it how it generates a response: “My responses are generated based on statistical patterns in the data I was trained on and may not always reflect the nuances of real-life situations.”</p><p>Basically, it’s saying the information that it provides could be made-up.</p><p><strong>The LLM for Everything Approach</strong></p><p>Professor Potts discussed two approaches to using LLMs — one that he calls the “LLMs for everything” approach and the other that he calls “retrieval-augmented in-context learning.”</p><p>With LLMs for everything, there is no separate retriever to find relevant information — the LLM does everything. The model acts as both a knowledge store and a language model. As he states, it just becomes “a big opaque model.”</p><p>He says that if you have a “big opaque model” (a black box), that you really can’t trust it because you don’t know where the data came from that provided your answer. You now have a trustworthiness and explainability issue.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/463/1*TlLWGMrYSaN2F04LuKNYlg.png" /><figcaption><a href="https://docs.google.com/presentation/d/1WPYaLEEVJJI_-DOzjudeVoYpl_y0yUi1kWs0VFBnba4/edit#slide=id.g1c79e641885_1_813">Source </a>— LLM for everything approach where you have a black-box language model that isn’t explainable.</figcaption></figure><p>What he’s pointing out here is what I see as the crux of the issue — LLMs, like ChatGPT, aren’t knowledge models, they’re language models.</p><p>That’s how they should be used — as language models, and we should leverage their strengths, such as synthesizing information.</p><p>That’s where a retrieval-augmented approach comes in.</p><p><strong>Retrieval-Augmented NLP</strong></p><p>With the retrieval augmented approach, Professor Potts discusses how a retriever finds relevant information in a large corpus by using neural search to find documents most similar to a query. This relevant information is then fed to an LLM where it can synthesize the information.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/525/1*R7A1sJFzm7ULGKXnaEcPHg.png" /><figcaption><a href="https://docs.google.com/presentation/d/1WPYaLEEVJJI_-DOzjudeVoYpl_y0yUi1kWs0VFBnba4/edit#slide=id.g1c79e641885_1_813">Source</a> — Retrieval-augmented approach where a retriever finds documents similar to a query and passes them to an LLM to synthesize a response.</figcaption></figure><p>As Professor Potts states, you can take the relevant documents and “synthesize them into a single answer, maybe meeting the user’s information need directly.” One benefit of this is the user no longer needs to read through multiple different retrieved documents to find the answer. The LLM can synthesize the information from those relevant documents into a single answer.</p><p>Not only does this simplify the user’s life, but it also helps address the issue of explainability.</p><p>With a retriever, you know where the results come from. You know the source of the relevant information that was fed to the LLM to generate the response. This helps address the explainability and trustworthiness issues associated with LLMs.</p><p><strong>Vector Databases: A Path to Explainability</strong></p><p><a href="https://medium.com/gsi-technology/vector-databases-made-easy-51483c4e44d7">Vector databases</a> are a popular way to implement the retriever function. For example, an enterprise can index their own data in a vector database and use it to retrieve relevant information to feed an LLM. This could help a manager with a task like having to sift through various data sources such as emails, notes, and reports to find specific information.</p><p>The vector database could retrieve the information and feed it to an LLM. The manager could then prompt the LLM with tasks such as synthesizing the information from the various sources into a single response or summarizing the information from each of the sources.</p><p>A vector database can provide the document IDs for the documents that it retrieved — so you know the exact sources of information that the LLM is using.</p><p>This makes LLMs less of a black box by allowing for better explainability, transparency, and trustworthiness. As mentioned previously, these are critical items when considering using LLMs in areas such as healthcare, finance, and law, where users need to know how a model came up with its response.</p><p><strong>Getting Rid of the Black Box</strong></p><p>As Professor Potts stated, the “LLMs for everything” approach leads to a “big opaque model” which is a black box that really can’t be trusted or explained.</p><p>His proposed “retrieval-augmented in-context learning” approach addresses the explainability issue. It adds a retriever to feed an LLM relevant information. Since the source of this information is known, you now have a model that is more transparent and trustworthy.</p><p>So for your next NLP application, make it more explainable, trustworthy, and less of a black box by using a <a href="https://www.gsitechnology.com/sites/default/files/brochures/GSIT-Neural-Search-Simplified.pdf">vector database</a> as a retriever to feed your LLM relevant, and transparent data. Your users will thank you for it.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b8215696a659" width="1" height="1" alt=""><hr><p><a href="https://medium.com/gsi-technology/making-large-language-models-more-explainable-b8215696a659">Making Large Language Models More Explainable</a> was originally published in <a href="https://medium.com/gsi-technology">GSI Technology</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Unlocking the Benefits of SAR Back Projection]]></title>
            <link>https://medium.com/gsi-technology/unlocking-the-benefits-of-sar-back-projection-fe6be5afc606?source=rss----12ba45b97995---4</link>
            <guid isPermaLink="false">https://medium.com/p/fe6be5afc606</guid>
            <category><![CDATA[sensors]]></category>
            <category><![CDATA[disaster-response]]></category>
            <category><![CDATA[radar]]></category>
            <category><![CDATA[satellite-technology]]></category>
            <category><![CDATA[environmental-issues]]></category>
            <dc:creator><![CDATA[Pat Lasserre]]></dc:creator>
            <pubDate>Thu, 15 Dec 2022 23:33:10 GMT</pubDate>
            <atom:updated>2022-12-19T18:51:53.021Z</atom:updated>
            <content:encoded><![CDATA[<p>Synthetic Aperture Radar (SAR) has many important applications, for example: military surveillance; border security; and monitoring environmental issues like deforestation, flooding, oil spills, changes in arctic sea ice, and wildfire progression.</p><p>Back projection is considered the ideal SAR algorithm, but its adoption has been limited to date because of the computational expense of running it on CPU or GPU.</p><p>In this post, we’ll present <a href="https://www.gsitechnology.com/SAR">a solution</a> that addresses the computational expense of back projection — allowing it to be more widely adopted.</p><h4><strong>How SAR Works</strong></h4><p>SAR is a form of radar that uses microwaves to create high-resolution images.</p><p>SAR radars are mounted on moving airborne platforms, creating the effect of a large virtual antenna that provides high resolution. The radars emit microwaves towards a surface, and the intensity of the reflected waves is used to create the images.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*m9IYOBsGODQuDRZt6pQclg.png" /><figcaption>SAR satellite image example</figcaption></figure><h4><strong>SAR Benefits</strong></h4><p>Two key advantages that SAR sensors have over optical sensors is that they can work both day and night and in all kinds of weather (with cloud penetration being particularly advantageous).</p><p>To create an image, optical sensors (passive) measure solar light reflected from objects. That means that they will have issues with darkness, adverse weather conditions, clouds, smoke, etc. As noted in <a href="https://arxiv.org/pdf/2207.13184.pdf">this paper</a>, “such limitations produce an information gap during or post a disaster event delaying the planning and the course of action needed to save lives.”</p><p>As noted in <a href="https://medium.com/gsi-technology/getting-started-with-satellite-data-processing-7653dbd4fc1e">this post</a>, SAR sensors don’t suffer from these issues because the lengths of the microwaves used by SAR “are much longer than those of visible or infrared light, making the sensors functional in all lighting conditions and able to penetrate through clouds and dust.”</p><p>SAR can also work on many moving platforms with diverse imaging areas, resolution ranges, and imaging ranges. For example, it can work on imaging ranges from 1000km (satellites) to 100m (quadcopters). Resolution, which is the area covered by an individual image pixel, can range from 100 meters per pixel for huge areas like oceans or deserts to sub meter per pixel for high-accuracy applications covering small and medium fixed areas.</p><h4><strong>SAR Spotlight-mode for High Resolution</strong></h4><p>For SAR applications that require high resolution, SAR spotlight-mode is often used. It works by transmitting successive waves at a fixed target over a limited area as the antenna moves past the target.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*biOfcY8cMp9YMdX_aK989A.png" /><figcaption>SAR Spotlight-mode transmits waves over a fixed limited area to capture high-resolution images</figcaption></figure><h4><strong>Back Projection — the Ideal SAR Processing Algorithm</strong></h4><p>Back projection is the preferred SAR processing algorithm because of its many benefits:</p><p>· Supports a wide range of frequencies, including low frequencies, which provide better penetration into the target</p><p>· Supports low imaging range which is good for platforms like quadcopters and unmanned aerial vehicle (UAV)</p><p>· Compensates for platform motion with high maneuverability — good for quadcopters and UAV</p><p>Its adoption, however, has been limited by the computational expense of running it on traditional solutions based on CPU.</p><h4><strong>Parallelism — Addressing Back Projection’s Computational Expense</strong></h4><p>Creating a SAR image with the back projection algorithm takes a lot of computation. It can take thousands of pulses to create an image, and solutions that use CPU work sequentially, pixel-by-pixel in the grid of each pulse.</p><p>Taking that sequential approach on thousands of pulses carries a high computational expense that leads to prohibitive costs and power usage.</p><p>A better approach is to work on all the pixels in the grid for each pulse in <strong>parallel. </strong>That is what <a href="https://www.gsitechnology.com/sites/default/files/AppNotes/GSIT-SAR-AppBrief.pdf">GSI Technology’s APU</a> does.</p><p>By enabling parallel processing, the APU allows back-projection-based SAR images to be produced cost-effectively and power-efficiently. It also reduces the processing time from a few minutes to a few seconds.</p><p>For the case of a large area SAR image being processed in one second at high resolution, the APU uses, on average, <a href="https://ir.gsitechnology.com/news-releases/news-release-details/gsi-technologys-nrt-sar-image-processing-acceleration-system">88% less power</a> than CPU or GPU systems.</p><h4><strong>Non-Linear Arithmetic Support</strong></h4><p>Another benefit that the APU brings to back projection is its ability to efficiently handle non-linear functions such as COS, SIN, SQRT, and table lookup, which are key functions of back projection. CPUs and GPUs do not efficiently handle non-linear operations.</p><p>For your next SAR project, look for <a href="https://www.gsitechnology.com/SAR">a solution</a> that provides parallel processing and non-linear arithmetic support — so you can take advantage of back projection, the ideal SAR image-formation algorithm.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fe6be5afc606" width="1" height="1" alt=""><hr><p><a href="https://medium.com/gsi-technology/unlocking-the-benefits-of-sar-back-projection-fe6be5afc606">Unlocking the Benefits of SAR Back Projection</a> was originally published in <a href="https://medium.com/gsi-technology">GSI Technology</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Natural Language Processing … for Image Search?]]></title>
            <link>https://medium.com/gsi-technology/natural-language-processing-for-image-search-41fadd74221b?source=rss----12ba45b97995---4</link>
            <guid isPermaLink="false">https://medium.com/p/41fadd74221b</guid>
            <category><![CDATA[elasticsearch]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[opensearch]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[neural-search]]></category>
            <dc:creator><![CDATA[Pat Lasserre]]></dc:creator>
            <pubDate>Fri, 10 Jun 2022 16:37:52 GMT</pubDate>
            <atom:updated>2023-03-23T21:14:54.066Z</atom:updated>
            <content:encoded><![CDATA[<h3>Natural Language Processing … for Image Search?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*spcFhswtD9BMSFeMFCdByw.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@nate_dumlao?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Nathan Dumlao</a> on <a href="https://unsplash.com/s/photos/unsure?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>Natural language processing (NLP) is a major part of search — so much so that it is even being used in image search applications.</p><p>For example, <a href="https://blog.google/products/search/introducing-mum/">Google said</a>, when talking about its MUM model, “Eventually, you might be able to take a photo of your hiking boots and ask, ‘can I use these to hike Mt. Fuji?’ MUM would understand the image and connect it with your question to let you know your boots would work just fine. It could then point you to a blog with a list of recommended gear.” This makes MUM multimodal because it understands both text and images.</p><p>In this post, I’ll show how <a href="https://medium.com/gsi-technology/leveraging-word2vec-for-more-than-text-a5659d680207">vector embeddings</a> outperform keyword search for multimodal text-to-image search. I’ll also discuss a solution that allows you to leverage your existing OpenSearch installation to quickly and easily create a text-to-image search application.</p><h4>The Limitations of Keyword Search</h4><p>Previously, when using text to search for relevant images, one would perform keyword search using the image captions to compare against the text query. This meant the image itself wasn’t even being used in the search.</p><p>One problem with this is there could be relevant images that don’t have captions. This could result in the images not being returned as candidates, even though they are relevant.</p><p>Another problem with keyword search is it could omit images with captions that don’t share many keywords with the query but are in fact relevant images. This could impact business in e-commerce applications because sellers often don’t enter the most descriptive text, so even if their item is exactly what the buyer is looking for, it might not be returned as a candidate.</p><p>Also, as shown in <a href="https://blog.muves.io/multilingual-and-multimodal-vector-search-with-hardware-acceleration-2091a825de78">this post</a>, keyword search has limited understanding of user intent and could return irrelevant images even if there are “multiple matching terms between the query and the result.” As shown below, it incorrectly returned an image where the caption matched the keywords <strong><em>eating fish,</em></strong> but it missed the main search term <strong><em>bear</em></strong>.</p><p><strong>Query:</strong> <em>A bear </em><strong><em>eating fish</em></strong><em> by a river</em></p><p><strong>Result:</strong> <em>heron </em><strong><em>eating fish</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/0*1ngwrgdiyKY5u7YC" /><figcaption>An irrelevant search result returned using keyword search for the query “<em>A bear eating fish by a river.” </em><a href="https://blog.muves.io/multilingual-and-multimodal-vector-search-with-hardware-acceleration-2091a825de78"><em>Source</em></a></figcaption></figure><h4>Vector Embeddings — The Key to Text-to-Image Search</h4><p>To address the previously mentioned keyword search limitations, we can use a multilingual <a href="https://huggingface.co/sentence-transformers/clip-ViT-B-32-multilingual-v1">CLIP model</a> to generate vector embeddings. <a href="https://openai.com/blog/clip/">CLIP </a>was created by OpenAI, and they state that it “efficiently learns visual concepts from natural language supervision.” Basically, CLIP maps text and images to the same embedding space where they can be compared for similarity.</p><p>As we discussed in a previous <a href="https://medium.com/gsi-technology/scalable-semantic-vector-search-with-elasticsearch-e79f9145ba8e">post</a>, vector embeddings better understand the searcher’s intent and the contextual meaning of the query. Instead of simply matching the keywords, it takes into consideration what the words mean and not just the words themselves.</p><p>An example of that can be seen in the image below. In this case, vector embeddings were used instead of keywords. The same query about a bear eating a fish was used, but unlike the keyword approach that returned an irrelevant image, vector embeddings returned a relevant image.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/897/1*nqtWp39FbJzaV9d5EXj0Mg.png" /><figcaption><em>A relevant search result using vector embeddings for the query “nehir kenarında balık yiyen ayı” (a bear eating fish by a river — Turkish)</em></figcaption></figure><p>Not only did vector embedding return a relevant image, but the vector embeddings approach also showed that it understands multiple languages, in this example, Turkish.</p><p>Vector embeddings can also improve recall. Recall is important because it can impact a company’s business. For example, in e-commerce, sellers often either don’t enter very descriptive text, don’t use the right keywords, or they might enter incorrect text descriptions. In these cases, keyword search could prevent a product from being returned as a match, even if it actually is. This means a missed business opportunity for the seller.</p><p>Vector embeddings address this recall issue because even though the text descriptions were poor in those examples above, if there were relevant images that went with them, the vector embeddings of the images would allow those images to be returned as matches. Thus, the seller is no longer penalized for entering poor product descriptions, or even no descriptions.</p><h4>Easily Add Vector Embedding Search to Your OpenSearch</h4><p>As we wrote about in <a href="https://medium.com/gsi-technology/vector-databases-made-easy-51483c4e44d7">this post</a>, GSI Technology’s <a href="https://www.gsitechnology.com/sites/default/files/brochures/GSIT-Neural-Search-Simplified.pdf">OpenSearch k-NN plugin</a> allows users to easily add production-grade vector embedding search to their search pipeline. They can leverage their current OpenSearch installation rather than having to learn new software for one of the other vector search options out there. This saves them valuable time and resources.</p><p><a href="https://medium.com/u/95a0ba753977">Dmitry Kan</a> and <a href="https://medium.com/u/68d57f395c31">Aarne Talman</a> recently published a great <a href="https://blog.muves.io/multilingual-and-multimodal-vector-search-with-hardware-acceleration-2091a825de78">blog post</a> where they explained how they used our OpenSearch k-NN plugin as part of their search stack to easily create a text-to-image search application.</p><p>In addition to saving developers valuable time and resources, our OpenSearch k-NN plugin allows for <a href="https://medium.com/gsi-technology/high-performance-billion-scale-similarity-search-686b2c354ee6">billion-scale</a> neural search and addresses one of the key limitations of native OpenSearch — namely it’s lack of pre-filter support for nearest neighbor vector search.</p><p>Pre-filtering on metadata is used in many search applications. For example, product metadata such as item description, item title, category, color, or brand are often used as pre-filters to a search query.</p><p><a href="https://opensearch.org/docs/latest/search-plugins/knn/approximate-knn/">The OpenSearch website states</a>: “Because the native library indices are constructed during indexing, it is not possible to apply a filter on an index and then use this search method. All filters are applied on the results produced by the approximate nearest neighbor search.” This means that native OpenSearch only supports post-filtering of the approximate nearest neighbor results and doesn’t support pre-filtering</p><p>As mentioned in one of our <a href="https://medium.com/gsi-technology/vector-databases-made-easy-51483c4e44d7">previous posts</a>, post-filtering is problematic because it has a high likelihood of returning far fewer results than the intended k-nearest neighbors. In fact, it could lead to zero results being returned. This leads to an unsatisfying user experience since very few, or no, relevant results might be returned for a particular search query.</p><p><a href="https://www.gsitechnology.com/sites/default/files/brochures/GSIT-Neural-Search-Simplified.pdf">GSI’s OpenSearch plugin</a> supports pre-filtering, and even supports range filtering. For example, if somebody was searching for shirts, in addition to using common filters such as brand, style, size, and color, they could also add a range filter, for example, to limit the search to shirts in the range between $55 and $85.</p><h4>Summary</h4><p>This post showed some of the advantages of vector embedding search over keyword search — for example, better understanding user intent and improving recall in e-commerce applications where sellers either don’t enter very descriptive text, don’t use the right keywords, or enter incorrect text descriptions. Ultimately, these vector embedding advantages lead to improved business for sellers.</p><p>We also presented our OpenSearch k-NN plugin that allows users to easily add production-grade vector embedding search to their search pipeline — saving them valuable time and resources. The plugin also provides billion-scale search along with strong filtering capability.</p><p>If you want to try out our <a href="https://www.gsitechnology.com/sites/default/files/brochures/GSIT-Neural-Search-Simplified.pdf">OpenSearch k-NN plugin</a>, please contact us at opensearch@gsitechnology.com.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=41fadd74221b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/gsi-technology/natural-language-processing-for-image-search-41fadd74221b">Natural Language Processing … for Image Search?</a> was originally published in <a href="https://medium.com/gsi-technology">GSI Technology</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How Neural Search is Being Used in Production]]></title>
            <link>https://medium.com/gsi-technology/how-neural-search-is-being-used-in-production-340b1601a51a?source=rss----12ba45b97995---4</link>
            <guid isPermaLink="false">https://medium.com/p/340b1601a51a</guid>
            <category><![CDATA[open-distro]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[elasticsearch]]></category>
            <category><![CDATA[vector-search]]></category>
            <dc:creator><![CDATA[Pat Lasserre]]></dc:creator>
            <pubDate>Mon, 11 Apr 2022 22:29:11 GMT</pubDate>
            <atom:updated>2023-03-23T21:11:28.645Z</atom:updated>
            <content:encoded><![CDATA[<p>Neural search (also commonly referred to as vector search or semantic search) is hot — it seems like almost every day I see a new article written about it. For example, I recently saw these articles from <a href="https://cloud.google.com/blog/topics/retail/retail-search-product-discovery-increased-conversion-shopping-ecommerce-global-launch">Google</a>, <a href="https://www.datanami.com/2022/03/15/home-depot-finds-diy-success-with-vector-search/">Home Depot</a>, and <a href="https://www.insideradio.com/podcastnewsdaily/spotify-update-suggests-one-ingredient-to-podcast-engagement-starts-with-search-results/article_9c749ff8-a9fa-11ec-ba1c-938d86814b9d.html">Spotify</a>.</p><p>That’s why I shouldn’t be too surprised by the number of people who have reached out to us to see if our <a href="https://www.gsitechnology.com/sites/default/files/brochures/GSIT-Neural-Search-Simplified.pdf">Elasticsearch and OpenSearch</a> k-NN plugins can accelerate their neural search applications.</p><p>In this post, I’ll briefly present a few of those applications, many of which need to search through large amounts of unstructured data — ranging from hundreds of millions to billions of items.</p><h3>The Move to Neural Search</h3><p>Traditional search engines use an inverted index along with something like TF-IDF or BM25 for ranking. They use sparse vectors, based on literal keywords, to find similar/relevant information.</p><p>As we wrote about in a <a href="https://medium.com/gsi-technology/scalable-semantic-vector-search-with-elasticsearch-e79f9145ba8e">previous blog post</a>, these traditional approaches are limited in their understanding of user intent. For example, they could omit documents that don’t share many keywords with the query but are relevant documents.</p><p>Search is moving beyond this simple keyword-based search, to neural search, where user intent is better captured.</p><p><a href="https://medium.com/gsi-technology/scalable-semantic-vector-search-with-elasticsearch-e79f9145ba8e">Neural search</a> uses neural networks to better understand user intent and go beyond traditional exact-match keyword search.</p><h3>Neural Search Applications</h3><p>Here are just a few of the neural search applications that people have talked to us about:</p><h4><strong>Prior Art Search</strong></h4><p>Traditionally, IP experts have spent a lot of time analyzing documents to come up with relevant keywords for a prior art patent search. Some of the keywords can be very subtle and can be missed if done manually.</p><p>Also, with datasets in this space growing rapidly, oftentimes in the billions of documents, relying solely on human analysis is not realistic or scalable.</p><p>That’s why IP experts are turning to neural search to efficiently sift through those billions of documents. Neural search provides the ability to quickly search through patent data by focusing on context, and not just keywords, to provide relevant references.</p><p>Figure 1 below is an example from a paper that shows how neural search can be used in prior art discovery.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*yBuQbiHMnju3qFYm1B-88w.png" /><figcaption>Figure 1. Neural search in prior art discovery. A language model is used to compare embeddings of a query to embeddings from patent documents to determine if a patent is relevant. (source: <a href="https://www.semanticscholar.org/paper/Patent-prior-art-search-using-deep-learning-model-Kang-Lee/b35b6bad97055e45c587a580b98974a05bdacf17"><strong>Patent prior art search using deep learning language model</strong></a><strong>)</strong></figcaption></figure><h4><strong>Recruitment and Talent Acquisition</strong></h4><p>In this application, candidates and job descriptions are mapped to a common embedding space to find potential fits. This can be used to recommend and rank candidates for a particular position, find similar candidates, or to find similar positions.</p><p>This application typically has large data requirements, on the order of hundreds of millions of documents.</p><h4><strong>Semantic Video Transcription</strong></h4><p>Have you ever experienced the frustration of watching a video and later wanted to find a specific topic in the video, but you couldn’t remember exactly where it was in the video? If so, then video transcription with semantic search can save you from having to sift through the whole video to find it.</p><p>Basically, what this application does is break the transcribed video down into paragraphs that are semantically indexed. A neural search between the query and this semantically indexed content then helps you find what you’re looking for.</p><h4><strong>Targeted Advertising</strong></h4><p>This is where companies like Google and Facebook analyze online behavior to categorize individuals. They then use this information to match groups of finely categorized people to individualized advertisements — creating a targeted advertisement. This is done by creating embeddings for target customers and advertisements and mapping them to a common embedding space such that a similarity search can be performed to find the right match.</p><h3><strong>Your Applications</strong></h3><p>If you’re using neural search in your application, I’d love for you to share a bit about it in the comments section.</p><p>Also, if you would like to see if our <a href="https://www.gsitechnology.com/sites/default/files/brochures/GSIT-Neural-Search-Simplified.pdf">Elasticsearch or OpenSearch plugins</a> can help accelerate your neural search application, please contact us at <a href="/service/mailto:elasticsearch@gsitechnology.com">elasticsearch@gsitechnology.com</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=340b1601a51a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/gsi-technology/how-neural-search-is-being-used-in-production-340b1601a51a">How Neural Search is Being Used in Production</a> was originally published in <a href="https://medium.com/gsi-technology">GSI Technology</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Semantic Vector Search — Taking the Leap from Keyword Search]]></title>
            <link>https://medium.com/gsi-technology/semantic-vector-search-taking-the-leap-from-keyword-search-18855e96854a?source=rss----12ba45b97995---4</link>
            <guid isPermaLink="false">https://medium.com/p/18855e96854a</guid>
            <category><![CDATA[elasticsearch]]></category>
            <category><![CDATA[opensearch]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[semantic-search]]></category>
            <category><![CDATA[nearest-neighbor-search]]></category>
            <dc:creator><![CDATA[Pat Lasserre]]></dc:creator>
            <pubDate>Wed, 24 Nov 2021 17:57:37 GMT</pubDate>
            <atom:updated>2023-03-23T20:50:26.297Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/323/1*WxlMYsvA3HHnwnT3CSvnUQ.png" /><figcaption>Photo by <a href="https://unsplash.com/@benjaminwedemeyer?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Benjamin Wedemeyer</a> on <a href="https://unsplash.com/s/photos/jumping-in-water?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><h3>Neural Search — Taking the Leap from Keyword Search</h3><p><a href="https://medium.com/gsi-technology/scalable-semantic-vector-search-with-elasticsearch-e79f9145ba8e">Semantic vector search</a> is a topic that’s getting a lot of attention, and many search developers are wondering if, and how, they should add it to their search solution.</p><p>In a recent Haystack LIVE! Meetup titled <a href="https://www.meetup.com/Haystack-Search-Relevance-Online/events/281374982/">Evolving from Keyword to Neural Search</a>, <a href="https://medium.com/u/51b74d7c71f4">Branden Chan</a> of Deepset presented their <a href="https://www.deepset.ai/haystack">Haystack NLP framework</a> and was asked what approach he would recommend for adding semantic vector search to a platform based on bag-of-words (keyword) search.</p><p>In this post, I will review Branden’s recommendation and propose a way to implement his proposal.</p><h4>Traditional Search</h4><p>Traditional search engines, like Elasticsearch, rely heavily on a bag-of-words (keyword) approach that uses an inverted index along with TF-IDF or BM25 ranking functions. A sparse vector, based on literal keyword matching, is used to find similar/relevant information.</p><p>As we wrote about in a <a href="https://medium.com/gsi-technology/scalable-semantic-vector-search-with-elasticsearch-e79f9145ba8e">previous blog post</a>, this keyword approach has some limitations in terms of understanding user intent. For example, it could omit documents that don’t share many keywords with the query but are relevant documents.</p><h4>Combining Keyword and Semantic Search</h4><p>Recognizing these limitations, many search developers have been wondering if they should add dense vector semantic search to their platform to better capture user intent. In fact, one of the attendees of Branden Chan’s recent talk titled <a href="https://www.meetup.com/Haystack-Search-Relevance-Online/events/281374982/"><em>Evolving from Keyword to Neural Search</em></a> asked Braden for his thoughts about that.</p><p>The attendee mentioned that they have a content server, and they are currently using a bag-of-words (BoW) approach for search. They said that sometimes their users know exactly what they’re searching for, but other times their search is exploratory and, in that case, if you’re limited to a BoW approach, it’s hard to understand exactly what the user wants.</p><p>He said that he is interested in seeing if dense vectors could help him better understand user intent. He mentioned that he couldn’t switch to a dense vector solution “overnight,” due to implementation reasons and concerns of how it could change performance over time. He was interested in Branden’s advice on how to incrementally take advantage of dense vector search to make search better.</p><p>Branden said that you could use the existing sparse (keyword) score and combine it with a dense (semantic) vector score. He suggested passing the query to both a dense and sparse path, with each path having a score that you could weight.</p><p>Branden said you could take a cautious path of lightly weighting the dense score initially and then incrementally bumping it up and experimenting to see how to optimally weight it.</p><h4>How to Add Semantic Search Incrementally</h4><p>One way a user could implement Branden’s incremental approach is to use one of our <a href="https://www.gsitechnology.com/sites/default/files/AppNotes/GSIT-Elasticsearch-Plugin-AppBrief.pdf">Elasticsearch</a> or <a href="https://www.gsitechnology.com/sites/default/files/brochures/GSIT-Neural-Search-Simplified.pdf">OpenSearch </a>k-NN plugins along with the user’s current Elasticsearch or OpenSearch installations.</p><p>That would allow the user to leverage the powerful keyword search (sparse vector) capabilities of Elasticsearch and OpenSearch and combine it with semantic vector search (dense vector) of one of GSI’s k-NN plugins.</p><p>Installing the plugins is easy, and they allow for vector similarity search to be run as simply as any standard Elasticsearch query. The Elasticsearch k-NN plugin provides similarity search results in the standard Elasticsearch format, so a user could follow Branden’s advice of combining the sparse and dense vector scores. The user could lightly weight the dense score (k-NN result) initially and then incrementally bump it up and experiment to see how to optimally weight it.</p><p>For more information about GSI’s k-NN plugins, contact us at <a href="/service/mailto:elasticsearch@gsitechnology.com">elasticsearch@gsitechnology.com</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=18855e96854a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/gsi-technology/semantic-vector-search-taking-the-leap-from-keyword-search-18855e96854a">Semantic Vector Search — Taking the Leap from Keyword Search</a> was originally published in <a href="https://medium.com/gsi-technology">GSI Technology</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Vector Databases Made Easy]]></title>
            <link>https://medium.com/gsi-technology/vector-databases-made-easy-51483c4e44d7?source=rss----12ba45b97995---4</link>
            <guid isPermaLink="false">https://medium.com/p/51483c4e44d7</guid>
            <category><![CDATA[elasticsearch]]></category>
            <category><![CDATA[nearest-neighbor-search]]></category>
            <category><![CDATA[similarity-search]]></category>
            <category><![CDATA[vector-search-engine]]></category>
            <category><![CDATA[opensearch]]></category>
            <dc:creator><![CDATA[Pat Lasserre]]></dc:creator>
            <pubDate>Mon, 11 Oct 2021 17:16:08 GMT</pubDate>
            <atom:updated>2023-03-23T20:58:01.415Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*EZpNrySLIQldiw7J" /><figcaption>Photo by <a href="https://unsplash.com/@pablogamedev">Pablo Arroyo</a> on <a href="https://unsplash.com/">Unsplash</a></figcaption></figure><p>I was talking with <a href="https://medium.com/u/95a0ba753977">Dmitry Kan</a>, Principal AI Scientist at Silo AI, and he mentioned that one of the big benefits of our <a href="https://www.gsitechnology.com/sites/default/files/brochures/GSIT-Neural-Search-Simplified.pdf">Elasticsearch and OpenSearch</a> k-NN plugins is that they allow users to easily add a production-grade vector similarity search database to their search pipeline.</p><h4>Leveraging Your Current Elasticsearch and OpenSearch Installation</h4><p>Dmitry said that a lot of search teams already use Elasticsearch and AWS, so it would be easier for them to just use our <a href="https://www.gsitechnology.com/sites/default/files/brochures/GSIT-Neural-Search-Simplified.pdf">Elasticsearch and OpenSearch</a> k-NN plugins with their current Elasticsearch and OpenSearch installations rather than having to learn new software for one of the available vector databases out there. Additionally, as we mentioned in a <a href="https://medium.com/gsi-technology/scalable-semantic-vector-search-with-elasticsearch-e79f9145ba8e">previous post</a>, users said that it would be simpler for them to use use an approximate nearest neighbor (ANN) solution directly in Elasticsearch rather than trying trying to integrate and productionize one of the popular open-source vector search libraries into their search platform.</p><p>Dmitry mentioned that while there are a few <a href="https://dmitry-kan.medium.com/milvus-pinecone-vespa-weaviate-vald-gsi-what-unites-these-buzz-words-and-what-makes-each-9c65a3bd0696">vector database options</a>, such as Milvus, Pinecone, Vespa, and Weaviate, those options require a user to learn a new software platform. Many search teams are thin on resources, so having to learn new software in order to add vector similarity search to their application might not be their top choice.</p><p>While popular open-source vector search libraries, like NMSLIB or Faiss, provide good results for nearest neighbor benchmarks, they aren’t easy to productionize. They are just libraries, not full vector similarity search systems, so a user would still need to build a distributed system around them that scales. They would also have to manage the indexing of those distributed systems, which isn’t easy.</p><p>Also, building the graphs and indexes for graph-based libraries like NMSLIB is slow and memory-intensive. It is computationally expensive and takes a lot of memory to store each node’s neighbors. This limits the scalability of those libraries.</p><p>On the other end of this ease-of-use spectrum are our <a href="https://www.gsitechnology.com/sites/default/files/brochures/GSIT-Neural-Search-Simplified.pdf">Elasticsearch and OpenSearch</a> k-NN plugins. Installing the plugins is easy, and they allow for vector similarity search to be run as simply as any standard Elasticsearch query. The Elasticsearch plugin provides similarity search results in the standard Elasticsearch format, so a user doesn’t have to learn new software. They simply need to install the plugins, which use the existing query interfaces. This allows for a simple-to-use, ready-for-production vector similarity search solution that helps preserve end users’ engineering resources.</p><p>The <a href="https://www.gsitechnology.com/sites/default/files/brochures/GSIT-Neural-Search-Simplified.pdf">GSI Elasticsearch k-NN plugin</a> also addresses the previously mentioned scaling and indexing issues seen with many of the open-source vector search libraries because it leverages two of Elasticsearch’s key strengths — horizontal scaling and index management. Elasticsearch is a distributed system that easily scales horizontally, and since the plugin uses the core Elasticsearch dense_vector field type and index mapping, there is no need to reindex documents.</p><h4>Extending Elasticsearch and OpenSearch</h4><p>The k-NN plugins also extend the functionality of Elasticsearch and OpenSearch by addressing some of their limitations.</p><p>As was noted in a <a href="https://medium.com/gsi-technology/scalable-semantic-vector-search-with-elasticsearch-e79f9145ba8e">previous post</a>, Elasticsearch doesn’t support approximate nearest neighbor search (ANN) — it does an exhaustive “match_all” search using cosine similarity to measure similarity against all items in the list. Cosine similarity is relatively computationally expensive, so that’s why they first run a restrictive query to limit the number of vectors they measure similarity against. This restricts the Elastic nearest neighbor search to very small datasets.</p><p>The GSI Elasticsearch k-NN plugin fills this gap in the Elastic offering by providing an approximate nearest neighbor solution that scales to billions of vectors. This opens up nearest neighbor search using Elasticsearch to many more applications and use cases since it is not limited to small databases like the base Elastic offering.</p><p>The GSI <a href="https://www.gsitechnology.com/sites/default/files/brochures/GSIT-Neural-Search-Simplified.pdf">OpenSearch </a>k-NN plugin also addresses one of the key limitations of OpenSearch — namely it’s lack of pre-filter support for nearest neighbor vector search. Pre-filtering on metadata is a common requirement in search applications. For example, product metadata such as item description, item title, category, and brand, are often used as pre-filters to a search query.</p><p>The developers of OpenSearch (previously Open Distro for Elasticsearch) <a href="https://opendistro.github.io/for-elasticsearch-docs/docs/knn/approximate-knn/">state</a>: “Because the graphs are constructed during indexing, it is not possible to apply a filter on an index and then use this search method. All filters will be applied on the results produced by the approximate nearest neighbor search.” This means that they only support post-filtering of the approximate nearest neighbor results.</p><p>Post-filtering, however, is problematic because it has a high likelihood of returning far fewer results than the intended k-nearest neighbors. For example, if a user wanted to return the 10 nearest neighbors (k=10) to an image in a visual search application, after applying a filter, such as price or color, perhaps only 2 of those 10 nearest neighbors remain. In fact, it could lead to zero results being returned. This leads to an unsatisfying user experience since very few, or no, relevant results are returned for a particular search query. To try to increase the probability of returning relevant results, you could increase the number of nearest neighbors returned (k), but this comes with a tradeoff of increased latency. Also, depending on how restrictive the filter is, this still might lead to few, or no, relevant results being surfaced.</p><p>GSI’s <a href="https://www.gsitechnology.com/sites/default/files/brochures/GSIT-Neural-Search-Simplified.pdf">OpenSearch k-NN plugin</a> doesn’t have this restriction — it allows for pre-filtering of documents prior to running an approximate k-NN vector similarity search.</p><h4>Conclusion</h4><p>Today, vector similarity search users have many options, including open-source vector search libraries, vector database platforms, and search and analytics platforms like Elasticsearch and OpenSearch.</p><p>The open-source vector search libraries require a user to handle the complex tasks needed to productionize them— for example, building a distributed system that scales horizontally and managing the indexing of those distributed systems.</p><p>The vector database platforms require a user to learn a new software platform, which isn’t an attractive proposition for a resource-constrained team.</p><p>Elasticsearch and OpenSearch have many users that have already invested the time and resources to build a robust, production-grade search application around those platforms. GSI’s <a href="https://www.gsitechnology.com/sites/default/files/brochures/GSIT-Neural-Search-Simplified.pdf">Elasticsearch and OpenSearch</a> k-NN plugins allow those users to leverage that investment by providing them to a way to easily add a production-grade vector similarity search database to those platforms while helping them to preserve their engineering resources.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=51483c4e44d7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/gsi-technology/vector-databases-made-easy-51483c4e44d7">Vector Databases Made Easy</a> was originally published in <a href="https://medium.com/gsi-technology">GSI Technology</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Elasticsearch Nearest Neighbor Search Options]]></title>
            <link>https://medium.com/gsi-technology/a-comparison-of-elasticsearch-k-nn-solutions-be8d668812e?source=rss----12ba45b97995---4</link>
            <guid isPermaLink="false">https://medium.com/p/be8d668812e</guid>
            <category><![CDATA[semantic-search]]></category>
            <category><![CDATA[elasticsearch]]></category>
            <category><![CDATA[vector-search]]></category>
            <category><![CDATA[nearest-neighbor-search]]></category>
            <category><![CDATA[vector-search-engine]]></category>
            <dc:creator><![CDATA[Pat Lasserre]]></dc:creator>
            <pubDate>Tue, 22 Jun 2021 17:39:50 GMT</pubDate>
            <atom:updated>2021-09-23T18:51:42.146Z</atom:updated>
            <content:encoded><![CDATA[<p><a href="https://medium.com/u/95a0ba753977">Dmitry Kan</a>, Principal AI Scientist at Silo AI, <a href="https://towardsdatascience.com/speeding-up-bert-search-in-elasticsearch-750f1f34f455">recently wrote a blog post</a> where he compared four different approaches to BERT vector search with Elasticsearch using k-NN.</p><p>The four approaches that he compared were: native Elasticsearch, Elastiknn (an external plugin), Open Distro (now OpenSearch) for Elasticsearch k-NN plugin, and <a href="https://www.gsitechnology.com/sites/default/files/AppNotes/GSIT-Elasticsearch-Plugin-AppBrief.pdf">GSI’s Elasticsearch k-NN plugin</a>.</p><p>As seen in Figure 1 below, GSI’s solution proved to have the best performance coming in with a latency of 92.6ms, well below the latencies for the other solutions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/602/1*AuSdmZXZrGQPnNhwlsZ89A.png" /><figcaption>Figure 1. — Latency vs. number of documents searched</figcaption></figure><p>Latency, however, is only part of the story. In addition to having a latency advantage, GSI’s plugin also provides advantages in other areas such as offering multimodal searching with filters, it is compatible with the Elasticsearch dense_vector field, which means that previously indexed Elasticsearch dense_vectors do not need to be reindexed, and it allows for batch search, which provides higher throughput. This blog post will go into a little more detail about those other advantages.</p><h4><strong>Multimodal Search</strong></h4><p>Multimodal search, where for example images and text are combined to form a powerful search, is a rapidly emerging trend. Retail segments, such as fashion and home design, are one particular driver of multimodal search because they rely heavily on visual search since style is often difficult to describe using text.</p><p>In addition to visual search, text search is also a required part of the solution because product information, such as item description, item title, category, and brand are generally used as pre-filters to a visual search. Thus, what is needed is a solution that allows for multimodal search using filters — most competing solutions, however, are not well-suited to handling this.</p><p>For example, <a href="https://discuss.opendistrocommunity.dev/t/approximate-k-nn-with-pre-filter/4494">one user of Open Distro for Elasticsearch asks</a>, “Is it possible to use a pre-filter (on a keyword field for example) and then do approximate k-NN on the remaining documents so that I always get back exactly k documents?”</p><p>The response from the Open Distro team was, “Unfortunately it is not possible to do approximate-knn on filtered documents.”</p><p><a href="https://opendistro.github.io/for-elasticsearch-docs/docs/knn/approximate-knn/">The reason for this restriction on filtering</a> is explained by this excerpt from the Open Distro for Elasticsearch documentation:</p><p>“Because the graphs are constructed during indexing, it is not possible to apply a filter on an index and then use this search method. All filters will be applied on the results produced by the approximate nearest neighbor search.”</p><p>GSI’s Elasticsearch k-NN plugin doesn’t have these restrictions — it allows for pre-filtering of documents prior to an approximate k-NN. This means that the GSI plugin allows for multimodal search using filters.</p><h4><strong>No Need to Reindex Documents</strong></h4><p>Another advantage of the GSI plugin is that it is compatible with the Elasticsearch dense_vector field. The plugin uses Elasticsearch’s dense_vector field type and index mapping, which means no reindexing of documents is needed to use the plugin.</p><p>The same can’t be said for most competing solutions.</p><p>For example, as can be seen from this section of the <a href="https://opendistro.github.io/for-elasticsearch/blog/odfe-updates/2020/04/Building-k-Nearest-Neighbor-(k-NN)-Similarity-Search-Engine-with-Elasticsearch/">Open Distro for Elasticsearch GitHub page</a>:</p><p>“First, we added a new field type, knn_vector, using the Mapper plugin, to represent the vectors as a an array of floating point numbers in a document … We also introduced a new Apache Lucene codec, KNNCodec, to add a new index file format for storing and retrieving the vectors and make Apache Lucene aware of the graphs built by NMSLIB.”</p><p>Adding these custom field type(s) and mappings means that a user who already has Elasticsearch indices using the dense_vector field type needs to change their index mapping and reindex their documents. This reindexing can be costly since many other vector plugins use graph-based approaches (such as HNSW from NMSLIB) for search, which suffer from slow indexing and memory-intensive indexing.</p><p>Elastiknn also requires a custom mapping as mentioned in their <a href="https://elastiknn.com/api/#mappings">documentation</a>: “Before indexing vectors, you first define a mapping specifying a vector datatype, an indexing model, and the model’s parameters.”</p><p>As seen in Figure 2 below, GSI’s Elasticsearch k-NN plugin uses Elasticsearch’s dense_vector field type and index mapping, so no reindexing of documents is needed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/278/1*muVWJ7VY8QfMPDcq5so5xg.png" /><figcaption>Figure 2. The GSI Elasticsearch plugin mapping for the dense_vector field type is the same as Elasticsearch’s, so no reindexing of documents is needed to use the plugin.</figcaption></figure><h4>Batch Search for Higher Throughput</h4><p>Batch search, where multiple search requests are processed as a group (batch), allows for higher throughput when compared to processing requests on a query-by-query basis.</p><p>An example of where batch search is useful is an e-commerce website where multiple customers are accessing a common database. In this case, user queries can be combined into a batch and processed together in order to provide high throuput search results.</p><p>Unfortunately, not all Elasticsearch k-NN plugins work well in batch mode. For example, Elastiknn wrote “if you need high-throughput nearest neighbor search for periodic batch jobs, there are several faster and simpler methods.”</p><p>GSI’s Elasticsearch k-NN plugin is one of those methods that offers high-throughput batch processing, and it is a feature that has received good feedback from users.</p><h4><strong>Conclusion</strong></h4><p>GSI’s Elasticsearch k-NN plugin is a high-performance vector search solution that provides low latency, the ability for multimodal search using filters, uses Elasticsearch’s dense_vector field type and index mapping, which means no reindexing of documents is needed to use the plugin, and offers high-throughput batch processing.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=be8d668812e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/gsi-technology/a-comparison-of-elasticsearch-k-nn-solutions-be8d668812e">Elasticsearch Nearest Neighbor Search Options</a> was originally published in <a href="https://medium.com/gsi-technology">GSI Technology</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>