<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Piyush Gupta on Medium]]></title>
        <description><![CDATA[Stories by Piyush Gupta on Medium]]></description>
        <link>https://medium.com/@pgupta?source=rss-296cd0537989------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*bbMdeII8TVVHUBJt.</url>
            <title>Stories by Piyush Gupta on Medium</title>
            <link>https://medium.com/@pgupta?source=rss-296cd0537989------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 06:48:57 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@pgupta/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Pi day exercise: Calculating pi with expressions!]]></title>
            <link>https://medium.com/aerospike-developer-blog/pi-day-exercise-calculating-pi-with-expressions-248426963fda?source=rss-296cd0537989------2</link>
            <guid isPermaLink="false">https://medium.com/p/248426963fda</guid>
            <category><![CDATA[pi]]></category>
            <category><![CDATA[aerospike]]></category>
            <category><![CDATA[expression]]></category>
            <dc:creator><![CDATA[Piyush Gupta]]></dc:creator>
            <pubDate>Thu, 14 Mar 2024 13:39:11 GMT</pubDate>
            <atom:updated>2024-03-14T13:39:11.271Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*63YNeugu_fixWowT.png" /></figure><p><strong>The Monte Carlo method is an effective technique for calculating probabilities.</strong></p><p>In this fun blog, whose concept was “engineered” by our very own <a href="https://www.linkedin.com/in/jpollack/">Joshua Pollack</a>, we demonstrate how to use <a href="https://aerospike.com/docs/server/guide/expressions">expressions in Aerospike</a> to calculate the value of Pi using a Monte Carlo technique.</p><p>While the example is fun and rather contrived, it highlights an underlying technique of using expressions in Aerospike to compute incoming data values for thresholds and apply conditional executions to real-time situations such as <a href="https://aerospike.com/solutions/use-cases/fraud-prevention/">fraud detection</a>.</p><h4>What is Pi?</h4><p><a href="https://www.piday.org/learn-about-pi/">Pi</a> (often represented as the Greek letter π) is “one of the most well-known mathematical constants [and] is the ratio of a circle’s circumference to its diameter. For any circle, the distance around the edge is a little more than three times the distance across. Typing π into a calculator and pressing ENTER will yield the result 3.141592654, not because this value is exact, but because a calculator’s display is often limited to 10 digits.”</p><h4>Calculating the value of Pi</h4><p>The <a href="https://aws.amazon.com/what-is/monte-carlo-simulation/">Monte Carlo method</a> to compute the value of Pi uses the probability of a random point whose x and y coordinates are between -1.0 and 1.0 falling within the enclosing square of side 2.0 versus within the circle at center (0.0, 0.0) with radius r = 1.0.</p><p>The probability of being in the circle versus the encompassing square is π*r^2/(2r)^2. This article describes this simple technique quite nicely.</p><p>The technique reduces to:<br> Estimated value of Pi = 4 * number of points inside the circle / total number of points.</p><p>We can apply the ratio to just the top quadrant of the circle and rectangle.</p><p>We can use the Java random number generator to produce x and y coordinates of a point between 0.0 and 1.0.</p><p>We will do this computation in a single Aerospike record that we initialize as follows:</p><pre>AerospikeClient client = new AerospikeClient(&quot;127.0.0.1&quot;, 3000);<br><br>//Initialize our single aggregation record, center of circle <br><br>Key key1 = new Key(&quot;test&quot;, &quot;points&quot;, 1); <br>Bin xcoord = new Bin(&quot;x&quot;, 0.0);<br>Bin ycoord = new Bin(&quot;y&quot;, 0.0);<br>Bin inCount = new Bin(&quot;inCount&quot;, 1.0);<br>Bin totalCount = new Bin (&quot;count&quot;, 1.0);<br>WritePolicy wPolicy = new WritePolicy();<br>client.put(wPolicy, key1, xcoord, ycoord, inCount, totalCount );<br>//Check<br>System.out.println(client.get(null, key1));</pre><p>Output:</p><pre>(gen:1),(exp:448491015),(bins:(x:0.0),(y:0.0),(inCount:1.0),(count:1.0))</pre><p>As we insert the next random values of the (x,y) pair, we will use expressions to compute whether the value lies within the circle or outside and in the same write transaction, update the inCount and the total count.</p><p>We use another expression-based call to get the estimated value of Pi (expected to be 3.14xx).</p><pre>//Insert a record using a class based construct<br>class Point {<br>    <br>  public void insertPoint (Key key1, double xval, double yval, boolean bCheck) {<br>    Bin xcoord = new Bin(&quot;x&quot;, xval);<br>    Bin ycoord = new Bin(&quot;y&quot;, yval);<br>    WritePolicy wPolicy1 = new WritePolicy();<br>      <br>    Expression inCircleExp = Exp.build(<br>       Exp.cond(<br>        Exp.le(<br>          Exp.add(<br>            Exp.mul(Exp.floatBin(&quot;x&quot;), Exp.floatBin(&quot;x&quot;)), <br>            Exp.mul(Exp.floatBin(&quot;y&quot;), Exp.floatBin(&quot;y&quot;))<br>          ),<br>          Exp.val(1.0)<br>        ),<br>       Exp.add(Exp.floatBin(&quot;inCount&quot;), Exp.val(1.0)),<br>       Exp.floatBin(&quot;inCount&quot;)<br>       ));<br>        <br>    Expression totalCountExp = Exp.build(Exp.add(Exp.floatBin(&quot;count&quot;), Exp.val(1.0)));<br> <br>    Record record = client.operate( wPolicy1, key1,   <br>          Operation.put(xcoord),<br>          Operation.put(ycoord),                   <br>          ExpOperation.write(&quot;inCount&quot;, inCircleExp, ExpWriteFlags.DEFAULT),<br>          ExpOperation.write(&quot;count&quot;, totalCountExp, ExpWriteFlags.DEFAULT)<br>                                  );                 <br>    if(bCheck){<br>      System.out.println(client.get(null, key1));<br>    }   <br>  }<br>    <br>  public void getPi(Key key1){<br>      WritePolicy wPolicy = new WritePolicy();<br>      Expression piExp = Exp.build(<br>          Exp.mul(Exp.val(4.0), <br>                  Exp.div(Exp.floatBin(&quot;inCount&quot;),Exp.floatBin(&quot;count&quot;))<br>                 ));<br>      <br>      Record record = client.operate(wPolicy, key1, ExpOperation.read(&quot;pi&quot;, piExp, ExpReadFlags.DEFAULT));<br>      System.out.println(&quot;Pi : &quot; + record.getValue(&quot;pi&quot;));<br>  }<br>  <br>}</pre><p>Let’s try this code construct:</p><pre>//Check<br>Point p = new Point();<br>p.insertPoint(key1, 0.71, 0.71, true);  //Out of circle, in rectangle</pre><p>Output:</p><pre>(gen:2),(exp:448490510),(bins:(x:0.71),(y:0.71),(inCount:1.0),(count:2.0))</pre><p>Let’s scale it up and insert 10,000 random points. This takes an imperceptible amount of time on Aerospike.</p><pre>//Insert random points with x,y between 0.0 and 1.0<br><br>Random rnd1 = new Random(0);  <br><br>for(int i=1;i&lt;10000;i++){<br>  double xval = rnd1.nextDouble();<br>  double yval = rnd1.nextDouble();<br>  p.insertPoint(key1, xval, yval, false);  <br>}</pre><p>And now for our estimate of Pi:</p><pre>p.getPi(key1);</pre><p>Output:</p><pre>Pi : 3.1424</pre><p>Happy Pi Day!</p><p>In the above example, getPi() is shown here as a separate call. However, it can be made part of the return value in the write api (insertPoint()).</p><p>Instead of our pair of x and y, if you had 100s of such parameters that you would want to compute a net score to make a go/nogo decision, the above code constructs can be easily extended. In this example, we had no need to retain the 10,000 random values. However, if the data needs to be retained, consider using one of our collection data types, such as a Map or a List.</p><h4>Get the most out of your data with Aerospike Expressions</h4><p>To learn more, watch our <a href="https://aerospike.com/resources/webinars/devchat-aerospike-expressions/">Optimizing query performance with Aerospike Expressions</a> webinar with Chief Developer Advocate Tim Faulkes.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=248426963fda" width="1" height="1" alt=""><hr><p><a href="https://medium.com/aerospike-developer-blog/pi-day-exercise-calculating-pi-with-expressions-248426963fda">Pi day exercise: Calculating pi with expressions!</a> was originally published in <a href="https://medium.com/aerospike-developer-blog">Aerospike Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to retrieve the key in a key-value store]]></title>
            <link>https://medium.com/aerospike-developer-blog/how-to-retrieve-the-key-in-a-key-value-store-c09244ac919e?source=rss-296cd0537989------2</link>
            <guid isPermaLink="false">https://medium.com/p/c09244ac919e</guid>
            <category><![CDATA[values]]></category>
            <category><![CDATA[aerospike]]></category>
            <category><![CDATA[keys]]></category>
            <dc:creator><![CDATA[Piyush Gupta]]></dc:creator>
            <pubDate>Wed, 13 Mar 2024 17:55:09 GMT</pubDate>
            <atom:updated>2024-03-13T17:55:09.956Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*g0oOthLgOAqme6Ib.png" /></figure><p><strong>Learn how to retrieve user-specified keys in Aerospike using single record reads.</strong></p><p>In this blog, you’ll learn how to retrieve the user-specified key in Aerospike, a key-value store.<br> Aerospike is designed from the ground up to support extremely low latency create, read, update, and delete (<a href="https://aerospike.com/resources/tech-videos/crud-ops/">CRUD</a>) operations. In Aerospike, applications store data as individual records that are uniformly distributed across all the nodes of the database cluster. A user-specified key is associated with each record along with the application data. Below, we will walk you through the steps necessary to retrieve your user-specified key in a single record read operation.</p><h4>Keys in Aerospike</h4><p>As mentioned above, Aerospike stores user data in individual records, which are accessible by a user-specified “key.” Keys can only be a string, integer, or a byte array.</p><pre>[USER SPECIFIED KEY] → [USER DATA in one or more &quot;bins&quot;]</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/827/0*tourBffBB9wrUCI_.png" /></figure><p>Applications access the data via the <a href="https://aerospike.com/developer/client">Aerospike client library</a>, which computes a 20-byte “digest” by hashing the user-supplied key information of set name, user key value, and user key type.</p><p>The client library sends the 20-byte digest as the key of record to the server. So, the record in the server should read:</p><pre>[20 Byte Digest] → [USER DATA in one or more &quot;bins&quot;]</pre><h4>Storing keys in records</h4><p>What if two keys hash to the same digest value, resulting in a hash collision?</p><p>Aerospike uses the RIPEMD-160 hashing algorithm, which has a statistically near-zero probability of hash collision. By default, Aerospike does not store the user-supplied key with the saved record. However, applications have the option to store the user-specified key along with the record via the <a href="https://aerospike.com/docs/server/guide/policies">write policy</a>.</p><pre>[20 Byte Digest ] → [USER DATA in one or more &quot;bins&quot;] + [User Key (string / int / blob)]</pre><p>The approach above ensures that the server verifies the hash against the stored key during CRUD operations. Any subsequent updates where a hash collision is detected, the application will receive a KEY_MISMATCH error.</p><h4>Key mismatch explained</h4><p>Consider the case in a namespace test, where a user-specified key of abc in set testSet1 generates a particular digest, digest1. When this record is first created, Aerospike uses digest1 as the internal key of that record but also stores the user-specified key of abc with the record.</p><p>Hypothetically, in the same namespace test, consider the user-specified key of xyz in set testSet2, which also happens to hash to the same value, digest1. (This has near zero probability, practically speaking.)</p><p>If the application tries to create or update a record with digest1 and with user-specified key xyz, Aerospike will access the record stored against digest1 and compare the stored key abc against the incoming transaction&#39;s user-specified key of xyz and discover that they are different. A KEY_MISMATCH exception will be thrown in this case.</p><h4>User key stored with the record</h4><p>The curious may want to know: Is it possible to read this user-specified stored key back in the application?</p><p>For data modeling purposes and for queries (primary or secondary index queries that retrieve qualifying records from the entire namespace), the user stored key can be retrieved back to the client. This is useful since the user may want to then update individual records selectively, as part of their use case.</p><p>However, for single key reads the user key is not returned to the client, as you must have had the key to be able to read the record in the first place. This type of read can be done by passing either the user key and the set name or the digest.</p><p>In certain operational situations, users have asked if it is possible to read back the user stored key on an individual record read. They stored their key with the record and would now like to retrieve it. These are situations where users have the 20-byte digest of a record, but they do not know the user key. The digest was recorded in the server logs. An example of this would be when turning on <a href="https://support.aerospike.com/s/article/How-To-use-rw-client-logging-to-identify-a-hot-digest">rw-client logging</a> to identify the digests of “hot-keys,” records that are being updated very frequently.</p><p>The read APIs don’t provide this functionality upfront via the policies since this is not a typical data modeling need. However, Aerospike does provide <a href="https://aerospike.com/docs/server/guide/expressions">expressions</a> that can access the user stored key.</p><h4>What are Aerospike Expressions?</h4><p>An expression is a strongly typed domain-specific language designed for manipulating and comparing bins and record metadata. There is no “programming language” per se; an expression is built with code. It runs on the nodes in the cluster, not on the client.</p><p>Aerospike supports three types of expressions:</p><ol><li>Filter expressions (introduced in version 5.2.0)</li><li><a href="https://aerospike.com/products/features/cross-datacenter-replication-xdr/">Cross Datacenter Replication</a> filter expressions (introduced in version 5.3.0)</li><li>Operation expressions (introduced in version 5.6.0)</li></ol><p>Filter and operation expressions can be specified on a per-call basis.</p><h4>Key retrieval code example</h4><p>Below is an example code snippet demonstrating how to retrieve a key using the Java client.</p><p>Creating the record initially:</p><pre>AerospikeClient client = new AerospikeClient(&quot;127.0.0.1&quot;, 3000);<br>Key key = new Key(&quot;test&quot;, &quot;testset&quot;, &quot;key1&quot;);<br>WritePolicy wPolicy = new WritePolicy();<br><br>wPolicy.sendKey = true;  //Store our user key on the server<br><br>Bin b0 = new Bin(&quot;b00&quot;, Value.get(28));<br>client.put(wPolicy, key, b0);</pre><p>Now, assume we just have the 20-byte digest and would like to retrieve our key:</p><p>We can use the following key constructor:</p><pre>// Initialize key from namespace, digest, optional set name, and optional userKey.<br>Key(String namespace, byte[] digest, String setName, Value userKey)</pre><p>Let’s find the digest of the record we created above, for purposes of this demo.</p><pre>byte[] recDigest = new byte[20];<br>recDigest =  Key.computeDigest(&quot;testset&quot;, Value.get(&quot;key1&quot;)) ;</pre><p>Next, create a new key object using the digest-based constructor.</p><pre>Key keyByDigest = new Key(&quot;test&quot;, recDigest, null, Value.NULL);</pre><h4>About Value.NULL</h4><p>The constructor allows specifying the user-specified key in the last argument as a “Value” type object. For example, Value.get(&quot;myKey&quot;). This is provided for the rare case where the application wants to do an update with the key generated using the digest, and the set name is not known but still wants to update with sendKey=true. In our case, we don&#39;t know the key and are trying to retrieve it in an operate() call, which is like a write, although we will only perform a read via expressions. Hence, we pass Value.NULL for the user-specified key and use sendKey=false.</p><p>We should be able to read the same record using either “key” or “keyDigest.” Let’s validate.</p><pre>System.out.println(&quot;Record: &quot;+ client.get(null, key)); <br>System.out.println(&quot;Record via Digest: &quot;+ client.get(null, keyByDigest));</pre><p>Output:</p><pre>Record: (gen:12),(exp:447740032),(bins:(b00:28))<br>Record via Digest: (gen:12),(exp:447740032),(bins:(b00:28))</pre><h4>Case 1: key type known</h4><p>Now, using the digest-based key, let’s retrieve the stored key from the server. Since we are specifying the digest and Value.NULL for the userKey in our recDigest constructor, we must set thewPolicy.sendKey = false; to avoid a KEY_MISMATCH error.</p><pre>wPolicy.keySend = false;<br>Expression recKeyExp = Exp.build(Exp.key(Exp.Type.STRING));<br>Record record = client.operate( wPolicy, keyByDigest,   <br>          ExpOperation.read(&quot;reckey&quot;, recKeyExp, ExpReadFlags.DEFAULT) <br>         );<br>System.out.println(&quot;Record Key: &quot; + record.getValue(&quot;reckey&quot;));</pre><p>Recall we mentioned that user-specified keys can be of type integer, string, or byte-array. Here, although we don’t know our key, we know it is a string type. This is specified using Exp.Type.STRING when building the expression for the key.</p><p>Output:</p><pre>Record Key: key1</pre><h4>Case 2: key type unknown</h4><p>What if we don’t know the key type? We can check for all the three allowable types. Although, in this case, we must use ExpReadFlags.EVAL_NO_FAIL instead of ExpReadFlags.DEFAULT so that the failed expressions with type mismatch don&#39;t generate an Exception.</p><p>Example code is shown below.</p><pre>wPolicy.sendKey = false;<br>Expression recKeyExpStr = Exp.build(Exp.key(Exp.Type.STRING));<br>Expression recKeyExpInt = Exp.build(Exp.key(Exp.Type.INT));<br>Expression recKeyExpBlob = Exp.build(Exp.key(Exp.Type.BLOB));<br><br>Record record = client.operate( wPolicy, keyByDigest,   <br>          ExpOperation.read(&quot;reckeyStr&quot;, recKeyExpStr, ExpReadFlags.EVAL_NO_FAIL),<br>          ExpOperation.read(&quot;reckeyInt&quot;, recKeyExpInt, ExpReadFlags.EVAL_NO_FAIL),<br>          ExpOperation.read(&quot;reckeyBlob&quot;, recKeyExpBlob, ExpReadFlags.EVAL_NO_FAIL)  <br>         );<br>System.out.println(&quot;Record Key String: &quot; + record.getValue(&quot;reckeyStr&quot;));<br>System.out.println(&quot;Record Key Integer: &quot; + record.getValue(&quot;reckeyInt&quot;));<br>System.out.println(&quot;Record Key BLOB: &quot; + record.getValue(&quot;reckeyBlob&quot;));</pre><p>Output:</p><pre>Record Key String: key1<br>Record Key Integer: null<br>Record Key BLOB: null</pre><h4>Unlock the power of expressions in Aerospike for enhanced data operations</h4><p>Expressions are a very powerful tool in Aerospike. They can be used as filters for selecting records to return in primary or secondary index queries, as well as compute on a record’s existing data or metadata. It can then return the result of the computation as a read or write it to a new bin. The possibilities are only limited by your creativity!</p><p>To learn more, watch our <a href="https://aerospike.com/resources/webinars/devchat-aerospike-expressions/">Optimizing query performance with Aerospike Expressions</a> webinar with Chief Developer Advocate Tim Faulkes.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c09244ac919e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/aerospike-developer-blog/how-to-retrieve-the-key-in-a-key-value-store-c09244ac919e">How to retrieve the key in a key-value store</a> was originally published in <a href="https://medium.com/aerospike-developer-blog">Aerospike Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Sorted Results from a Secondary Index Query — Part II]]></title>
            <link>https://medium.com/aerospike-developer-blog/sorted-results-from-a-secondary-index-query-part-ii-de17f43f7fec?source=rss-296cd0537989------2</link>
            <guid isPermaLink="false">https://medium.com/p/de17f43f7fec</guid>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Piyush Gupta]]></dc:creator>
            <pubDate>Wed, 02 Oct 2019 16:27:40 GMT</pubDate>
            <atom:updated>2020-04-06T17:57:25.541Z</atom:updated>
            <content:encoded><![CDATA[<h3>Sorted Results from a Secondary Index Query — Part II</h3><p>Aerospike, a distributed NoSQL database, stores records distributed evenly across all nodes of the cluster using a hash value derived from the user’s key. While this scheme gives awesome performance at scale for record reads and writes, when running secondary index queries on a set of records in a namespace, it presents the challenge of getting sorted results.</p><p>In <a href="https://medium.com/@pgupta/sorted-results-from-a-secondary-index-query-in-aerospike-9e05117a68ef">Part I</a>, we explored using Map data type to sort results from a Secondary Index query in Aerospike.</p><p>In this blog, Part II, we continue the same example using the <em>List</em> instead of the <em>Map</em> data type for sorting. Using the ordered List data type, we can append the results as they are retrieved from the secondary index query run on records in a persistent namespace (data on disk) into a scratch record in a pure in-memory namespace. The appended results are automatically inserted in the sort order in the scratch record based on the first item in the list. We can then retrieve the sorted results, paginating as necessary using the List Operation APIs for consumption by the client.</p><p>Maps can be used if the sort key value is unique for each record and you are sure that the value associated with the key will not be overwritten. A list of the list value type, as shown in this example, allows you to sort based on the first value of list element, as long as each list value has the same number of entries. You can opt to keep or remove duplicate values.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/747/1*uSLcS2zHG_PNNaBCzoSL9A.png" /></figure><p><em>Example:</em> We have data on disk on people — name, house#, street name, city, county — millions of records. We want to search for all the people on a given street by doing a partial match on street name. However, we want the result set sorted and paginated by house number. Multiple people may live in the same house, so multiple record entries may be present for the same house number.</p><ul><li>Create the Secondary Index (SI) query <em>or</em> scan the entire data set and filter the results to a desired predicate filter:</li></ul><p><em>Example data:</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hthhV8yHmk2plxDf-H8BWQ.png" /><figcaption>Starting Data Set Sample</figcaption></figure><pre>Statement stmt = new Statement();<br> stmt.setSetName(sn); // “people”<br> stmt.setNamespace(ns); // “test”, a disk storage namespace<br> //If SI filtering is needed ...<br> stmt.setIndexName(“county_idx”);<br> stmt.setFilter(Filter.equal(“county”,”Santa Clara”));</pre><pre>stmt.setPredExp(<br> PredExp.<em>stringBin</em>(“street”),<br> //any prefix- any suffix partial string match, <br> //ignore case or newline<br> PredExp.<em>stringValue</em>(“.*”+street+”.*”),<br> PredExp.<em>stringRegex</em>(RegexFlag.<em>ICASE</em> | RegexFlag.<em>NEWLINE</em>)<br> );</pre><ul><li>Consume the results in a call back, one record at a time and store back in an in-memory namespace for sorting and paginating:</li></ul><p><em>SI on county query, records returned where </em><strong><em>street = “mary”</em></strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*g22ot9Aag9YD-iDyP-_XGA.png" /><figcaption>Qualifying records after filtering on SI and Predicate Filter</figcaption></figure><pre>RecordSet recordSet = <em>client</em>.query(queryPolicy, stmt);<br> <br>...<br>...<br>while (recordSet.next()) {<br>  record = recordSet.getRecord();<br>  vl.add(record.getString(“housenum”));<br>  vl.add(record.getString(“street”));<br>  vl.add(record.getString(“city”));<br>  vl.add(record.getString(“county”));<br>  vl.add(record.getString(“name”));<br>  tempRec = <em>client</em>.operate(null, tempKey,<br>           ListOperation.<em>append</em>(lPolicy, “tempBin”,Value.<em>get</em>(vl)));<br>  vl.clear();<br> }</pre><ul><li>Read back a sub-set (paginate) or all results from the in-memory namespace, sorted by desired sort key:</li></ul><p><em>Sorted results stored as a list of lists in in-memory namespace:</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/705/1*deYEFzcRKxEbRt3ytuyInQ.png" /></figure><pre>for(int i=0; i&lt;size; i++) {<br>    List&lt;?&gt; lItem = (List&lt;?&gt;)(ll.get(i));<br>    str = “”;<br>    for(int j=0; j&lt;5; j++) {<br>      if(lItem.get(j)!=null) {str+=lItem.get(j).toString()+” “;}<br>    }<br>  lPeople.add(str);<br> }</pre><p>(Total number of records is equal to <em>size</em>. To paginate, say in sets of 10, loop over <em>i</em> in sets of 10, till <em>size</em> is exhausted.)</p><p>With <a href="https://www.aerospike.com/apidocs/java/com/aerospike/client/policy/QueryPolicy.html">fail on cluster change</a> and <a href="https://www.aerospike.com/docs/guide/consistency.html">Strong Consistency mode</a> guarantees, this can be enhanced to account with code exceptions for timeouts and consistent performance under cluster network events.</p><p><em>Putting it together… sample code for the method is shown below:</em></p><pre>public ArrayList&lt;String&gt; getPeopleByStreetUsingLists(String ns, String sn, String street, String queryId) {</pre><pre> ArrayList&lt;String&gt; lPeople = new ArrayList&lt;String&gt;();<br> if(street.isEmpty()) return lPeople;<br> if(queryId.isEmpty()) return lPeople;</pre><pre> Record record = null;<br> Statement stmt = new Statement();<br> stmt.setSetName(sn); // “people”<br> stmt.setNamespace(ns); // “test”, a disk storage namespace<br> //If SI filtering is needed ...<br> stmt.setIndexName(“county_idx”);  //SI pre-built<br> stmt.setFilter(Filter.equal(“county”,”Santa Clara”));</pre><pre> stmt.setPredExp(<br> PredExp.<em>stringBin</em>(“street”),<br> //any prefix- any suffix partial string match, <br> //ignore case or newline<br> PredExp.<em>stringValue</em>(“.*”+street+”.*”),<br> PredExp.<em>stringRegex</em>(RegexFlag.<em>ICASE</em> | RegexFlag.<em>NEWLINE</em>)<br> );</pre><pre> QueryPolicy queryPolicy = null;<br> RecordSet recordSet = <em>client</em>.query(queryPolicy, stmt);<br> Key tempKey = new Key(“scratch”, “temp”, queryId);</pre><pre> ListPolicy lPolicy = new ListPolicy(ListOrder.<em>ORDERED</em>,<br>                      ListWriteFlags.<em>DEFAULT</em>);<br> //We will sort people data values as list with <br> //first item in the list being housenum.<br> //Must use append() with ORDERED, needed to retrieve sorted values.</pre><pre> int size = 0;<br> Record tempRec = null;<br> List&lt;String&gt; vl = new ArrayList&lt;String&gt;();</pre><pre><em> client</em>.delete(null, tempKey);</pre><pre> while (recordSet.next()) {<br>  record = recordSet.getRecord();<br>  vl.add(record.getString(“housenum”));<br>  vl.add(record.getString(“street”));<br>  vl.add(record.getString(“city”));<br>  vl.add(record.getString(“county”));<br>  vl.add(record.getString(“name”));<br>  tempRec = <em>client</em>.operate(null, tempKey,<br>           ListOperation.<em>append</em>(lPolicy, “tempBin”,Value.<em>get</em>(vl)));<br>  vl.clear();<br> }</pre><pre> if(tempRec!=null) {<br>  size = tempRec.getInt(“tempBin”);<br>  //Use size to paginate if desired. If expected size <br>  //is within 128KiB boundary, entire record can be retrieved.<br>  //Single server read operation fetches sorted data.</pre><pre>  tempRec = <em>client</em>.operate(null, tempKey,<br>          ListOperation.<em>getByIndexRange</em>(“tempBin”,0,size,<br>          ListReturnType.<em>VALUE</em>));</pre><pre>  List&lt;?&gt; ll = tempRec.getList(“tempBin”);<br>  String str;</pre><pre>  for(int i=0; i&lt;size; i++) {<br>    List&lt;?&gt; lItem = (List&lt;?&gt;)(ll.get(i));<br>    str = “”;<br>    for(int j=0; j&lt;5; j++) {<br>      if(lItem.get(j)!=null) {str+=lItem.get(j).toString()+” “;}<br>    }<br>  lPeople.add(str);<br> }</pre><pre> if(lPeople.size()==0) {<br>  lPeople.add(“No match found.”);<br> }<br> return lPeople;<br>}</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=de17f43f7fec" width="1" height="1" alt=""><hr><p><a href="https://medium.com/aerospike-developer-blog/sorted-results-from-a-secondary-index-query-part-ii-de17f43f7fec">Sorted Results from a Secondary Index Query — Part II</a> was originally published in <a href="https://medium.com/aerospike-developer-blog">Aerospike Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Sorted Results from a Secondary Index Query in Aerospike]]></title>
            <link>https://medium.com/aerospike-developer-blog/sorted-results-from-a-secondary-index-query-in-aerospike-9e05117a68ef?source=rss-296cd0537989------2</link>
            <guid isPermaLink="false">https://medium.com/p/9e05117a68ef</guid>
            <category><![CDATA[secondary-index]]></category>
            <category><![CDATA[data-modeling]]></category>
            <category><![CDATA[aerospike]]></category>
            <category><![CDATA[database]]></category>
            <dc:creator><![CDATA[Piyush Gupta]]></dc:creator>
            <pubDate>Tue, 24 Sep 2019 02:04:15 GMT</pubDate>
            <atom:updated>2019-11-08T02:04:07.828Z</atom:updated>
            <content:encoded><![CDATA[<h3>Sorted Results from a Secondary Index Query in Aerospike — Part I</h3><p>Aerospike, a distributed NoSQL database, stores records distributed evenly across all nodes of the cluster using a hash value derived from the user’s key. While this scheme gives excellent performance at scale for reading and writing records, when running secondary index queries on a set of records in a namespace, it presents the challenge of getting <em>sorted</em> results to the client.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZpyqUMGJh7YjQGla5bCpxg.png" /></figure><p>For many use cases, users need the query results sorted. For example, in a data set of residents of cities, where each resident’s information is an individual record stored in a persistent namespace on disks, we may want to find all residents on a particular street, sorted by their house numbers.</p><p>Natively, while you can run this kind of <a href="https://www.aerospike.com/docs/guide/predicate.html">Secondary Index query combined with predicate filtering</a> in Aerospike, as of version 4.6, you will get the matching records in no particular order. The onus then falls on the client to accumulate the results and sort, and optionally paginate. This requires both compute and memory resources on the client side.</p><p>However, Aerospike has two neat options at our disposal to let Aerospike do this work for us. <em>First, Aerospike offers in-memory namespaces where a record can grow literally to whatever size allowed by the available memory. There are ingress and egress limitations of 128MiB in the network socket buffers, but for our purpose, we will update the record incrementally and not retrieve the entire record if it exceeds 128MiB (we can paginate on retrieval). Second, the Map data type in Aerospike takes key:value inserts and can be configured to store and index them by key order. Map Operation APIs are available to extract the associated value in key sort order, either all or paginated by count.</em></p><p><strong>Example: </strong>We have data on people — name, house#, street name, city, county — millions of records. We want to search for all the people on a given street by doing a partial match on street name. However, we want the result set sorted and paginated by house number. Multiple people may live in the same house, so multiple record entries may be present for the same house number.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hthhV8yHmk2plxDf-H8BWQ.png" /><figcaption>Sample Data Set</figcaption></figure><p>We will take each record on disk extracted using a predicate filtered query and store in a scratch record, in an in-memory namespace as a map data type (key-value pair) where the key is carefully constructed</p><pre>//Select key for desired sort order<br> k = record.getString(“housenum”)+ record.getString(“name”); </pre><p>to yield the sort order that we need.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*g22ot9Aag9YD-iDyP-_XGA.png" /><figcaption>Records selected based on Secondary Index and Predicate Filtering</figcaption></figure><p>The scratch record (in-memory) is deleted prior to, and can also be deleted after, each run.</p><p>The queryId passed by the user is used as the primary key for the scratch record to keep each query’s scratch space separate from others. Users can devise efficient ways to create unique queryIds based on their use case.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/789/1*5tK3Q4VpYuSH87SNTGlsog.png" /><figcaption>Using Map Type to store query results in an in-memory namespace</figcaption></figure><p>Using Map APIs, we can easily retrieve results, from the scratch record, sorted by index and we can also limit by count (paginate).</p><p>With <a href="https://www.aerospike.com/apidocs/java/com/aerospike/client/policy/QueryPolicy.html">fail on cluster change</a> and <a href="https://www.aerospike.com/docs/guide/consistency.html">Strong Consistency mode</a> guarantees, this can be enhanced with code exceptions for timeouts and deliver consistent performance even under cluster network events.</p><p>The key idea is to utilize an in-memory namespace without record size limitation, move the storage and compute to the server and exploit the capabilities of the Map and List APIs. Alternatively, one can also use sorted lists data type to achieve a similar implementation, whatever works better for the data model at hand.</p><p>In <a href="https://medium.com/@pgupta/sorted-results-from-a-secondary-index-query-part-ii-de17f43f7fec">Part II</a>, we will explore the List data type implementation for sorting the result set.</p><p><strong>Sample implementation is shown below:</strong></p><pre>public ArrayList&lt;String&gt; getPeopleStrings(String ns, String sn, String street, String queryId) {</pre><pre>  ArrayList&lt;String&gt; lPeople = new ArrayList&lt;String&gt;();<br>  if(street.isEmpty()) return lPeople;<br>  if(queryId.isEmpty()) return lPeople;</pre><pre>Record record = null;<br>  Statement stmt = new Statement();<br>  stmt.setSetName(sn);  // &quot;people&quot;<br>  stmt.setNamespace(ns); // &quot;test&quot;, a disk storage namespace<br>  // If SI filtering is needed, add here.<br>  stmt.setIndexName(&quot;county_idx&quot;);<br>  stmt.setFilter(Filter.equal(&quot;county&quot;,&quot;Santa Clara&quot;));<br>  stmt.setPredExp(<br>  //match part of the string (street name) passed<br>  //by the user on record bin &quot;street&quot;<br>  PredExp.stringBin(&quot;street&quot;),<br>  PredExp.stringValue(&quot;.*&quot;+street+&quot;.*&quot;),<br>  PredExp.stringRegex(RegexFlag.ICASE | RegexFlag.NEWLINE)<br>  );</pre><pre>  QueryPolicy queryPolicy = null;<br>  RecordSet recordSet = client.query(queryPolicy, stmt); <br>  Key tempKey = new Key(&quot;scratch&quot;, &quot;temp&quot;, queryId);<br>  //scratch is an in-memory namespace</pre><pre>  MapPolicy mPolicy = new MapPolicy(MapOrder.<em>KEY_ORDERED</em>,<br>                      MapWriteFlags.<em>DEFAULT</em>);<br>  //Must use KEY_ORDERED to retrieve values in sorted key order.</pre><pre>  int size = 0;<br>  Record tempRec = null;<br>  String k;<br>  String v;<br>  client.delete(null, tempKey);</pre><pre>  while (recordSet.next()) { <br>    record = recordSet.getRecord();<br>    v = record.getString(&quot;housenum&quot;)+&quot; &quot;+<br>        record.getString(&quot;street&quot;) +&quot; &quot;+<br>        record.getString(&quot;city&quot;)+&quot; &quot;+<br>        record.getString(&quot;county&quot;)+&quot; [&quot; +<br>        record.getString(&quot;name&quot;) +&quot;]&quot;;</pre><pre>    //We reduced v to a string for simplicity -<br>    //could be kept as a list of items.</pre><pre>    //Select key for desired sort order<br>    k = record.getString(&quot;housenum&quot;)+ record.getString(&quot;name&quot;);</pre><pre>    tempRec = client.operate(null,  tempKey,<br>              MapOperation.put(mPolicy, &quot;tempBin&quot;,Value.get(k),<br>              Value.get(v)));</pre><pre>    //save values to scratch space in key order<br>    //mPolicy is KEY_ORDERED<br>  }</pre><pre>  if(tempRec!=null) {<br>    size = tempRec.getInt(&quot;tempBin&quot;);</pre><pre>    //Use size to paginate if desired. If expected size<br>    //is less than 128MiB, entire record can be retrieved.  <br>    //Sorted page data is fetched in a single read operation.</pre><pre>    tempRec = <em>client</em>.operate(null,  tempKey,<br>              MapOperation.<em>getByIndexRange</em>(&quot;tempBin&quot;,0,size,<br>              MapReturnType.<em>VALUE</em>));</pre><pre>    List&lt;?&gt; ll = tempRec.getList(&quot;tempBin&quot;);</pre><pre>    for(int i=0; i&lt;size; i++) {<br>      lPeople.add((String)ll.get(i));<br>    }<br>  }<br>  return lPeople;<br>}</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9e05117a68ef" width="1" height="1" alt=""><hr><p><a href="https://medium.com/aerospike-developer-blog/sorted-results-from-a-secondary-index-query-in-aerospike-9e05117a68ef">Sorted Results from a Secondary Index Query in Aerospike</a> was originally published in <a href="https://medium.com/aerospike-developer-blog">Aerospike Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>