<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ravi Chauhan on Medium]]></title>
        <description><![CDATA[Stories by Ravi Chauhan on Medium]]></description>
        <link>https://medium.com/@ravichauhan03?source=rss-f18ad48a2325------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*LeVh88u74OubAIQrS30WpA.jpeg</url>
            <title>Stories by Ravi Chauhan on Medium</title>
            <link>https://medium.com/@ravichauhan03?source=rss-f18ad48a2325------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 17:44:37 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ravichauhan03/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Declarative authorization with Spring-Security @Secured annotation]]></title>
            <link>https://medium.com/@ravichauhan03/declarative-authorization-with-spring-security-secured-annotation-ace1c748feeb?source=rss-f18ad48a2325------2</link>
            <guid isPermaLink="false">https://medium.com/p/ace1c748feeb</guid>
            <category><![CDATA[spring-security]]></category>
            <category><![CDATA[decisionvoter]]></category>
            <category><![CDATA[authorization]]></category>
            <category><![CDATA[repeatable-annotations]]></category>
            <category><![CDATA[secured]]></category>
            <dc:creator><![CDATA[Ravi Chauhan]]></dc:creator>
            <pubDate>Tue, 07 Jul 2020 05:17:19 GMT</pubDate>
            <atom:updated>2020-09-24T08:27:56.251Z</atom:updated>
            <content:encoded><![CDATA[<p>Authors: <a href="https://medium.com/u/f18ad48a2325">Ravi Chauhan</a> &amp; <a href="https://medium.com/u/ac9f65816ee1">Agarwalshubhi</a></p><h3><strong>Prerequisites</strong></h3><ol><li>Basic understanding of spring-security framework</li><li>Worked on Spring-boot applications</li></ol><h3>Introduction</h3><p>Spring Security is one of the most advanced frameworks available today that enables developers to enforce authorization at a method-level and type-level as mentioned in the Secured Interface below:</p><pre>@Target({<strong>ElementType.METHOD</strong>, <strong>ElementType.TYPE</strong>})<br>@Retention(RetentionPolicy.RUNTIME)<br>@Inherited<br>@Documented<br>public @interface Secured {<br>    String[] value();<br>}</pre><p>@Secured annotations work well when you need to check the privileges like role &amp; permissions. But when we ask a bigger question that, is @Secured annotation self-sufficient to perform fine-grained authorization with the current implementation considering the dynamic nature of method invocations based on various arguments?</p><p>The answer to this is, maybe not. But why?</p><p>With the growing scale in today’s era and humongous scale, just authentication along with role &amp; permissions-based authorization is not sufficient enough. There is a rapid need arising for finer access control. This finer access control can have any engine running behind the scenes to check access control. But, can we have a simpler and easier way to look at it? Can we use the existing features to make it future-ready? Yes, we can.</p><p>Now the next question is, can we achieve this using the already existing spring security paradigm?</p><p>Below code-snippet shows how this @Secured annotation is used in the code for authorization purpose. This is also called as authorization based on pre-invocation handling. A pre-invocation decision on whether the invocation is allowed to proceed further is made by the “AccessDecisionManager”</p><pre><strong>#1 <br>@Secured ({&quot;ROLE_USER&quot;, &quot;ROLE_ADMIN&quot;})</strong><br>public void addUser(String name, String pwd);</pre><pre><strong>OR </strong></pre><pre><strong>#2</strong><br><strong>@Secured([&quot;{key1:value1, key2:value2}&quot;])</strong> // any format which we want to parse</pre><p>Note that, in the above code snippet, you see 2 ways of passing arguments to @Secured. Are both of these supported by spring-security out of the box? No, it is not.</p><p>The first one comes handy where you can pass roles or permissions. But, in the second one, you see a JSON. Can that be parsed by not making any other changes or extended implementations? No, it is not possible.</p><p>Therefore, the below section describes the changes required to make it possible and parse any Array of JSON/String that we want which comes as input from the @Secured annotation argument. Please note, the arguments which are accepted are in the format of String array as you can see below:</p><pre>public @interface Secured {<br>    <strong>String[]</strong> value();<br>}</pre><p>There are 2 other annotations provided by spring-security, where you can also give SpEL expressions in the parameters which is not possible in case of @Secured annotation.</p><pre><strong>@PreAuthorize</strong>(&quot;isAuthenticated()&quot;)<br>public String unlock() {  <br>    return &quot;you are inside the locker&quot;;  <br>}</pre><pre><strong>@PostAuthorize</strong>(&quot;returnObject.ownerName == authentication.name&quot;)<br>	public Owner getOwner();<br>Note*: Spring-security provides built-in keyword i.e. returnObject</pre><h3><strong>Architecture</strong></h3><p>To make the implementation more clear, you should have some understanding of what you are trying to do here. Therefore, the basic authorization architecture in spring-security is shown in the below diagram which also highlights where your custom code fits in.</p><p>Below diagram depicts the workflow of how custom authorization can be performed in spring with @Secured annotation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DPd4tGIOBSirjy9XKHbBQQ.png" /><figcaption>Decision Manager workflow</figcaption></figure><p>If you look closely, “CustomDecisionVoter” is our custom implementation for fetching the arguments from @Secured annotation and process them to provide the right vote to access or deny the respective invocation.</p><h3>Implementation</h3><p>To start with the implementation, you need to have the custom implementation of “GlobalMethodSecurityConfiguration”. This is required to register your “CustomDecisionVoter” in the application container’s “AccessDecisionManager”. Also note, you can choose the desired implementation for managing decisions i.e. one of “AffirmativeBased” OR “ConsensusBased” OR “UnanimousBased”. You can find more details on this under spring documentation <a href="https://docs.spring.io/spring-security/site/docs/4.2.x/reference/html/authz-arch.html#authz-access-decision-manager">here</a>.</p><pre>@EnableGlobalMethodSecurity(<br>    securedEnabled = true<br>)<br>public class CustomMethodSecurityConfig extends GlobalMethodSecurityConfiguration {<br>    @Autowired<br>    private CustomDecisionVoter customDecisionVoter;<br><br>    public MethodSecurityConfig() {<br>    }<br><br>    @Bean<br>    protected AffirmativeBased accessDecisionManager() {<br>        List&lt;AccessDecisionVoter&lt;?&gt;&gt; voters = new ArrayList();<br>        <strong>//you can register as many voters as you want</strong><br>        <strong>voters.add(this.customDecisionVoter)</strong>;<br>        AffirmativeBased decisionManager = new AffirmativeBased(voters);<br>        decisionManager.setAllowIfAllAbstainDecisions(false);<br>        return decisionManager;<br>    }<br>}</pre><p>Note that, the flag <strong>secureEnabled</strong>, the value is set to true so that you can use the @Secured annotation of spring-security. To enable the @PreAuthorize &amp; @PostAuthorize, there is a different flag as shown below:</p><pre>@EnableGlobalMethodSecurity(<br><strong>prePostEnabled = true</strong>,securedEnabled = true,jsr250Enabled = true)<br>public class MethodSecurityConfig extends GlobalMethodSecurityConfiguration {}</pre><ul><li><strong><em>prePostEnabled</em></strong> property enables Spring Security pre/post annotations.</li><li><strong><em>securedEnabled</em></strong> property determines if the <em>@Secured</em> annotation should be enabled.</li></ul><p>Sometimes, it might so happen that your application is already implementing the custom “WebSecurityConfigurerAdapter”. So in that case, you do not need to add it again else you will start facing issues. The reason behind that is, the<em> “</em>WebSecurityConfigurerAdapter” <em>is already annotated with @order(100) and @Order</em> annotation on each static class indicates the order in which the configurations will be considered to find one that matches the requested URL.</p><p><strong>Note that, the <em>order</em> value for each class must be unique.</strong></p><p>Below code-snippet can be used just in case you wanted to define multiple entry-points. Just ensure that you are adding the order appropriately.</p><pre>@Configuration<br>@ConditionalOnMissingBean({WebSecurityConfigurerAdapter.class})<br><strong>@Order(</strong><strong>Ordered.LOWEST_PRECEDENCE)</strong><br>public class CustomWebSecurityConfigurerAdapter extends <strong>WebSecurityConfigurerAdapter</strong> {<br>    public CustomWebSecurityConfigurerAdapter() {<br>    }<br><br>    protected void configure(HttpSecurity httpSecurity) throws Exception {<br>        httpSecurity.formLogin().disable();<br>        httpSecurity.csrf().disable();<br>    }<br>}</pre><p>So you are all set so far with enabling “CustomDecisionVoter” but not written the logic for it. The below code snippet will help you do that.</p><pre>@Component<br>public class CustomDecisionVoter implements AccessDecisionVoter&lt;Object&gt; {<br>    ....<br><br>    public boolean supports(ConfigAttribute configAttribute) {<br>        return true;<br>    }<br><br>    public boolean supports(Class&lt;?&gt; aClass) {<br>        return true;<br>    }<br><br>    public int vote(Authentication authentication, Object securedObject, Collection&lt;ConfigAttribute&gt; methodConfigs) {<br>        <br>        // custom implementation goes here</pre><pre>        ......</pre><pre>        //return 0/1;</pre><pre>    }</pre><p>Focus on the vote function in the above snippet, you can get your arguments passed to @Secured annotation from the methodConfigs argument shown below is an example:</p><pre>Object[] args = methodConfigs.toArray();<br>for(Object object : args){<br>     // process the object or convert to any Java Domain form<br>}</pre><p>The vote function returns either a ‘0’ or ‘1’ which means “AccessDenied” or “AccessGranted”. In case of “AccessDenied”, you need to handle the exception in your ExceptionHandler for returning appropriate response or spring will throw the default exception “AccessDenied” and in case of “AccessGranted” it will be a pass-through and the invocation will happen seamlessly.</p><h3>Extensions</h3><p>Great! we had done our implementation for using @Secured in our own customized way of passing arguments and using them to return the respective decisions.</p><p>The next section now describes the way we can also create our own annotations and add them to method arguments level or method level to get a better control of the way we want to pass the arguments for getting the decision based on specific inputs instead of reading through all the method arguments.</p><pre>@Secured([&quot;k1:v1, k2:v2&quot;])<br>public void securedMethod(<strong>@AuthorizationInput(name=&quot;input1&quot;) String securityParameter</strong>){<br>     //Method implementation goes here...<br>     ....<br>}</pre><p>What are we doing in above snippet is, we are creating our own annotations for method argument level. For example:</p><pre>@Target({ElementType.PARAMETER, ElementType.FIELD, ElementType.METHOD})<br>@Retention(RetentionPolicy.RUNTIME)<br>@Documented<br>public @interface AuthorizationInput {<br>    String name() default &quot;input&quot;;<br>    String value() default &quot;test&quot;;<br>}</pre><p>Now, along with the parameters that have been defined in the @Secured annotation, we can write some reflection code to read through all the annotated fields as we want and use those as parameters to get the final decision instead of reading through all the arguments and having a performance hit for processing. You can read specific annotated fields with this approach. Also, you can read about <strong>@Repeatable</strong> annotations to do effective processing of your inputs annotated by custom annotations.</p><p>For example:</p><pre>@Target({ElementType.PARAMETER, ElementType.FIELD, ElementType.METHOD})<br>@Retention(RetentionPolicy.RUNTIME)<br>@Documented<br><strong>@Repeatable(AuthorizationInputs.class)</strong><br>public @interface AuthorizationInput {<br>    String name() default &quot;&quot;;<br><br>    String value() default &quot;&quot;;<br>}</pre><pre>@Target({ElementType.METHOD})<br>@Retention(RetentionPolicy.RUNTIME)<br>@Documented<br>public @interface AuthorizationInputs {<br>    AuthorizationInput[] value();<br>}</pre><p>Find below how you can pick up the custom annotated arguments in your vote method:</p><pre>public int vote(Authentication authentication, Object object, Collection&lt;ConfigAttribute&gt; methodConfigs) {</pre><pre>   // @Secured argument can be fetched <br>   // from methodConfigs as mentioned above</pre><pre>  try {<br>        MethodInvocation mi = (MethodInvocation)object;<br>        Parameter[] params = mi.getMethod().getParameters();<br>        Object[] arguments = mi.getArguments();</pre><pre>        for(int i = 0; i &lt; params.length; ++i) {<br>             Parameter param = params[i];<br>             ....<br>        }<br>    } catch (Exception e) {<br>       e.printStackTrace();<br>    }<br>    <br>    // decision making implementation</pre><pre>   return 0/1;<br>}</pre><p>That’s it. This type of implementation gives you much finer control over what can be the inputs and how you can make decisions based on inputs and get to an authorization decision.</p><h3>Summary and learning</h3><ol><li>Using existing spring-security filter chain helps in doing existing + custom authorization in a much more powerful way.</li><li>A lot of boilerplate code is avoided.</li><li>You have full control on the input you want to pass as arguments to @Secured. “k1:v1, k2:v2” is just an example for demonstration.</li><li>Custom annotations for method/parameter/field level gave much better flexibility and control on the passed input.</li><li>You decide on any number of DecisionVoters’s you want to implement.</li><li>Clean code, easy to understand and is extensible.</li></ol><p>Finally, you are all set to test your declarative authorization based on spring-security annotation.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ace1c748feeb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Import data from csv to DynamoDB using AWS SDK for Python (Boto 3)]]></title>
            <link>https://medium.com/@ravichauhan03/import-data-from-csv-to-dynamodb-using-aws-sdk-for-python-boto-3-642b25fcfd3d?source=rss-f18ad48a2325------2</link>
            <guid isPermaLink="false">https://medium.com/p/642b25fcfd3d</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[import]]></category>
            <category><![CDATA[dynamodb]]></category>
            <category><![CDATA[csv]]></category>
            <dc:creator><![CDATA[Ravi Chauhan]]></dc:creator>
            <pubDate>Wed, 13 Nov 2019 17:48:25 GMT</pubDate>
            <atom:updated>2019-11-13T17:48:25.115Z</atom:updated>
            <content:encoded><![CDATA[<p>In this post, we will see how to import data from csv file to AWS DynamoDB. There is a lot of information available in bits and pieces for various different data type import in dynamo but most of them mention the crude way of data import using Boto3 and not via csv format. This blog details out a simpler way to import bulk data from csv file to DynamoDB.</p><p><strong>Prerequisites</strong>:</p><ol><li><a href="https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html">AWS CLI</a> and <a href="https://medium.com/@rajnish_notes/boto-dynamodb-python-client-library-37624a572fed">setting up credentials</a></li><li>Python 3.x</li><li><a href="https://github.com/boto/boto3">Boto3</a> Installation</li></ol><p>Below are some screenshots from the pyCharm IDE, to help you understand the format of how you can specify the AWS credentials, format of test csv and python utility code using boto3.</p><p><strong>Specifying the credentials:</strong></p><p>Credentials are also specified as csv format. The 1st line dictates the field names which which be used for establishing connection with DynamoDb. The 2nd line will contain the actual values.</p><p>Please note, password can be left blank as access_key and access_secret is available.</p><p>for e.g:<em><br>test_dynamodb_user,,AYYYQ2KBPXXXXXXKZZZM,2WEsdtc+DfLk7sdfkDHj5OtbiU3IzXYZ/LpE98sDh,https://1234567890.signin.aws.amazon.com/console</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BIrCn6UAQS2fsO61lFYvSA.png" /><figcaption>AWS Credentials csv format</figcaption></figure><p><strong>How does the data look like:</strong></p><p>Once we have the csv file available with us, we need to append 2 line on top of the file which is mandatory for the utility to process the file.</p><p>1st line : Headers or the column names we want to keep in DynamoDB Table.</p><p>2nd line : Currently, 3 data-types are handled i.e. string, string_set and integer.</p><p>Note that, persisting string_set from csv to DynamoDB is not straight forward and the reason, why I felt to share this learning with all of you.</p><p>And, the data field in string_set is delimited by pipe(‘|’). You can also modify based on your need.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*su8H-hHWbOSl-90tWm0SDA.png" /><figcaption>Test Data format</figcaption></figure><p>Below, is the screenshot for the utility code and will write data specifically to us-west-2 region of your configured account.</p><p>Please note that, you can refer the code in the GitHub repository for which the link is provided at the bottom of the page.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JLa-snN0uyTtef33QwANBg.png" /><figcaption>Python csv import utility Code</figcaption></figure><p>Before we execute the below command, please ensure that the DynamoDB table is in place in the AWS account you are using to insert data. The table must have the same column names as mentioned in the data csv file.</p><p>Please note, any mismatch in column name(s) will result in error.</p><p><strong>&gt;&gt;&gt; python importToDynamo.py &lt;csv data file&gt; &lt;table name&gt;</strong></p><p>for e.g:</p><p><strong>&gt;&gt;&gt; python importToDynamo.py test.csv test</strong></p><p>You can refer the code in the below GitHub repository:</p><p><a href="https://github.com/ravichauhan03/ImportUtilityDynamoDB">ravichauhan03/ImportUtilityDynamoDB</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=642b25fcfd3d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Journey to migrate TB’s of data from on-premise Oracle — >DynamoDB]]></title>
            <link>https://medium.com/@ravichauhan03/journey-to-migrate-tbs-of-data-from-on-premise-oracle-dynamodb-c53aff83a90a?source=rss-f18ad48a2325------2</link>
            <guid isPermaLink="false">https://medium.com/p/c53aff83a90a</guid>
            <category><![CDATA[migration]]></category>
            <category><![CDATA[aws-dms]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Ravi Chauhan]]></dc:creator>
            <pubDate>Mon, 14 Oct 2019 17:20:23 GMT</pubDate>
            <atom:updated>2020-01-06T16:03:31.844Z</atom:updated>
            <content:encoded><![CDATA[<h3>Journey to migrate TB’s of data from on-premise Oracle — &gt;DynamoDB</h3><p>It’s a truism that we should always try to choose the right technology or tool for the job we need to do. In today’s era, it is really exciting to see technologies that can be fully managed just on a click and you can completely focus on the development efforts. But the question is:</p><p><strong>Is it that simple?</strong></p><p>So, to give away some more context to the problem, we had huge amount of data, in fact more than ~30TB. The data is completely non-transactional. Since the service which is saving this data is a legacy one, the data is getting stored in an on-premise ORACLE. So we decided to use the right data-store technology to migrate this data to, and chose a No-SQL data-store. The reason for this choice will become clear shortly. We also were ready to go through a multi-step process, crunching numbers at each step and deciding what to do next.</p><p>Many of you might be wondering why someone may ever need to move data from a relational store to a No-SQL store. The reasons for us wanting to do something like this is mentioned below:</p><ol><li>The data which we were dealing with is non-transactional and immutable</li><li>We were looking for a data-store which can scale and has high availability</li><li>We wanted to plan for scalability and avoid rework</li><li>We had no complex queries or joins</li></ol><p><strong>How does the data look like?</strong></p><p>The data-set consisted of immutable records, mainly user activity. These are preserved for a longer duration for compliance, legal and other auditing reasons. The data records are stored against the user id unique key.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wKjSiXOMjpk13m8ewf__Fw.png" /></figure><p>As we already know that the primary key in DynamoDB can have 2 parts as shown below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*t0BVu0yYqyRtHpTDRWwJVQ.png" /></figure><p>We chose the combination of user_id and created_date as the primary key for our data-set. The reason for this choice is, this enables us to obtain high cardinality in the primary key because the timestamp will be different for each row.</p><p><strong>Why we chose DynamoDB?</strong></p><p>Of course, there are multiple reasons for this choice but some of the critical ones are listed below:</p><ol><li>We were looking for a write-intensive database</li><li>We wanted a fully managed NoSQL data store. Also, we needed a data store where minimal effort is required for cross-region replication and which supports streaming</li><li>We wanted to control access to our data</li><li>We also wanted a data store that can support our organic data growth year on year</li><li>We had 95% writes and rest were reads, and hence needed good write performance</li><li>The cardinality of the primary key is already very high, as explained above, so partition didn’t seem to be a problem (We expected a uniform distribution across partitions)</li></ol><p>Keeping the above factors in mind, and given that we were already in our AWS journey, DynamoDB seemed to fit the bill.</p><p><strong>What were the post data-store selection problems?</strong></p><p>In the No-SQL world, this kind of real-world data is still a bit vague. When we started digging more deeply into the data, we figured out the below details:</p><ol><li>We needed additional filters on top of the primary key on other attributes as well</li><li>We did not know how well our reads will perform and whether it will scale or not (With the filters)</li><li>We noticed some out-liars, where data was skewed in the sense that some users/partitions had 100’s of GB’s data. We missed some skewed data for some users earlier. As seen in the graph below, some users have a huge amount of data[multiple millions of records] for a short period. This questioned our choice of sort key[whether to take only the date part or full timestamp]. After closer analysis, we chose the complete timestamp for larger variability or high cardinality.[<strong><em>Refer Graph-1 Image</em></strong>]</li><li>We were wary of hot partitions, both during data migration and at real-time when the load is high <br>[<strong><em>Yes, we saw this problem during the data migration but not during real-time. This can be avoided to some extent by picking up records for various partitions in the same batch and then calling insert to dynamo-DB.</em></strong> <strong><em>We might need to reconcile the data per use-case basis if we do this.</em></strong>]</li><li>According to the documentation, the RCU’s and WCU’s gets divided equally between all partitions. Therefore, as the data volume increases and new partitions are created, RCU and WCU per partition go down. To preserve throughput, one is obliged to add more RCU and WCUs, at extra cost. <br>[<strong><em>Earlier, this was the case, which is nicely explained in the </em></strong><a href="https://www.google.com/url?sa=t&amp;rct=j&amp;q=&amp;esrc=s&amp;source=web&amp;cd=3&amp;cad=rja&amp;uact=8&amp;ved=2ahUKEwiv69KGr_HjAhWJWX0KHSJyDcQQFjACegQIDRAI&amp;url=https%3A%2F%2Fread.korzh.cloud%2Faws-dynamodb-partitions-and-key-design-56688bee8502&amp;usg=AOvVaw3z3Q0ewiqR4TLvYW19ovxz"><strong><em>article</em></strong></a><strong><em>. But, now AWS has smartly made the default setting to be adaptive, hence your unused capacity from one partition can be used somewhere else. Also, it’s always good to have the buffer capacity provisioned, just-in-case the auto-provisioning does not happen instantly.</em></strong>]</li><li>We wondered whether DynamoDB 10GB partition limit applies here, as mentioned in the documentation. <br>[<strong><em>Learnt from exploration that, it only applies when you have an LSI (Local Secondary Index) on your table. If there is no LSI on your table, you can go beyond per partition limit in DynamoDB i.e. 10GB</em></strong>]</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iphkDOCMYTw_XSAEmN3oCA.png" /><figcaption>Graph-1: The left side of the graph shows the out-liars[Users with skewed data]</figcaption></figure><p>Moving further, we now wanted something reliable which we can use to migrate this <strong>hefty amount of data in a limited time</strong>. So, we started exploring options for moving data from on-premise Oracle to DynamoDB. Finally, we nailed it down to 3 options as mentioned below:</p><ol><li>Using AWS DMS Service.</li><li>Writing code to migrate data.</li><li>Using snowball.</li></ol><p>Since we were on tight deadlines for shutting down our on-premise data centers, we thought that AWS DMS was the most viable option. We decided to do a POC on it and verify the numbers. It was obvious that writing code from scratch to migrate such amount of data is going take a lot of time and effort and will require rigorous testing. Hence, we started doing POC with AWS DMS, keeping in mind that we need to answer the following questions:</p><ol><li>How does it handle failure events?</li><li>Does it have a retry? If yes, how many times? What happens to the data record after that?</li><li>How will we do the data reconciliation?</li><li>What happens if DynamoDB Throttles?</li><li>What are the cost implications?</li><li>How can we achieve this in limited time to migrate the complete data?</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MNue1TG17N8-tY9vOTa4sQ.png" /><figcaption>Basic architecture depicting the entities during the migration of data.</figcaption></figure><p>Since we had monthly partitions on our on-premise ORACLE table, we started moving partition by partition to our RDS store. Because of the huge volume of data, we created 3 RDS instances of 6TB each and started moving the data to those RDS instances partition by partition to achieve some parallelism(will be explained below) via Direct Connect from our on-premise store.</p><p><strong>Step by step explanation of our POC and conclusion:</strong></p><p>Initial configuration during our POC:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/432/1*M5GabF5p-hhLaZxnpZeIXw.png" /></figure><p>Observations:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/650/1*Xcg_EAcMXeBO_M8acqKRIQ.png" /></figure><p>Obviously, with this rate, we would never have been able to achieve the complete migration by the timeline we were looking at. We tried playing with some of the configurations present on AWS console for DMS which unfortunately did not help.</p><p>So, what next? We wanted to achieve more performance and hence we started using <a href="https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-using.html">AWS CLI</a>[Command Line Interface] to change the default performance settings with proper guidance from the AWS support.</p><p>What settings did we change from CLI?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/948/1*AfhJP9zA2fV1IdMNG7AsGw.png" /><figcaption>Performance Settings</figcaption></figure><p>Good news! With the aforementioned changes, we were able to verify that Dynamo DB was hitting the 1000 WCU limit. After a couple of iterations, where we ensured that we could reach the correct value without being throttled, we changed it to 6K and observed that other resources were being utilized optimally. We also observed that the data migration rate increased multiple folds, i.e. ~ 300K records/min.</p><p>Was this speed enough for us?</p><p>No, we would still have missed our deadlines with this speed. We wanted to tune the performance settings of AWS DMS further. We then tried to increase the parameters to these below-mentioned numbers:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/692/1*pjyHZoQMYdTbdKZ5IhLHlQ.png" /></figure><p>Did it change anything? Of course, we just doubled our migration rate.</p><p>But still, this was not sufficient. The migration percentage to hours ratio for just one of the partitions was 1% per hour. We wanted more speed. So far we did not see any significant improvement.</p><p>What did we do next?</p><p>We started thinking on the lines of parallel-load, increasing more threads, parallel-tasks, etc.</p><p>We started exploring options for parallel-load and figured from AWS support that it is not supported for source=oracle and target=DynamoDB. Therefore, option 1 is nipped in the bud.</p><p>Next steps….</p><p>We did some more data analysis, POC results analysis and then performed below steps:</p><ol><li>We wanted to run parallel tasks. Therefore we wanted some filters on the date to be able to pick up a particular set and start a task. But we did not have an index on the date in our oracle DB. So, we indexed our date field in RDS, to be able to run multiple DMS tasks using a filter for the date.</li><li>We further tuned the performance settings to:</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/690/1*2KrNpgAqj9B80eX_n7Ipxg.png" /></figure><p>Again, we rerun the POC with the above settings and started the DMS task and this time the numbers came out right. We have found our migration plan.</p><p>Final Migration plan:</p><ol><li>We had 3 RDS instances for entire data migration. Each partition was imported in its own table name as “<em>Data_Table_Name_&lt;month&gt;</em>”. This helped to create DMS tasks for each partition table.</li><li>We ran 1 DMS task/table with 128 threads and r4.8xlarge replication instance</li><li>We provisioned Dynamo DB with 25K WCU’s to start with and we increased it as and when required.</li></ol><p>Did we make any changes to the plan? Yes, we were observing this migration very closely and were analyzing the number as we went along. Based on the analysis we kept changing the WCU’s and load threads accordingly so as to be able to use the resources optimally. Since there is cost associated with every resource we consumed on AWS, this made sure that we optimized cost.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ehOpsUNQrtmGnipL9cp-fQ.png" /></figure><p>What about the reconciliation of data? Did the data records match exactly?</p><p>This is a mandatory step after any data migration. It’s a validation that tells us if the data is migrated properly or not. With this amount of data, one cannot obviously match character by character, but we did a random sampling of data and validated it. Also, we did not discard the previous data until we were confident that we had all the data in our brand new store.</p><p>We knew beforehand, from AWS documentation, that DynamoDB updates the record count every 6 hours. So, we waited and then checked the record counts. Alas, records did not match. This actually scared the crap out of us. After doing all of the above, we didn’t see a count match from source, RDS, and the destination, DMS records migrated and eventually written to DynamoDB.</p><p>We tried figuring out what went wrong. We cross-checked our data and figured out that DMS has some issues related to refresh of STS token. But since for us the count mismatch was not much initially, we thought we could move that data manually and will not need to rerun the whole partition migration.</p><p>We also ran into some more interesting things with AWS DMS as listed:</p><ol><li>DMS does not log any customer data value in logs and it doesn’t let one change the logging level for already running tasks for more transparency during the run. They have an option of enabling debugging which might give some details. But even if that is possible, given the size of the data, it will be a nightmare to analyze that huge file. Hence, we didn’t consider that option.</li><li>We then spoke to AWS support and they acknowledged the fact, there is some issue with DMS while refreshing credentials and they applied a fix patch on our replication instances used by DMS and finally were able to do a count match and get the details of records which were missed out by DMS without enabling debug log feature.</li><li>We did see some throttling at DynamoDB, but DMS has a feature of setting maximum no of retry in-case of failures for replaying a record.</li></ol><p>Summary &amp; Learning:</p><ol><li>Always analyze your data carefully and be watchful with the data store selection.</li><li>Be very clear about your use-cases. In the NoSQL world, you need to know your use-cases. You just cannot force fit a new use-case directly with the old model. Even if you did, you will compromise either on cost or on performance.</li><li>You have to be really crystal clear about your key selection process and data stored against that key bucket and what operations are you going to perform.</li><li>If writing service from scratch, think about the usability in the future. This means, how much data are you going to generate per Partition key. You might end up many limitations by going with ease of cloud management as there are many restrictions as well.</li><li>If moving an old ghost service and a lot of data with that to some cloud-managed NoSQL store, keep in mind that you have to do a lot of math before you do the actual start. Think about the complete migration plan and time if you are limited to that as well.</li><li>You might not be able to have the same performance for all your use-cases. Therefore, think about the priority that you want to serve and think again if you made the right choice of data store decision.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c53aff83a90a" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>