<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Meenakshi Kumari on Medium]]></title>
        <description><![CDATA[Stories by Meenakshi Kumari on Medium]]></description>
        <link>https://medium.com/@meenakshi_kumari?source=rss-831710d34eb6------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*CR-QJn8OvfEaUCtq.</url>
            <title>Stories by Meenakshi Kumari on Medium</title>
            <link>https://medium.com/@meenakshi_kumari?source=rss-831710d34eb6------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 22:36:11 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@meenakshi_kumari/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[High Performance Rate Limiting at Databricks: A Visual Guide]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@meenakshi_kumari/high-performance-rate-limiting-at-databricks-a-visual-guide-5b4c9e35eb2a?source=rss-831710d34eb6------2"><img src="https://cdn-images-1.medium.com/max/634/1*uYedMOzRn45Ospc9_Cq4KA.png" width="634"></a></p><p class="medium-feed-snippet">Understanding how Databricks scaled their rate limiting system to handle millions of requests with 10x latency improvements</p><p class="medium-feed-link"><a href="https://medium.com/@meenakshi_kumari/high-performance-rate-limiting-at-databricks-a-visual-guide-5b4c9e35eb2a?source=rss-831710d34eb6------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@meenakshi_kumari/high-performance-rate-limiting-at-databricks-a-visual-guide-5b4c9e35eb2a?source=rss-831710d34eb6------2</link>
            <guid isPermaLink="false">https://medium.com/p/5b4c9e35eb2a</guid>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[system-design-concepts]]></category>
            <category><![CDATA[token-bucket-algorithm]]></category>
            <category><![CDATA[rate-limit]]></category>
            <dc:creator><![CDATA[Meenakshi Kumari]]></dc:creator>
            <pubDate>Sat, 04 Jul 2026 05:22:34 GMT</pubDate>
            <atom:updated>2026-07-04T05:22:34.264Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Why CockroachDB Uses LSM Trees (And Why B-Trees Fail at Scale)]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-snippet">In this post we will walk through exactly how LSM trees work - from the in-memory MemTable that absorbs every write, to the immutable SST</p><p class="medium-feed-link"><a href="https://medium.com/@meenakshi_kumari/lsm-trees-demystified-how-cockroachdb-and-rocksdb-store-your-data-3a5ec87dfae4?source=rss-831710d34eb6------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@meenakshi_kumari/lsm-trees-demystified-how-cockroachdb-and-rocksdb-store-your-data-3a5ec87dfae4?source=rss-831710d34eb6------2</link>
            <guid isPermaLink="false">https://medium.com/p/3a5ec87dfae4</guid>
            <category><![CDATA[lsm-tree]]></category>
            <category><![CDATA[cockroachdb]]></category>
            <category><![CDATA[backend-engineering]]></category>
            <category><![CDATA[distributed-system-design]]></category>
            <category><![CDATA[database-internals]]></category>
            <dc:creator><![CDATA[Meenakshi Kumari]]></dc:creator>
            <pubDate>Tue, 28 Apr 2026 13:30:02 GMT</pubDate>
            <atom:updated>2026-04-28T13:55:00.432Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[How CockroachDB Stores Data: KV Engine Explained with Real Internals]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-snippet">Foundation: Everything Is KV</p><p class="medium-feed-link"><a href="https://medium.com/@meenakshi_kumari/crdb-key-value-storage-deep-dive-dee0f2d088b5?source=rss-831710d34eb6------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@meenakshi_kumari/crdb-key-value-storage-deep-dive-dee0f2d088b5?source=rss-831710d34eb6------2</link>
            <guid isPermaLink="false">https://medium.com/p/dee0f2d088b5</guid>
            <category><![CDATA[pebble]]></category>
            <category><![CDATA[kv-store]]></category>
            <category><![CDATA[index]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[cockroachdb]]></category>
            <dc:creator><![CDATA[Meenakshi Kumari]]></dc:creator>
            <pubDate>Tue, 28 Apr 2026 12:01:26 GMT</pubDate>
            <atom:updated>2026-04-28T13:52:57.912Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Interview with GOJEK]]></title>
            <link>https://medium.com/@meenakshi_kumari/interview-with-gojek-d56d9f3f1e73?source=rss-831710d34eb6------2</link>
            <guid isPermaLink="false">https://medium.com/p/d56d9f3f1e73</guid>
            <category><![CDATA[gojek]]></category>
            <category><![CDATA[interview]]></category>
            <dc:creator><![CDATA[Meenakshi Kumari]]></dc:creator>
            <pubDate>Sat, 14 Aug 2021 09:08:22 GMT</pubDate>
            <atom:updated>2021-08-14T09:08:22.655Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*fkjlRmJZsBxgTpouRhuwdQ.gif" /></figure><p>In this blog, I’m sharing my interview experience with GoJek. Many people have asked me about the process, team and culture of the company.</p><p>One day was watching youtube and came across this <a href="https://www.youtube.com/watch?v=7YNlkYqzqYI">TVF video</a> and got to know about job openings in GOJEK. I applied and my resume got shortlisted This blog will take you through my interview journey.</p><p><strong>Resume Selection</strong></p><p>I knew that GoJek focuses more on open sources and our blogs. I had written a few blogs and was doing some coding in ruby and kept my Github account updated. I did mention my blogs and GitHub link in my resume. (Trust me it will help you alot😀.)</p><ol><li>Try to make a one-page resume.</li><li>Highlight only recent and important points.</li><li>Explain your projects in short and mention only relevant information.</li></ol><p>Luckily I got a call from the recruiter from their team and was asked about my experience and they explained to me their requirement. And then my first round of code pairing was arranged.</p><p><strong>Code Pair Round</strong></p><p>It was my first round and was of two hours long. They gave me a ticket management system and I was given 20 min of time to read the whole problem.</p><p>Tips:</p><ol><li>Read the problem carefully. At the start, it looked very intimidating as it was 3–4 pages long. But take a deep breath😋 and keep a pen and notepad with you and start jotting down the points which are will help you in designing your approch.</li><li>The problem was very well defined and if you have any doubts ask interviewers.</li><li>You need to show your OOPs concept and clean code. You can code in any language you are comfortable with.</li><li>Keep on expressing whatever approach you are following and whatever you are thinking to do. Just to keep the panel in sync. This also helps you if you are going off track or doing any mistakes. Pannel is very helpful. and trust me you will learn a lot from them.</li><li>Try to go step by step as you will not be able to finish the whole problem but try to incorporate as much as possible.</li></ol><p><strong>Database and System Design Round</strong></p><p>My next two rounds were arranged after a week of my first round. Each interview was an hour-long(may exceed depends). I faced some network glitches but the interviewer was very cooperative and patient. In my second round, I was asked about my previous projects and had a very good discussion regarding one of my project role. Then he asked me to design a database for a system like Book my Show. He asked me to design a very simple DB and also asked a few SQL queries. The interviewer was very understanding and clarified any doubt which I had.</p><p>In my third interview, he asked me few questions related to my resume and some question related to ruby and goroutine. Then he asked me to system design for Twitter.</p><p>Tips: Try not to complicate the problem statement and keep on expressing whatever is your next step.</p><p><strong>Bar Raiser Round</strong></p><p>My last round was the bar raiser round which was with the hiring manager of the team for which I was interviewed. You will be asked about your current work, what you do in your spare time, and how is your learning process. They wanted to see wheater you are the right fit for the company and their team.</p><p>In this whole process, I learned a lot during the process. The team was very cooperative and they mail and call to check for your availability and will always get back to you if you have any doubts. Things went well and I got the offer call☺️. Will be joining the company next month. I’m very excited to start my journey as I was always fascinated with Gojek. In the Rails conference, I used to solve the questions to get the swags of Gojek 😋. I’m glad that I’ll be part of this super app team.</p><p>Update: It’s been 8 months of working here in GoJek. I have joined the identity team here. It’s been an amazing journey so far. I have learnt so much in these past months. Trust me you won’t regret joining this organization. Glad to be part of this team :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d56d9f3f1e73" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[LRU with Ruby]]></title>
            <link>https://medium.com/@meenakshi_kumari/lru-with-ruby-a0ffb5599496?source=rss-831710d34eb6------2</link>
            <guid isPermaLink="false">https://medium.com/p/a0ffb5599496</guid>
            <category><![CDATA[lru-cache]]></category>
            <category><![CDATA[lru]]></category>
            <category><![CDATA[datastrucutre]]></category>
            <category><![CDATA[ruby]]></category>
            <dc:creator><![CDATA[Meenakshi Kumari]]></dc:creator>
            <pubDate>Fri, 23 Oct 2020 15:44:23 GMT</pubDate>
            <atom:updated>2020-10-23T15:44:23.889Z</atom:updated>
            <content:encoded><![CDATA[<p>Have implemented a Cache replacement policy — Least Recent Policy with Ruby.</p><h4><strong>The cache can hold a fixed number of items, in LRU we remove the least recently used item in the cache to make room for the new item.</strong></h4><p>The data structure used here is a Hash for fast lookup and Doubly Linked List(DLL) for the fast removal of the least recently used items.</p><p>Hash key’s values will be nodes within the doubly linked list. In DLL the least recently used items will be head and the most recently used will be tail. The node will contain the value of the node and previous and next node object address.</p><pre># Node structure for the DLL</pre><pre>class Node<br>  attr_accessor :prev_node, :next_node, :value<br>  def initialize(value, prev_node, next_node)<br>    @value = value<br>    @next_node = next_node<br>    @prev_node = prev_node<br>  end<br>end</pre><p>Now a single node of the DLL is defined, now have to declare and initialize the LRU class where will be making head and tail as nil. And will initialize the lookup table as an empty hash and the current size of the table as 0. Will be accepting the maximum size of the cache from the user. Code is given below:</p><pre>class Lru<br>  attr_accessor :head, :tail, :current_size, :max_size</pre><pre>  def initialize(max_size)<br>    @head = nil<br>    @tail = nil<br>    @current_size = 0<br>    @max_size = max_size<br>    @lookup_table = {}<br>  end<br>end</pre><p>The LRU class will have two methods get() and set() which are defined as follows.</p><p><strong>get(x)</strong>: Gets the value of the key x if the key exists in the cache otherwise returns -1. This method is implemented below:</p><p>If the key is at the tail then will return its value otherwise we need to move that node to the tail as it is recent used item.</p><pre>def get(key)<br>  result = @lookup_table[key]<br>  return -1 unless result</pre><pre>  #result is at tail then return value as it is recently used item<br>  return result.value if @tail == result</pre><pre>  if result.prev_node == nil<br>    @head = result.next_node<br>    @head.prev_node = nil<br>  else<br>    result.prev_node.next_node = result.next_node<br>    result.next_node.prev_node = result.prev_node<br>  end<br>  @tail.next_node = result<br>  result.next_node = nil<br>  result.prev_node = @tail<br>  @tail = result<br>  result.value<br>end</pre><p><strong>set(x,y)</strong>: Inserts the value if the key x is not already present. If the cache reaches its capacity it invalidates the least recently used item before inserting the new item. s the value of the key x if the key exists in the cache otherwise returns -1. This method is implemented below:</p><pre>def set(key, val)<br>  @current_size += 1<br>  # if current size exceeds, remove least recent item (i.e. head)<br>  if @current_size &gt; @max_size <br>    # remove least recent key values pair from lookup table<br>    @lookup_table.delete(@lookup_table.key(@head))      <br>    @head = @head.next_node # then change the head of DLL<br>    @head.prev_node = nil<br>  end<br>  new_node = Node.new(val, @tail, nil)<br>  if @tail == nil # when new node is very first element in the Dll<br>    @head = new_node<br>  else<br>    @tail.next_node = new_node<br>  end<br>  @tail = new_node<br>  @lookup_table[key] = new_node<br>end</pre><p>For the whole code, you can check on <a href="https://github.com/meenakshikumari/LRU-cache">Github</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a0ffb5599496" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[My Journey from No SQL to SQL(Part 3)-The final touch up]]></title>
            <link>https://medium.com/@meenakshi_kumari/https-medium-com-meenakshi-89695-my-journey-from-nosql-to-sql-part3-9d7e2ae65afc?source=rss-831710d34eb6------2</link>
            <guid isPermaLink="false">https://medium.com/p/9d7e2ae65afc</guid>
            <category><![CDATA[data-migration]]></category>
            <category><![CDATA[ruby-on-rails]]></category>
            <category><![CDATA[postgresql]]></category>
            <category><![CDATA[sequels]]></category>
            <category><![CDATA[mongodb]]></category>
            <dc:creator><![CDATA[Meenakshi Kumari]]></dc:creator>
            <pubDate>Sun, 24 Mar 2019 05:19:32 GMT</pubDate>
            <atom:updated>2019-03-24T05:19:32.987Z</atom:updated>
            <content:encoded><![CDATA[<p>In my <a href="https://medium.com/@meenakshi_89695/https-medium-com-meenakshi-89695-bon-voyage-to-new-chapter-in-project-part-2-da1d282f8773">last blog</a> i had shared my experience of data importing rake task. In this blog i’ll cover the changes after all the data import on PostgreSQL side and problems faced before launching updated project.</p><h3>Changes after data import on PostgreSQL side</h3><p>After all the data import tasks were done and data was migrated their were still few of the cases which were not handled by sequel scripts. Few of the cases are as follows:</p><ol><li>We had a field in a table which stored hash with key as string and value as integer, but as we have hstore as data type so by default it stores value in string only. Therefore we have to change our code accordingly that whenever we are reading this record the value should get converted into integer.</li><li>We had a hash record in a table which stored key as id for other associated table and some string value. But as our id were mongoid, therefore we have to write a rake task to update the mongoid to postgres id.</li><li>One of the major issue we faced was with a array type field containing date elements. Here we have to store this record in relation-ids of that particular table and on postgresql side we dealt with it. We updated_column by splitting the array and mapping it to_date.</li><li>We had fields storing image url containing unique mongo_id, which we had to replace with postgresql id.</li><li>One of the issue was related to Single Table Inheritence (STI), we had a situation were we have a table which has STI relation with another table. So for mongo data both the table contains “_type” field values but when a new activerecord was made i.e. a new record in postgres side was setting the child table with type field but not the parent table, so the type field was set to nil for parent. So in schema we have to set the default value for type field to the parent.</li><li>Mongo embedded relations was also had a major change as active record don’t have such association, therefore all the embedded relation were changed to has_one belongs_to association. And as embedded table creates a object for it’s parent, so on postgres side we have to build a new object for that embedded child explicitly.</li><li>Used money-rails gem to deal with the money fields and handled the code changes accordingly.</li></ol><h3><strong>Conclusion</strong></h3><p>After 3–4 months of hard work we finally achieved a system with latest versions of Ruby and Rails and database migrated to Postgres. The speed of our system have increased to almost 50% and even the database size have also reduced from 12GB to 7GB.</p><p>I have tried my best to cover my all experience of this major transition in these blogs. Hope i have done justice with your time. Thanks for your patience.</p><p>For more such interesting stuff stay connected with me :) Thanks for your time.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9d7e2ae65afc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[My journey from NoSQL to SQL(Data Transfer)]]></title>
            <link>https://medium.com/@meenakshi_kumari/https-medium-com-meenakshi-89695-bon-voyage-to-new-chapter-in-project-part-2-da1d282f8773?source=rss-831710d34eb6------2</link>
            <guid isPermaLink="false">https://medium.com/p/da1d282f8773</guid>
            <category><![CDATA[data-migration]]></category>
            <category><![CDATA[mongo]]></category>
            <category><![CDATA[sequels]]></category>
            <category><![CDATA[postgresql]]></category>
            <category><![CDATA[ruby-on-rails]]></category>
            <dc:creator><![CDATA[Meenakshi Kumari]]></dc:creator>
            <pubDate>Wed, 21 Nov 2018 13:26:00 GMT</pubDate>
            <atom:updated>2020-09-28T14:16:23.311Z</atom:updated>
            <content:encoded><![CDATA[<p>In my <a href="https://medium.com/@meenakshi_89695/my-journey-from-nosql-to-sql-56b5882d049f">last blog</a>, I have shared my experience of gem changes and preparation of schema for PostgreSQL database. In this blog i’ll be covering the data transfer process from MongoDB to PostgreSQL database without any inconsistency. And will also cover what all challenges we faced and their solutions.</p><h3>Data transfer from MongoDB to PostgreSQL database.</h3><p>As our project was updated with PostgreSQL and database schema prepared, the next hurdle in-front of us was how can we import the data of MongoDB to PostgreSQL without affecting the data quality and disturbing the old data.</p><p>After lot of research, we found gem <strong>“</strong><a href="http://sequel.jeremyevans.net/rdoc/"><strong>Sequel</strong></a><strong>”</strong>.<strong> </strong>It is a simple, flexible, and powerful SQL database access toolkit for Ruby. It basically has ability to connect to different type of databases (in our case MongoDB &amp; PostgreSQL), read/write to those connected databases etc. SEQUEL can do all this because it includes a comprehensive Object Relational Mapping (ORM) layer for mapping records to Ruby objects and handling associated records. We used it like connect to destination PostgreSQL DB to write data which is read from our MongoDB.</p><p>So let’s start with the data transfer, steps which were followed are:</p><ol><li>Switch to the MongoDB branch and there we need to install sequel and pg gem. <em>In my previous blog I had mentioned that we had two separate branches for the MongoDB and PostgreSQL code in the same GitHub project repository. All this data transfer was done on the MongoDB branch.</em></li><li>Now we have to write rake tasks for data mapping. I’ll be taking the previous blog’s table example i.e. <a href="https://medium.com/p/56b5882d049f#131c">Address</a>. You have already seen the table schema in the previous section, now i’ll show how to write rake task for data import for this particular table using Sequel. In the below rake task following this will be covered:</li></ol><ul><li><strong>Database connection</strong> is established using :</li></ul><pre>DB = Sequel.postgres(‘database_name’, user: ‘username’, password:     ‘password’, host:‘localhost’, port: port_number)</pre><ul><li><strong>Extensions for </strong><a href="http://sequel.jeremyevans.net/rdoc-plugins/files/lib/sequel/extensions/pg_hstore_rb.html"><strong>pg_hstore</strong></a><strong> and </strong><a href="http://sequel.jeremyevans.net/rdoc-plugins/files/lib/sequel/extensions/pg_array_rb.html"><strong>pg_array</strong></a> are added to support the PostgreSQL hstore and array type to Sequel.</li></ul><pre>DB.extension :pg_array<br>DB.extension :pg_hstore</pre><ul><li>Loop over table in batches (for fast execution) and then make a hash containing <strong>one to one mapping</strong> for MongoDB record fields with respective PostgreSQL table fields.</li><li>In below rake task you will see “safe_task” method, this is to handle exceptions and throw error message. Below is the code for safe_task;</li></ul><pre>def safe_task(&amp;proc)<br>  begin<br>    yield<br>    return true<br>  rescue StandardError =&gt; e<br>    puts &quot;Exception Message: #{e.message}&quot;<br>    puts &quot;Exception Class: #{e.class.name}&quot; <br>  end<br>  false<br>end</pre><ul><li><strong>Handle associations</strong>: In our table, address belongs_to company and company has_many address. To store company_id inside address table we have to import company records before address data. And then we will fetch associated company_id from the PostgreSQL company table and update the address table accordingly and also we will be keeping the mongo_id for the same company in relation_ids field of address.</li></ul><pre>## Retrieve associated company data from the PostgreSQL database<br>company = DB[:companies].select(:id).where(mongo_id: address.company_id.to_s).first</pre><ul><li><strong>Hstore and array fields</strong> are firstly mapped directly to their PostgreSQL fields and as they cannot be directly inserted into the tables through Sequel, we have handled them as shown in the following part of code:</li></ul><pre>## Delete keys who has empty array values <br>mapping.delete_if { |key, value| value.class == Array &amp;&amp; value.empty? }</pre><pre>## Edit hash values        <br>mapping.each do |key, value| <br>  Hash.include?(value.class) &amp;&amp; mapping.update(key =&gt; Sequel.hstore(value)) <br>end</pre><ul><li>In the last we have to insert the mapping hash inside the PostgreSQL database.</li></ul><pre>record_no = DB[:addresses].insert(mapping)</pre><p>Below is the consolidated rake task code for the Address table;</p><pre>desc &#39;Address  Data Migrate&#39; <br>task :addresses =&gt; :environment do</pre><pre>  ## Establish Connection<br>  ## Add extension for hstore and array </pre><pre>  ## Loop over address table and start one-to-one field mapping<br>    ## Retrieve associated company data from the PostgreSQL database<br> <br>    task = safe_task do<br>      mapping = {  <br>        mongo_id:    address.id.to_s, ## Actual table mongo_id<br>        ## Map PostgreSQL fields: MongoDB fields<br>        company_id:  company[:id], ## Storing PostgreSQL parent_id</pre><pre>        ## For storing associated table mongo_id<br>        relation_ids: {  <br>          company_id: ## Store company mongo_id<br>        }<br>      }</pre><pre>      ## Handle hstore and array fields(if any) before insertion</pre><pre>      ## Insert record          <br>      record_no = DB[:addresses].insert(mapping)<br>    end<br>  end<br>end</pre><p>You have observed following things in the above rake task:</p><ul><li>“mongo_id” and “relation_ids” fields are used as was mentioned in my <a href="https://medium.com/p/56b5882d049f#4251">previous blog</a>, and we are storing the mongodb id into mongo_id for the record and associated tables mongo_id in relation_ids for future reference. So if anything goes wrong while data transfer task it can be handled while checking these fields and also for cross checking consistency of the data records which are transferred from MongoDB to PostgreSQL.</li></ul><p><em>TIP: You can print number of the records which are imported with their respective mongo_id’s and reason of the failed record insertions(if any) just to keep track of the task status.</em></p><p><strong>NOTE: We needed to store the PostgreSQL id of the parent table into their child tables, so the data for parent is needed to be populated first in the database that’s why we started with the rake tasks of parent tables and then moved on to their children and the process continues till the leaf tables. And the same sequence we followed while running these rake tasks.</strong></p><p><strong>One of the biggest challenge we faced was</strong>: we had a table in our system with more then 10 millions of records, and our rake task was taking more then 30 hours for data importing and was getting break at many points due to ‘cursor not found error’ and “SIGHUP” as we were trying to fetch the data in batches over the huge table records. It wasn’t getting imported in a single flow as it was taking to much time to load the whole table. We even tried loading in batches but that also didn&#39;t helped .</p><p><strong>Solution</strong>: Initially we were iterating over the whole table and then mapping their fields and then inserting. Instead of this we started with the up to down approach. Let me explain following example;</p><pre>class Grandparent &lt; ApplicationRecord<br>  has_many :parents<br>end</pre><pre>class Parent &lt; ApplicationRecord<br>  has_many   :children<br>  belongs_to :grandparent<br>end</pre><pre>class Child &lt; ApplicationRecord<br>  belongs_to :parent<br>end</pre><p>Let’s say we have to transfer ‘Child’ table records from MongoDB to PostgreSQL database. So instead of directly looping over the ‘Child’ table, we will start with the grandparent table and then for that particular grandparent we will loop over their parents and then proceed to their children. Have a look into below pseudo code for more explanation;</p><pre>desc &#39;Child  Data Migrate&#39; <br>task :children =&gt; :environment do</pre><pre>  puts &quot;Rake task &#39;children&#39; start time #{Time.now}&quot;</pre><pre>  ## Establish Connection</pre><pre>  ## Add extension for hstore and array </pre><pre>  total_count = Child.count<br>  success_count = 0</pre><pre>  Grandparent.each do |grand_parent|<br>    grandparent.parents.each do |parent|<br>      parent.children.each do |child|<br>        ## Map all the fields of Child table and then insert into DB<br>      end<br>    end<br>  end<br>end</pre><p>We followed the above approach and this reduced the time for data import to 5 hours. And we were done with our data transfer. Sequel really made this whole procedure very easy for us.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*-qdHlTZTBY_-xelL7RJueg.gif" /></figure><p>After this phase we faced many issues related to the updated versions of Ruby on Rails and changed database, which i’ll cover in my next part of this <a href="https://medium.com/@meenakshi_kumari/https-medium-com-meenakshi-89695-my-journey-from-nosql-to-sql-part3-9d7e2ae65afc">blog</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=da1d282f8773" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[My journey from NoSQL to SQL]]></title>
            <link>https://medium.com/@meenakshi_kumari/my-journey-from-nosql-to-sql-56b5882d049f?source=rss-831710d34eb6------2</link>
            <guid isPermaLink="false">https://medium.com/p/56b5882d049f</guid>
            <category><![CDATA[ruby-on-rails]]></category>
            <category><![CDATA[data-migration]]></category>
            <category><![CDATA[mongodb]]></category>
            <category><![CDATA[postgresql]]></category>
            <dc:creator><![CDATA[Meenakshi Kumari]]></dc:creator>
            <pubDate>Wed, 21 Nov 2018 13:24:51 GMT</pubDate>
            <atom:updated>2020-09-28T14:18:41.632Z</atom:updated>
            <content:encoded><![CDATA[<p>My project is an B2B(business-to-business) website, where vendors can sell their products to shopkeepers directly or with the help of agents and vice-versa. When I started with this project, it had MongoDB database(NoSQL) along with Rails(v4.1.1) framework written in Ruby(v2.1.0).</p><p>MongoDB is a fast NoSQL database. Unfortunately, it’s wasn’t the cure for all the performance troubles. Many issues regarding our site unavailability were reported which was caused due to slow querying on associations and indexes by MongoDB. One particular case was: we were exporting reports in our site, which was retrieving data from many associated and embedded documents in database, which was very slow process. Data update and create tasks were taking more time because of complex transactions over highly associated data. MongoDB is not ACID compliance , consistency and availability are incompatible in Mongo due to the CAP theorem.<em>(tip: </em><a href="https://dzone.com/articles/how-acid-mongodb"><em>MongoDB ACID compiliance</em></a><em>, </em><a href="http://www.sitepoint.com/sql-vs-nosql-differences/"><em>NoSQL vs SQL</em></a><em>)</em></p><p>So we wanted to switch to database which had transaction support and is ACID compliance. To enhance our website speed and availability, our team decided to migrate project database to PostgreSQL. It is an object-relational database management system (ORDBMS) with an emphasis on extensibility and also supported NoSQL features. Along with database migration we also upgraded our Ruby on Rails versions.</p><p>After this successful migration, our site availability and resilience improved as PostgreSQL performed much better for indexes and joins and our service became faster and snappier as a result. And also our database size reduced since it stores information more efficiently.</p><p>I’ll be sharing my experience of project migration in following series of blogs:</p><ol><li><a href="https://medium.com/p/56b5882d049f#4251">Gem changes and Preparation of schema for PostgreSQL database.</a></li><li><a href="https://medium.com/p/da1d282f8773#2550">Data transfer from MongoDB to PostgreSQL database, without any inconsistency.</a></li><li><a href="https://medium.com/@meenakshi_kumari/https-medium-com-meenakshi-89695-my-journey-from-nosql-to-sql-part3-9d7e2ae65afc">Problems faced before launching updated project</a>.</li></ol><p>In this blog i’ll be explaining how to update Gemfile and also the schema designing for PostgreSQL from the MongoDB.</p><p>So come along with me on my journey of this migration.</p><h3>Gem changes and Preparation of schema for PostgreSQL database.</h3><p><strong>NOTE: We had two separate branches for the MongoDB and PostgreSQL code in the same GitHub project repository.</strong></p><p>We have to update our project Gemfile to PostgreSQL by replacing all mongo related gems by pg gems, for eg:</p><ul><li>mongoid with pg</li><li>mongoid_tree with ltree_hierarchy</li><li>mongoid_search with pg_search</li><li>mongoid_observers with rails_observers</li><li>mongoid_audit with audited</li><li>carrierwave_mongoid with carrierwave</li></ul><p>Next step was, preparation of a schema for our PostgreSQL database from the MongoDB collection. Replace mongoid.yml file with database.yml file and create database using <strong>rake db:create</strong> command. We have to make several changes in the data type, relations, etc in PostgreSQL database, some of them are as follows:</p><ol><li>Symbol type field of MongoDB document was changed to string and while retrieving the data from DB it was to be converted to_sym explicitly.</li><li>MongoDB has ‘embeds_one ’, ‘embeds_many’ relation which was converted to ‘has_one’ ‘has_many’’ relation in PostgreSQL. For example:</li></ol><pre>##### MONGODB CODE #####<br>class Company<br>  embeds_many :addresses<br>end</pre><pre>##### POSTGRESQL CODE #####<br>class Company &lt; ApplicationRecord<br>  has_many :addresses, dependent: :destroy<br>end</pre><p>3. For has_and_belongs_to relation, a third table in schema was created in PostgreSQL, where both the tables ids was stored. For example:</p><pre>##### MONGODB CODE #####<br>class Company<br>  has_and_belongs_to_many :users<br>end</pre><pre>class User<br>  has_and_belongs_to_many :companies<br>end</pre><pre>##### POSTGRESQL CODE #####<br>class Company &lt; ApplicationRecord<br>  has_and_belongs_to_many :users, association_foreign_key: &#39;user_id&#39;, <br>    join_table: &#39;companies_users&#39;<br>end</pre><pre>class User &lt; ApplicationRecord<br>  has_and_belongs_to_many :companies, join_table: &#39;companies_users&#39;<br>end</pre><pre>##### POSTGRESQL SCHEMA FOR COMPANIES_USERS TABLE #####<br>create_table &quot;companies_users&quot;, force: :cascade do |t|<br>  t.bigint &quot;company_id&quot;<br>  t.bigint &quot;user_id&quot;<br>  t.index [&quot;company_id&quot;], name: &quot;index_companies_users_on_company_id&quot;<br>  t.index [&quot;user_id&quot;], name: &quot;index_companies_users_on_user_id&quot;<br>end</pre><p><strong>NOTE: </strong><a href="http://www.mccartie.com/2016/12/05/rails-5.1.html"><strong>Primary key default was changed from Integer to BIGINT for PostgreSQL from rails 5.1</strong></a><strong>.</strong></p><p><em>TIP: To check that correct data is imported from mongodb to PostgreSQL, we stored the mongo_id of the imported mongo record in a string field named </em><strong><em>mongo_id </em></strong><em>and mongo_ids of all the associated tables of that record, in field called </em><strong><em>relation_ids </em></strong><em>which is of type hstore. So if anything goes wrong while data transfer task it can be handled while checking these fields and also for cross checking consistency of the data records which are transferred from MongoDB to PostgreSQL. Both of these records are for future reference and they can be removed later when you are sure about the imported data.</em></p><p>Likewise we had to write migrations for each table in order to prepare our schema. Sample example for migration and corresponding schema table of Address is as follows:</p><pre>##### MONGODB MODEL #####<br>class Address<br>  field :flat_no, type: Integer<br>  field :pincode, type: Symbol<br>  field :city,  type: String<br>  field :state,  type: String, default: ‘’</pre><pre>  belongs_to :company<br>end</pre><pre>##### POSTGRESQL MIGRATION #####<br>class CreateAddresses &lt; ActiveRecord::Migration[5.2]<br>  def change<br>    create_table :addresses do |t|<br>      t.integer     :flat_no<br>      t.string      :pincode<br>      t.string      :city <br>      t.string      :state , default: ‘’</pre><pre>      ## newly introduced fields ##<br>      t.string      :mongo_id, default: ‘’<br>      t.hstore      :relation_ids, default: {}</pre><pre>      t.timestamps</pre><pre>      t.belongs_to  :company, index: true<br>    end<br>  end<br>end</pre><p>Now our schema was ready and we were all geared up for next phase for this procedure which was data transfer from MongoDB to PostgreSQL database which i’ll explain in my next part of the blog.</p><p>Do read my next blog <a href="https://medium.com/@meenakshi_89695/https-medium-com-meenakshi-89695-bon-voyage-to-new-chapter-in-project-part-2-da1d282f8773">here</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=56b5882d049f" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>