<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Rajneesh Mitharwal on Medium]]></title>
        <description><![CDATA[Stories by Rajneesh Mitharwal on Medium]]></description>
        <link>https://medium.com/@rajneesh?source=rss-2fb93e0f01e5------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*vkm6tM_oo1dLyRMaZrXXew.png</url>
            <title>Stories by Rajneesh Mitharwal on Medium</title>
            <link>https://medium.com/@rajneesh?source=rss-2fb93e0f01e5------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 11:11:51 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@rajneesh/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Storing hundreds of millions of simple keys in 282 MB with Redis]]></title>
            <link>https://medium.com/the-work-we-do-hikeapp/storing-hundreds-of-millions-of-simple-keys-in-282-mb-with-redis-2142b7c52cc9?source=rss-2fb93e0f01e5------2</link>
            <guid isPermaLink="false">https://medium.com/p/2142b7c52cc9</guid>
            <category><![CDATA[redis]]></category>
            <dc:creator><![CDATA[Rajneesh Mitharwal]]></dc:creator>
            <pubDate>Thu, 25 Jun 2015 16:53:51 GMT</pubDate>
            <atom:updated>2016-01-20T07:14:15.486Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TLfl-TrTdvmAodQUFhfBFw.png" /></figure><p>To start off with, I am Rajneesh, an engineer at Hike Messenger. Today, hike is the fastest growing messaging app in India.</p><p>In this blog I will be talking about how important it is to choose right <strong><em>Redis </em></strong><em>data structure</em><strong><em> </em></strong>— In-memory key–values store for best performance and minimal memory usages.</p><p>We are using Redis extensively at Hike and it is core of our caching layer backed by persistent databases like MySql, MongoDB, Hbase etc. We also use Redis as persistent storage for some of use cases.</p><p>For one of our use cases we tried different different approaches and Redis data structures to store data with minimum memory consumption without compromising performance. And the results motivated me to share my findings with the community. So lets first understand our use case.</p><p>Hike is mobile messaging application where one hike user can send unlimited Instant Messages to another hike user as well as non-hike user via SMS. At the same time, there are <a href="https://en.wikipedia.org/wiki/Telecom_Regulatory_Authority_of_India">TRAI</a> rules &amp; regulations about sending SMS to DND (Do Not Disturb) users in India. So we need to query whether a given user ( <a href="http://en.wikipedia.org/wiki/MSISDN">MSISDN</a> with only digits) is on DND or not with minimum latency and we decided to use redis for it. There are approximately 250 millions users registered on DND in India. <strong><em>Well that’s very huge data to deal with. isn’t it?</em></strong></p><p>A very straight forward approach would have been using simply key — value store in following manner</p><pre>&gt; redis-cli set 9199717880XX 1<br>OK<br>&gt; redis-cli exists 9199717880XX<br>(integer) 1 </pre><p>but <a href="http://instagram-engineering.tumblr.com/post/12202313862/storing-hundreds-of-millions-of-simple-key-value"><strong>Instagram’s blog on redis</strong></a><strong> </strong>and Redis’s own <a href="http://redis.io/topics/memory-optimization"><strong>documentation</strong></a> on the same saved our time and we tried to use hashes to store this whole data.</p><p>Hashes in Redis are dictionaries that are can be encoded in memory very efficiently; the Redis setting ‘<em>hash-zipmap-max-entries</em>’ configures the maximum number of entries a hash can have while still being encoded efficiently. We didn’t modify this value ( which is 64 by default for Redis server &gt;= 2.6 ) as setting higher value would have caused HSET commands to higher CPU usages.</p><p>We decided to make the buckets of maximum 10000 fields in a hash with the assumption that all users in 0–9999 range with common starting prefixes won’t be DND users and average size of a bucket will be nearly 500.</p><p>Data in Redis for the same msisdn example looks like following</p><pre>&gt; redis-cli hset 91997178 80XX 1<br>OK<br>&gt; redis-cli hexists 91997178 80XX<br>(integer) 1</pre><p>Using this approach, we were able to store the whole data in 14GB memory on Amazon EC2 m3.xlarge instance. But storing <em>dummy value as 1</em> with each entry means a lot of wastage of costly memory. So we looked at a Redis bitmap where you can manipulate bits at given position. We found that this perfectly suits our use case where we simply need to check the presence of a member.</p><p>So taking the same example again our data looks like following</p><pre>&gt; redis-cli setbit 91997178 80XX 1<br>(integer) 0<br>&gt; redis-cli getbit 91997178 80XX<br>(integer) 1</pre><p>Results of using this approach were amazing. We are now able to store same data in just 282 MB. <strong><em>Sound’s great, right ? </em></strong><em>and memory consumption will not increase linearly as number of DND users increases. Currently there are </em>189,692 buckets so we are able to store ~1.8 Billion users’ information using the same memory :-).</p><p>Time complexity of member presence check ( <a href="http://redis.io/commands/getbit">GITBIT</a>) and add/delete of member (<a href="http://redis.io/commands/setbit">SETBIT</a>) is still O(1).</p><p>One may suggest using <a href="http://redis.io/topics/data-types-intro#sets">Redis Sets</a> but our requirement wasn’t taking advantages of any set operations and don’t you think it is wastage of memory also where keys contain<em> </em><strong><em>common prefixes?</em></strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2142b7c52cc9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/the-work-we-do-hikeapp/storing-hundreds-of-millions-of-simple-keys-in-282-mb-with-redis-2142b7c52cc9">Storing hundreds of millions of simple keys in 282 MB with Redis</a> was originally published in <a href="https://medium.com/the-work-we-do-hikeapp">The work we do at hike messenger</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How we built our entire testing environment in one box using Docker]]></title>
            <link>https://medium.com/the-work-we-do-hikeapp/hike-server-in-a-box-one-box-dev-testing-environment-ff80ac5d6e5d?source=rss-2fb93e0f01e5------2</link>
            <guid isPermaLink="false">https://medium.com/p/ff80ac5d6e5d</guid>
            <category><![CDATA[hike-messenger]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[testing]]></category>
            <dc:creator><![CDATA[Rajneesh Mitharwal]]></dc:creator>
            <pubDate>Wed, 06 May 2015 15:04:08 GMT</pubDate>
            <atom:updated>2016-01-20T07:16:40.944Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kHZ-bVe_DRxLZOY0Dvmbwg.jpeg" /></figure><h2><strong>Building our entire testing environment in one box using Docker</strong></h2><p>To start off with, I’m Rajneesh, an engineer at <a href="http://get.hike.in/">Hike Messenger</a>. Today, hike is the fastest growing messaging app in India.</p><p>In this blog I will cover how we created a ‘Server in a Box’; a one-box sandbox environment using <a href="https://www.docker.com/">Docker</a> that runs the entire hike setup consisting of 20+ services. Not only did this simplify life for developers, it has helped us move faster than ever before.</p><h3><strong>Problems We Were Trying to Solve</strong></h3><h4>1. <strong><em>New Environment Setup</em></strong></h4><p>Our local development environment setup was a total nightmare. Hike architecture consists of various services; Redis, Mongodb, Mysql, Python, Java amongst others. Configuring all these services in terms of the right versions, right drivers, the order in which they should be brought up, was a error prone process and simply put, really tedious and time consuming. It would require almost a day or more if any platform specific issue surfaced. Every single developer environment inside hike was almost always different than the other even if you tried to list down each and every step (Ideally it’s not possible in real word!) which lead to issues like <em>“It works on my laptop but don’t know why it’s not working on yours!”.</em></p><h4><strong><em>2. Code Testing</em></strong></h4><p>We have a central QA staging environment, where QA keeps running the tests and verifying the stage of the system. When the team was small, this doubled up as a dev environment as well. But as we started moving faster with more developers (which meant more experiments), maintaining the health of this environment became a big challenge. Any break in this setup would cost us hours and severely affect our productivity. We really needed a sandbox environment, where they could test their changes without affecting others’ productivity.</p><h4><strong><em>3. Resource Utilization</em></strong></h4><p>It would have a provisioning of at least 3–4 under utilized <em>AWS</em> instances (plus a lot of effort from dev-ops) for an isolated environment setup per developer or per team. A VM based approach would’ve led to resource wastage, given the duplication of system bin, library files and low CPU utilization. The list goes on….</p><p>The solution to all of these problems was to have a completely automated way of creating an isolated dev environment which could be created on demand. After evaluating multiple options, we finally chose Docker. Docker contained all the dependencies, the order in which services should start, the versions of packages to be pulled &amp; built and would run on a single VM.</p><h3><strong>What is Docker?</strong></h3><p>Docker is a tool that makes it super easy to create, deploy, and run applications by using containers. Docker allows a developer to bring together an application with all of it’s parts(such as libraries and other dependencies) and ship it all out as one package.</p><p>By doing so, the developer can ensure that the application will run on any other Linux machine regardless of any customized settings.</p><p>In a way, Docker is a bit like a virtual machine. But unlike a virtual machine, which creates a whole virtual operating system, Docker allows applications to use the same <em>Linux</em> kernel as the system that they’re running on. It only requires applications to be shipped with things not already running on the host computer. This gives a significant performance boost and reduces the size of the application.</p><h4><em>Version Control and Component Re-use</em></h4><p>You can track successive versions of a container, inspect differences, or roll-back to previous versions. Container re-use components from the preceding layers, which makes them noticeably lightweight. Sounds like Github right ? Yep, <strong>Docker is an open source project as well.</strong></p><h4>Key elements we were looking for ?</h4><ul><li><strong><em>Isolated</em></strong>: No other project should affect it and it shouldn’t affect any other projects. For example, if one project uses Python 2.7, that shouldn’t affect another project which uses Python 3.4.</li><li><strong><em>Repeatable</em></strong>: Write it once and have it work each time. I don’t want to spend time tweaking it for each user.</li><li><strong><em>Production Replica: </em></strong>Be as close to production as possible.</li><li><strong><em>One Box environment: </em></strong>Easy to start/stop and manage.</li></ul><blockquote>All that sounds great. But how do we achieve this?</blockquote><h4>Here’s What We Did</h4><p>We used <a href="http://boot2docker.io/">Boot2Docker</a> (A VM host to run docker daemon as it only works on <em>Linux</em> kernel) for Mac OS X and <a href="http://www.fig.sh/yml.html">Fig</a>, a orchestration tool for building, launching and linking Docker containers that moves all the configuration required to orchestra Docker into a simple clear <strong>fig.yml</strong> file. We also used a simple bash script over fig commands to increase usability.</p><p>We automated each and every step to setup the environment and we can now start our stack by running a single bash script. Now each developer has a 16 GB machine where boot2docker VM is using memory around 5 GB. The number of containers we can run on this is close to 25 including db containers with minimal setup.</p><p>Here’s a sample Dockerfile:</p><pre>FROM python:2.7.9<br>COPY requirements.txt requirements.txt<br>RUN pip install --quite -r requirements.txt</pre><p>A sample fig file:</p><pre>redisdb:<br>    image: redis:2.8.19</pre><pre>apiserver:<br>    build: .<br>    command: python apiserver.py<br>    volumes:<br>      - .:/code<br>    ports:<br>      - &quot;:8080&quot;<br>    links:<br>       - redisdb</pre><p>Now one can simply run <em>fig up </em>and your api server is running. In future if you want to scale your api server then you can do the following:</p><pre>$ fig scale apiserver=2</pre><p>And voila, you have two services running for api server</p><h4>More Info on Docker</h4><p>To get, the best place is <a href="https://docs.docker.com/">Docker’s official document</a> and their online interactive <a href="https://www.docker.com/tryit/">tutorial</a> . But here’s a list of some commands I personally find very useful:</p><pre>$ docker ps  <br>Lists all Docker processes running</pre><pre>$ docker help<br>To list all available options </pre><pre>$ docker exec -it &lt;container-id&gt; bash <br>To go inside already running container</pre><pre>$ docker rm &lt;container-id&gt;<br>To remove a container</pre><pre>$ docker rm $(docker ps -a -q)<br>To remove all containers</pre><pre>$ docker rmi &lt;container-id&gt;<br>To remove images of the given container</pre><pre>$ docker rmi $(docker ps -a -q)<br>To remove all images </pre><p>You can read about how to setup boot2docker on Mac OS X <a href="http://viget.com/extend/how-to-use-docker-on-os-x-the-missing-guide">here</a></p><p><strong>Other Notes:</strong></p><ul><li>Tooling for Docker is also a blossoming set of options.</li><li><a href="http://fig.sh/">Fig</a> (now named <a href="https://docs.docker.com/compose/">docker compose</a>) is now owned by Docker, and is an easy to use orchestration tool for building, launching and linking Docker containers.</li><li>Docker is also in a race to build their own ecosphere of tools around the core Docker engine. Such as <a href="https://github.com/docker/swarm">Swarm</a> for clustering and <a href="https://github.com/docker/machine">Machine</a> for container and VM management.</li><li><a href="https://github.com/citadel/citadel">Citadel</a>, <a href="http://panamax.io/">Panamax</a>, <a href="https://github.com/clusterhq/flocker">Flocker</a> and <a href="http://shipyard-project.com/">Shipyard</a> lets you compose, deploy and/or manage Docker containers and clusters in many different ways. <a href="https://github.com/progrium/dokku">Dokku</a> lets you easily deploy your Git based applications into Docker containers.</li></ul><p><em>Every week I seem to be finding new and amazing tools for Docker and containers in general.</em></p><h4>Conclusion</h4><p>Docker is a truly amazing project and represents a huge leap forward for advanced systems administration. It’s very powerful and has many use cases beyond what I’ve discussed here. My focus for evaluating Docker has been primarily around server setups delivering web applications.</p><p>We found that it’s great to setup for a staging environment but taking it to production will need a lot more expertise.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ff80ac5d6e5d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/the-work-we-do-hikeapp/hike-server-in-a-box-one-box-dev-testing-environment-ff80ac5d6e5d">How we built our entire testing environment in one box using Docker</a> was originally published in <a href="https://medium.com/the-work-we-do-hikeapp">The work we do at hike messenger</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>