<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Kushal Singh on Medium]]></title>
        <description><![CDATA[Stories by Kushal Singh on Medium]]></description>
        <link>https://medium.com/@kush712?source=rss-2afc659b7fa4------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*sp7ooF_ZZaQ_VoETcp55OA.jpeg</url>
            <title>Stories by Kushal Singh on Medium</title>
            <link>https://medium.com/@kush712?source=rss-2afc659b7fa4------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 08:46:42 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@kush712/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Nginx debugging and major issues - Solved]]></title>
            <link>https://medium.com/uc-engineering/nginx-debugging-and-major-issues-solved-b33eed3b8624?source=rss-2afc659b7fa4------2</link>
            <guid isPermaLink="false">https://medium.com/p/b33eed3b8624</guid>
            <category><![CDATA[elb]]></category>
            <category><![CDATA[shellscripting]]></category>
            <category><![CDATA[dns]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[nginx]]></category>
            <dc:creator><![CDATA[Kushal Singh]]></dc:creator>
            <pubDate>Mon, 02 Jul 2018 20:58:46 GMT</pubDate>
            <atom:updated>2018-07-20T07:45:35.550Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/950/1*1O4yhq7DNsDQgQuiuDLIdg.png" /></figure><p>We use Nginx as a reverse proxy which communicates with our backend and frontend servers. In our micro-service architecture, all the services run as docker containers deployed under ECS cluster. Nginx redirects traffic to our internal load balancer at a specific port which then forwards it to the service specific containers. Details of our deployment architecture are mentioned in a different <a href="https://medium.com/urbanclap-engineering/evolving-our-infrastructure-with-docker-ecs-60ac26e77783">post</a>.</p><p>Many a times we faced issues where API requests would start taking more time, our servers would become unreachable. <strong>5xx errors</strong> would spike on our dashboards. The same would be found from Nginx server&#39;s error logs.</p><p>In all such situations the only resort used to be to restart Nginx. And magically all issues would get fixed.</p><p>With basic debugging we found that upstream servers had been all healthy and receiving proper traffic. This encouraged us to go deeper at Nginx level and fix issues.</p><p>Following are a few scenarios we faced and action steps that were taken:</p><h3><strong><em>Error 1: “no live upstreams while connecting to……”</em></strong></h3><p>With the default configuration, Nginx behaves as a load-balancer and performs healthchecks on the upstream servers.</p><p>When for certain requests your upstream server fails to respond or times out, Nginx marks the upstream server as unreachable and stops redirecting any further traffic.</p><h3><strong>Solution</strong></h3><pre><br>upstream sm_url {</pre><pre>  server LOAD_BALANCER_DOMAIN_NAME: <strong>max_fails</strong>=0;</pre><pre>}<br># <strong>fail_timeout</strong> is the time interval during which if health-check fails for #<strong>max_fails </strong>times then the upstream server is considered as unhealthy. Default : 10sec, 1 respectively.</pre><p>Setting these to 0 disables health-check and all traffic is always routed to the upstream servers.</p><p><strong>Note : </strong>With any change in Nginx configuration don&#39;t forget to test the changes:</p><pre>sudo nginx -t</pre><h3>Error 2: “upstream timed out (110: Connection timed out) while connecting to upstream,…..”</h3><p>1) First of all it is important to log the upstream ip address on which connection is failing.</p><p><strong><em>Change</em> : </strong>set $<a href="http://nginx.org/en/docs/http/ngx_http_upstream_module.html?&amp;_ga=2.244258316.801072342.1530342434-46912000.1522311347#var_upstream_addr">upstream_addr</a> in confs/ngnix_log.conf</p><p><a href="https://gist.github.com/Kushal712/b49906379c22074acbbd6b463f9f6b16">Sample log config</a></p><p><strong><em>Corresponding error log:</em></strong></p><pre>“2018/03/17 07:59:34 [error] 17028#17028: *77553273 upstream timed out (110: Connection timed out) while connecting to upstream, client: 42.107.148.218, server: <a href="http://www.urbanclap.com,/">www.urbanclap.com,</a> request: “POST /api/v1/providers/cd/notify HTTP/1.1”, upstream: “http://x.x.x.x:80/api/v1/providers/cd/notify&quot;, host: “www.urbanclap.com&quot;</pre><p>2) Now we need to check the load balancer’s ip address which should receive above traffic.</p><pre>root@<strong>app6.nginx[Prod] </strong>nslookup LOAD_BALANCER_DOMAIN_NAME</pre><pre>Server: 172.31.0.2</pre><pre>Address: 172.31.0.2#53</pre><pre>Non-authoritative answer:</pre><pre>Name: LOAD_BALANCER_DOMAIN_NAME</pre><pre>Address: <strong>52.74.x.x</strong></pre><pre>Name: LOAD_BALANCER_DOMAIN_NAME</pre><pre>Address: <strong>54.255.x.x</strong></pre><p>Above output shows that the upstream address <strong>52.220.146.83</strong> is different from the allocated ip addresses of the load balancer. This means Nginx is not redirecting traffic to the actual ip address.</p><p><strong><em>After debugging, we found the following points:</em></strong></p><p>a) AWS <a href="https://aws.amazon.com/documentation/elastic-load-balancing/">ELB</a> keeps more than one server which maps to a load balancer. With changes in traffic it can either scale up or scale down these servers. In such cases there is change in the load-balancer&#39;s DNS -&gt; ip mapping.</p><p>b) Nginx resolves the DNS -&gt; ip mapping only at the time of reload/restart and for subsequent requests it reads it from cache until reload/restart.</p><p>c) We can specify <a href="http://nginx.org/en/docs/http/ngx_http_upstream_module.html?&amp;_ga=2.244258316.801072342.1530342434-46912000.1522311347#resolve">resolver</a> ie DNS server either in nginx.conf or by default it reads from /etc/resolv.conf file.</p><h3>Solution</h3><p>For Nginx to correctly resolve the load-balancer&#39;s domain name with the right ip address, we first tried changing Nginx Config from following sources -</p><p><strong><em>Approach 1 (didn&#39;t work):</em></strong></p><p><a href="https://www.nadeau.tv/nginx-proxy_pass-dns-cache/">Reference1</a>, <a href="https://www.sanyamkapoor.com/blog/nginx-dns/">Reference2</a></p><p>These changes might be effective for your config structure, but apparently they didn&#39;t work for us. Since we had already spent enough time debugging these issues, we went ahead with a very simple approach:</p><p><strong><em>Approach 2 (worked):</em></strong></p><p>Keep a periodic script which compares the current ip address of load balancer and the ones being used by Nginx. If they aren’t same then reload Nginx. Also, send slack notifications whenever such event happens.</p><pre>#!/bin/bash</pre><pre>PATH=/usr/sbin:/usr/bin:/sbin:/bin</pre><pre>ips=`nslookup xx-xxx-xxx-xxxx.ap-southeast-1.elb.amazonaws.com | grep &quot;Address&quot; | awk -F&#39;: &#39; &#39;{ if (NR != 1) print $2}&#39;`</pre><pre>echo $ips &gt; /tmp/latest</pre><pre>a=`cat /tmp/latest | tr &quot; &quot; &quot;\n&quot;|sort|tr &quot;\n&quot; &quot; &quot;`</pre><pre>b=`cat /tmp/old | tr &quot; &quot; &quot;\n&quot;|sort|tr &quot;\n&quot; &quot; &quot;`</pre><pre>if [ &quot;$a&quot; == &quot;$b&quot; ];</pre><pre>then</pre><pre>    echo &quot;Same&quot;</pre><pre>else</pre><pre>    echo &quot;Different&quot;</pre><pre>    text=&#39;ALBs ip is changed. Old: &#39;$b&#39; New: &#39;$a&#39; Reloading Nginx.&#39;</pre><pre>    echo $text</pre><pre>    curl -X POST --data-urlencode &#39;payload={&quot;channel&quot;: &quot;#alerts-devops&quot;, &quot;username&quot;: &quot;alb-ip-change(prod-nginx)&quot;, &quot;text&quot;: &quot;&#39;&quot;$text&quot;&#39;&quot;, &quot;icon_emoji&quot;: &quot;:rube:&quot;}&#39; https://hooks.slack.com/services/xxxxx/yyyyy/zzzzzz</pre><pre>    echo $a &gt; /tmp/old</pre><pre>    service nginx reload</pre><pre>fi</pre><p><strong><em>Alerts from above script:</em></strong></p><pre><strong>alb-ip-change(prod-nginx)</strong>:<br>ALBs ip is changed. Old: 52.76.100.149 54.251.189.195 New: 54.251.189.195 54.254.230.33 Reloading Nginx.</pre><p><strong><em>Frequency of alerts/ip changes:</em></strong></p><p>We get these alerts or Nginx reloads or ip address changes every 3–5 days.</p><p>This setup has worked quite well for us and so far we haven&#39;t received any similar issue !</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b33eed3b8624" width="1" height="1" alt=""><hr><p><a href="https://medium.com/uc-engineering/nginx-debugging-and-major-issues-solved-b33eed3b8624">Nginx debugging and major issues - Solved</a> was originally published in <a href="https://medium.com/uc-engineering">Urban Company – Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Evolving our Infrastructure with Docker & ECS]]></title>
            <link>https://medium.com/uc-engineering/evolving-our-infrastructure-with-docker-ecs-60ac26e77783?source=rss-2afc659b7fa4------2</link>
            <guid isPermaLink="false">https://medium.com/p/60ac26e77783</guid>
            <category><![CDATA[aws-ecs]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[deployment]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[continuous-integration]]></category>
            <dc:creator><![CDATA[Kushal Singh]]></dc:creator>
            <pubDate>Thu, 10 Aug 2017 13:20:52 GMT</pubDate>
            <atom:updated>2017-08-10T13:53:53.201Z</atom:updated>
            <content:encoded><![CDATA[<p>With growing team and feature sets, the biggest challenge we started to face in terms of developer productivity was how to build and release functionalities in an independent and a consistent manner. To launch an application into production, a large context was shared between developers and ops engineers(configuration, operational characteristics, system dependencies, file paths, permissions, etc). Therefore, there was an increasing dependency on the Platform &amp; DevOps teams which led to delays in the development cycle.</p><p>In our earlier infrastructure, we had a separate deployment stack for each service. This consisted of load-balancer, auto-scaling group, <a href="http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIs.html">AMI</a>, deployment scripts etc. Few of the complexities we faced in this architecture were:</p><ul><li>Every service required their specific tools and dependencies which were included in AMIs.</li><li>We had to setup logging and alerting again on the new machines.</li><li>New deployment scripts were to be written with every service.</li></ul><p>Therefore, provisioning a new infrastructure for a new service was time consuming and not scalable.</p><p>A couple of months back we relooked into our architecture and realized that we can heavily save the provisioning time and infra cost by having a single consistent architecture which manages all services. Docker was the way to go. Now, we have retired most of our old instances and their launch configs. Services now run as docker containers.</p><p>With this blog, I will be sharing our learnings which would be helpful on your way to dockerizing your production systems.</p><h3><strong>Build &amp; Deployment process for individual services</strong></h3><p>With docker, now we have a single deployment script for all services. Most of the dependencies like (node, wget, td-agent) are kept as a part of a single base docker image. The service owner includes any other dependency in the respective dockerFile which is used in final <a href="https://docs.docker.com/engine/reference/commandline/image_build/#options">docker build</a> process.</p><p>Following diagram explains the build and deployment process for a service.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*EDq_tuYoxmwis7bn." /></figure><h3><strong>Overall architecture</strong></h3><p>In the new architecture, we run our services under <a href="http://docs.aws.amazon.com/AmazonECS/latest/developerguide/Welcome.html">ECS</a> cluster. Some of the noteworthy points are:</p><ul><li>Instances get attached to the cluster through auto-scaling groups having scale-up and scale-down policies set appropriately.</li><li>Services that get launched in the cluster are set to have a minimum and maximum count of containers. Auto-scaling is also set at service level with thresholds on memory or CPU utilization depending upon the type of service.</li><li>Each service has an <a href="http://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_definitions.html">active task definition</a> which specifies the CPU units, memory, container to instance port binding, docker image path for the service.</li><li>We specify a dynamic host to container port mapping i.e. 0:xxxx . This means docker dynamically allocates a host port while application runs on xxxx port inside the container. With this we are able to deploy multiple containers of same type on same machine.</li></ul><p>The diagram below shows the overall infrastructure under ECS cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/788/0*MfJCpnw4MTm9nsll." /></figure><h3><strong>Reducing your docker image size</strong></h3><p>Keeping your docker image size under control has a lot of practical benefits. It makes things faster, more portable and less prone to breaks. Also, the build process takes lesser time which reduces the deployment time as well. With that in mind, let’s talk about a few steps that we took to reduce our initial image size from 1GB to ~250MB.</p><ol><li><strong>Remove legacy modules:</strong> We removed a lot of legacy modules which were no more in use. These were initially meant for client side rendering like Sass, compass, bower, grunt, Ruby, etc.</li><li><strong>Keeping package.json clean:</strong><a href="https://www.npmjs.com/package/npm-check"><strong> </strong></a>Using <a href="https://www.npmjs.com/package/npm-check">npm-check</a> module, we identified a lot of unused node modules. By moving these modules under devDependencies and then running <a href="https://futurestud.io/tutorials/how-to-install-npm-dependencies-for-production-environment">npm install — production</a> during deployment, we were able to get rid of them.</li><li><strong>.dockerignore file: </strong>Folders like .git, .tmp, bin, logs, test-run etc were added to <a href="https://docs.docker.com/engine/reference/builder/#dockerignore-file">.dockerignore</a> file.</li><li><strong>Club commands together: </strong>Since each line in dockerfile is <a href="http://blog.replicated.com/refactoring-a-dockerfile-for-image-size/">committed as a separate layer</a> in the final docker image, it’s better to club multiple commands together. For example:</li></ol><pre>`<br>RUN apt-get update -y \</pre><pre>&amp;&amp; rm /bin/sh &amp;&amp; ln -s /bin/bash /bin/sh \</pre><pre>&amp;&amp; apt-get install wget curl -y \</pre><pre>&amp;&amp; curl -L https://toolbelt.treasuredata.com/sh/install-ubuntu-trusty-td-agent2.sh | sh<br>`</pre><h3><strong>How much memory and CPU does my application need?</strong></h3><p>It’s important to figure out the right set of resources needed by your application. We at UrbanClap have a parallel staging ECS cluster where we first load-test our application with the peak traffic and tune its CPU and memory requirements before launching it on production. Since AWS <a href="http://docs.aws.amazon.com/AmazonECS/latest/developerguide/cloudwatch-metrics.html">cloud-watch</a> provides us with historical plots of Utilised memory(CPU)/ Reserved memory(CPU) (image 1.3), one can easily tweak these values with a central configuration file. For this purpose, we keep a configuration file which is used at the time of deployment. <a href="https://gist.github.com/Kushal712/485534de5efbf0eebab2658cb6fd9568">A sample configuration file</a> which we use at UrbanClap has been provided here for convenience.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/920/0*YbeuW6p45Z1B5BPf." /></figure><h3><strong>Logging and persistence</strong></h3><p>Since docker runs its containers as a separate machine, therefore its logs are not visible to the instance running the container. Once a container goes away it also removes all its logs. To solve this, we created <a href="http://container-solutions.com/understanding-volumes-docker/">mount points</a> from our instance into container before starting one. These logs then gets created on the instance thereby keeping a persistent storage. We further pipeline these logs to our <a href="https://techstricks.com/elasticsearch-fluentd-kibana-efk-setup-guide/">EFK</a> stack and s3 through <a href="https://docs.treasuredata.com/articles/td-agent">td-agent</a> clients and crons respectively.</p><h3>Code revert</h3><p>When things break it&#39;s important to be able to rollback your changes to the previous build as soon as possible.</p><p>Under the new architecture, every deployment for a service releases a new version of task definition which has a new docker image linked to it. With a single click in ECS management console we can revert back to a previous version of task definition. New docker containers get launched and old ones are terminated automatically.</p><p>Apart from removing the complexity in architecture, docker has also helped us in other ways:</p><ol><li>Uniform build and deployment process across services and across environments(dev, stage, prod).</li><li>Isolation of applications: Each container has its own resource isolated from other containers. For example, one version of node can be used by one application and other for other application on the same instance.</li><li>No more AMI changes: Developers can install their required packages on their own and give a final docker image for deployment.</li><li>With a couple of docker build and run commands anyone(QA, product, Devs) having a docker image can bring the service up and running regardless of OS/platform they use.</li><li>One time logging and alerting at instance level.</li><li>Version control of docker images and easy rollback.</li></ol><h3><strong>Bottomline</strong></h3><p>Docker comes with a lot more benefits than shared in this article. It helps us achieve a faster development, application isolation and consistency among service deployments, therefore leading to a much better devOps experience.</p><p>It has now become easier to support greater number of developers and application scale with minimum dependency on platform/DevOps.</p><p>With all these architectural changes and its benefits, we feel more bullish on separating out our core services and getting them deployed as docker containers.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=60ac26e77783" width="1" height="1" alt=""><hr><p><a href="https://medium.com/uc-engineering/evolving-our-infrastructure-with-docker-ecs-60ac26e77783">Evolving our Infrastructure with Docker &amp; ECS</a> was originally published in <a href="https://medium.com/uc-engineering">Urban Company – Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>