<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Soumyadip De on Medium]]></title>
        <description><![CDATA[Stories by Soumyadip De on Medium]]></description>
        <link>https://medium.com/@soumyadipde?source=rss-1c3da4cc8a76------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*I1YbGX674lzyukAI.</url>
            <title>Stories by Soumyadip De on Medium</title>
            <link>https://medium.com/@soumyadipde?source=rss-1c3da4cc8a76------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 10:45:12 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@soumyadipde/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Docker Enterprise Edition Review— The Good, the Bad, the Ugly & the Fair!]]></title>
            <link>https://medium.com/@soumyadipde/docker-enterprise-edition-the-good-the-bad-the-ugly-the-fair-9a78ca532c25?source=rss-1c3da4cc8a76------2</link>
            <guid isPermaLink="false">https://medium.com/p/9a78ca532c25</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[containerization]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[container-orchestration]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Soumyadip De]]></dc:creator>
            <pubDate>Mon, 18 Jun 2018 14:34:11 GMT</pubDate>
            <atom:updated>2018-06-22T10:32:57.733Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QzaZK1vsCKeidqDG3z1MlA.png" /><figcaption>Yeah you got the reference right — the cactus here symbolizing fair!</figcaption></figure><p>At the time of writing of this article, <strong>Containerization</strong> technology has already established itself as one of the most efficient way to build, ship and deploy applications and acted as a bridge to enable traditional application to be Cloud deploy-able. <a href="http://www.docker.com/"><strong>Docker</strong></a><strong> </strong>is the most popular containerization technology available at the moment.</p><blockquote><strong>Containerization</strong> is a method by which the application is broken down into easily manageable sizes, functional and individually packaged, complete with all the dependencies added to them.</blockquote><p>Although being the leader in containerization, Docker still lags to other players in choice of managing of containers i.e., <strong>Orchestration</strong>. Container orchestration is a major part of container deployment architecture as it manages the container lifetime, deployment strategy, versions among other aspects.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BG6olFNxyra9LS-NFTQciw.png" /><figcaption>Container Orchestration Trend</figcaption></figure><p>As you can see, <a href="https://kubernetes.io/"><strong>Kubernetes</strong></a> has a major market lead in orchestration compared to Docker counterpart <strong>Docker Swarm</strong>. It has been developed by <strong>Google </strong>and now maintained by <a href="https://www.cncf.io/"><strong>CNCF</strong></a>. There are lot of enterprise solutions based on Kubernetes itself such as <a href="https://www.openshift.com/">Openshift</a>, <a href="https://www.ibm.com/cloud/container-service">IBM Cloud,</a> <a href="https://cloud.google.com/kubernetes-engine/">GKE</a>, <a href="https://aws.amazon.com/eks/">EKS </a>etc. In the latest <a href="https://www.docker.com/kubernetes">version </a>of Docker, Kubernetes is included as an orchestration option.</p><p>Before going into the facts, let me give you a brief history of our journey to Docker and <strong>Docker Enterprise Edition (EE)</strong>. We, at <a href="https://www.oecloud.io/">oecloud.io</a>, have been using Docker for more than 2 years and faced ups and downs along with its journey up to this point. Our focus is to have an efficient CI-CD pipeline for our products as well as secure delivery of products to customers. We bought Docker Enterprise Edition on Nov ’17, around 7 months before writing of this article. So if you are planning to buy Docker EE, our experience might be helpful to you.</p><h3>Why did we buy Docker EE?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/289/1*xCCBSd4ljM8wtSfzBh8R0w.png" /></figure><p>We decided to buy Docker Enterprise Edition based on the following reasons:</p><ol><li>We needed our docker image delivery to be secure and properly authorized for the customer end. <strong>Docker Trusted Registry (DTR)</strong> is well-fitted solution for this purpose.</li><li>For multiple project within same organization, we needed access control for resources (Networks/ Containers/ Services/ Volumes). <strong>Universal Control Plane (UCP)</strong> is providing this Out-of-the-box.</li><li>We needed automatic CI-CD from developer commit up to image deployment to public cloud. <strong>DTR web-hook integrations </strong>made it possible to add customized hooks for CI-CD.</li><li>While using Docker CE, maintaining swarm cluster was not very smooth drive as the orchestration issues were not easy to track and fix. <strong>Enterprise Support</strong> on Docker environment enabled us to have a stable swarm cluster.</li></ol><h3>Docker EE — The Good!</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/468/1*465sdHmry2FWQNaIaJpk1w.jpeg" /></figure><p>Docker EE has quite a good number of features that makes it great investment for Enterprises. This includes,</p><ol><li><strong><em>Authentication and Authorization</em>: <br></strong> Docker <em>UCP </em>has its own built-in authentication mechanism and integrates with <em>LDAP </em>services. It also has <em>role-based access control (RBAC)</em>, so that you can control who can access and make changes to your swarm and applications. (<a href="https://docs.docker.com/datacenter/ucp/2.2/guides/access-control/">Learn more</a>)</li><li><strong><em>Secure Image delivery:<br></em></strong>Docker <em>DTR </em>has a built in security scanner that can be used to discover what versions of software are used in your images. It scans each layer and aggregates the results to give you a complete picture of what you are shipping as a part of your stack. Most importantly, it co-relates this information with a <em>vulnerability database </em>that is kept up to date through <a href="https://docs.docker.com/ee/dtr/admin/configure/set-up-vulnerability-scans/">periodic updates</a>. This gives you <a href="https://docs.docker.com/ee/dtr/user/manage-images/scan-images-for-vulnerabilities/">unprecedented insight into your exposure to known security threats</a>. (<a href="https://docs.docker.com/ee/dtr/#security-scanning">Learn more</a>)</li><li><strong><em>Trusted image delivery and deployment:<br></em></strong> DTR ships with <a href="https://docs.docker.com/notary/getting_started/"><em>Notary</em></a><em> </em>built in so that you can use <a href="https://docs.docker.com/engine/security/trust/content_trust/"><em>Docker Content Trust</em></a> to sign and verify images. With Docker UCP you can enforce applications to only use Docker images signed by UCP users you trust. When a user tries to deploy an application to the cluster, UCP checks if the application uses a Docker image that is not trusted, and won’t continue with the deployment if that’s the case. (<a href="https://docs.docker.com/ee/ucp/admin/configure/run-only-the-images-you-trust/">Learn more</a>)</li></ol><h3>Docker EE — The Bad!</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/466/1*VcVN2XTjmC_N9mridLO_Ug.jpeg" /></figure><p>Although we are quite happy with the Docker EE subscription, there are a few features unavailable here that other orchestration option such as Openshift offers. Some of them are,</p><ol><li><strong><em>Host OS support not included:<br></em></strong>When having a production instance running, you need to cover all your components with enterprise support. While docker has support for its certified images, it lacks support for host OS that <em>Openshift (dedicated)</em> includes in its subscription. (<a href="https://www.openshift.com/products/dedicated/">Learn more</a>)</li><li><strong><em>Documentation could have been better:<br></em></strong>There have been multiple times when we had to implement some features on UCP/DTR and the documents weren’t really that clear enough for implementation. May be things got improved by now and Docker would surely be working on improving this, but it is a point worth mentioning.</li><li><strong><em>Support is still in learning stage:</em></strong><br>Experience with support have been a mixed bag for us. Some of the issues are resolved promptly with proper explanation and some of them are stalled with unclear justification. So it depends upon whether you faced a common issue or you faced something rare.</li></ol><h3>Docker EE — The Ugly!</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/465/1*QFBzBW6YtAC1HSwmM38UtA.jpeg" /></figure><p>Here come the features that we expected Docker EE to have but somehow it has failed to our expectations. We have communicated these to our Docker spokesperson for improvement and waiting to have in coming releases. These include,</p><ol><li><strong><em>Not CLI friendly:</em></strong><br>Docker UCP/DTR components have CLI to install/upgrade but lacks CLI for feature enable/disable. This makes it hard to automate DTR/UCP feature control. As an example, we can’t turn on vulnerability scan of an image on DTR or create users on UCP through CLI.</li><li><strong><em>Upgradation issues:</em></strong><br>Although docker tried to have exhaustive details on how to upgrade UCP/DTR along with known issues list, we still faced lot of issues while upgrading and had to raise support tickets to solve that. There was one time when we upgraded DTR but support told we can’t upgrade DTR without UCP being upgraded. This should be alerted while executing upgrade command, not by support person!</li><li><strong><em>Kubernetes rbac support:</em></strong><br>This drawback comes with latest feature to support Kubernetes (k8s) on Docker EE platform. Although it supports most of the k8s components, it does not support k8s rbac components and uses its own UCP RBAC authentication. So far so good, but the fact that we need to add manually all the k8s roles to UCP really tested our patience! (<a href="https://docs.docker.com/ee/ucp/authorization/migrate-kubernetes-roles/">Learn more</a>)</li></ol><h3>Docker EE — The Fair!</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/530/1*n0BRS-gS4jIH3W_XdtF66Q.jpeg" /></figure><p>Now you would be wondering why this point came into picture as it is deviating from the movie reference! But I wanted to make this article a balanced review of Docker EE and thus added another positive view point to balance two negatives. These are the features Docker EE really shined in our case,</p><ol><li><strong><em>Latest security patches:</em></strong><br>Out of all the orchestration platform available, Docker EE provides the latest security patches for its container platform that takes months to come on other platforms. This may be a deciding factor for some Enterprises where security is the prime concern.</li><li><strong><em>Minimizing vendor lock-in:</em></strong><br>While other platforms tend to lock you into specific Linux distribution or their cloud platform, with Docker EE users break that vendor lock-in and get to choose the platforms and cloud they want. This also is a major plus point for Enterprise CIOs as they should always avoid any form of vendor lock-in.</li><li><strong><em>Kubernetes/Docker swarm integration:</em></strong><br>There is hardly any orchestration product that offers reliability of kubernetes as well as simplicity of Docker swarm side by side as offered in Docker EE 2.0. This has really been a game changer for Docker in terms of their market place and puts a vast option for its users to choose from.</li></ol><h4>Conclusion:</h4><p>I hope this article would prove helpful in deciding your choice to buy Docker Enterprise Edition. These points are entirely my viewpoint while implementing Docker EE components for my company and does not reflect my company’s viewpoint on Docker EE. Also you have to take account that as docker releases more and more versions, the points mentioned on this article may become more and more irrelevant!</p><p>You can connect with me on my linkedin profile <a href="https://www.linkedin.com/in/soumyadip-de-724315a0/">here</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9a78ca532c25" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Monitoring in Docker Stacks - It’s that easy with Prometheus!]]></title>
            <link>https://medium.com/@soumyadipde/monitoring-in-docker-stacks-its-that-easy-with-prometheus-5d71c1042443?source=rss-1c3da4cc8a76------2</link>
            <guid isPermaLink="false">https://medium.com/p/5d71c1042443</guid>
            <category><![CDATA[monitoring]]></category>
            <category><![CDATA[oe-cloud]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[docker-swarm]]></category>
            <category><![CDATA[prometheus]]></category>
            <dc:creator><![CDATA[Soumyadip De]]></dc:creator>
            <pubDate>Sat, 17 Jun 2017 21:56:01 GMT</pubDate>
            <atom:updated>2018-02-22T18:02:27.513Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*DYByMm8lAvTuRuPmAiOZbw.png" /><figcaption>Until this comes into rescue...let’s have some open-source tools save you!</figcaption></figure><p>You might have heard about <a href="https://www.docker.com/what-docker">docker</a> and how it is revolutionizing every corner of software industry — be it in SDLC process or microservices architecture. Having a self-sufficient container for each of your logically independent modules makes your software resilient to traffic spikes and emergency software patches.</p><p>But as life has taught us already, <em>Everything comes at a cost!</em> Having multiple ephemeral containers — created and destroyed as per load — make it very hard to keep track and monitor applications. Monitoring has been a major concern for containers from the start causing many solutions to come forward in recent past.</p><p>There are few paid solutions are available addressing Container monitoring, of which one of the most popular would be <a href="https://www.datadoghq.com/">Datadog</a>. But as we, folks at <a href="https://www.oecloud.io/">OeCloud.io</a>, believe in open-source for quick access to innovation, we looked up for solutions in open-source communities and found <a href="https://prometheus.io/">Prometheus</a>, backed by <a href="https://cncf.io/">Cloud Native Computing Foundation</a>, to be a good fit for our need.</p><blockquote>Now that docker daemon also supporting exporting metrics to prometheus in latest version (<a href="https://docs.docker.com/engine/admin/prometheus/">link</a>), it’s a matter of time before most of us in community adopts prometheus.</blockquote><p>Now what made docker to choose prometheus over other open-source alternatives comes down to how prometheus collects, stores and represent metrics.</p><h4><strong>Features</strong>:</h4><ul><li>a multi-dimensional <a href="https://prometheus.io/docs/concepts/data_model/">data model</a> (time series identified by metric name and key/value pairs)</li><li>a <a href="https://prometheus.io/docs/querying/basics/">flexible query language</a> to leverage this dimensionality</li><li>no reliance on distributed storage; single server nodes are autonomous</li><li>time series collection happens via a pull model over HTTP</li><li><a href="https://prometheus.io/docs/instrumenting/pushing/">pushing time series</a> is supported via an intermediary gateway</li><li>targets are discovered via service discovery or static configuration</li><li>multiple modes of graphing and dashboarding support</li></ul><h4>Components:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/744/1*NPhGxil0syw1DLREuzWc6w.png" /><figcaption>Prometheus Architecture (Yes, copy-pasted from official docs!)</figcaption></figure><p>Now that introduction with our friend Prometheus is settled, let’s guide you through each of its components to know it better.</p><ol><li><strong>Exporters</strong>: Prometheus server takes help from various exporters to collect metrics (Pull mechanism). In case of Docker stack/swarm, we need two kind of exporters,<br>I. <em>Node-exporter</em>: This is used to collect Node/Server related information such as server CPU/Memory/Storage/Network utilization. This is supported by prometheus community. (<a href="https://github.com/prometheus/node_exporter">link</a>)<br>II. <em>Container-exporter</em>: Now this is our saviour for monitoring docker containers. At the time of writing this article, metrics from docker engine are not detailed enough. So we use Google community supported cAdvisor for container metrics. It is well detailed and each container CPU/Memory/Storage/Network utilization can be collected from it. (<a href="https://github.com/google/cadvisor">link</a>)<br>III. <em>Custom-exporter</em>: You can use custom exporters for various components you use in your eco-system like HAProxy, Apache, Elastic-search etc. You can find your specific tool exporter if someone already built it or you can build on your own following this <a href="https://prometheus.io/docs/instrumenting/writing_exporters/">guide</a>.</li><li><strong>Prometheus Server</strong>: The heart of prometheus monitoring system which pulls metrics from exporters and stores in sophisticated local storage subsystem which has powerful querying support. It has simple UI where you can check status of each exporter endpoint and test queries. (<a href="https://github.com/prometheus/prometheus">link</a>)</li><li><strong>Grafana</strong>: Grafana is a visualization component that supports querying to prometheus and show various graphs depending on query. It recently included alerting feature which can also come handy but there is separate component in prometheus to handle alerts. (<a href="https://github.com/grafana/grafana">link</a>)</li><li><strong>Alertmanager</strong>: This is a component where prometheus server pushes pre-configured alerts via configuration file. Alertmanager handles routing of alerts to intended recipients, stacks duplicate alerts and suppresses alert based on conditions. (<a href="https://github.com/prometheus/alertmanager">link</a>)</li></ol><h4><strong>Configuration:</strong></h4><p>Now the part comes where you’ll be guided on how to befriend with prometheus for docker stacks monitoring.</p><ol><li><em>Node-exporter</em>: You can run global docker service of node-exporter exposing 9000 port of container.</li></ol><pre>docker service create --name node-exporter --mode global -p 9100:9100 prom/node-exporter</pre><p>2. <em>cAdvisor</em>: cAdvisor is available as a docker image and can be run as a docker service.</p><pre>docker service create --name cAdvisor --mode global -p 8080:8080 google/cadvisor</pre><blockquote>But when we tried to implement it, we faced a major <a href="https://github.com/google/cadvisor/issues/771">issue</a> in running cAdvisor as container which forced us to install cAdvisor as a application in host machine and run as a service (<a href="https://github.com/google/cadvisor/blob/master/docs/development/build.md">guide</a>).</blockquote><p>3. <em>Alertmanager</em>: Alertmanager can run as docker service in replicated mode exposing 9093 port. Alertmanager needs a configuration file that can be made following this <a href="https://prometheus.io/docs/alerting/configuration/">guide</a>. The following command assumes you have alertmanager.conf in currrent directory.</p><p>First create overlay network for communication,</p><pre>docker network create -d overlay prometheus_network</pre><p>Then service creation,</p><pre>docker service create \<br>--name alertmanager \<br>-p 9093:9093 \<br>--network prometheus_network \<br>--mount\<br>   target=/config/alermanager.conf,\<br>   source=./alertmanager.conf,type=bind \<br>prom/alertmanager:v0.6.2 -config.file=/config/alermanager.conf</pre><p>4. <em>Prometheus server</em>: You need to pass a configuration file to promethues server like this,</p><pre>global:<br>  scrape_interval: 15s<br>  evaluation_interval: 15s</pre><pre>rule_files:<br> - &quot;alert.rules&quot;</pre><pre>scrape_configs:<br> - job_name: docker-engine-metrics<br>   static_configs:<br>    - targets: [&#39;10.X.X.1:9323&#39;,&#39;10.X.X.2:9323&#39;]</pre><pre>- job_name: container-metrics<br>  static_configs: <br>   - targets: [&#39;10.X.X.1:8080&#39;,&#39;10.X.X.2:8080&#39;]</pre><pre>- job_name: node-metrics<br>  static_configs:<br>   - targets: [&#39;10.X.X.1:9100&#39;,&#39;10.X.X.2:9100&#39;]</pre><p>Alertmanager rules also needs to be added as a file as alert.rules file. Sample alert.rules file can be as,</p><pre># Alert for any instance that is unreachable for &gt;5 minutes.</pre><pre>ALERT InstanceDown<br>IF up == 0<br>FOR 5m<br>LABELS { severity = &quot;page&quot; }<br>ANNOTATIONS {<br>summary = &quot;Instance {{ $labels.instance }} down&quot;,<br>description = &quot;{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 5 minutes.&quot;</pre><p>Now for running prometheus server assuming configuration file is in ./prometheus/prometheus.conf,</p><pre>docker service create \<br>--name cep-prometheus-server \<br>--network prometheus_network \<br>--mount target=/config,source=./prometheus,type=bind \<br>prom/prometheus:v1.6.3 \<br> -config.file=/config/prometheus.conf \<br> -alertmanager.url=http://alertmanager:9093</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BS4zC95G157zgM308N7GLQ.png" /><figcaption>Prometheus server targets screen (No, it is not supposed to be down!)</figcaption></figure><p>5. <em>Grafana</em>: Grafana also comes as docker image and can be run as docker service.</p><pre>docker service create --name grafana-server grafana/grafana:4.2.0</pre><p>Setting up dashboards can be found at this <a href="http://docs.grafana.org/guides/basic_concepts/">link</a>. Pre-built dashboards can be found on this <a href="https://grafana.com/dashboards">link</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_G70RIiJl49a8ZSXdYoXDw.png" /><figcaption>Grafana dashboard (Yes, we are under-utilizing our servers by miles!)</figcaption></figure><p>For this guide, the dashboard being used is <a href="https://grafana.com/dashboards/179">this</a>. To group containers based on stack/service, you can use docker labels on containers to group them. For instance, network output of containers can be grouped by service-name having “your-app” in it by following query,</p><pre>sort_desc(sum by (container_label_com_docker_swarm_service_name) (rate(container_network_transmit_bytes_total{image!=&quot;&quot;,container_label_com_docker_swarm_service_name=~&quot;.*your-app.*&quot;}[1m] ) ))</pre><p>Here ‘Metric lookup’ would be <em>container_network_transmit_bytes_total </em>and ‘Legend format’ as <em>{{ container_label_com_docker_swarm_service_name }}.</em></p><p>If you want to group by stack namespace, that can also be done by replacing service-name filter with <em>container_label_com_docker_stack_namespace.</em></p><h4>Conclusion:</h4><p>Prometheus has been serving us nice for some time and you may need to go through prometheus docs to tweak it as per your need such as metrics retention period or persistence of data.</p><p>Some may not agree with the heading containing <em>“..that easy!’’</em> following this guide. For them, I would suggest to go to our open-source contribution on <a href="https://github.com/EdgeVerve/cep-provision"><strong>Github</strong></a><strong> </strong>which automates docker swarm cluster setup along with monitoring and logging components. The automation steps for monitoring include,</p><ol><li>Generation of configuration file(s) from host ips and installation</li><li>Opening necessary ports of firewall on each of the hosts</li><li>Deploy necessary components for monitoring</li><li>Copy pre-built dashboards in grafana for quick setup</li><li>Provide URL to access grafana/prometheus/alertmanager (like <a href="https://grafana.your-domain.com)/">https://grafana.your-domain.com)</a> using HAProxy router.</li></ol><p>So you’ll be able to get things up and running in no time and focus more on your application rather than infrastructure setup!</p><p>You can connect with me on my linkedin <a href="https://www.linkedin.com/in/soumyadip-de-724315a0/">profile</a>.. Happy Monitoring!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5d71c1042443" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>