<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Codefresh on Medium]]></title>
        <description><![CDATA[Stories by Codefresh on Medium]]></description>
        <link>https://medium.com/@codefresh?source=rss-563ec507cbd3------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*DhdOwEFrKA3K1MDPC83hqA.png</url>
            <title>Stories by Codefresh on Medium</title>
            <link>https://medium.com/@codefresh?source=rss-563ec507cbd3------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 14:49:24 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@codefresh/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[My Process Used Minimal Memory, and My Docker Memory Usage Exploded]]></title>
            <link>https://medium.com/containers-101/my-process-used-minimal-memory-and-my-docker-memory-usage-exploded-f759bed2b06f?source=rss-563ec507cbd3------2</link>
            <guid isPermaLink="false">https://medium.com/p/f759bed2b06f</guid>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[memories]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[pipeline]]></category>
            <category><![CDATA[code]]></category>
            <dc:creator><![CDATA[Codefresh]]></dc:creator>
            <pubDate>Thu, 12 Nov 2020 11:30:45 GMT</pubDate>
            <atom:updated>2020-11-12T11:30:45.795Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*q3j3YYei_gHTq4Z5hKG82w.png" /></figure><p>The Docker infrastructure abstracts a lot of aspects of the creation of images and running them as containers, which we usually do not know about nor interact with. One of those aspects is the handling of the filesystem inside the container.</p><p>This post is a case study on how we discovered that writing large amounts of data inside a container has side effects with memory caching. Initially, we thought that we had an issue with our source code, but this was never the case.</p><p>In this post, we will see:</p><ul><li>A walk-through on how you can reproduce the memory issue that we encountered</li><li>How we discovered a temporary workaround</li><li>The steps we followed to find a proper solution</li></ul><p>If you want to know more about the way Docker abstraction behaves and you always had unexplained questions about large memory usage, then this post is for you.</p><h3>How Docker handles Memory allocation</h3><p>Before we dive into our specific scenario, we want to give an overview of how memory allocation and usage works in Docker.</p><p>Docker does not apply memory limitations to containers by default. The Host’s Kernel Scheduler determines the capacity provided to the Docker memory. This means that in theory, it is possible for a Docker container to consume the entire host’s memory.</p><p>One way to control the memory used by Docker is by setting the runtime configuration flags of the ’docker run command’. There are specific scenarios when you will want to set these limitations.</p><p>Generally, you want to prevent the container from consuming too much of the host machine’s memory. In the case of Linux, once the Kernel detects that it will run out of memory, it will throw an ’Out Of Memory Exception’ and will kill processes to free up memory.</p><p>Usually, those processes are ordered by priority, which is determined by the OOM (Out of Memory) killer. However, if the wrong process is killed, the entire system might crash.</p><p>Contrary to popular belief, even if you set a memory limit to your container, it does not automatically mean that the container will only consume the allocated memory. You can try this out yourself. The following example allocates 60mb of memory inside of a container with 50mb. You would expect the OOME to kill the process. However, it does not. To try it out, run:</p><pre>docker run — memory 50m — rm -it progrium/stress — vm 1 — vm-bytes 62914560 — timeout 1s</pre><p>To prevent this from happening, you have to define the memory-swap in addition to the memory by setting it equal to the memory:</p><pre>docker run — memory 50m — memory-swap 50m — rm -it progrium/stress — vm 1 — vm-bytes 62914560 — timeout 1s</pre><p>Note that this will only work if your Docker has <a href="https://docs.docker.com/install/linux/linux-postinstall/">cgroup memory and swap accounting</a> enabled.</p><p>The official Docker documentation provides more details on the steps that you can take to restrict the container’s memory usage; here are the main points:</p><ul><li>Perform tests to understand the memory requirements of your application before placing it into production.</li><li>Ensure that your application runs only on hosts with adequate resources.</li><li>Limit the amount of memory your container can use, as described in the official Docker documentation.</li><li>Be mindful when configuring swap on your Docker hosts. Swap is slower and less performant than memory but can provide a buffer against running out of system memory.</li><li>Consider converting your container to a<a href="https://docs.docker.com/engine/swarm/services/"> service</a>, and using service-level constraints and node labels to ensure that the application runs only on hosts with enough memory.</li></ul><p>More information can be found in the <a href="https://docs.docker.com/config/containers/resource_constraints/">official Docker documentation for runtime options</a>.</p><h3>Observing a memory issue with our ETL script</h3><p>This section provides an overview of our script and details how you can recreate the memory bloat using Codefresh and the YAML file provided.</p><p>The issue appeared when we ran an ETL (extract-transform-load) application inside a container.</p><p>The details of the business purpose behind the application are not that important. In short, this is the process that our application was supposed to follow:</p><ol><li>Pull quite a lot of open-source data into memory</li><li>Convert the data and write it into a single file on the filesystem</li><li>Next, upload the file to Google Cloud and populate it into BigQuery</li></ol><p>The resulting file consists of about 20 million lines, so we expected the process to take some time.</p><p>However, we observed that after a while, we received a memory indication warning. The container then slowed down and finally crashed.</p><h3>Reproducing the issue with Node.js</h3><p>Just to confirm that our application code is not at fault here we reproduce the error with a simple script running in Node.js, image node:12.13.0 :</p><pre>&#39;use strict&#39;;</pre><pre>const fs = require(&#39;fs&#39;);</pre><pre>const sameFile =process.env.SAME_FILE;</pre><pre>//node --max-old-space-size=400 ./checkBloat.js</pre><pre>const main = async () =&amp;gt; {</pre><pre>let totalWrite = 0;</pre><pre>let fileCnt = 0;</pre><pre>let numFiles = 1;</pre><pre>while(true) {</pre><pre>const filename = sameFile ? &#39; file&#39; : `file${numFiles}`;</pre><pre>const theFile = fs.openSync(filename, &#39;w&#39;);</pre><pre>for(let i=1; i&amp;lt;10000000+1;i++){ fileCnt=i; fs.writeSync(theFile, `${i}) =================================================================&amp;gt;${Math.random()}`);</pre><pre>totalWrite+=1;</pre><pre>if (totalWrite%100000===0) {</pre><pre>console.log(`${filename} ${numFiles}) cnt: ${fileCnt})total written: ${totalWrite} `);</pre><pre>}</pre><pre>}</pre><pre>console.log(`CLOSE FILE } `);</pre><pre>fs.closeSync(theFile);</pre><pre>numFiles++;</pre><pre>};</pre><pre>};</pre><pre>Promise.resolve()</pre><pre>.then( async () =&amp;gt; {</pre><pre>await main();</pre><pre>});</pre><p>This script reproduces the effect of the memory climbing up close to the limit of the underlying machine and staying there. This could continue until the device filesystem is full. Then, we would try to log into a Unix shell and flush the memory, in which case it would crash.</p><h3>Reproducing the issue with a Python script</h3><p>To compare with our Node.js script, we ran the same logic in Python. This would verify the fact that the application code is not a fault, but instead something is wrong with the container runtime.</p><p>We rewrote the NodeJs script in Python. The Python script would seem to work in the beginning but at some point, there was no more room to write the data to the output file and we would receive the following error message: OSError: [Errno 28] No space left on device</p><pre>import os</pre><pre>import sys</pre><pre>from collections import defaultdict</pre><pre>lines = 0</pre><pre>file_cnt = 0</pre><pre>d=defaultdict(list)</pre><pre>for k, v in ((k.lstrip(&#39;-&#39;), v) for k,v in (a.split(&#39;=&#39;) for a in sys.argv[1:])):</pre><pre>d[k].append(v)</pre><pre>print (d)</pre><pre>limit = int(d.get(&#39;limit&#39;)[0]) if  d.get(&#39;limit&#39;)  else None</pre><pre>same_file =  False if not d.get(&#39;bloat&#39;) or d.get(&#39;bloat&#39;)[0] in [&#39;true&#39;, &#39;on&#39;] else True</pre><pre>stats =  True if not d.get(&#39;stats&#39;) or d.get(&#39;stats&#39;)[0] in [&#39;true&#39;, &#39;on&#39;] else False</pre><pre>print (f&#39;argv = {sys.argv} limit: {limit} same_file {same_file}  stats {stats}&#39;)</pre><pre># create the file encoded for the yaml</pre><pre>os.system(&#39;FNAME=&quot;write_lots.py&quot; &amp;amp;&amp;amp; cat &quot;$FNAME&quot; | python -m base64 &amp;gt; &quot;$FNAME.b64&quot;&#39;)</pre><pre>while(limit is None or lines&amp;lt; limit) : file_cnt += 1 filename= f&#39;file&#39; if same_file else f&#39;bloat{file_cnt}&#39; f = open(f&#39;{filename}&#39;, &quot;w&quot;) for x in range(1,1000*1000+1): f.write(f&#39;{file_cnt}) {x}: write line {lines} ============================================&amp;gt; \n&#39;);</pre><pre>lines += 1</pre><pre>if (lines%(1000*1000)==0):</pre><pre>print(f&#39;Written {file_cnt} {filename}) {x}: write line {lines}&#39;)</pre><pre>f.close()</pre><pre>print(f&#39;{file_cnt}) close {filename} written {x} total {lines} &#39;);</pre><pre>if (stats):</pre><pre>try:</pre><pre>os.system(&#39;&#39;&#39;ps auxww | grep -E &quot;python [w]rite_lots|[P]ID &quot; &#39;&#39;&#39;)</pre><pre>os.system(&#39;free&#39;)</pre><pre>except(e):</pre><pre>pass</pre><p>The python code has options to use the same file/stats and sizes as our original source code.</p><h3>Reproducing in the Codefresh platform</h3><p>Instead of running the script separately, you can also use the following codefresh.yml file with the python code embedded. Note that the long string is the embedded Python code.</p><pre>version: &quot;1.0&quot;</pre><pre>stages:  []</pre><pre>steps:</pre><pre>run:</pre><pre>title: &quot;Running process&quot;</pre><pre>image: python</pre><pre>commands:</pre><pre>- ls</pre><pre>- pwd</pre><pre>- FNAME=&quot;write_lots.py&quot;</pre><pre>- echo &#39; content was created by bash&amp;gt; FNAME=&quot;write_lots.py&quot; &amp;amp;&amp;amp; cat &quot;$FNAME&quot; | python -m base64 &amp;gt; &quot;$FNAME.b64&quot; &#39;</pre><pre>- &amp;gt;-</pre><pre>echo  &quot;</pre><pre>aW1wb3J0IG9zCmltcG9ydCBzeXMKZnJvbSBjb2xsZWN0aW9ucyBpbXBvcnQgZGVmYXVsdGRpY3QK</pre><pre>bGluZXMgPSAwCmZpbGVfY250ID0gMApkPWRlZmF1bHRkaWN0KGxpc3QpCmZvciBrLCB2IGluICgo</pre><pre>ay5sc3RyaXAoJy0nKSwgdikgZm9yIGssdiBpbiAoYS5zcGxpdCgnPScpIGZvciBhIGluIHN5cy5h</pre><pre>cmd2WzE6XSkpOgogICAgICBkW2tdLmFwcGVuZCh2KQpwcmludCAoZCkKbGltaXQgPSBpbnQoZC5n</pre><pre>ZXQoJ2xpbWl0JylbMF0pIGlmICBkLmdldCgnbGltaXQnKSAgZWxzZSBOb25lCnNhbWVfZmlsZSA9</pre><pre>ICBGYWxzZSBpZiBub3QgZC5nZXQoJ2Jsb2F0Jykgb3IgZC5nZXQoJ2Jsb2F0JylbMF0gaW4gWyd0</pre><pre>cnVlJywgJ29uJ10gZWxzZSBUcnVlCnN0YXRzID0gIFRydWUgaWYgbm90IGQuZ2V0KCdzdGF0cycp</pre><pre>IG9yIGQuZ2V0KCdzdGF0cycpWzBdIGluIFsndHJ1ZScsICdvbiddIGVsc2UgRmFsc2UKZmlsZXNp</pre><pre>emUgPSAgaW50KGZsb2F0KGQuZ2V0KCdmaWxlc2l6ZScpWzBdKSoxMDAwMDAwMCkgaWYgZC5nZXQo</pre><pre>J2ZpbGVzaXplJykgZWxzZSAgMTAwMDAwMDAKcHJpbnQgKGYnYXJndiA9IHtzeXMuYXJndn0gbGlt</pre><pre>aXQ6IHtsaW1pdH0gc2FtZV9maWxlIHtzYW1lX2ZpbGV9ICBzdGF0cyB7c3RhdHN9IGZpbGVzaXpl</pre><pre>IHtmaWxlc2l6ZX0nKSAKIyBjcmVhdGUgdGhlIGZpbGUgZW5jb2RlZCBmb3IgdGhlIHlhbWwKb3Mu</pre><pre>c3lzdGVtKCdGTkFNRT0id3JpdGVfbG90cy5weSIgJiYgY2F0ICIkRk5BTUUiIHwgcHl0aG9uIC1t</pre><pre>IGJhc2U2NCA+ICIkRk5BTUUuYjY0IicpCndoaWxlKGxpbWl0IGlzIE5vbmUgb3IgbGluZXM8IGxp</pre><pre>bWl0KSA6CiAgIGZpbGVfY250ICs9IDEKICAgZmlsZW5hbWU9ICBmJ2ZpbGUnIGlmIHNhbWVfZmls</pre><pre>ZSBlbHNlIGYnYmxvYXR7ZmlsZV9jbnR9JyAKICAgZiA9IG9wZW4oZid7ZmlsZW5hbWV9JywgInci</pre><pre>KQogICBmb3IgeCBpbiByYW5nZSgxLGZpbGVzaXplKzEpOgogICAgICBmLndyaXRlKGYne2ZpbGVf</pre><pre>Y250fSkge3h9OiB3cml0ZSBsaW5lIHtsaW5lc30gPT09PT09PT09PT09PT09PT09PT09PT09PT09</pre><pre>PT09PT09PT09PT09PT09PT0+IFxuJyk7CiAgICAgIGxpbmVzICs9IDEKICAgICAgaWYgKGxpbmVz</pre><pre>JSgxMDAqMTAwMCk9PTApOgogICAgICAgICBwcmludChmJ1dyaXR0ZW4ge2ZpbGVfY250fSB7Zmls</pre><pre>ZW5hbWV9KSB7eH06IHdyaXRlIGxpbmUge2xpbmVzfScpCiAgIGYuY2xvc2UoKQogICBwcmludChm</pre><pre>J3tmaWxlX2NudH0pIGNsb3NlIHtmaWxlbmFtZX0gd3JpdHRlbiB7eH0gdG90YWwge2xpbmVzfSAn</pre><pre>KTsKICAgaWYgKHN0YXRzKToKICAgICAgdHJ5OgogICAgICAgICBvcy5zeXN0ZW0oJycncHMgYXV4</pre><pre>d3cgfCBncmVwIC1FICJweXRob24gW3ddcml0ZV9sb3RzfFtQXUlEICIgJycnKQogICAgICAgICBv</pre><pre>cy5zeXN0ZW0oJ2ZyZWUnKQogICAgICBleGNlcHQoZSk6CiAgICAgICAgIHBhc3MKCgoK</pre><pre>&quot; &amp;gt; &quot;$FNAME.b64&quot;</pre><pre>- ls</pre><pre>- cat &quot;$FNAME.b64&quot; | python -m base64 -d &amp;gt; &quot;$FNAME&quot;</pre><pre>- ls -ltr</pre><pre>- echo &quot;the code&quot;</pre><pre>- cat &quot;$FNAME&quot;</pre><pre>- date</pre><pre>- echo &quot;no bloat&quot;</pre><pre>- python &quot;$FNAME&quot; limit=400000000 bloat=off stats=on filesize=1</pre><pre>- date</pre><pre>- echo &quot;would bloat&quot;</pre><pre>- python &quot;$FNAME&quot; limit=400000000 bloat=on stats=on filesize=1</pre><pre>- echo done</pre><pre>- date</pre><p><strong>To use this in your Codefresh pipeline, follow these steps.</strong></p><ul><li>First set-up your <a href="https://codefresh.io/docs/docs/getting-started/create-a-codefresh-account/">Codefresh account.</a></li><li>Next, create a project and set-up a pipeline within the project.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*IvH-o5xMPqFzojTw.png" /></figure><ul><li>When you create the pipeline, make sure to turn the “add git example” off.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*Tf0xzMpK8Wqjdvge.png" /></figure><ul><li>It will provide a template codefresh.yml file by default. Replace this file with the YAML file above. This will show some warnings but should not throw any errors.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*TJnm1ylC6OkOhc9r.png" /></figure><ul><li>Next, run the pipeline.</li><li>You can view the output in the Codefresh Console.</li></ul><p>The pipeline runs the code twice. We have added a command-line option to our script (called <strong>bloat</strong>) which decides if we use a single file for output (our original design) or multiple files but with the same name (we will explain the choice in the next section).</p><p>The pipeline will still crash with an out of memory error.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*w3YmvLECTyqRHU0P.png" /></figure><h3>Kubernetes analysis</h3><p>Even though the problem seems to be Docker related, our initial analysis was started with Kubernetes (Codefresh is running pipeline inside Docker containers on a Kubernetes cluster).</p><p>To debug the issue we ran the following command in order to get a shell inside the container.</p><pre># NS=the namespace NAME= the container name</pre><pre>kubectl -n $NS exec — stdin — tty $NAME — /bin/bash</pre><p>We looked at the buffer/cache by running the <a href="https://linuxize.com/post/free-command-in-linux/">Linux free</a> command. This showed growing memory usage:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*BVrMIY7P5eSFKnuv.png" /></figure><p>Next, we tried removing all the code and focused on the process of creating the file to see whether it is a problem with the code or the file generation and usage. In this case, the same thing happened and the memory still increased.</p><p>We cross-checked whether it was the code or creating the file by writing into /dev/null. Dev/null is a file that is present in every Linux OS, also known as the null device on UNIX systems, and sometimes called ‘the black hole’. It is only used to write, and whatever is written into dev/null will be immediately disregarded.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/255/0*gNydtaTaiIaZI_69.png" /></figure><p>So the assumption is that writing into this file should not have any effect on the memory used.</p><p>However, using /dev/null did not change anything. The memory also bloated in this case. We now knew the issue is related to writing the file.</p><h3>First approach: clearing the memory cache</h3><p>Now that we know what the issue was, we had to find a solution or workaround.</p><p>The next step that most developers would take is searching online for answers (i.e. the stack overflow approach). Thus, we looked for combined issues with the buff/cached and the free command.</p><p>This gave us the following search results:</p><ul><li>“buff/cache — The combined memory used by the kernel buffers and page cache and slabs. This memory can be reclaimed at any time if needed by the applications.”</li></ul><p>These commands force/trigger to clear the cache, which should usually clear automatically when the need arises. To try this out, either of the following commands would work:</p><ul><li>one way: echo 1 &gt; /proc/sys/vm/drop_caches</li><li>Another, less hacky: sysctl -w vm.drop_caches=1</li></ul><p>The screenshot below shows the size of the cache before and after.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*ayiO3pkghue9L7Qb.png" /></figure><p>Next, we used a one-liner script in the shell to keep the process alive and the cache low:</p><ul><li>while sleep 60; do sysctl -w vm.drop_caches=1 ; echo .; done</li></ul><p>When we were calling the script in the shell on the running container, the process did finish like expected. However, while running the script in combination with our own process, it still failed. In this case, we used the following command:</p><pre>sysctl -w vm.drop_caches=1</pre><pre>=&amp;gt; sysctl: setting key “vm.drop_caches”: Read-only file system</pre><p>Note that our Kubernetes cluster used spot instances. Thus, there might be other reasons for the crash than memory bloat. The best practice, especially when using spot instances, is to build the process as incremental work. If the process fails, restarting it will not make it work the second time around. In this case, the work would have to be down-scaled again.</p><p>If you are interested in the topic, you can find several suggestions online related to increasing proc/sys/vm/<a href="https://docs.gluster.org/en/latest/Administrator%20Guide/Linux%20Kernel%20Tuning/#vmvfs_cache_pressure">vfs_cache_pressure</a>.</p><h3>Second approach: Using Multiple Files instead of one</h3><p>Remember how we mentioned above that the file we wanted to transfer had over 20 million lines of data? Our second thought was that something along executing the file could have messed with our process, or that the file was simply too large. A colleague suggested writing many files instead of executing a single big one. Thus we divided the content across multiple files. At first, we used the following naming pattern: job1.json, job2.json…</p><p>It turned out, using multiple files was a lot better in various ways. Each file would run in sequential order. This allowed us to see the progress of the job and play with partial data.</p><p>But unfortunately, the process still failed in the end and we were back to square one.</p><p>Third approach: Using a single file multiple times</p><p>Our last approach was simply using the same filename “job” for each batch of work instead of naming every file differently.</p><p>In this case, the kernel clears the cached filesystem buffer nodes when the file is deleted/truncated.</p><p>Finally, this solved the issue and our ETL application finished successfully! Hurray!</p><p>One way to avoid writing large amounts of data to the file system is to write the data to a different destination like an external bucket service which is not cached by the Linux file system.</p><h3>Final thoughts</h3><p>When we started the experiments, we expected to see deterministic behavior. We expected the process to run as defined in the source code and produce the same output every time.</p><p>What we saw instead was a different behavior every time we ran the script.</p><p>Running our script always produced a different result depending on cluster type, memory allocation, and timing. This made our process quite unpredictable and added difficulty in determining the issues. Every time we would change one of the variables before running the code, the outcome would be different. It could result in a crash, the memory limit was reached, a slow down of our CPU, or we would receive an OSError: [Errno 28] No space left on device.</p><p>Overall, there is a lot more going on behind the Docker infrastructure abstraction which our script does not reveal.</p><p>While we did not find a definite cause of the problem, we found a workaround. Here are the main takeaways:</p><ul><li>When using Docker, don’t write large amounts of data into your filesystem. Instead, write it directly into external services.</li><li>If you still have to store data at an intermediate location, use a limited memory space and overwrite/delete the data once it is no longer needed. By flushing the buffer/cache we can avoid a memory bloat.</li><li>Do not reuse the same file to record an extensive amount of data.</li><li>Lastly, you should understand the implications of using spot instances for your Kubernetes cluster.</li></ul><p>Please let us know in the comments whether this story was interesting to you and whether you would like to read more of this type. Also, please share with us any questions or suggestions for us and for the community.</p><p>Remember — You can clap up to 50 times! Make sure that others see this post, too!</p><p>If you are new to Codefresh <a href="https://codefresh.io/codefresh-signup/?utm_source=Blog&amp;utm_medium=Post&amp;utm_campaign=dockermemoryexplosion">Create Your Free Account today</a> and try out the memory bloat!</p><p>Author: Saffi Hartal — Backend Engineer at Codefresh</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f759bed2b06f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/containers-101/my-process-used-minimal-memory-and-my-docker-memory-usage-exploded-f759bed2b06f">My Process Used Minimal Memory, and My Docker Memory Usage Exploded</a> was originally published in <a href="https://medium.com/containers-101">Container Hub</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Observability: From Push to Production]]></title>
            <link>https://medium.com/containers-101/observability-from-push-to-production-14ce8f2c01cf?source=rss-563ec507cbd3------2</link>
            <guid isPermaLink="false">https://medium.com/p/14ce8f2c01cf</guid>
            <category><![CDATA[production]]></category>
            <category><![CDATA[observability]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Codefresh]]></dc:creator>
            <pubDate>Fri, 10 Jul 2020 14:59:39 GMT</pubDate>
            <atom:updated>2020-07-24T19:43:47.221Z</atom:updated>
            <content:encoded><![CDATA[<p><strong><em>Editors note: We recorded a webinar on this very topic. Scroll to the bottom of this post to view the webinar.</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VjDGqv-K-Ig26GwmVs_--A.png" /></figure><p>Developers are building and deploying to production with greater frequency. Elite organizations are deploying to production multiple times per day. All the while we continue to distribute our applications even wider with the adoption of micro-services, and global deployments. This consistent churn and increasing code complexity create the perfect storm that makes finding problems even harder.</p><p>How do you know the changes just committed actually deployed? How do you know the changes worked?</p><p>Today, the hardest thing about debugging systems is no longer understanding how the code runs but finding where in your system is the code with the problem. We have traditionally tried to solve this using monitoring platforms, that look at known dimensions for unknown state. This is called monitoring your known-unknowns. Debugging this way depends on your prior knowledge of the system, which may no longer have relevance with the current state because of new code.</p><p>Observability is how you explain unknown-unknowns. It’s about exploration and debugging instead of dashboards and pattern matching. Being able to ask a question of any dimension, with any cardinality, form a thesis, ask another question, confirm (or deny) that thesis, and continue asking more questions creating a core analysis loop.</p><p>Developers need to understand their application across any dimension, including unique ones. The observability platform needs to work with raw data, ensuring we aren’t aggregating away the details. This allows us to understand why our systems behave the way they do. Tracking down application issues, like requests originating from us-west-2, for new users, of tenant type enterprise, breaking on an autosave feature. These debugging paths are not pre-conceived. You will not find the answer on a dashboard created based on prior knowledge of a service to understand how new functionality behaves.</p><p>With a strong observability platform and CI/CD pipeline in place, we can combine the developer toolchain, and deploy with confidence. During the deployment phase, annotating your observability platform with markers and indicators allows users to understand the exact moment the change took place and can understand if our change actually worked as intended in real-time.</p><p><strong>Watch the webinar on the topic now:</strong></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FS7LNQaxGz-w%3Fstart%3D1%26feature%3Doembed%26start%3D1&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DS7LNQaxGz-w&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FS7LNQaxGz-w%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/24626bf65cb3a7c0fc870d08fbca1114/href">https://medium.com/media/24626bf65cb3a7c0fc870d08fbca1114/href</a></iframe><p>Honeycomb enables developers to gain observability into their applications. The data storage and query engine was built to work with raw events that have high cardinality dimensions and no pre-aggregation of the data. The platform is exploratory by nature allowing developers to create a thesis about their application’s behavior and ask iterative questions to validate that thesis honoring the core analysis loop. Combined with integration into the Codefresh pipeline, organizations can deploy to production with confidence multiple times a day.</p><p><em>Author: Pierre Tessier, Sales Engineer at Honeycomb.io</em></p><p><em>Originally published at </em><a href="https://codefresh.io/webinars/observability-push-production/"><em>https://codefresh.io</em></a><em> on July 10, 2020.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=14ce8f2c01cf" width="1" height="1" alt=""><hr><p><a href="https://medium.com/containers-101/observability-from-push-to-production-14ce8f2c01cf">Observability: From Push to Production</a> was originally published in <a href="https://medium.com/containers-101">Container Hub</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Optimize Your CI/CD Pipeline with Codefresh & Lightstep]]></title>
            <link>https://medium.com/containers-101/optimize-your-ci-cd-pipeline-with-codefresh-lightstep-7e414d6748ab?source=rss-563ec507cbd3------2</link>
            <guid isPermaLink="false">https://medium.com/p/7e414d6748ab</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[cicd-pipeline]]></category>
            <category><![CDATA[continuous-delivery]]></category>
            <category><![CDATA[codefresh]]></category>
            <dc:creator><![CDATA[Codefresh]]></dc:creator>
            <pubDate>Mon, 18 May 2020 03:26:14 GMT</pubDate>
            <atom:updated>2020-07-24T19:46:29.889Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/693/1*xYBHgge05CgUrLKf7fzm7A.png" /></figure><p>How do you know if the changes that you’re making to your microservices are having the desired impact to your end-users? What happens after a deploy? Did your app’s performance remain the same? Were any bugs introduced? Do you need to roll back?</p><p>With Codefresh and Lightstep, you’re able to more quickly, more accurately, and more easily ensure that any commit and build results in a better experience for your end-users.</p><p><strong>Learn How to Use Codefresh and Lightstep to Monitor and Rollback Deployments</strong></p><p>In five simple steps, we’ll show you how to deploy an app using Helm and Codefresh, and <a href="https://docs.lightstep.com/paths/codefresh-path">automatically what matters most with Lightstep </a>.</p><ol><li>Create a version tag (known as attributes in OpenTelemetry) in your instrumentation.</li><li>View the marker in a service involved in the deploy to see performance before and after.</li><li>Create a comparison time period to view differences in latency, find system attributes that show correlation to that latency, and compare operation performance, helping you to pinpoint the issue.</li><li>Confirm your hypothesis.</li><li>Rollback your changes and confirm that issues have been fixed.</li></ol><p><strong>Dive Deep into the Lightstep and Codefresh integration</strong></p><p>In this webinar, you’ll learn how Codefresh’s easy deployments and Lightstep’s observability can combine to allow you to ensure that every new deployment is more performant, more stable, and more robust than your last.</p><p>You’ll learn how to easily set up continuous delivery of your code with Codefresh, and how to measure the impact of a deployment in Lightstep. In addition, you’ll see how easy it is to detect regressions and roll back to a healthy and stable state using Lightstep’s Service Health for Deployments.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FAnTXOCGYS0A%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DAnTXOCGYS0A&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FAnTXOCGYS0A%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/1c64d7e733f5a889fd248565bb028789/href">https://medium.com/media/1c64d7e733f5a889fd248565bb028789/href</a></iframe><p><em>Author: David Klein, Head of Growth at Lightstep</em></p><p><em>Originally published at </em><a href="https://codefresh.io/webinars/optimize-ci-cd-pipeline-codefresh-lightstep/"><em>https://codefresh.io</em></a><em> on May 18, 2020.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7e414d6748ab" width="1" height="1" alt=""><hr><p><a href="https://medium.com/containers-101/optimize-your-ci-cd-pipeline-with-codefresh-lightstep-7e414d6748ab">Optimize Your CI/CD Pipeline with Codefresh &amp; Lightstep</a> was originally published in <a href="https://medium.com/containers-101">Container Hub</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Docker Buildkit Support in Codefresh]]></title>
            <link>https://medium.com/containers-101/docker-buildkit-support-in-codefresh-5c4e8be7d813?source=rss-563ec507cbd3------2</link>
            <guid isPermaLink="false">https://medium.com/p/5c4e8be7d813</guid>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[codefresh]]></category>
            <category><![CDATA[buildkit]]></category>
            <category><![CDATA[cicd]]></category>
            <dc:creator><![CDATA[Codefresh]]></dc:creator>
            <pubDate>Mon, 11 May 2020 22:43:37 GMT</pubDate>
            <atom:updated>2020-07-24T17:50:56.945Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/699/1*afvn65NGZwXhtnJJydZdGA.png" /></figure><p>Codefresh has now added support for Docker Buildkit. If you don’t know what Buildkit is, and you’d like to learn more, read on! Otherwise, you can skip to the example.</p><h3>What is Buildkit?</h3><p>Buildkit is a feature offered in Docker <a href="https://docs.docker.com/develop/develop-images/build_enhancements/">since version 18.06 </a>. Essentially, it swaps out the traditional image builder with a new builder inside the Docker engine, providing for much faster, optimized Docker builds.</p><p>Let’s discuss some of Buildkit’s advantages:</p><ul><li><strong>Parallel Execution: </strong>Traditionally, layers in a Dockerfile are built sequentially, one at a time. With Buildkit enabled, you gain several speed improvements, as different layers of your image can now be built in parallel.</li><li><strong>Cache Efficiency and Flexibility: </strong>Buildkit will pull layers of the previously built image from a registry as opposed to a local image. Buildkit also gives you the ability to embed cache information into the image itself.</li><li><strong>Secret Storage: Buildkit allows secrets files to be mounted as a part of the build process. These secrets are kept in-memory and not stored within the image.</strong></li><li><strong>SSH Forwarding: SSH agents are mounted into the build without adding the private keys to the image</strong></li><li><strong>Improved Logging: Progress indicators are provided for each step of the Docker build, with options to see the full container log output, which is very useful for debugging your builds.</strong></li></ul><h3>How to Use Buildkit with Docker</h3><p>First, let’s go into how to use vanilla Buildkit outside of Codefresh.</p><p>To enable Buildkit, you need to export an environment variable:</p><pre>export DOCKER_BUILDKIT=1</pre><p>Another way to enable Buildkit, if you don’t wish to export an environment variable, is by using running the following:</p><pre>DOCKER_BUILDKIT=1 docker build .</pre><p>You can also set Buildkit to be your default builder, which is recommended if you build a lot of images, by adding to /etc/docker/daemon.json the following:</p><pre>{ &quot;features&quot;: { &quot;buildkit&quot;: true } }</pre><h3>Secret Storage with Buildkit</h3><p>Accessing private repositories or resources has always been an issue with the traditional Docker build engine. Even if you removed secret files after use, the secrets could end up in the metadata of the image. With Docker 18.09, a new -secrets flag was added that allows users to pass secret information to a Docker build, without storing that information in the final image.</p><p>Note that the usage of “secrets” here is in terms of temporary secrets needed as part of the build, i.e. private SSH keys needed for cloning a private repository. Do not use this functionality for permanent secrets needed to deploy an application, such as database credentials.</p><p>For example, we can create a secret file:</p><pre>echo “my-test-secret” &amp;lt; secret.txt</pre><p>Our Dockerfile will consist of the following:</p><pre># this allows us to use the new Dockerfile syntax<br># syntax = docker/dockerfile:1.0-experimental</pre><pre>FROM alpine</pre><pre># shows secret from default secret location:</pre><pre>RUN --mount=type=secret,id=mysecret cat /run/secrets/mysecret</pre><p>Then, to build the image, run:</p><pre>docker build -t buildkit-secret-demo --secret id=mysecret,src=secret.txt .</pre><p>This command passes the secret.txt file to the RUN command in the Dockerfile, mounting it at /run/secrets (the default secret location).</p><p>If you navigate to the /run/secrets directory in the container, you may be shocked to find your secrets file there, but it is only a blank, 0-byte file.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*qv0Q0qzyXMTbusKq.png" /></figure><h3>SSH with Buildkit</h3><p>Oftentimes, you will need to access a private repository via SSH as a part of your build. Of course, one option is to embed private keys into your image, but if you do that, it obviously cannot be shared.</p><p>Buildkit simplifies this by providing an --ssh flag that can forward your private key to the build engine.</p><p>Say our Dockerfile consists of the following:</p><pre># syntax=docker/dockerfile:experimental<br>FROM alpine</pre><pre># install ssh client and git</pre><pre>RUN apk add --update openssh-client git</pre><pre># download public key for github.com</pre><pre>RUN mkdir -p -m 0600 ~/.ssh &amp;amp;&amp;amp; ssh-keyscan github.com &amp;gt;&amp;gt; ~/.ssh/known_hosts</pre><pre># clone our private repository</pre><pre>RUN --mount=type=ssh git clone <a href="/service/mailto:git@github.com">git@github.com</a>:anna-codefresh/buildkit-demo.git</pre><p>You can then build this image by running:</p><pre>DOCKER_BUILDKIT=1 docker build --ssh default -t buildkit-ssh-demo .</pre><p>The git clone operation will now successfully clone the private repository, provided you have your private key saved into the local, default ssh-agent. If you have it loaded elsewhere, you can also specify its location by running:</p><pre>docker build --ssh github=path/to/id_rsa -t buildkit-ssh-demo .</pre><p>And modifying your Dockerfile SSH mount to:</p><pre>RUN --mount=type=ssh,id=github git clone <a href="/service/mailto:git@github.com">git@github.com</a>:anna-codefresh/buildkit-demo.git</pre><h3>Codefresh Built-in Buildkit Support</h3><p>Codefresh now supports the use of Buildkit within a pipeline’s build steps — all capabilities you have using Buildkit with Docker, you have within Codefresh.</p><p>Now that you have an understanding of what Buildkit is, and how to use it with Docker commands, let’s go over how to use the same functionality from a Codefresh pipeline.</p><p>The easiest way to enable Buildkit support in your pipeline, is by from within your build step.</p><pre>version: &#39;1.0&#39;</pre><pre>steps:<br>  BuildMyImage:<br>    title: Building My Docker image<br>    image_name: my-app-image<br>    type: build<br>    arguments:<br>      buildkit: true</pre><p>However, using any of the buildkit features enables it by default.</p><p>Note, that Buildkit is an opt-in feature, and optional in Codefresh. You can use both Buildkit and non-Buildkit steps without any limitations, from within the same pipeline.</p><h3>SSH Example Using Codefresh and Buildkit</h3><p>Here I will guide you on how to easily use Buildkit’s SSH feature within a Codefresh pipeline.</p><p>We will first store our SSH key as a <a href="https://codefresh.io/docs/docs/configure-ci-cd-pipeline/shared-configuration/">Shared Secret </a>(shared meaning we can share this secret amongst different pipelines and projects). From the left-hand panel, navigate to <strong>Account Settings </strong>&gt; <strong>Shared Configuration</strong>. Select the <strong>Add Shared Values</strong> button and from the drop-down, click on <strong>Shared Secret</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/758/0*Bvi1-uOLSb2ni9WX.png" /></figure><p>Give your secret a name and click <strong>Save</strong>. Give it a key, i.e., <em>GITHUB_SSH_KEY </em>and paste your SSH key into the value field and click <strong>Save</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*GKh0I8YW-XP5QeVK.png" /></figure><p>Now, let’s define a pipeline using our Shared Secret and the <a href="https://codefresh.io/docs/docs/configure-ci-cd-pipeline/pipelines/#using-the-inline-pipeline-editor">Inline YAML editor </a>. To the right of the in-line editor, select the <strong>Variables</strong> tab, the three dots in the upper-right corner, and then <strong>Add Shared Configuration</strong>. Select the Shared Configuration we created earlier:</p><p>Below is a YAML example of the pipeline. Of course, you will need to modify the clone operation <a href="https://github.com/anna-codefresh/buildkit-demo/blob/07be61319ba353dfbe5df03f8b9647d821bd4ab9/Dockerfile#L15">in the Dockerfile </a>of the demo repo to your own private repository in order for this pipeline to run.</p><pre># More examples of Codefresh YAML can be found at<br># <a href="https://codefresh.io/docs/docs/yaml-examples/examples/">https://codefresh.io/docs/docs/yaml-examples/examples/</a></pre><pre>version: 1.0</pre><pre>stages:<br>  - clone<br>  - setup<br>  - build</pre><pre>steps:<br>  clone:<br>    title: Cloning repository...<br>    type: git-clone<br>    stage: clone<br>    arguments:<br>      repo: <a href="https://github.com/anna-codefresh/buildkit-demo.git">https://github.com/anna-codefresh/buildkit-demo.git</a><br>      revision: master<br>      <br>  setup_ssh_key:<br>    title: Setting up SSH key...<br>    type: freestyle<br>    stage: setup<br>    arguments:<br>      image: alpine:latest<br>      commands:<br>        - mkdir /codefresh/volume/keys<br>        - echo &quot;${GITHUB_SSH_KEY}&quot; | tr  &#39;,&#39; &#39;\n&#39; &amp;gt; /codefresh/volume/keys/github_rsa<br>        - cat /codefresh/volume/keys/github_rsa<br>      <br>  build:<br>    title: Building Docker Image,..<br>    type: build<br>    stage: build<br>    working_directory: ${{clone}}<br>    arguments:<br>      image_name: buildkit-ssh-demo<br>      tag: latest<br>      dockerfile: Dockerfile<br>      buildkit: true<br>      progress: tty<br>      ssh:<br>        - github=/codefresh/volume/keys/github_rsa</pre><p>This pipeline does the following:</p><ol><li>Uses a <a href="https://codefresh.io/docs/docs/codefresh-yaml/steps/freestyle/">freestyle step </a>to write our SSH key from a shared configuration to the Codefresh Volume</li><li>Builds the Docker image <a href="https://codefresh.io/docs/docs/codefresh-yaml/steps/build/#buildkit-support">using Buildkit </a>, forwarding our private key to the SSH agent</li><li>Builds the Docker image <a href="https://codefresh.io/docs/docs/codefresh-yaml/steps/build/#buildkit-support">using Buildkit</a>, forwarding our private key to the SSH agent</li></ol><p>Note the improved logging Buildkit provides, with timestamps and improved output at each step of the Docker build:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*EDW4hsLeTS4GBfgZ.png" /></figure><p>To recap, let’s take a look at our Dockerfile:</p><pre># syntax=docker/dockerfile:experimental</pre><pre>FROM alpine</pre><pre># install ssh client and git</pre><pre>RUN apk add --no-cache openssh-client git</pre><pre># download public key for github.com</pre><pre>RUN mkdir -p -m 0600 ~/.ssh &amp;amp;&amp;amp; ssh-keyscan github.com &amp;gt;&amp;gt; ~/.ssh/known_hosts</pre><pre># clone our private repository</pre><pre>RUN --mount=type=ssh,id=github git clone <a href="/service/mailto:git@github.com">git@github.com</a>:anna-codefresh/buildkit-private-demo.git</pre><p>Our Dockerfile downloads the public key from Github and adds it to our known_hosts. Using the public key, it passes our private SSH key to the second RUN command (and only this specific RUN command, nothing else), allowing us to access our private repository, all without storing any confidential information inside the image itself.</p><h3>Conclusion</h3><p>While you can use Buildkit’s secret storage for things like SSH keys, check out our new <a href="https://codefresh.io/docs/docs/configure-ci-cd-pipeline/secrets-store/">Secrets Storage support </a>! Using the new Secret Store, you can keep all kinds of secrets within your Kubernetes cluster, without storing them as a shared configuration in Codefresh, as we saw in the previous example. This helps alleviate security concerns, keeping your data safe and protected.</p><p>Buildkit is the next generation of Docker image building, with improved efficiency and security of your Docker builds. It is definitely the recommended way to go if you are building a lot of images. As it is in its early stages, it still has some growing up to do, but hopefully from this article, you can see its many advantages (all of which we did not describe here).</p><p>If you’d like to learn more about Buildkit and its many advantages not covered by this article, see some of the <a href="https://github.com/moby/buildkit">official documentation</a>.</p><p>Or, for more information on Codefresh Buildkit Support, see the Codefresh <a href="https://codefresh.io/docs/docs/codefresh-yaml/steps/build/#buildkit-support">documentation</a>.</p><p>New to Codefresh? <a href="https://codefresh.io/codefresh-signup/?utm_source=Blog&amp;utm_medium=Post&amp;utm_campaign=13keyFeatures">Sign up for your free account today!</a></p><p><em>Author: Anna Baker, Technical writer&amp; Developer Evangelist for Codefresh</em></p><p><em>Originally published at </em><a href="https://codefresh.io/docker-tutorial/announcing-docker-buildkit-support/"><em>https://codefresh.io</em></a><em> on May 11, 2020.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5c4e8be7d813" width="1" height="1" alt=""><hr><p><a href="https://medium.com/containers-101/docker-buildkit-support-in-codefresh-5c4e8be7d813">Docker Buildkit Support in Codefresh</a> was originally published in <a href="https://medium.com/containers-101">Container Hub</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[13 Key Features Every Modern CI/CD Tool Should Contain]]></title>
            <link>https://medium.com/containers-101/13-key-features-every-modern-ci-cd-tool-should-contain-ed05ea807304?source=rss-563ec507cbd3------2</link>
            <guid isPermaLink="false">https://medium.com/p/ed05ea807304</guid>
            <category><![CDATA[continuous-integration]]></category>
            <category><![CDATA[continuous-delivery]]></category>
            <category><![CDATA[continuous-deployment]]></category>
            <category><![CDATA[cicd]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Codefresh]]></dc:creator>
            <pubDate>Fri, 08 May 2020 17:05:30 GMT</pubDate>
            <atom:updated>2020-07-24T17:50:03.116Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yJD_x0sdSi3vK_f48xFrVQ.png" /></figure><p>DevOps methodologies have become a huge staple of the Software Development Lifecycle. As such, more and more companies have been adopting the need for Continuous Integration/Continuous Delivery (CI/CD) tools. Building a successful CI/CD workflow can be a tedious process that requires your team to participate in a large amount of preparation and planning — but it doesn’t have to be that way.</p><p>There are a growing number of CI/CD tools available on the market today, so much that your team may be overwhelmed in narrowing down what to use. In this blog post, we hope to help simplify your choices by breaking down the 13 key features every modern CI/CD tool should contain:</p><ol><li>Docker-based Architecture Right From the Start</li><li>Cloud and Version Control Agnosticism</li><li>Pipeline Creation with Standardized Definitions</li><li>Graphical Pipeline View</li><li>Parallel Steps</li><li>Standardized Plugin Mechanism (Docker-based)</li><li>Configuration Options Through Both Code and UI</li><li>Reusable Pipelines for Microservices</li><li>Live Pipeline Debugging with Breakpoints</li><li>Native Support for Kubernetes, Helm, and Docker</li><li>Saas, On-prem, and Hybrid Installation Methods</li><li>Zero Config, Distributed Caching</li><li>Monorepo Support</li></ol><h3>Docker-based Architecture Right From the Start</h3><p>A modern CI/CD tool should have Docker-based architecture since its inception. Many people consider “Docker-based” to mean the usage of Docker as a deployment package, but the usage of Docker-based in this sense means using Docker for the build tooling itself. Traditionally, in VM-based CI/CD tools, you had to make sure all of your build tooling (and correct versions) were pre-installed on the virtual machine in order to run the pipeline. Tools were static and used for all pipelines.</p><p>With Docker-based CI/CD tooling, only Docker needs to be pre-installed on your build nodes — nothing else. Your build tools are now isolated to each pipeline, and you can dynamically pop them in or out based on your needs. Dockerizing the CI/CD architecture itself allows your team to create and reuse building blocks to make pipeline creation much faster and easier, cutting the amount of time and cost it takes to get your CI/CD builds up and running:</p><ul><li>Docker is the de-facto standard for delivery and not tied to any vendor</li><li>Any Docker image is a potential step for a pipeline</li><li>Create pipelines quickly from existing images</li><li>Extending a pipeline step requires Docker knowledge and nothing more</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*wxRGerJpggvhVLY9.png" /></figure><p>While a number of tools have strategically shifted to a “Docker-based” approach, Codefresh offered this functionality right from the start from the heart of its architecture. All Codefresh pipelines use Docker images in one form or another, whether it be by using them as runtime tools, or creating them as deployment artifacts. Codefresh has always required that all tools used in a pipeline be Dockerized, and each step in the pipeline runs in the context of its own Docker image:</p><h3>Cloud and Version Control Agnosticism</h3><p>Many CI/CD tools limit which providers you can use for both version control and cloud providers. A good CI/CD tool should be cloud and version control agnostic — it should:</p><ul><li>Play well with any Git or Cloud provider</li><li>Give you a great amount of flexibility on how you choose to version control and deploy your code</li><li>Have no vendor lock-in</li></ul><p>Many tools do not offer such agnosticism with version control, and are tied to their own version control systems. The same can be said for Cloud agnosticism as well. Some CI/CD tools from major Cloud Service Providers can be examples of this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ItV6J3xe9v431ull.png" /></figure><p>One of the biggest advantages of Codefresh is the fact that it offers native support (and great documentation!) for <strong>any</strong> or provider, giving you a great amount of flexibility on how you build your pipelines. With Codefresh, there is no vendor-lock in and you are free to choose the tools best-suited for your needs.</p><p>Oftentimes, companies use multiple different cloud and git providers, both on-premise and in the cloud, so offering cloud and git agnosticism is essential.</p><h3>Pipeline Creation with Standardized Definitions</h3><p>Using a standardized YAML approach for programmatic deployments is important. Some tools offer non-standardized approaches, offering pipeline configuration in different modes, such as scripted, procedural Groovy pipelines. Scripted pipelines do not provide a strict and pre-defined structure, which introduces a lot of complexity when it comes to building out your pipelines, making them complicated and hard to manage. In addition, scripted pipelines force operators into learning a full programming language just to pick up and implement the CI/CD system.</p><p>Codefresh simplifies this by offering only one mode for constructing pipelines: <a href="https://codefresh.io/docs/docs/codefresh-yaml/what-is-the-codefresh-yaml/">declarative YAML </a>. This keeps your pipelines (and the documentation supporting it):</p><ul><li>Standardized and easy to pick up and implement, speeding up the time it takes to get from source code to deployment.</li><li>Managed via source file and able to go through a standard PR / code review process, meaning the state of your code and your pipeline are always aligned</li><li>Reusable, as YAML can be used as building blocks to distribute to other projects</li></ul><p>In addition to a one-size-fits-all approach to pipeline configuration, a modern CI/CD tool should allow for easy, global, one-time authentication/integrations for Docker registries and Kubernetes clusters. With a modern CI/CD tool, the less manual configurations, the better, as it:</p><ul><li>Enables your DevOps team to get up and running, focusing on building out their pipelines as opposed to dealing with messy configuration options</li><li>Reduce the amount of time it takes to administer authentication options within your pipeline</li><li>Makes CI/CD easy for everyone to use without requiring everyone to learn a new language to create pipelines</li></ul><p>Codefresh provides this capability automatically to all pipelines. You authenticate one time through your global account settings to Docker registries (without the need to run docker login), Kubernetes clusters (without the need to run kubectl commands), Secret repositories, Helm repositories, and so on — once you configure this one time, authentication through the pipeline is all handled automatically for you.</p><h3>Graphical Pipeline View</h3><p>Any modern CI/CD tool should provide an intuitive user interface, complete with a graphical pipeline view of your workflow. Most CI/CD tools offer this, but the user interface/experience can be lacking in critical build information and modern design principles.</p><p>At Codefresh, we provide a user-friendly experience when it comes to viewing your pipelines graphically. Your pipeline steps can even be organized into different stages — they are completely customizable. You define as many stages as you wish, and define what steps are grouped under each stage. Our graphical pipeline view provides:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*4SToU4zfUnA64iHY.png" /></figure><ul><li><strong>Any</strong> sort of stakeholder an overview of your pipeline and what is occurring at each step</li><li>The YAML, logs, state, and metrics of the build as a whole or as a part of any particular step</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hqBNy_OdmsfbmvsJ.png" /></figure><p>Getting a holistic view of your pipeline is extremely important, the larger and more complex your pipeline gets.</p><p>The ability to see the metrics of your pipeline allows you to instantly view bottlenecks and optimize slow build times without any guesswork.</p><h3>Parallel Steps</h3><p>Most common pipeline steps run sequentially, meaning each step in the pipeline is executed individually, one at a time. The ability to run steps in parallel as opposed to sequentially is a very important feature of any modern CI/CD tool, as it can dramatically decrease the amount of time it takes for your pipeline to complete.</p><p>For example, consider a sequential pipeline with 5 steps, each step running its own set of tests: unit tests, fast tests, slow tests, smoke tests, and integration tests. For simplicity, let’s say each set of tests takes 1 minute to run. In a sequential pipeline, the build time would be a total of 5 minutes. Parallelizing all of these steps would cut the time it takes the pipeline to run down to 1 minute, reducing the time your engineers wait for a build to finish. CI/CD tools that offer parallelization are beneficial in:</p><p>Most modern tools offer this capability, but only at a basic level of splitting a single task into parallel tasks Codefresh takes it one step further, by allowing you to create a custom graph of dependencies between parallel steps, giving you maximum flexibility on any complex flow.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/716/0*5Q0ApGbgHvFXTD3i.png" /></figure><p>For example, at Codefresh, to run steps in parallel, we support two modes: one for parallelizing your entire pipeline, and one for parallelizing only specific portions of it.</p><p>The advantage of parallelism within Codefresh is that we offer support for an advanced parallel mode with full step definitions and their explicit dependencies, allowing for complex <a href="https://codefresh.io/docs/docs/yaml-examples/examples/fan-in-fan-out/">fan-in, fan-out configurations </a>as seen in the diagram above.</p><h3>Standardized Plugin Mechanism (Docker-based)</h3><p>Many CI/CD tools support a marketplace of plugins, but the method of creating one is not standardized. A modern CI/CD tool should have a standardized, Docker-based mechanism for plugin creation. This allows:</p><ul><li>Developers to create their own plugins without special knowledge of the CI/CD tool’s API or specific language used for that platform.</li><li>Every team member to create plugins in a standardized environment, i.e. solving the issue of “it works on my machine”</li><li>Reuse of existing plugins from other teams</li><li>The ability to write a plugin in <strong>any</strong> programming language, as long as it is packaged in a Docker image</li><li>Any public Dockerhub image is a potential pipeline step within your plugin</li></ul><p>At the base of every plugin, Codefresh uses a Docker image, simplifying the plugin creation process. And, since Codefresh has a Docker-based build architecture from its heart, you can easily plug-and-play with any plugin from our marketplace into your pipelines.</p><p>Take a look at <a href="https://codefresh.io/steps/">any of our plugins </a>(they are all open-source!) and you will see that they are composed of a Dockerfile, making it easy for developers to extend and add their own, without any background knowledge of our API. In addition, plugins are scoped per pipeline, so you have no dependency hell to monitor between plugins, or a need to monitor which versions are compatible with your platform version.</p><h3>Configuration Options Through Both Code and UI</h3><p>A modern CI/CD tool should have both Configuration as Code as well as Graphical User Interface options. Many tools offer configuration through code, but lack in UI configurations, or vise versa. Allowing a means of both offers:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/375/0*Dql3IsWnJ_agdH9R.png" /></figure><p>Codefresh offers both types of configurations for many options. For example, we offer an in-line editor you can use directly from the UI to perform dry-runs of your pipeline, and once your pipeline is ready, you can move it to source control and use it from there. The pipeline can even be in a different repository from your application’s source, allowing for maximum flexibility. Moreover, you can define variables as a part of the YAML itself, or, we give you the flexibility to define these variables as a part of the pipeline settings, through the UI:</p><p>Developers and beginners tend to work with the GUI, while Ops teams and advanced users tend to work with configuration as code. Offering a means to support both keeps teams happy and is vital to any organization.</p><h3>Reusable Pipelines for Microservices</h3><p>A modern CI/CD tool should have a mechanism in place to reuse pipelines for Microservice applications. Most microservices have a similar pipeline pattern in place. Since those similarities exist, it should be possible to reuse a single pipeline across many microservices.</p><p>One of the ground-breaking features of Codefresh is that pipelines are not bound to specific Git repositories, meaning you can re-use them amongst different microservices by just adding more triggers. This allows for simplified:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/662/0*2JKUqX8KGSNyrv2h.png" /></figure><p>Codefresh offers a variety of triggers, including Git, Dockerhub, Azure, Quay, Artifactory, Cron, and the Codefresh API/CLI.</p><p>Take for example, the above diagram. The pipeline process is the same for all microservices, and four microservices already exist (and thus, four triggers). When a new microservice is created, the Ops team simply adds a new trigger to the existing pipeline, and the work is finished. This speeds up the amount of time needed to create a new project, compared to other solutions where bootstrapping a new microservice can take days to complete.</p><h3>Live Pipeline Debugging with Breakpoints</h3><p>The ability to troubleshoot a failed build is absolutely essential when it comes to an efficient CI/CD tool. Many CI/CD tools offer simple debugging with SSH, enabling users to debug a failed build from the builder node.</p><p>Codefresh takes a unique approach to pipeline debugging, on the other hand, by being the <strong>first ever </strong>CI/CD solution to make <a href="https://codefresh.io/docs/docs/configure-ci-cd-pipeline/debugging-pipelines/#using-the-debugger-terminal">pipeline debugging live </a>:</p><ul><li>Live debugging is more user-friendly and the way most developers are already accustomed to debugging their code</li><li>It makes the CI/CD creation process easier for engineers to test, troubleshoot, and fix their pipelines</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/449/0*Ll8hPcv915yR1BWQ.png" /></figure><p>Similar to the way developers debug applications live in their IDEs, you can place breakpoints within your pipeline steps to inspect the live states of your build.</p><p>Once the pipeline pauses at your breakpoint, you have the ability to run any commands you wish to understand the state of your container, making creating, modifying, and testing your pipelines much easier.</p><h3>Native Support for Kubernetes, Helm, and Docker</h3><p>This is a growing area of interest, as many companies wish to adopt cloud native approaches. Many tools offer integrations with Kubernetes, Helm, and Docker through the use of plugins, but few have native support for all three. In fact, Codefresh works out-of-the-box with Kubernetes, Helm, and Docker, without the need of any special plugins or configurations. It is the only CI/CD solution that provides native Kubernetes, Helm, and Docker dashboards.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*dzIsnYG5kCL-ztnD.png" /></figure><p><em>Kubernetes dashboard with full traceability into your clusters.</em></p><ul><li>Full traceability into your clusters, as you can monitor what images are currently running, and trace them all the way back to an individual commit.</li><li>The ability to create and edit services, deployments, and more</li><li>Visibility across clusters all in one place, along with their state, status, and any errors</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*K6eKwq0ihAOYuWmp.png" /></figure><p><em>Built-in Helm chart repository view, with one-click installation methods.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*wfaI5Jdeo9ya4T0Z.png" /></figure><p><em>Helm release dashboard, where you can view the status of currently deployed releases.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*x6U3-94Lcq84Hc79.png" /></figure><p><em>Helm promotion dashboard, giving you an overview of where each release exists in your development lifecycle, and the ability to promote different releases to different environments.</em></p><ul><li>A built-in Codefresh-hosted chart repository with built-in, one-click installation</li><li>View the status of currently deployed releases</li><li>One-click rollbacks of your production Kubernetes cluster</li><li>A rolling version history (with diffs) of various chart versions</li><li>Drag-and-drop Helm release promotions to different environments</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*v62v-0bh91DhpbWW.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*SGz9J9tFEKwMgUxM.png" /></figure><p><em>Docker Image Dashboard with Meta-data</em></p><ul><li>Details such as the git branch, commit message, and hash that created it, date of creation as well as any tags</li><li>You can also select any image and look at its individual metadata</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*kcENCR4UtVkPuvy-.png" /></figure><p>From your first commit to your production deployment, Codefresh offers full traceability of your software development lifecycle. You can view all phases of development within a single, unified platform. With other CI/CD solutions, you don’t receive the benefit of this holistic view and have to investigate using other platforms just to trace what is going on with a release.</p><h3>SaaS, On-prem, and Hybrid Installation Methods</h3><p>There is always a debate as to whether or not to host the CI/CD system yourself, or offload the work to a cloud-based/SaaS service. There are pros and cons to each approach, namely, offloading the work can be beneficial and faster, as you can access a shared pool of the CI/CD tool’s hosted resources to run your pipelines. But, with this approach, there are often security concerns around code/data accessibility and many companies feel safer running their pipelines from their own machines. The downside to this approach is that any and all maintenance is now your responsibility.</p><p>Any modern CI/CD tool should not lock its customers down into one installation method, but give customers flexibility. In fact, to solve the debate between on-prem vs. SaaS solutions, Codefresh offers the best of both worlds, by offering a <a href="https://codefresh.io/docs/docs/enterprise/behind-the-firewall/">Hybrid installation model </a>. With this mode:</p><ul><li>The Codefresh UI runs on the Codefresh infrastructure</li><li>Actual builds happen via the customer’s location (behind the firewall, from a Kubernetes cluster).</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*A5ZHfgg6BZlvwClQ.png" /></figure><p>This keeps security concerns at bay, while still leaving all the heavy lifting and maintenance of the platform up to Codefresh.</p><p>With the Hybrid mode, you can take advantage of specific custom infrastructure within Codefresh, such as GPU nodes and other specialized hardware. You can also use SAAS and Hybrid simultaneously, so some pipelines can run in the cloud, while others run behind your firewall.</p><h3>Zero Config, Distributed Caching</h3><p>A modern CI/CD tool should contain a multitude of caching mechanisms that take place as a part of your pipelines. These caching mechanisms should be:</p><ul><li>Automatic, and require little to no configuration</li><li>Distributed across all build nodes or pipeline steps</li></ul><p>This allows for decreased network costs, improved responsiveness, and increased hardware performance — all in all, reducing your build times. Lengthy build times are bad for your team for various reasons:</p><ul><li>Reduction in commit frequency</li><li>Due to the cost in time it takes to build, developers will put off building and submit code in larger batches of work, creating a larger surface area for change</li><li>This leads to more difficult integration and a greater risk for merge conflicts</li><li>Small improvements become costly, leading to increased life cycle time overall</li></ul><p>Distributed caching is a great way to mitigate the risks of slow build times, and Codefresh offers several types:</p><ul><li>Mimics the way Docker Layer Caching behaves locally on your workstation — when building images, Docker will cache intermediate layers making future builds much faster</li><li>Docker Layer Caching is distributed, meaning all build nodes in your pipelines share the same cache. Any available node can pick up your next pipeline build, as all of them share access to previously cached Docker layers.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/861/0*PX61CcRGJuIw4r82.png" /></figure><ul><li>Many CI solutions make you manually configure what folder you want to cache</li><li>Codefresh offers an internal Docker volume, under /codefresh/volume that is automatically cached for subsequent builds of the same pipeline</li></ul><h3>Monorepo Support</h3><p>Monorepo-style projects take an approach where:</p><ul><li>Multiple projects are committed to the same repository</li><li>Projects depend upon one another and can share a common codebase</li><li>When changes are made, only specific projects affected by that change are rebuilt/retested, as opposed to rebuilding the entire monorepo itself</li></ul><p>The advantages of monorepos are:</p><ul><li>Developers only need to update and check-out one repository</li><li>Single source of truth for all versions of your code and their dependent projects</li><li>Atomic commits: changes to multiple projects can be checked in as one commit</li><li>Merge code only once, as opposed to multiple times across multiple repositories</li><li>Simplification of merge conflict handling</li></ul><p>Monorepo support should be a key feature of any modern CI/CD tool, as it greatly decreases the amount of builds occuring in your daily development cycles. If you use a CI/CD tool without monorepo support, you will:</p><ul><li>Go through the entire build/test/deploy process for the entire repository when any changes are introduced in any one microservice</li><li>Have to create custom in-house tools that mimic the same functionality Codefresh offers out-of-the-box</li></ul><p>Transitioning to a monorepo from a multi-repo approach may require you to rethink how you structure your CI/CD process. You only need to build the artifacts affected by that change, as you are no longer building a single application, i.e. triggering pipelines to execute only when changes happen in specific folders.</p><p>Codefresh simplifies this process for you, by providing easy-to-use monorepo support. In the trigger of each pipeline, you define a glob expression that maps to the project’s files. Only when the expression is matched, will the pipeline execute, meaning you can define pipelines that run a sequence of operations only on the microservices you specify, despite them being located in the same repository.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/538/0*FPvyjUsKQGYfuY4Q.jpg" /></figure><p>In the above diagram, we have four projects in our mono-repo. We create two separate pipelines, that execute on different triggers based on their glob expressions.</p><h3>Conclusion</h3><p>I hope this list was helpful in providing you with the information you need to choose an efficient, modern CI/CD tool. Obviously, you will need to select a tool that best fits your needs as a team, but the items listed here are what I consider to be a bare-minimum in terms of what your selected product has to offer. Technology moves very fast, and it’s important that your CI/CD tool of choice is up-to-date and evolves with new development trends and practices.</p><p>New to Codefresh? <a href="https://codefresh.io/codefresh-signup/?utm_source=Blog&amp;utm_medium=Post&amp;utm_campaign=13KeyFeaturesAModernCICDToolShouldContain">Sign up for a free account today</a>!</p><p><em>Author: Anna Baker, Technical writer&amp; Developer Evangelist for Codefresh</em></p><p><em>Originally published at </em><a href="https://codefresh.io/continuous-deployment/15588/"><em>https://codefresh.io</em></a><em> on May 8, 2020.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ed05ea807304" width="1" height="1" alt=""><hr><p><a href="https://medium.com/containers-101/13-key-features-every-modern-ci-cd-tool-should-contain-ed05ea807304">13 Key Features Every Modern CI/CD Tool Should Contain</a> was originally published in <a href="https://medium.com/containers-101">Container Hub</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using Codefresh to Deploy a Windows Server Application to Google Kubernetes Engine]]></title>
            <link>https://medium.com/containers-101/using-codefresh-to-deploy-a-windows-server-application-to-google-kubernetes-engine-aa169c1b9207?source=rss-563ec507cbd3------2</link>
            <guid isPermaLink="false">https://medium.com/p/aa169c1b9207</guid>
            <category><![CDATA[cicd]]></category>
            <category><![CDATA[google-kubernetes-engine]]></category>
            <category><![CDATA[codefresh]]></category>
            <category><![CDATA[gke]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Codefresh]]></dc:creator>
            <pubDate>Mon, 27 Apr 2020 09:23:50 GMT</pubDate>
            <atom:updated>2020-05-01T17:15:18.564Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DsHHu4rSk9tSWhlnhPlgsw.png" /></figure><p>While Kubernetes has been traditionally used with Linux workloads, the desire to run Windows applications is an important need for many organizations that have critical applications running on Windows Server.</p><p>Docker has already offered support for native Windows containers, so the next missing piece would be Windows node support in Kubernetes clusters. Google Cloud has recognized this gap and is now offering <a href="https://cloud.google.com/blog/products/containers-kubernetes/run-windows-server-containers-on-gke">Windows support for Kubernetes clusters</a>.</p><p>Windows clusters use a mix of Linux and Windows nodes. The Linux nodes run critical services of the cluster itself, but the Windows nodes are fully available to run Windows applications. The cluster is exposing the standard Kubernetes API which means that all <a href="https://codefresh.io/docs/docs/deploy-to-kubernetes/deployment-options-to-kubernetes/">Codefresh features</a> are available even in the case of Windows clusters without any extra configuration.</p><h3>Creating a Windows cluster on GKE</h3><p>To create a Windows cluster you can follow the <a href="https://cloud.google.com/kubernetes-engine/docs/how-to/creating-a-cluster-windows">official documentation pages</a>. The easiest is by running all commands in the Google Cloud console:</p><pre>gcloud beta container clusters create windows-on-gcp --enable-ip-alias --num-nodes=2 --release-channel=rapid</pre><pre>gcloud container node-pools create dozer --cluster=windows-on-gcp --image-type=WINDOWS_SAC --enable-autoupgrade --machine-type=n1-standard-2</pre><p>Once that is done you can see the cluster in the Google cloud console (notice the two kinds of node pools)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/878/1*Xa3ourJ8VCnHhjme3J8Jzw.png" /></figure><p>You can also connect to this cluster like any other cluster and run all familiar kubectl commands</p><p>This concludes the setup in Google Cloud. Let’s move now to the Codefresh part.</p><h3>Connecting the cluster to Codefresh</h3><p>Codefresh has native support for connecting to Google Cloud clusters and the fact that this cluster is running Windows nodes doesn’t make any difference. You can follow <a href="https://codefresh.io/docs/docs/deploy-to-kubernetes/add-kubernetes-cluster/#adding-gke-cluster">the standard documentation</a> to connect your cluster centrally to your Codefresh account.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*A7KS70Uf2XRRzeZzo3rxzA.png" /></figure><p>Once the cluster is connected you will see its nodes in the preview window. The cluster will also be available for all other dashboards in Codefresh such as the <a href="https://codefresh.io/docs/docs/deploy-to-kubernetes/manage-kubernetes/">service dashboard</a> or the <a href="https://codefresh.io/docs/docs/deploy-to-kubernetes/environment-dashboard/">environment dashboard</a>.</p><h3>Creating a deployment pipeline</h3><p>Next, we are going to deploy a Windows application on the cluster. You can see the example deployment at <a href="https://github.com/todaywasawesome/windows-k8s-demo">https://github.com/todaywasawesome/windows-k8s-demo</a>. We will create the following pipeline:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*I2fDHd2odhH8S9B-O7JbMg.png" /></figure><p>The pipeline has only two steps:</p><ol><li>First, it checks out the deployment manifest</li><li>Then it uses the <a href="https://codefresh.io/docs/docs/deploy-to-kubernetes/kubernetes-templating/">Codefresh deployment step</a> to start the release.</li></ol><p>Here is the full pipeline definition.</p><pre>version: &quot;1.0&quot;<br>stages:<br>  - &quot;clone&quot;<br>  - &quot;build&quot;<br>  - &quot;deploy&quot;<br>steps:<br>  clone:<br>    title: &quot;Cloning repository&quot;<br>    type: &quot;git-clone&quot;<br>    repo: &quot;todaywasawesome/windows-k8s-demo&quot;<br>    revision: &quot;${{CF_BRANCH}}&quot;<br>    git: &quot;todaywasawesome&quot;<br>    stage: &quot;clone&quot;<br>  deploytowindows:<br>    title: Deploy App to Windows Kubernetes<br>    image: codefresh/cf-deploy-kubernetes:master<br>    stage: &quot;deploy&quot;<br>    commands:<br>      - /cf-deploy-kubernetes ${{CF_VOLUME_PATH}}/windows-k8s-demo/dotnet-framework.yaml<br>    environment:<br>      - KUBECONTEXT=windows-on-gcp@FirstKubernetes<br>      - KUBERNETES_NAMESPACE=default<br>      - MY_REPLICA_NUMBER=3<br>      - KUBERNETES_DEPLOYMENT_TIMEOUT=360</pre><p>For more details see the <a href="https://codefresh.io/docs/docs/codefresh-yaml/what-is-the-codefresh-yaml/">Codefresh YAML reference</a>.</p><h3>Inspecting the deployment</h3><p>Once the pipeline has finished you can see your deployment in the service dashboard:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*e5MQmiVOfZCHa-dezvQ_ew.png" /></figure><p>The deployment manifest also includes a service definition to expose the sample application. You can click on the endpoint URL to see the application:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4XPYg8_TuGv1iOW1-QA74w.png" /></figure><p>That’s it! A Windows application running on a Google Kubernetes cluster on a Windows node. Because of the power of the Kubernetes API, the deployment process is essentially the same as deploying a Linux application.</p><p>For more details see the Google <a href="https://cloud.google.com/kubernetes-engine/docs/how-to/creating-a-cluster-windows">documentation for Windows on Kubernetes</a>.</p><p>New to Codefresh? <a href="https://codefresh.io/codefresh-signup/?utm_source=Blog&amp;utm_medium=Post&amp;utm_campaign=imageApiRoadmaphttps://codefresh.io/codefresh-signup/?utm_source=Blog&amp;utm_medium=Post&amp;utm_campaign=windowsgcp">Create Your Free Account Today</a>!</p><p><em>Kostis Kapelonis is a software engineer/technical-writer dual class character. He lives and breathes automation, good testing practices and stress-free deployments.</em></p><p><em>Originally published at </em><a href="https://codefresh.io/kubernetes-tutorial/windows-gke-codefresh/"><em>https://codefresh.io</em></a><em> on April 27, 2020.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=aa169c1b9207" width="1" height="1" alt=""><hr><p><a href="https://medium.com/containers-101/using-codefresh-to-deploy-a-windows-server-application-to-google-kubernetes-engine-aa169c1b9207">Using Codefresh to Deploy a Windows Server Application to Google Kubernetes Engine</a> was originally published in <a href="https://medium.com/containers-101">Container Hub</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Making the Most of Helm 3]]></title>
            <link>https://medium.com/containers-101/making-the-most-of-helm-3-b52f2e0cb00e?source=rss-563ec507cbd3------2</link>
            <guid isPermaLink="false">https://medium.com/p/b52f2e0cb00e</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[containers]]></category>
            <category><![CDATA[helm]]></category>
            <category><![CDATA[helm-3]]></category>
            <dc:creator><![CDATA[Codefresh]]></dc:creator>
            <pubDate>Fri, 24 Apr 2020 23:27:23 GMT</pubDate>
            <atom:updated>2020-07-24T17:47:24.672Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nm0G5DH9iuk-sNm0QMUlfA.png" /></figure><p><strong><em>Editors Note: We did a webinar on this very topic! Scroll to the bottom of this post to watch the webinar.</em></strong></p><p>Building upon the success of Helm 2, Helm 3 has recently been released and the server-side component, Tiller, is finally gone! Helm works out-of-the-box with Coderesh, so releasing your Helm 3 applications is as easy as pie. In this blog post, you will learn about viewing Helm releases, and monitoring Helm environments. Still using Helm 2? Not to worry! With a click of a button, in Codefresh, you can manage both Helm 2 and Helm 3 clusters simultaneously!</p><h4>What is Helm?</h4><p>Helm is the package manager for Kubernetes. It’s very similar to apt/yum/homebrew. If you’re familiar with the way these tools work with operating systems, the same analogy can be made for Helm, with Kubernetes being the “operating system.” In this way, Helm greatly simplifies the process of creating, managing, and deploying applications using Helm Charts.</p><h4>What is a Helm Chart?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/304/0*8au6OH1AUEawJiIH.png" /></figure><p>In Helm, a chart is basically just a collection of manifests that describe a related Kubernetes resource. There are two main components to a chart: templates and values. These templates and values go through a template rendering engine, producing a manifest that is easily digestible by Kubernetes.</p><p>Charts are very similar to RPM and DEB packages for Linux, making it easy for developers to distribute and consume/install applications via different repositories.</p><h4>Why use Helm?</h4><p>Kubernetes can be difficult to manage with all the objects you need to maintain (ConfigMaps/Secrets, pods, services, etc.). Helm manages all of this for you, packaged into one simple chart, where you can even manage which portions of your application should be configurable. In addition, Helm also maintains a versioned history of every chart installation. If something goes wrong, you can simply call “helm rollback.” Similarly, if you need to upgrade a chart, you can simply call “helm upgrade.”</p><p>Combine this with a CI/CD tool, like Codefresh, and you can easily integrate extra actions into your pipelines, i.e. before an installation of a chart, you could run some integration tests, or after an upgrade has finished, you can trigger a notification.</p><h4>Helm 2 vs Helm 3</h4><p>There were many changes introduced in the new release of Helm, namely:</p><ul><li><strong>Removal of Tiller:</strong> Tiller was the server-side component of Helm 2. When installing Helm 2, you also needed to install Tiller on your Kubernetes cluster. Tiller’s main responsibility was to allow multiple Helm operators to work with the same set of releases. However, Role-Based Access Control was enabled by default in Kubernetes 1.6, making Tiller difficult to manage. Configurations would be granted that enabled users to have a broad range of permissions they weren’t intended to have. Now, with RBAC enabled by default, Helm no longer needs to control who installs what.</li><li>Security is now on a per-user basis and can be configured via the kubeconfig file</li><li>Chart installation information is stored in Kubernetes itself (previously this was saved to Tiller)</li><li>Release names are now scoped to the release namespace, as opposed to the Tiller namespace. Previously, if you wanted to have two releases with the same name, this was not possible since in Helm 2 releases were scoped to the Tiller namespace. Now, you can have two releases with the same name, provided they are in different namespaces.</li><li><strong>3-way Strategic Merge Patching: </strong>Helm 2 took only a two-way approach for merge patching, meaning on an upgrade or rollback, only the old chart and the new chart were considered. If someone were to go in and modify the live state of the application via kubectl, for instance, and apply the upgrade/rollback, no changes would be applied as the old chart and “new” chart would be the same. In Helm 3, for merge patching, the old chart, the live state of the application, and the new chart are considered.</li><li><strong>Secrets as the default storage driver for release information</strong>: previously, in Helm 2, release information was stored in ConfigMaps, which were stored encrypted, making it difficult to access/apply. Helm 3 has moved release information over to Secrets, so now, to retrieve release information Helm can simply access the secret, decrypt it, and apply it.</li><li><strong>JSON Schema Chart Validation:</strong> if a chart maintainer wants to impose specific value types on a chart, they may — all they need to do is include a JSONSchema file along with their chart, and the validation tool will automatically be run on helm install/upgrade/lint. Other notable mentions:</li><li><strong>Templates do not use Lua in Helm 3.</strong> There was some talk over the past year that Lua would be used for templating, and that work was not released with version 3, but will likely be a part of subsequent versions.</li><li>In Helm 2, if you attempted to install a release in a namespace that did not exist, it would create the namespace for you. Helm 3 now follows kubectl convention by throwing an error message if you attempt to create a release where the namespace does not exist.<strong>Namespaces not automatically created for releases:</strong></li><li>There are many other changes between Helm 2 and 3, and you can read about them on their official <a href="https://helm.sh/docs/faq/">documentation </a>.</li></ul><h4>Native Support for Helm within Codefresh</h4><p>We know developers want to get started right away, so we have several Helm integrations within Codefresh!</p><h4>Helm Repository</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*a8-8CQPWD7itT-in.png" /></figure><p>A default managed repo is created automatically for every account and you can add other external repositories from this view as well.</p><p>Take special note of the <strong>Install</strong> button, which allows you to manually install a chart with the click of a button. This is very handy for non-technical users who wish to spin up a demo environment .</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/810/0*DqlGhATyAxHvC8hD.png" /></figure><p>By default, the install button uses the Helm command <strong>helm install </strong>behind the scenes, but if for example, you wanted to trigger integration tests as a part of the installation process, you could create a pipeline and select it from the drop-down menu as shown in the screenshot below.</p><h4>Helm Release Dashboard</h4><p>From the Helm Release dashboard, you can view the current status of your cluster, currently deployed releases, and previous versions of installed charts. You can even rollback to previous revisions. Similar to installations, you can also define your own pipelines for test/rollbacks/delete operations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*PcaXchwgV68JorHr.png" /></figure><p>To toggle between Helm 2 or Helm 3 releases, simply click the gear icon next to your cluster, and select whether your releases are Helm 2 or Helm 3. For Helm 2 releases, you will need to define the namespace Tiller was installed on.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*L4Q4mWo4lDACotp-.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/829/0*Y-M_6qrZcb8xIFC9.png" /></figure><p>Clicking on a specific release will take you to its details, showing you the individual services that make up your application, and their status.</p><p>The <strong>History </strong>tab will show you current and previous revisions, and you can further expand a revision and view a <strong>helm diff</strong> to see which files were changed between specific releases. You can even rollback to a specific version of a chart by selecting the <strong>rollback</strong> button.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*qfP9PSv40BezjUK6.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_xKyqBAagqQzR3h7.png" /></figure><h4>Helm/Kubernetes Environment Dashboard</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8UxZ_hD85cyUaT2h.png" /></figure><p>Codefresh also provides a Helm/Kubernetes Environments dashboard, where you can see cluster status information, in addition to what your pipelines are doing. This view is useful for project leads to get an update on their environments, or for a developer to come see the status of a build after a PR has been merged.</p><h4>Helm Promotion Dashboard</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*M4Filt2wqkuv6DLm.png" /></figure><p>Codefresh also provides a Helm promotion dashboard, that can allow you to track your applications as they move through your infrastructure. Each column in the dashboard represents either a different cluster or a different namespace. So, the possibilities are endless when it comes to how you want to configure the dashboard specifically. You can also create more than one dashboard so each team can cater to their own needs.</p><p>You can see from this example board, we have 3 environments corresponding to a cluster, load-testing, staging, and production. Once an application is ready to be promoted to the next environment, you can simply drag-and-drop it to the corresponding column, therefore promoting it.</p><h4>Start your Helm journey here!</h4><p>The ability to deploy entire application stacks allows developers to focus their work on writing code, and less on deployment environments. Helm is the way to go for packaging and deploying Kubernetes applications, and Codefresh is the only CI/CD platform to offer native support for it!</p><p>If you enjoyed this blog post, check out our associated webinar, <a href="https://codefresh.io/events/making-helm-3-codefresh/">Making the Most of Helm 3 with Codefresh</a>.</p><p>New to Codefresh? <a href="https://codefresh.io/codefresh-signup/?utm_source=Blog&amp;utm_medium=Post&amp;utm_campaign=makingTheMostOfHelm3">Create Your Free Account Today</a>!</p><p><em>Author: Anna Baker, Technical writer&amp; Developer Evangelist for Codefresh</em></p><p><em>Originally published at </em><a href="https://codefresh.io/helm-tutorial/making-helm-3/"><em>https://codefresh.io</em></a><em> on April 24, 2020.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b52f2e0cb00e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/containers-101/making-the-most-of-helm-3-b52f2e0cb00e">Making the Most of Helm 3</a> was originally published in <a href="https://medium.com/containers-101">Container Hub</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deploying Your Applications Using Codefresh, Google Cloud Platform, and Google Kubernetes Engine]]></title>
            <link>https://medium.com/containers-101/deploying-your-applications-using-codefresh-google-cloud-platform-and-google-kubernetes-engine-5c1576a245fc?source=rss-563ec507cbd3------2</link>
            <guid isPermaLink="false">https://medium.com/p/5c1576a245fc</guid>
            <category><![CDATA[gke]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[codefresh]]></category>
            <category><![CDATA[gcp]]></category>
            <dc:creator><![CDATA[Codefresh]]></dc:creator>
            <pubDate>Fri, 10 Apr 2020 16:48:24 GMT</pubDate>
            <atom:updated>2020-04-10T16:48:24.780Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*T2VYZ_h7euCMYQm9imYEoQ.png" /></figure><h3>Introduction</h3><p>Kubernetes offers scalability and reliability for your container-based applications. Combine this with GKE, and you now eliminate the need to install or operate clusters on your own. A huge plus to using GKE is that you now will be running Kubernetes in a GCP environment, therefore you can utilize all the handy integrations Google has to offer.</p><p>Codefresh simplifies the process even more, by automating the process of getting your code built, tested, and deployed.</p><p>Let’s take a look at how easy it is to deploy your application to Kubernetes using Codefresh, GCP and GKE.</p><p>Note that if you sign-up using this tutorial, you will receive $500 in GCP credits!</p><h3>Prerequisites</h3><ul><li>A free GCP Account</li><li>A free Codefresh account</li></ul><h3>Step 1 — Create a Cluster in GKE</h3><p>In the GCP Console, navigate to the <strong>Compute</strong> section in the left-hand panel, and select <strong>Kubernetes Engine</strong>. Click on <strong>Create Cluster</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/889/0*DDVgaZyN5edqSIlA.png" /></figure><p>You can also use the <a href="https://cloud.google.com/sdk/gcloud/reference/container/clusters/create">Google Cloud CLI</a> to create a cluster, if you are not using GCP’s UI.</p><h3>Step 2 — Connect Codefresh to Google Kubernetes Engine</h3><p>Login to Codefresh. On the left-hand panel, select <strong>Account Settings &gt; Integrations</strong>. Under Kubernetes, select <strong>Configure</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*AFkY_GOqvqAi0eRS.png" /></figure><p>From the <strong>Add Provider</strong> dropdown, select <strong>Google Cloud Platform</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/612/0*yMRUF2gZ4Tr_28QB.png" /></figure><p>Click on <strong>Authenticate</strong> and sign in using the Gmail account linked to GCP.</p><h3>Step 3 — Add Your Clusters to Codefresh</h3><p>Expand the GCP row and click the <strong>Add Cluster </strong>button. Select your <strong>Project ID </strong>and <strong>Cluster</strong> from the dropdown menus.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/395/0*l_3jX_ZuvtUIYmjE.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*BgB9ovGUYcM9fVoP.png" /></figure><p>After you have added your cluster, you will be able to see all of your clusters, their names, IPs, and statuses once you expand the <strong>Show Cluster Nodes</strong> link.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*kXZ8UDWYt2U4eAvq.png" /></figure><h3>Step 4 — Deploy!</h3><p>Integrating your GKE cluster with Codefresh was as simple as that! Now, we need to deploy. Codefresh offers several ways of deploying your Docker image to your Kubernetes cluster:</p><ul><li>Using the Codefresh GUI to deploy to Kubernetes <a href="https://codefresh.io/docs/docs/getting-started/deployment-to-kubernetes-quick-start-guide/">on demand</a></li><li>Using the <a href="https://codefresh.io/docs/docs/codefresh-yaml/steps/deploy/">deploy step</a> in a pipeline</li><li>Using the <a href="https://codefresh.io/docs/docs/deploy-to-kubernetes/kubernetes-templating/">cf-deploy-kubernetes image</a> (this method offers a templating mechanism used for passing common parameters to your Kubernetes manifests, such as Docker image names/tags, number of replicas, service labels, configmaps, etc.)</li><li>Using<a href="https://codefresh.io/docs/docs/deploy-to-kubernetes/custom-kubectl-commands/"> custom kubectl commands</a> within freestyle steps</li><li>Using <a href="https://codefresh.io/docs/docs/getting-started/helm-quick-start-guide/">Helm</a> as a package manager</li></ul><p>In addition, you can view your deployment environments using Codefresh’s Environment Dashboard, by navigating to the <strong>DevOps Insights </strong>section in the left-hand panel, and selecting <strong>Environments</strong>. From here, you can see an overview of your Helm and Kubernetes deployments, monitoring your clusters and pipelines.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*EtNQV6jTpX-c3jDu.png" /></figure><h3>Conclusion</h3><p>And there you have it! Now, you can see how easy it is to integrate GCP/GKE with Codefresh and you are ready to utilize the three within your pipelines. Check out <a href="https://codefresh.io/docs/docs/getting-started/deployment-to-kubernetes-quick-start-guide/">this quick start guide</a> on more information on how to deploy to Kubernetes.</p><p>New to Codefresh?<a href="https://codefresh.io/codefresh-signup/?utm_source=Blog&amp;utm_medium=Post&amp;utm_campaign=imageApiRoadmaphttps://codefresh.io/codefresh-signup/?utm_source=Blog&amp;utm_medium=Post&amp;utm_campaign=gcpgke"> Create Your Free Account Today</a>!</p><p><em>Anna Baker is a Software Engineer/Technical Writer. She previously worked at Red Hat and is passionate about the open source community. In her free time, she enjoys drawing and cooking dishes from all over the world.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5c1576a245fc" width="1" height="1" alt=""><hr><p><a href="https://medium.com/containers-101/deploying-your-applications-using-codefresh-google-cloud-platform-and-google-kubernetes-engine-5c1576a245fc">Deploying Your Applications Using Codefresh, Google Cloud Platform, and Google Kubernetes Engine</a> was originally published in <a href="https://medium.com/containers-101">Container Hub</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Making the Business Case for DevOps — Codefresh]]></title>
            <link>https://medium.com/containers-101/making-the-business-case-for-devops-codefresh-a13bcb1c17d4?source=rss-563ec507cbd3------2</link>
            <guid isPermaLink="false">https://medium.com/p/a13bcb1c17d4</guid>
            <category><![CDATA[calculator]]></category>
            <category><![CDATA[business]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Codefresh]]></dc:creator>
            <pubDate>Wed, 01 Apr 2020 15:48:53 GMT</pubDate>
            <atom:updated>2020-07-24T15:57:24.794Z</atom:updated>
            <content:encoded><![CDATA[<h3>Making the Business Case for DevOps</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7gJRGbMPkuIMZAbcUFez8A.png" /></figure><p>Engineering and “the business” don’t always speak the same language. That’s why in this blog post and <a href="https://codefresh.io/events/making-business-case-devops/">webinar</a>, we want to show you how to bridge the gap using the language “the business” understands: cold hard cash.</p><p>When talking to the business value, here are <strong>four areas that DevOps provide value for your organization</strong> with associated calculators so you can get an exact value to share with your organization</p><p><strong>1.) Avoiding downtime</strong></p><p>Avoiding downtime is vital, but how do you calculate the value? A different way to think about it is how much value Annual Revenue/Minutes in a year = $$/minute, we see the value of mere minutes of downtime. <em>uptime</em> provides. In many organizations, if engineering is down, sales, marketing, etc. also cannot do their jobs. So if we look at Annual Revenue/Minutes in a year = $$/minute, we see the value of mere minutes of downtime.</p><p><a href="https://codefresh.outgrow.us/5e84e0a3386456713080c41a"><strong>Enter your own numbers here</strong></a><strong>.</strong></p><p><em>Yes, this is a simple calculation, and some may say it is an overestimate. But consider that this could actually be an underestimation because this does not factor in the perceived lack of reliability customers experience during this downtown and therefore, the future ramifications it could have on your business.</em></p><p><strong>2.) Deliver Faster</strong></p><p>If you want to improve your uptime, you should also improve your code release time.</p><p>Every software change has a value associated with it and every line of code an engineer writes delivers business value. But that code has to be delivered. If it is sitting and waiting to be deployed, it is rapidly decreasing the value of that change.</p><p>For example, if there is a code change that is going to save the business $100M, and you deliver the code 3 days faster than planned, you made the business an extra $822K.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*Dl5iuppz3-UqCq5w.png" /></figure><p><em>Estimating all the business value you have in a year like we have done above can be challenging. To get a rough estimate, take the total cost of your engineering salaries *8 (this was calculated from a sample set of organizations; if you are at a higher-performance organization this multiplier would be higher.)</em></p><p><a href="https://codefresh.outgrow.us/5e84d92a7715d971e1b06268"><strong>Use this calculator below to do your own calculation</strong></a></p><p>3.) Saving Engineering Time</p><p>Engineers teams are usually one of the most expensive parts of a business. Their time is valuable, so if you can cut just 5 min off build times, you can save a large amount of money.</p><p><a href="https://codefresh.outgrow.us/5e83bdb37715d971e1ac5075"><strong>Enter your estimates into this live calculator and see for yourself</strong></a></p><p>4.) Reducing Infrastructure Overhead</p><p>30% of infrastructure budgets are spent on idle machines, so moving to Cloud-Native technologies can help you reduce your overhead cost and in the process, you can reduce downtime and time to delivery. More ways to save:</p><ul><li>Spot Instances: Can reduce cloud costs 90%, requires great architecture and failover</li><li>Auto-scaling: Add nodes as needed, toss when done</li><li>Tighter Packing: Packing smaller microservices is more efficient than horizontally scaled monoliths</li></ul><h4>Build vs. Buy</h4><p>A CI/CD platform like Codefresh can help you reduce your lead time to deploy and in turn save you money. In fact, we have built our entire platform for this purpose with features like built-in caching, shared volumes, live debugging, and numerous other <a href="https://codefresh.io/features/">features</a> to help you deploy faster. Use this <a href="https://baremetrics.com/build-vs-buy">Build vs Buy calculator</a> to see how much you could save.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*1qJSHd-UFPoOwGGC.jpg" /></figure><p>Open your FREE Codefresh account today with UNLIMITED builds at <a href="https://codefresh.io/">Codefresh.io.</a></p><p><em>Originally published at </em><a href="https://codefresh.io/webinars/making-business-case-devops/"><em>https://codefresh.io</em></a><em> on April 1, 2020.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a13bcb1c17d4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/containers-101/making-the-business-case-for-devops-codefresh-a13bcb1c17d4">Making the Business Case for DevOps — Codefresh</a> was originally published in <a href="https://medium.com/containers-101">Container Hub</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Three Ways to Create Docker Images for Java]]></title>
            <link>https://medium.com/containers-101/three-ways-to-create-docker-images-for-java-e139805ecb7f?source=rss-563ec507cbd3------2</link>
            <guid isPermaLink="false">https://medium.com/p/e139805ecb7f</guid>
            <category><![CDATA[java]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Codefresh]]></dc:creator>
            <pubDate>Thu, 05 Mar 2020 18:46:23 GMT</pubDate>
            <atom:updated>2020-03-05T18:46:23.367Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*GakxFERLCJ1Ql6l4eAJl5g.png" /></figure><p>Long before Dockerfiles, Java developers worked with single deployment units (WARs, JARs, EARs, etc.). As you likely know by now, it is best practice to work in micro-services, deploying a small number of deployment units per JVM. Instead of one giant, monolithic application, you build your application such that each service can run on its own.</p><p>This is where Docker comes in! If you wish to upgrade a service, rather than redeploying your jar/war/ear to a new instance of an application server, you can just build a new Docker image with the upgraded deployment unit.</p><p>In this post, I will review 3 different ways to create Docker images for Java applications. If you want to follow along feel free to clone my repository at <a href="https://github.com/annabaker/docker-with-java-demos">https://github.com/annabaker/docker-with-java-demos</a>.</p><h3>Prerequisites</h3><ul><li>Docker is <a href="https://docs.docker.com/install/">installed</a></li><li>Maven is <a href="https://maven.apache.org/install.html">installed </a>(for example #1)</li><li>You have a simple Spring Boot application (I used the <a href="https://spring.io/guides/gs/spring-boot/#scratch">Spring Initializr</a> project generator with a Spring Web dependency)</li></ul><h3>First way: Package-only Build</h3><p>In a package-only build, we will let Maven (or your build tool of choice) control the build process.</p><p>Unzip the Spring Initializr project you generated as part of the prerequisites. In the parent folder of your Spring Boot application, create a Dockerfile. In a terminal, run:</p><pre>$ unzip demo.zip<br>$ cd demo<br>$ nano Dockerfile</pre><p>Paste the following and save:</p><pre># we will use openjdk 8 with alpine as it is a very small linux distro<br>FROM openjdk:8-jre-alpine3.9</pre><pre># copy the packaged jar file into our docker image<br>COPY target/demo-0.0.1-SNAPSHOT.jar /demo.jar</pre><pre># set the startup command to execute the jar<br>CMD [&quot;java&quot;, &quot;-jar&quot;, &quot;/demo.jar&quot;]</pre><ul><li>The FROM layer denotes which parent image to use for our child image</li><li>The COPY layer will copy the local jar previously built by Maven into our image</li><li>The CMD layer tells Docker the command to run inside the container once the previous steps have been executed</li></ul><p>Now, let’s package our application into a .jar using Maven:</p><pre>$ mvn clean package</pre><p>…and then build the Docker image. The following command tells Docker to fetch the Dockerfile in the current directory (the period at the end of the command). We build using the username/image name convention, although this is not mandatory. The -t flag denotes a Docker tag, which in this case is 1.0-SNAPSHOT. If you don’t provide a tag, Docker will default to the tag :latest.</p><pre>$ docker build -t anna/docker-package-only-build-demo:1.0-SNAPSHOT .</pre><p>To run the container from the image we just created:</p><pre>$ docker run -d -p 8080:8080 anna/docker-package-only-build-demo:1.0-SNAPSHOT</pre><p>-d will run the container in the background (detached mode), and -p will map our local port 8080 to the container’s port of 8080.</p><p>Navigate to localhost:8080, and you should see the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/498/0*FBZFDGEEOHnhvjZs.png" /></figure><p>Once you are satisfied with your testing, stop the container.</p><pre>$ docker stop &lt;container_id&gt;</pre><h3>Pros to this approach:</h3><ul><li>Results in a light-weight Docker image</li><li>Does not require Maven to be included in the Docker image</li><li>Does not require any of our application’s dependencies to be packaged into the image</li><li>You can still utilize your local Maven cache upon application layer changes, as opposed to methods 2 and 3 which we will discuss later</li></ul><h3>Cons to this approach:</h3><ul><li>Requires Maven and JDK to be installed on the host machine</li><li>The Docker build will fail if the Maven build fails/is not executed beforehand — this becomes a problem when you want to integrate with services that automatically “just build” using the present Dockerfile</li></ul><h3>Second way: Normal Docker Build</h3><p>In a “normal” Docker build, Docker will control the build process.<br>Modify the previous Dockerfile to contain the following:</p><pre># select parent image<br>FROM maven:3.6.3-jdk-8</pre><pre># copy the source tree and the pom.xml to our new container<br>COPY ./ ./</pre><pre># package our application code<br>RUN mvn clean package</pre><pre># set the startup command to execute the jar<br>CMD [&quot;java&quot;, &quot;-jar&quot;, &quot;target/demo-0.0.1-SNAPSHOT.jar&quot;]</pre><p>Now, let’s build a new image as we did in Step 1:</p><pre>$ docker build -t anna/docker-normal-build-demo:1.0-SNAPSHOT .</pre><p>And run the container:</p><pre>$ docker run -d -p 8080:8080 anna/docker-normal-build-demo:1.0-SNAPSHOT</pre><p>Again, to test your container, navigate to localhost:8080. Stop the container once you are finished testing.</p><h3>Pros to this approach:</h3><ul><li>Docker controls the build process, therefore this method does not require the build tool or the JDK to be installed on the host machine beforehand</li><li>Integrates well with services that automatically “just build” using the present Dockerfile</li></ul><h3>Cons to this approach:</h3><ul><li>Results in the largest Docker image of our 3 methods</li><li>This build method not only packaged our app, but all of its dependencies and the build tool itself, which is not necessary to run the executable</li><li>If the application layer is rebuilt, the mvn package command will force all Maven dependencies to be pulled from the remote repository all over again (you lose the local Maven cache)</li></ul><h3>Multi-stage Build (The ideal way)</h3><p>With multi-stage Docker builds, we use multiple FROM statements for each build stage. Every FROM statement creates a new base layer, and discards everything we don’t need from the previous FROM stage.</p><p>Modify your Dockerfile to contain the following:</p><pre># the first stage of our build will use a maven 3.6.1 parent image<br>FROM maven:3.6.1-jdk-8-alpine AS MAVEN_BUILD</pre><pre># copy the pom and src code to the container<br>COPY ./ ./</pre><pre># package our application code<br>RUN mvn clean package</pre><pre># the second stage of our build will use open jdk 8 on alpine 3.9<br>FROM openjdk:8-jre-alpine3.9</pre><pre># copy only the artifacts we need from the first stage and discard the rest<br>COPY --from=MAVEN_BUILD /docker-multi-stage-build-demo/target/demo-0.0.1-SNAPSHOT.jar /demo.jar</pre><pre># set the startup command to execute the jar<br>CMD [&quot;java&quot;, &quot;-jar&quot;, &quot;/demo.jar&quot;]</pre><p>Build the image:</p><pre>$ docker build -t anna/docker-multi-stage-build-demo:1.0-SNAPSHOT .</pre><p>And then run the container:</p><pre>$ docker run -d -p 8080:8080 anna/docker-multi-stage-build-demo:1.0-SNAPSHOT</pre><h3>Pros to this approach:</h3><ul><li>Results in a light-weight Docker image</li><li>Does not require the build tool or JDK to be installed on the host machine beforehand (Docker controls the build process)</li><li>Integrates well with services that automatically “just build” using the present Dockerfile</li><li>Only artifacts we need are copied from one stage to the next (i.e., our application’s dependencies are not packaged into the final image as in the previous method)</li><li>Create as many build stages as you need</li><li>Stop at any particular stage on an image build using the –target flag, i.e.</li></ul><pre>docker build — target MAVEN_BUILD -t anna/docker-multi-stage-build-demo:1.0-SNAPSHOT .</pre><h3>Cons to this approach:</h3><p>If the application layer is rebuilt, the mvn package command will force all Maven dependencies to be pulled from the remote repository all over again (you lose the local Maven cache)</p><h3>Verification: How big are the images?</h3><p>In a terminal, run:</p><pre>docker image ls</pre><p>You should see something like the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/730/0*FQIrJmyS84ByY-YR.png" /></figure><p>As you can see, the multi-stage build resulted in our smallest image, whereas the normal build resulted in our largest image. This should be expected since the normal build included our application code, all of its dependencies, and our build tooling, and our multi-stage build contained only what we needed.</p><h3>Conclusion</h3><p>Of the three Docker image build methods we covered, Multi-stage builds are the way to go. You get the best of both worlds when packaging your application code — Docker controls the build, but you extract only the artifacts you need. This becomes particularly important when storing containers on the cloud.</p><ul><li>You spend less time building and transferring your containers on the cloud, as your image is much smaller</li><li>Cost — the smaller your image, the cheaper it will be to store</li><li>Smaller surface area, aka removing additional dependencies from our image makes it less prone to attacks</li></ul><p>Thanks for following along, and I hope this helps! You can find the source code for all three examples at <a href="https://github.com/annabaker/docker-with-java-demos">https://github.com/annabaker/docker-with-java-demos</a></p><p>In a future post, I will show you how to deal with caching issues specifically for Java builds.</p><p><em>Anna Baker was previously a software engineer and currently breaking into technical writing. She previously worked at Red Hat and is passionate about the open source community. In her free time, she enjoys drawing and cooking dishes from all over the world.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e139805ecb7f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/containers-101/three-ways-to-create-docker-images-for-java-e139805ecb7f">Three Ways to Create Docker Images for Java</a> was originally published in <a href="https://medium.com/containers-101">Container Hub</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>