<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Velotio Perspectives - Medium]]></title>
        <description><![CDATA[Thoughts and ideas on startups, enterprise software &amp; technology by the Velotio team. Learn more at www.velotio.com. - Medium]]></description>
        <link>https://medium.com/velotio-perspectives?source=rss----3b93934e4b3d---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Velotio Perspectives - Medium</title>
            <link>https://medium.com/velotio-perspectives?source=rss----3b93934e4b3d---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 09:03:03 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/velotio-perspectives" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Building a Progressive Web Application in React [With Live Code Examples]]]></title>
            <link>https://medium.com/velotio-perspectives/building-a-progressive-web-application-in-react-with-live-code-examples-1c1bde4d61f1?source=rss----3b93934e4b3d---4</link>
            <guid isPermaLink="false">https://medium.com/p/1c1bde4d61f1</guid>
            <category><![CDATA[progressive-web-app]]></category>
            <category><![CDATA[pwa-in-react]]></category>
            <dc:creator><![CDATA[Velotio Technologies]]></dc:creator>
            <pubDate>Wed, 25 Nov 2020 10:22:10 GMT</pubDate>
            <atom:updated>2020-11-25T10:22:10.305Z</atom:updated>
            <content:encoded><![CDATA[<h3>What is PWA:</h3><p>A Progressive Web Application or PWA is a web application that is built to look and behave like native apps, operates offline-first, is optimized for a variety of viewports ranging from mobile, tablets to FHD desktop monitors and more. PWAs are built using front-end technologies such as HTML, CSS, and JavaScript and bring a native-like user experience to the web platform. PWAs can also be installed on devices just like native apps.</p><p>For an application to be classified as a PWA, it must tick all of these boxes:</p><ul><li>PWAs must implement service workers. Service workers act as a proxy between the web browsers and API servers. This allows web apps to manage and cache network requests and assets</li><li>PWAs must be served over a secure network, i.e. the application must be served over HTTPS</li><li>PWAs must have a web manifest definition, which is a JSON file that provides basic information about the PWA, such as name, different icons, look and feel of the app, splash screen, version of the app, description, author, etc</li></ul><h3>Why build a PWA?</h3><p>Businesses and engineering teams should consider building a progressive web app instead of a traditional web app. Here are some of the most prominent arguments in favor of PWAs:</p><ul><li>PWAs are responsive. The mobile-first design approach enables PWAs to support a variety of viewports and orientation</li><li>PWAs can work on slow Internet or no Internet environment. App developers can choose how a PWA will behave when there’s no Internet connectivity, whereas traditional web apps or websites simply stop working without an active Internet connection</li><li>PWAs are secure because they are always served over HTTPS</li><li>PWAs can be installed on the home screen, making the application more accessible</li><li>PWAs bring in rich features, such as push notification, application updates, and more</li></ul><h3>PWA and React</h3><p>There are various ways to build a progressive web application. One can just use Vanilla JS, HTML, and CSS or pick up a framework or library. Some of the popular choices in 2020 are Ionic, Vue, Angular, Polymer, and of course <a href="https://www.velotio.com/engineering-blog/inline-function-performance-in-react-applications">React</a>, which happens to be my favorite front-end library.</p><h3>Building PWAs with React</h3><p>To get started, let’s create a PWA which lists all the users in a system.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8d757e7f283f5e8e72a0b87d073988dc/href">https://medium.com/media/8d757e7f283f5e8e72a0b87d073988dc/href</a></iframe><p>Next, we will replace the default <strong>App.js</strong> file with our own implementation.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4cf96fe68b8f8b6dfacb108548fac352/href">https://medium.com/media/4cf96fe68b8f8b6dfacb108548fac352/href</a></iframe><p>This displays a list of users fetched from the server.</p><p>Let’s also remove the logo.svg file inside the src directory and truncate the App.css file that is populated as a part of the boilerplate code.</p><p>To make this app a PWA, we need to follow these steps:</p><p><strong>1. Register service worker</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9c2f64012514d92721ab1abaf0e028a6/href">https://medium.com/media/9c2f64012514d92721ab1abaf0e028a6/href</a></iframe><p><strong>2. Update the manifest file</strong></p><p>Our modified manifest file looks like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ab4dd1ea6f57285e4fddc254507cc4ae/href">https://medium.com/media/ab4dd1ea6f57285e4fddc254507cc4ae/href</a></iframe><figure><img alt="PWA Splash Screen generated by config defined in manifest.json" src="https://cdn-images-1.medium.com/max/313/0*NgqrOsst5XrlLqac.png" /><figcaption>PWA Splash Screen</figcaption></figure><p>Here the display mode selected is “standalone” which tells the web browsers to give this PWA the same look and feel like that of a standalone app. Other display options include, “browser,” which is the default mode and launches the PWA like a traditional web app, and “fullscreen,” which opens the PWA in fullscreen mode — hiding all other elements such as navigation, the address bar, and the status bar.</p><p>The manifest can be inspected using Chrome dev tools &gt; Application tab &gt; Manifest.</p><p>1. <strong>Test the PWA:</strong></p><ul><li>To test a progressive web app, build it completely first. This is because PWA features, such as caching aren’t enabled while running the app in dev mode to ensure hassle-free development</li><li>Create a production build with: <strong>npm run build</strong></li><li>Change into the build directory: <strong>cd build</strong></li><li>Host the app locally: <strong>http-server or python3 -m http.server 8080</strong>‍</li><li>Test the application by logging in to<a href="http://localhost:8080/"> http://localhost:8080</a><strong>‍</strong></li></ul><p>2. <strong>Audit the PWA</strong>: If you are testing the app for the first time on a desktop or laptop browser, PWA may look like just another website. To test and audit various aspects of the PWA, let’s use <a href="https://developers.google.com/web/tools/lighthouse">Lighthouse</a>, which is a tool built by Google specifically for this purpose.</p><figure><img alt="PWA install on a mobile device" src="https://cdn-images-1.medium.com/max/313/0*aiqyUZcUMA8eibzG.png" /><figcaption>PWA on mobile</figcaption></figure><p>At this point, we already have a simple PWA which can be published on the Internet and made available to billions of devices. Now let’s try to enhance the app by improving its offline viewing experience.</p><p><strong>1. Offline indication</strong>: Since service workers can operate without the Internet as well, let’s add an offline indicator banner to let users know the current state of the application. We will use <a href="https://developer.mozilla.org/en-US/docs/Web/API/NavigatorOnLine/onLine">navigator.onLine</a> along with the “online” and “offline” window events to detect the connection status. <strong>‍</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a95d57652b2eb91d58cd079ad4f9ffec/href">https://medium.com/media/a95d57652b2eb91d58cd079ad4f9ffec/href</a></iframe><p>The easiest way to test this is to just turn off the Wi-Fi on your dev machine. Chrome dev tools also provide an option to test this without actually going offline. Head over to Dev tools &gt; Network and then select “Offline” from the dropdown in the top section. This should bring up the banner when the app is offline.</p><p><strong>2. Let’s cache a network request using a service worker</strong></p><p>CRA comes with its own service-worker.js file which caches all static assets such as JavaScript and CSS files that are a part of the application bundle. To put custom logic into the service worker, let’s create a new file called ‘custom-service-worker.js’ and combine the two.</p><ol><li>yarn add react-app-rewired</li><li>Update the package.json as follows:</li></ol><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fb7648da75c3b1697aa7c80aff9246f5/href">https://medium.com/media/fb7648da75c3b1697aa7c80aff9246f5/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f988a86c86859266d485999b2279fefb/href">https://medium.com/media/f988a86c86859266d485999b2279fefb/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0d34c2eb8befa1834fdc8dc83286d262/href">https://medium.com/media/0d34c2eb8befa1834fdc8dc83286d262/href</a></iframe><p>Your app should now work correctly in offline mode.</p><h3>Distributing and publishing a PWA</h3><p><strong>‍</strong>PWAs can be published just like any other website and only have one additional requirement, i.e. it must be served over HTTPS. When a user visits PWA from mobile or tablet, a pop-up is displayed asking the user if they’d like to install the app to their home screen.</p><h3>Conclusion</h3><p>Building PWAs with React enables engineering teams to develop, deploy, and publish progressive web apps for billions of devices using technologies they’re already familiar with. Existing React apps can also be converted to a PWA. PWAs are fun to build, easy to ship and distribute, and add a lot of value to customers by providing a native-live experience, better engagement via features, such as add to the home screen, push notifications, and more without any installation process.</p><p><em>Originally published at </em><a href="https://www.velotio.com/engineering-blog/building-pwa-in-react"><em>https://www.velotio.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1c1bde4d61f1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/velotio-perspectives/building-a-progressive-web-application-in-react-with-live-code-examples-1c1bde4d61f1">Building a Progressive Web Application in React [With Live Code Examples]</a> was originally published in <a href="https://medium.com/velotio-perspectives">Velotio Perspectives</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[An Introduction To Cloudflare Workers And Cloudflare KV store]]></title>
            <link>https://medium.com/velotio-perspectives/an-introduction-to-cloudflare-workers-and-cloudflare-kv-store-ad8c8abd5677?source=rss----3b93934e4b3d---4</link>
            <guid isPermaLink="false">https://medium.com/p/ad8c8abd5677</guid>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[cloudflare]]></category>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[cloudflare-workers]]></category>
            <category><![CDATA[cloudflare-kv]]></category>
            <dc:creator><![CDATA[Velotio Technologies]]></dc:creator>
            <pubDate>Wed, 19 Feb 2020 10:37:36 GMT</pubDate>
            <atom:updated>2020-07-04T10:21:10.340Z</atom:updated>
            <content:encoded><![CDATA[<h3>Cloudflare Workers</h3><p>This post gives a brief introduction to Cloudflare Workers and Cloudflare KV store. They address a fairly common set of problems around scaling an application globally. There are standard ways of doing this but they usually require a considerable amount of upfront engineering work and developers have to be aware of the ‘scalability’ issues to some degree. Serverless application tools target easy scalability and quick response times around the globe while keeping the developers focused on the application logic rather than infra nitty-gritty.</p><h3>Global responsiveness</h3><p>When an application is expected to be accessed around the globe, requests from users sitting in completely different time-zones should take a similar amount of time, ideally speaking. There can be multiple ways of achieving this depending upon how data-intensive the requests are and what those requests actually do.</p><p>Data-intensive requests are harder and more expensive to globalize but again not all the requests are data intensive. On the other hand, static requests like getting a documentation page or a blog post can be globalized by generating markup at build time and deploying them on a CDN.</p><p>And there are semi-dynamic requests. They render static content either with some small amount of data or their content change based on the timezone the request came from.</p><p>The above is a loose classification of requests but there are exceptions, for example, not all the static requests are presentational.</p><p>Serverless frameworks are particularly useful in scaling static and semi-static requests.</p><h3>Cloudflare Workers Overview</h3><p>Cloudflare worker is essentially a function deployment service. They provide a serverless execution environment that can be used to develop and <a href="https://www.velotio.com/engineering-blog/deploy-serverless-event-driven-python-applications-using-zappa">deploy </a>small(although not necessarily) and modular cloud functions with minimal effort.</p><p>It is very trivial to start with workers. First, let&#39;s install wrangler, which is the tool for managing Cloudfare Worker projects.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e6655726f5da2d2cecf56ae33cebf422/href">https://medium.com/media/e6655726f5da2d2cecf56ae33cebf422/href</a></iframe><p>Wrangler handles all the standard stuff for you like project generation from templates, build, config, publishing among other things.</p><p>A Worker primarily contains 2 parts: an event listener that invokes a worker and an event handler that returns a Response object. Creating a worker is as easy as adding an event listener to a button!!</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/af996cf1ae96c9592a8f68b579a6beae/href">https://medium.com/media/af996cf1ae96c9592a8f68b579a6beae/href</a></iframe><p>Above a simple hello world example. Wrangler can be used to build and get a live preview of your worker.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/07d935e46306feae61ccc22f71c6b2d7/href">https://medium.com/media/07d935e46306feae61ccc22f71c6b2d7/href</a></iframe><p>will build your worker. And</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8494d26bf981353a87cee436a5678b12/href">https://medium.com/media/8494d26bf981353a87cee436a5678b12/href</a></iframe><p>can be used to take a live preview on the browser. The preview is just meant to be used for testing(either by you or others). If you want the workers to be triggered by your own domain or a workers.dev subdomain, you need to publish it.</p><p>Publishing is fairly straightforward and requires very little configuration on both wrangler and your project.</p><h3>Wrangler Configuration</h3><p>Just create an account on Cloudflare and get API key. To configure wrangler, just do:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bae9925f68fd8e0f0b0cf6fa4a51b9fa/href">https://medium.com/media/bae9925f68fd8e0f0b0cf6fa4a51b9fa/href</a></iframe><p>it will ask for the registered email and API key and you are good to go.</p><p>To publish your worker on a workers.dev subdomain, just fill your account id in the wrangler.toml and hit wrangler publish The worker will be deployed and live at a generated workers.dev subdomain.</p><h3>Regarding Routes</h3><p>When you publish on a {script-name}.{subdomain}.workers.dev domain, the script or project associated with script-name will be invoked. There is no way to call a script just from {subdomain}.workers.dev.</p><h3>Worker KV</h3><p>Workers alone can’t be used to make anything complex without any persistent storage, that’s where Workers KV comes into the picture. Workers KV as it sounds is a low-latency high-volume key-value store designed for efficient reads.</p><p>It optimizes the read latency by dynamically spreading the most frequently read entries to the edges(replicated in several regions) and storing less frequent entries centrally.</p><p>Newly added keys(or a CREATE) are immediately reflected in every region while a value change in the keys(or an UPDATE) may take as long as 60 seconds to propagate depending upon the region.</p><p>Workers KV is only available to paid users of Cloudflare.</p><h3>Writing Data in Workers KV</h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b9294078c2e7dbb6efe420adf325b096/href">https://medium.com/media/b9294078c2e7dbb6efe420adf325b096/href</a></iframe><p>Now we can write KV pairs in this namespace! The following HTTP requests will do the same:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/eb72e1385d7a0b38fcee54ed4072f329/href">https://medium.com/media/eb72e1385d7a0b38fcee54ed4072f329/href</a></iframe><p>Here the NAMESPACE_ID is the same ID that we received in the last request. first-key is the key name and the My first value! is the value.</p><h3>Let’s Complicate Things a little</h3><p>Above overview just introduces the managed cloud workers with a ‘hello world’ app and basics of the Workers KV but now let’s make something more complicated. We will make an app that will tell how many requests have been made yet from your country. For example, if you pinged the worker from the US then it will return how many requests have been made so far from the US.</p><p>We will need:</p><ul><li>Some place to store the count of requests for each country.</li><li>Find from which country the Worker was invoked.</li></ul><p>For the first, we will use the Workers KV to store the count for every request.</p><h3>Let’s start</h3><p>First, we will create a new project using wrangler: wrangler generate request-count.</p><p>We will be making HTTP calls to write values in the Workers KV so lets add ‘node-fetch’ to the project:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bda0514102988c46bf7829496b36be45/href">https://medium.com/media/bda0514102988c46bf7829496b36be45/href</a></iframe><p>Now, how do we find from which country each request is coming from? The answer is the cf object that is provided with each request to a worker.</p><p>A cf object is a special object that is passed with each request and can be accessed with request.cf. This contains mainly region-specific information along with TLS and Auth information. The details of what is provided in the cf can be found <a href="https://developers.cloudflare.com/workers/reference/runtime/apis/fetch/#the-cf-object">here</a>.</p><p>As we can see from the documentation, we can get the country from</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/df9b196128718a26f0d7b7c225b9067d/href">https://medium.com/media/df9b196128718a26f0d7b7c225b9067d/href</a></iframe><p>The cf object is not correctly populated in the wrangler preview, you will need to publish your worker in order to test cf’s usage. An open issue mentioning the same can be found <a href="https://community.cloudflare.com/t/cf-object-broken-in-worker-preview/67758">here</a>.</p><p>Now, the logic is pretty straightforward here. When we get a request from a country for which we don’t have an entry in the Worker’s KV we make an entry with value 1 else we increment the value of the country key.</p><p>To use Workers KV we will need to create a namespace. A namespace as it sounds is just a collection of key-value pairs where all the keys have to be unique.</p><p>A namespace can be created under the KV tab in Cloudflare web UI by giving the name or using the API call above. You can also view/browse all of your namespaces from the web UI. Following API call can be used to read the value of a key from a namespace:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/31d9cf4ce84fd35029016432c5a43014/href">https://medium.com/media/31d9cf4ce84fd35029016432c5a43014/href</a></iframe><p>But, it is neither the fastest nor the easiest way. Cloudflare provides a better and faster way to read data from your namespaces. It’s a concept called binding. Each KV namespace can be bound to a worker script so that it is available in that script by the variable name it is bound. Any namespace can be bound with any worker. A KV namespace can be bound to a worker by going to the editing menu of a worker from the Cloudflare UI.</p><p>Following steps show how to bind a namespace to a worker:</p><p>Go to the edit page of the worker in Cloudflare web UI and click on the KV tab:</p><p>Then add a binding by clicking the ‘Add binding’ button.</p><p>You can select the namespace name and the variable name by which it will be bound. More details can be found <a href="https://developers.cloudflare.com/workers/reference/storage/reading-data/">here</a>. A binding that I’ve made can be seen in the above image.</p><p>That’s all we need to get this to work. Following is the relevant part of the script:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8f00c408a84abbc6d8e26ad6b656659d/href">https://medium.com/media/8f00c408a84abbc6d8e26ad6b656659d/href</a></iframe><p>In the above code, I bound the Requests namespace that we created by the requests variable that would be dynamically resolved when we publish.</p><p>The full source of this can be found <a href="https://github.com/farkaskid/request_count">here</a>.</p><p>This small application also demonstrates some of the practical aspects of the workers. For example, you would notice that the updates take some time to get reflected and response times of the workers is quick, especially when they are deployed on a .workers.dev subdomain <a href="https://request-count.farkaskid1.workers.dev/">here</a>.</p><p>Side note: You will have to recreate the namespace-worker binding everytime you deploy the worker or you do wrangler publish.</p><h3>Workers vs. AWS Lambda</h3><p>AWS Lambda has been a major player in the serverless market for a while now. So, how do Cloudflare Workers compare to it? Let’s see.</p><h3>Architecture:</h3><p>Cloudflare Worker uses something called `Isolates` instead of a container-based underlying architecture. `Isolates` is the technology that allows V8(Google Chrome’s JavaScript Engine) to run thousands of processes on a single server in an efficient and secure manner. This effectively translates into faster code execution and lower use of memory. More details can be found <a href="https://blog.cloudflare.com/cloud-computing-without-containers/">here</a>.</p><h3>Price:</h3><p>The above mentioned architectural difference allows Workers to be significantly cheaper than Lambda. While a Worker offering 50 milliseconds of CPU costs $0.50 per million requests, the equivalent Lambda costs $1.84 per million. A more detailed price comparison can be found <a href="https://blog.cloudflare.com/cloud-computing-without-containers/#billing">here</a>.</p><h3>Speed:</h3><p>Workers also show significantly better performance numbers than Lambda and Lambda@Edge. Tests run by Cloudflare claim that they are 441% faster than Lambda and 192% faster than Lambda@Edge. A detailed performance comparison can be found <a href="https://blog.cloudflare.com/serverless-performance-comparison-workers-lambda/">here</a>.</p><p>This better performance is also confirmed by <a href="https://serverless-benchmark.com/">serverless-benchmark</a>.</p><h3>Wrapping Up:</h3><p>So as we have seen, Cloudflare Workers along with the KV Store does make it very easy to start with a <a href="https://www.velotio.com/engineering-blog/aws-serverless-application">serverless application</a>. They provide fantastic performance while using less cost along with intuitive deployment. These properties make them ideal for making globally accessible serverless applications.</p><p>*****************************************************************</p><p><em>This post was originally published on </em><a href="https://www.velotio.com/engineering-blog/cloudflare-workers-and-cloudflare-kv-store"><strong><em>Velotio Blog</em></strong></a><strong><em>.</em></strong></p><p><a href="https://velotio.com/"><strong>Velotio Technologies</strong></a> is an outsourced software product development partner for technology startups and enterprises. We specialize in enterprise B2B and SaaS product development with a focus on artificial intelligence and machine learning, DevOps, and test engineering.</p><p><em>Interested in learning more about us? We would love to connect with you on our </em><a href="https://velotio.com/"><strong><em>Website</em></strong></a><em>, </em><a href="https://www.linkedin.com/company/velotio-technologies/"><strong><em>LinkedIn</em></strong></a><em> or </em><a href="https://twitter.com/velotiotech"><strong><em>Twitter</em></strong></a><em>.</em></p><p>***************************************************************</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ad8c8abd5677" width="1" height="1" alt=""><hr><p><a href="https://medium.com/velotio-perspectives/an-introduction-to-cloudflare-workers-and-cloudflare-kv-store-ad8c8abd5677">An Introduction To Cloudflare Workers And Cloudflare KV store</a> was originally published in <a href="https://medium.com/velotio-perspectives">Velotio Perspectives</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Getting Started With Kubernetes Operators (Golang Based) — Part 3]]></title>
            <link>https://medium.com/velotio-perspectives/getting-started-with-kubernetes-operators-golang-based-part-3-edcaf3315088?source=rss----3b93934e4b3d---4</link>
            <guid isPermaLink="false">https://medium.com/p/edcaf3315088</guid>
            <category><![CDATA[cloud-native]]></category>
            <category><![CDATA[golang-based-operators]]></category>
            <category><![CDATA[kubernetes-operator]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Velotio Technologies]]></dc:creator>
            <pubDate>Tue, 15 Oct 2019 12:37:29 GMT</pubDate>
            <atom:updated>2020-06-28T09:49:23.042Z</atom:updated>
            <content:encoded><![CDATA[<h3>Getting Started With Kubernetes Operators (Golang Based) — Part 3</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VNCIyfpuL7Barf8c3oDHDw.png" /></figure><h3>Introduction</h3><p>In the first, <a href="https://velotio.com/blog/2019/9/10/kubernetes-helm-operators-part-1"><strong>getting started with Kubernetes operators (Helm based)</strong></a><strong>,</strong> and the second part, <a href="https://velotio.com/blog/2019/9/17/kubernetes-ansible-operators-part-2"><strong>getting started with Kubernetes operators (Ansible based)</strong></a><strong>,</strong> of this Introduction to Kubernetes operators blog series we learned various concepts related to Kubernetes operators and created a Helm based operator and an Ansible based operator respectively. In this final part, we will build a Golang based operator. In case of Helm based operators, we were executing a helm chart when changes were made to the custom object type of our application, similarly in the case of an Ansible based operator we executed an Ansible role. In case of Golang based operator we write the code for the action we need to perform (reconcile logic) whenever the state of our custom object change, this makes the Golang based operators quite powerful and flexible, at the same time making them the most complex to build out of the 3 types.</p><h3>What We Will Build?</h3><p>The database server we deployed as part of our book store app in previous blogs didn’t have any persistent volume attached to it and we would lose data in case the pod restarts, to avoid this we will attach a persistent volume attached to the host (K8s worker nodes ) and run our database as an statefulset rather than a deployment. We will also add a feature to expand the persistent volume associated with the mongodb pod.</p><h3>Building the Operator</h3><h4>1. Set up the project:</h4><p><strong><em>operator-sdk new bookstore-operator — dep-manager=dep</em></strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/713144bc68a6180fa9bec3475a6403ea/href">https://medium.com/media/713144bc68a6180fa9bec3475a6403ea/href</a></iframe><p>The above command creates the bookstore-operator folder in our $GOPATH/src, here we have set the <strong>— dep-manage</strong>r as <strong>dep</strong> which signifies we want to use dep for managing dependencies, by default it uses go modules for managing dependencies. Similar to what we have seen earlier the operator sdk creates all the necessary folder structure for us inside the bookstore-operator folder.</p><h4>2. Add the custom resource definition</h4><p><strong><em>operator-sdk add api — api-version=blog.velotio.com/v1alpha1 — kind=BookStore</em></strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/340867a7dea35a3284b3d34c7f421f09/href">https://medium.com/media/340867a7dea35a3284b3d34c7f421f09/href</a></iframe><p>The above command creates the CRD and CR for the BookStore type. It also creates the golang structs (pkg/apis/blog/v1alpha1/bookstore_types.go) for <strong>BookStore</strong> types. It also registers the custom type (pkg/apis/blog/v1alpha1/register.go) with schema and generates deep-copy methods as well. Here we can see that all the generic tasks are being done by the operator framework itself allowing us to focus on building and object and the controller. We will update the spec of our BookStore object later. We will update the spec of BookStore type to include two custom types BookApp and BookDB.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/340867a7dea35a3284b3d34c7f421f09/href">https://medium.com/media/340867a7dea35a3284b3d34c7f421f09/href</a></iframe><p>Let’s also update the BookStore CR (blog.velotio.com_v1alpha1_bookstore_cr.yaml)</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/52dce6365bf22d3cc6d8338398533e33/href">https://medium.com/media/52dce6365bf22d3cc6d8338398533e33/href</a></iframe><h4>3. Add the bookstore controller</h4><p><strong><em>operator-sdk add controller — api-version=blog.velotio.com/v1alpha1 — kind=BookStore</em></strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/718e8d67ad5443c3d021cce09f506d14/href">https://medium.com/media/718e8d67ad5443c3d021cce09f506d14/href</a></iframe><p>The above command adds the bookstore controller (pkg/controller/bookstore/bookstore_controller.go) to the project and also adds it to the manager.</p><p>If we take a look at the add function in the <strong><em>bookstore_controller.go</em></strong> file we can see that a new controller is created here and added to the manager so that the manager can start the controller when it (manager) comes up, the <strong><em>add(mgr manager.Manager, r reconcile.Reconciler)</em> </strong>is called by the public function <strong><em>Add(mgr manager.Manager)</em> </strong>which also creates<strong> </strong>a new<strong> </strong>reconciler<strong> </strong>objects and passes it to the <strong><em>add</em> </strong>where the controller is associated with the reconciler, in the <strong><em>add</em> </strong>function we also set the type of object (BookStore) which the controller will watch.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/01362b7fd1a9648c2eff9d9df94726d5/href">https://medium.com/media/01362b7fd1a9648c2eff9d9df94726d5/href</a></iframe><p>This ensures that for any events related to any object of BookStore type, a reconcile request (a namespace/name key) is sent to the <strong><em>Reconcile</em> </strong>method associated with the reconciler object (<strong><em>ReconcileBookStore</em></strong>) here.</p><h4>4. Build the reconcile logic</h4><p>The<strong> </strong>reconcile logic is implemented inside the <strong><em>Reconcile</em> </strong>method of the reconciler object of the custom type which implements the reconcile loop.</p><p>As a part of our reconcile logic we will do the following</p><ol><li>Create the bookstore app deployment if it doesn’t exist.</li><li>Create the bookstore app service if it doesn’t exist.</li><li>Create the Mongodb statefulset if it doesn’t exist.</li><li>Create the Mongodb service if it doesn’t exist.</li><li>Ensure deployments and services match their desired configurations like the replica count, image tag, service port, size of the PV associated with the Mongodb statefulset, etc.</li></ol><p>There are three possible events that can happen with the BookStore object</p><ol><li><strong>The object got created: </strong>Whenever an object of kind BookStore is created we create all the k8s resources we mentioned above.</li><li><strong>The object has been updated:</strong> When the object gets updated then we update all the k8s resources associated with it.</li><li><strong>The object has been deleted:</strong> When the object gets deleted we don’t need to do anything as while creating the K8s objects we will set the `BookStore` type as its owner which will ensure that all the K8s objects associated with it gets automatically deleted when we delete the object.</li></ol><p>On receiving the reconcile request the first step if to lookup for the object.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fa8adaeeee72dd6a2e2ba790ea4c224f/href">https://medium.com/media/fa8adaeeee72dd6a2e2ba790ea4c224f/href</a></iframe><p>If the object is not found, we assume that it got deleted and don’t requeue the request considering the reconcile to be successful.</p><p>If any error occurs while doing the reconcile then we return the error and whenever we return non-nil error value then controller requeues the request.</p><p>In the reconcile logic we call the BookStore method which creates or updates all the k8s objects associated with the BookStore objects based on whether the object has been created or updated.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f0844be7b06dda256577d2e57c9609f6/href">https://medium.com/media/f0844be7b06dda256577d2e57c9609f6/href</a></iframe><p>The implementation of the above method is a bit hacky but gives an idea of the flow. In the above function, we can see that we are setting the BookStore type as an owner for all the resources <strong><em>controllerutil.SetControllerReference(c, bookStoreDep, r.scheme)</em></strong> as we had discussed earlier. If we look at the owner reference for these objects we would see something like this.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6d1cf2d95af5fe5bf38a4360ccda1061/href">https://medium.com/media/6d1cf2d95af5fe5bf38a4360ccda1061/href</a></iframe><h4>5. Deploy the operator and verify its working</h4><p>The approach to deploy and verify the working of the bookstore application is similar to what we did in the previous two blogs the only difference being that now we have deployed the Mongodb as a stateful set and even if we restart the pod we will see that the information that we stored will still be available.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Gh24PrQzTxzaTYswYqnScA.png" /></figure><h4>6. Verify volume expansion</h4><p>For updating<strong> </strong>the volume associated with the MongoDB instance we first need to update the size of the volume we specified while creating the bookstore object. In the example above I had set it to 2GB let’s update it to 3GB and update the bookstore object.</p><p>Once the bookstore object is updated if we describe the MongoDB PVC we will see that it still has 2GB PV but the conditions we will see something like this.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/acdacfc8d676941f4f08f2b7c7edadbb/href">https://medium.com/media/acdacfc8d676941f4f08f2b7c7edadbb/href</a></iframe><p>It is clear from the message that we need to restart the pod for the resizing of volume to reflect. Once we delete the pod it will get restarted and the PVC will get updated to reflect the expanded volume size.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_EFSbBsPtxcD87zxa16Hxg.png" /></figure><p>The complete code is available here: <a href="https://github.com/akash-gautam/bookstore-operator-golang">https://github.com/akash-gautam/bookstore-operator-golang</a></p><h3>Conclusion</h3><p>Golang based operators are built mostly for stateful applications like databases. The operator can automate complex operational tasks allow us to run applications with ease. At the same time building and maintaining it can be quite complex and we should build one only when we are fully convinced that our requirements can’t be met with any other type of operator. Operators are an interesting and emerging area in Kubernetes and I hope this blog series on getting started with it helps the readers in learning the basics of it.</p><p>*****************************************************************</p><p><em>This post was originally published on </em><a href="https://velotio.com/blog/2019/9/24/kubernetes-golang-operators-part-3"><strong><em>Velotio Blog</em></strong></a><strong><em>.</em></strong></p><p><a href="https://velotio.com/"><strong>Velotio Technologies</strong></a> is an outsourced software product development partner for technology startups and enterprises. We specialize in enterprise B2B and SaaS product development with a focus on artificial intelligence and machine learning, DevOps, and test engineering.</p><p><em>Interested in learning more about us? We would love to connect with you on our </em><a href="https://velotio.com/"><strong><em>Website</em></strong></a><em>, </em><a href="https://www.linkedin.com/company/velotio-technologies/"><strong><em>LinkedIn</em></strong></a><em> or </em><a href="https://twitter.com/velotiotech"><strong><em>Twitter</em></strong></a><em>.</em></p><p>*****************************************************************</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=edcaf3315088" width="1" height="1" alt=""><hr><p><a href="https://medium.com/velotio-perspectives/getting-started-with-kubernetes-operators-golang-based-part-3-edcaf3315088">Getting Started With Kubernetes Operators (Golang Based) — Part 3</a> was originally published in <a href="https://medium.com/velotio-perspectives">Velotio Perspectives</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Getting Started With Kubernetes Operators (Ansible Based) — Part 2]]></title>
            <link>https://medium.com/velotio-perspectives/getting-started-with-kubernetes-operators-ansible-based-part-2-472eb0d453b7?source=rss----3b93934e4b3d---4</link>
            <guid isPermaLink="false">https://medium.com/p/472eb0d453b7</guid>
            <category><![CDATA[cloud-native]]></category>
            <category><![CDATA[ansible]]></category>
            <category><![CDATA[kubernetes-operator]]></category>
            <category><![CDATA[ansible-based-operators]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Velotio Technologies]]></dc:creator>
            <pubDate>Wed, 09 Oct 2019 13:14:04 GMT</pubDate>
            <atom:updated>2020-07-04T10:28:11.420Z</atom:updated>
            <content:encoded><![CDATA[<h3>Getting Started With Kubernetes Operators (Ansible Based) — Part 2</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VNCIyfpuL7Barf8c3oDHDw.png" /></figure><h3>Introduction</h3><p>In the first part of this blog series, <a href="https://velotio.com/blog/2019/9/10/kubernetes-helm-operators-part-1"><strong>getting started with Kubernetes operators (Helm based)</strong></a>, we learned the basics of operators and build a Helm based operator. In this blog post, we will try out an Ansible-based operator. Ansible is a very popular tool used by organizations across the globe for configuration management, deployment, and automation of other operational tasks, this makes Ansible an ideal tool to build operators as with operators also we intend to eliminate/minimize the manual interventions required while running/managing our applications on Kubernetes. Ansible based operators allow us to use Ansible playbooks and roles to manage our application on Kubernetes.</p><p>Read the third part of this blog series, <a href="https://velotio.com/blog/2019/9/24/kubernetes-golang-operators-part-3"><strong>getting started with Kubernetes operators (Helm based)</strong></a><strong>, </strong>to build a Golang based operator.</p><h3>Operator Maturity Model</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yzSB5ATawlPZa63r86QQ_w.png" /><figcaption>Image source: <a href="https://github.com/operator-framework/operator-sdk/blob/master/doc/images/operator-capability-level.png">Github</a></figcaption></figure><p>Before we start building the operator let’s spend some time in understanding the operator maturity model. Operator maturity model gives an idea of the kind of application management capabilities different types of operators can have. As we can see in the diagram above the model describes five generic phases of maturity/capability for operators. The minimum expectation/requirement from an operator is that they should be able to deploy/install and upgrade application and that is provided by all the operators. Helm based operators are the simplest of all of them as Helm is Chart manager and we can do only install and upgrades using it. Ansible based operators can be more mature as Ansible has modules to perform a wide variety of operational tasks, we can use these modules in the Ansible roles/playbooks we use in our operator and make them handle more complex applications or use cases. In the case of Golang based operators, we write the operational logic ourselves so we have the liberty to customize it as per our requirements.</p><h3>Building an Ansible Based Operator</h3><h4>1. Let’s first install the operator sdk</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/73e03ce0f6cdc9c8b84163777f5e8885/href">https://medium.com/media/73e03ce0f6cdc9c8b84163777f5e8885/href</a></iframe><p>Now we will have the operator-sdk binary in the $GOPATH/bin folder.</p><h4>2. Setup the project</h4><p><strong><em>Operator-sdk new bookstore-operator — api-version=blog.velotio.com/v1alpha1 — kind=BookStore — type=ansible</em></strong></p><p>In the above command we have set the operator type as ansible as we want an ansible based operator. It creates a folder structure as shown below</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/90e56533a60e3ce81e3012128535cfa7/href">https://medium.com/media/90e56533a60e3ce81e3012128535cfa7/href</a></iframe><p>Inside the roles folder, it creates an Ansible role name `bookstore`. This role is bootstrapped with all the directories and files which are part of the standard ansible roles.</p><p>Now let’s take a look at the watches.yaml file:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0604444477f405921d4bb8e0263a970c/href">https://medium.com/media/0604444477f405921d4bb8e0263a970c/href</a></iframe><p>Here we can see that it looks just like the operator is going to watch the events related to the objects of BookStore kind and execute the ansible role bookstore. Drawing parallels from our helm based operator we can see that the behavior in both the cases are similar the only difference being that in case of Helm based operator the operator used to execute the helm chart specified in response to the events related to the object it was watching and here we are executing an ansible role.</p><p>In the case of ansible based operators, we can get the operator to execute an Ansible playbook as well rather than an ansible role.</p><h4>3. Building the bookstore Ansible role</h4><p>Now we need to modify the bookstore Ansible roles created for us by the operator-framework.</p><p>First we will update the custom resource (CR) file ( blog_v1alpha1_bookstore_cr.yaml) available at deploy/crd/ location. In this CR we can configure all the values which we want to pass to the bookstore Ansible role. By default the CR contains only the size field, we will update it to include other fields that we need in our role. To keep things simple, we will just include some basic variables like image name, tag, etc. in our spec.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ee6c959e9d918449f4ac878acdec0f97/href">https://medium.com/media/ee6c959e9d918449f4ac878acdec0f97/href</a></iframe><p>The Ansible operator passes the key values pairs listed in the spec of the cr as variables to Ansible. The operator changes the name of the variables to snake_case before running Ansible so when we use the variables in our role we will refer the values in snake case.</p><p>Next, we need to create the tasks the bookstore roles will execute. Now we will update the tasks to define our deployment. By default, an Ansible role executes the tasks defined at `/tasks/main.yml`. For defining our deployment we will leverage the <a href="https://docs.ansible.com/ansible/latest/modules/k8s_module.html">k8s</a> module of Ansible. We will create a Kubernetes deployment and service for our app as well as MongoDB.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8bbec0d75e19f42f05346f3dd662e2ca/href">https://medium.com/media/8bbec0d75e19f42f05346f3dd662e2ca/href</a></iframe><p>In the above file we can see that we have used the pullPolicy field defined in our cr spec as <strong><em>‘pull_policy’ </em></strong>in our tasks. Here we have used inline definition to create our k8s objects as our app is quite simple. For large applications creating objects using separate definition files would be a better approach.</p><h4>4. Build the bookstore-operator image</h4><p>The Dockerfile for building the operator image is already in our build folder we need to run the below command from the root folder of our operator project to build the image.</p><p><strong><em>‘operator-sdk build akash125/bookstore-operator:ansible’</em></strong></p><p>You can use your own docker repository instead of <strong><em>‘akash125/bookstore-operator’</em></strong></p><h4>5. Run the bookstore-operator</h4><p>As we have our operator image ready we can now go ahead and run it. The deployment file (operator.yaml under deploy folder) for the operator was created as a part of our project setup we just need to set the image for this deployment to the one we built in the previous step.</p><p>After updating the image in the operator.yaml we are ready to deploy the operator.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ec12675ebb418642fc0062f5de7edaff/href">https://medium.com/media/ec12675ebb418642fc0062f5de7edaff/href</a></iframe><blockquote><strong><em>Note: </em></strong><em>The role created might have more permissions then actually required for the operator so it is always a good idea to review it and trim down the permissions in production setups.</em></blockquote><p>Verify that the operator pod is in running state.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/734/1*bJWK9IXtVwOPrAy0Drk8Vw.png" /></figure><p>Here two containers have been started as part of the operator deployment. One is the operator and the other one is ansible. The ansible pod exists only to make the logs available to stdout in ansible format.</p><h4>6. Deploy the bookstore app</h4><p>Now we have the<strong> </strong>bookstore-operator running in our cluster we just need to create the<strong> </strong>custom resource for deploying our bookstore app.</p><p>First, we can create bookstore cr we need to register its crd</p><p><strong><em>‘kubectl delete -f deploy/crds/blog_v1alpha1_bookstore_crd.yaml’</em></strong></p><p>Now we can create the bookstore object</p><p><strong><em>‘kubectl delete -f deploy/crds/blog_v1alpha1_bookstore_cr.yaml’</em></strong></p><p>Now we can see that our operator has deployed our book-store app:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/915/1*Zs1chO-I8StQaowb8UcAeg.png" /></figure><p>Now let’s grab the external IP of the app and make some requests to store details of books.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*axwSUxG1ckpMN1SsynYWYQ.png" /></figure><p>Let’s hit the external IP on the browser and see if it lists the books we just stored:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a7rZu3IIZsUQxlsFLpKaSA.png" /></figure><p>We can see that our <strong><em>‘book-store’</em></strong> app is up and running.</p><p>The operator build is available <a href="https://github.com/akash-gautam/bookstore-operator-ansible">here</a>.</p><h3>Conclusion</h3><p>In this blog post, we learned how we can create an Ansible based operator using the operator framework. Ansible based operators are a great way to combine the power of Ansible and Kubernetes as it allows us to deploy our applications using Ansible role and playbooks and we can pass parameters to them (control them) using custom K8s resources. If Ansible is being heavily used across your organization and you are migrating to Kubernetes then Ansible based operators are an ideal choice for managing deployments. In the next blog, we will learn about Golang based operators.</p><p>*****************************************************************</p><p><em>This post was originally published on </em><a href="https://velotio.com/blog/2019/9/17/kubernetes-ansible-operators-part-2"><strong><em>Velotio Blog.</em></strong></a></p><p><a href="https://velotio.com/"><strong>Velotio Technologies</strong></a> is an outsourced software product development partner for technology startups and enterprises. We specialize in enterprise B2B and SaaS product development with a focus on artificial intelligence and machine learning, DevOps, and test engineering.</p><p><em>Interested in learning more about us? We would love to connect with you on our</em><a href="https://velotio.com/"><strong><em>Website</em></strong></a><em>, </em><a href="https://www.linkedin.com/company/velotio-technologies/"><strong><em>LinkedIn</em></strong></a><em> or </em><a href="https://twitter.com/velotiotech"><strong><em>Twitter</em></strong></a><em>.</em></p><p>*****************************************************************</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=472eb0d453b7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/velotio-perspectives/getting-started-with-kubernetes-operators-ansible-based-part-2-472eb0d453b7">Getting Started With Kubernetes Operators (Ansible Based) — Part 2</a> was originally published in <a href="https://medium.com/velotio-perspectives">Velotio Perspectives</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Getting Started With Kubernetes Operators (Helm Based) — Part 1]]></title>
            <link>https://medium.com/velotio-perspectives/getting-started-with-kubernetes-operators-helm-based-part-1-47c49efa61c2?source=rss----3b93934e4b3d---4</link>
            <guid isPermaLink="false">https://medium.com/p/47c49efa61c2</guid>
            <category><![CDATA[helm-based-operators]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[helm]]></category>
            <category><![CDATA[cloud-native]]></category>
            <category><![CDATA[kubernetes-operations]]></category>
            <dc:creator><![CDATA[Velotio Technologies]]></dc:creator>
            <pubDate>Wed, 25 Sep 2019 12:10:07 GMT</pubDate>
            <atom:updated>2020-07-04T10:32:11.602Z</atom:updated>
            <content:encoded><![CDATA[<h3>Getting Started With Kubernetes Operators (Helm Based) — Part 1</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VNCIyfpuL7Barf8c3oDHDw.png" /></figure><h3>Introduction</h3><p>The concept of operators was introduced by CoreOs in the last quarter of 2016 and post the introduction of operator framework last year, operators are rapidly becoming the standard way of managing applications on Kubernetes especially the ones which are stateful in nature. In this blog post, we will learn what an operator is. Why they are needed and what problems do they solve. We will also create a helm based operator as an example.</p><p>This is the first part of our Kubernetes Operator Series. In the second part, <a href="https://velotio.com/blog/2019/9/17/kubernetes-ansible-operators-part-2"><strong>getting started with Kubernetes operators (Ansible based)</strong></a><strong>,</strong> and the third part, <a href="https://velotio.com/blog/2019/9/24/kubernetes-golang-operators-part-3"><strong>getting started with Kubernetes operators (Golang based)</strong></a>, you can learn how to build Ansible and Golang based operators.</p><h3>What is an Operator?</h3><p>Whenever we deploy our application on Kubernetes we leverage multiple Kubernetes objects like deployment, service, role, ingress, config map, etc. As our application gets complex and our requirements become non-generic, managing our application only with the help of native Kubernetes objects becomes difficult and we often need to introduce manual intervention or some other form of automation to make up for it.</p><p>Operators solve this problem by making our application first-class Kubernetes objects that is we no longer deploy our application as a set of native Kubernetes objects but a custom object/resource of its kind, having a more domain-specific schema and then we bake the “operational intelligence” or the “domain-specific knowledge” into the controller responsible for maintaining the desired state of this object. For example, etcd operator has made etcd-cluster a first-class object and for deploying the cluster we create an object of Etcd Cluster kind. With operators, we are able to extend Kubernetes functionalities for custom use cases and manage our applications in a Kubernetes specific way allowing us to leverage Kubernetes APIs and Kubectl tooling.</p><p>Operators combine <a href="https://velotio.com/blog/2018/9/1/extending-kubernetes-api-with-crd">crds </a>and custom controllers and intend to eliminate the requirement for manual intervention (human operator) while performing tasks like an upgrade, handling failure recovery, scaling in case of complex (often stateful) applications, and make them more resilient and self-sufficient.</p><h3>How to Build Operators?</h3><p>For<strong> </strong>building and managing operators, we mostly leverage the <a href="https://github.com/operator-framework">Operator Framework</a> which is an open-source tool kit allowing us to build operators in a highly automated, scalable and effective way. Operator framework comprises of three subcomponents:</p><ol><li><strong>Operator SDK: </strong>Operator SDK is the most important component of the operator framework. It allows us to bootstrap our operator project in minutes. It exposes higher-level APIs and abstraction and saves developers the time to dig deeper into Kubernetes APIs and focus more on building the operational logic. It performs common tasks like getting the controller to watch the custom resource (cr) for changes etc as part of the project setup process.</li><li><strong>Operator Lifecycle Manager:</strong> Operators also run on the same Kubernetes clusters in which they manage applications and more often than not we create multiple operators for multiple applications. Operator lifecycle manager (OLM) provides us a declarative way to install, upgrade and manage all the operators and their dependencies in our cluster.</li><li><strong>Operator Metering:</strong> Operator metering is currently an alpha project. It records historical cluster usage and can generate usage reports showing usage breakdown by pod or namespace over arbitrary time periods.</li></ol><h3>Types of Operators</h3><p>Currently, there are three different types of operator we can build:</p><ol><li><strong>Helm based operators: </strong>Helm based operators allow us to use our existing helm charts and build operators using them. Helm based operators are quite easy to build and are preferred to deploy a stateless application using operator pattern.</li><li><strong>Ansible based Operator:</strong> Ansible based operator allows us to use our existing Ansible playbooks and roles and build operators using them. There are also easy to build and generally preferred for stateless applications.</li><li><strong>Go based operators: </strong>Go based operators are built to solve the most complex use cases and are generally preferred for stateful applications. In case of a Golang based operator, we build the controller logic ourselves providing it with all our custom requirements. This type of operators is also relatively complex to build.</li></ol><h3>Building a Helm based operator</h3><h4>1. Let’s first install the operator sdk</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/85d395edad717e4982dab6c4eb9f670b/href">https://medium.com/media/85d395edad717e4982dab6c4eb9f670b/href</a></iframe><p>Now we will have the operator-sdk binary in the $GOPATH/bin folder.</p><h4>2. Setup the project</h4><p>For building a helm based operator we can use an existing Helm chart. We will be using the <a href="https://github.com/akash-gautam/helmcharts/tree/master/book-store">book-store Helm chart</a> which deploys a simple python app and MongoDB instances. This app allows us to perform crud operations via. rest endpoints.</p><p>Now we will use the operator-sdk to create our Helm based bookstore-operator project.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/dca50dc8e2974d2ef30ad4e640503c8e/href">https://medium.com/media/dca50dc8e2974d2ef30ad4e640503c8e/href</a></iframe><p>In the above command, the bookstore-operator is the name of our operator/project. — kind is used to specify the kind of objects this operator will watch and — api-verison is used for versioning of this object. The operator sdk takes only this much information and creates the custom resource definition (crd) and also the custom resource (cr) of its type for us (remember we talked about high-level abstraction operator sdk provides). The above command bootstraps a project with below folder structure</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c9069bcae1d962bbfaeca6fb6b0aaa5a/href">https://medium.com/media/c9069bcae1d962bbfaeca6fb6b0aaa5a/href</a></iframe><p>We had discussed the operator-sdk automates setting up the operator projects and that is exactly what we can observe here. Under the build folder, we have the Dockerfile to build our operator image. Under the deploy folder, we have a crd folder containing both the crd and the cr. This folder also has operator.yaml file using which we will run the operator in our cluster, along with this we have manifest files for role, rolebinding and service account file to be used while deploying the operator. We have our book-store helm chart under helm-charts. In the watches.yaml file.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/82da3c716fc6b52634324a21671c4d2d/href">https://medium.com/media/82da3c716fc6b52634324a21671c4d2d/href</a></iframe><p>We can see that the bookstore-operator watches events related to BookStore kind objects and executes the helm chart specified.</p><p>If we take a look at the cr file under deploy/crds (velotio_v1alpha1_bookstore_cr.yaml) folder then we can see that it looks just like the values.yaml file of our book-store helm chart.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ce07ea286f5f1b090c4a8debc3316727/href">https://medium.com/media/ce07ea286f5f1b090c4a8debc3316727/href</a></iframe><p>In the case of Helm charts, we use the values.yaml file to pass the parameter to our Helm releases, Helm based operator converts all these configurable parameters into the spec of our custom resource. This allows us to express the values.yaml with a custom resource (CR) which, as a native Kubernetes object, enables the benefits of RBAC applied to it and an audit trail. Now when we want to update out deployed we can simply modify the CR and apply it, and the operator will ensure that the changes we made are reflected in our app.</p><p>For each object of `BookStore` kind the bookstore-operator will perform the following actions:</p><ol><li>Create the bookstore app deployment if it doesn’t exist.</li><li>Create the bookstore app service if it doesn’t exist.</li><li>Create a MongoDB deployment if it doesn’t exist.</li><li>Create a MongoDB service if it doesn’t exist.</li><li>Ensure deployments and services match their desired configurations like the replica count, image tag, service port, etc.</li></ol><h4>3. Build the Bookstore-operator Image</h4><p>The Dockerfile for building the operator image is already in our build folder we need to run the below command from the root folder of our operator project to build the image.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4ea5d4ef84e46a88ffd01a868c8042f0/href">https://medium.com/media/4ea5d4ef84e46a88ffd01a868c8042f0/href</a></iframe><h4>4. Run the Bookstore-operator</h4><p>As we have our operator image ready we can now go ahead and run it. The deployment file (operator.yaml under deploy folder) for the operator was created as a part of our project setup we just need to set the image for this deployment to the one we built in the previous step.</p><p>After updating the image in the operator.yaml we are ready to deploy the operator.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a1e5df6c1a1e307ef7af12102ea12042/href">https://medium.com/media/a1e5df6c1a1e307ef7af12102ea12042/href</a></iframe><blockquote><strong><em>Note: </em></strong><em>The role created might have more permissions then actually required for the operator so it is always a good idea to review it and trim down the permissions in production setups.</em></blockquote><p>Verify that the operator pod is in running state.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*e5dbqAm4wxOe3ntDgg6N0w.png" /></figure><h4>5. Deploy the Bookstore App</h4><p>Now we have the<strong> </strong>bookstore-operator running in our cluster we just need to create the<strong> </strong>custom resource for deploying our bookstore app.</p><p>First, we can create bookstore cr we need to register its crd.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9de406fb209d8c995f32717d59c33883/href">https://medium.com/media/9de406fb209d8c995f32717d59c33883/href</a></iframe><p>Now we can create the bookstore object.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/67be14f4c4576a6dc860cd5f00b24e5d/href">https://medium.com/media/67be14f4c4576a6dc860cd5f00b24e5d/href</a></iframe><p>Now we can see that our operator has deployed our book-store app.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*__yZGK1-pGV9rhuAngbCRg.png" /></figure><p>Now let’s grab the external IP of the app and make some requests to store details of books.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zgoRy9le9KXsf_bCaTjC_Q.png" /></figure><p>Let’s hit the external IP on the browser and see if it lists the books we just stored:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WukRMStRJn7I21rxGeLGEQ.png" /></figure><p>The bookstore operator build is available <a href="https://github.com/akash-gautam/bookstore-operator">here</a>.</p><h3>Conclusion</h3><p>Since its early days, Kubernetes was believed to be a great tool for managing stateless applications but managing stateful applications on Kubernetes was always considered difficult. Operators are a big leap towards managing stateful applications and other complex distributed, multi (poly) cloud workloads with the same ease that we manage the stateless applications. In this blog post, we learned the basics of Kubernetes operators and build a simple helm based operator. In the next installment of this blog series, we will build an <a href="https://velotio.com/blog/2019/9/17/kubernetes-ansible-operators-part-2"><strong>Ansible based Kubernetes operator</strong></a><strong> </strong>and then in the last blog we will build a full-fledged Golang based operator for managing stateful workloads.</p><p><strong>Related Reads:</strong></p><ul><li><a href="https://velotio.com/blog/2019/5/6/cicd-for-kubernetes-using-circleci-and-helm">Continuous integration and delivery (CI/CD) for Kubernetes using CircleCI and Helm</a></li><li><a href="https://velotio.com/blog/2018/9/1/extending-kubernetes-api-with-crd">Extending Kubernetes APIs with Custom Resource Definitions (CRDs)</a></li></ul><p>*****************************************************************</p><p><em>This post was originally published on </em><a href="https://velotio.com/blog/2019/9/10/kubernetes-helm-operators-part-1"><strong><em>Velotio Blog.</em></strong></a></p><p><a href="https://velotio.com/"><strong>Velotio Technologies</strong></a> is an outsourced software product development partner for technology startups and enterprises. We specialize in enterprise B2B and SaaS product development with a focus on artificial intelligence and machine learning, DevOps, and test engineering.</p><p><em>Interested in learning more about us? We would love to connect with you on our</em><a href="https://velotio.com/"><strong><em>Website</em></strong></a><em>, </em><a href="https://www.linkedin.com/company/velotio-technologies/"><strong><em>LinkedIn</em></strong></a><em> or </em><a href="https://twitter.com/velotiotech"><strong><em>Twitter</em></strong></a><em>.</em></p><p>*****************************************************************</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=47c49efa61c2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/velotio-perspectives/getting-started-with-kubernetes-operators-helm-based-part-1-47c49efa61c2">Getting Started With Kubernetes Operators (Helm Based) — Part 1</a> was originally published in <a href="https://medium.com/velotio-perspectives">Velotio Perspectives</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kubernetes CSI in Action: Explained with Features and Use Cases]]></title>
            <link>https://medium.com/velotio-perspectives/kubernetes-csi-in-action-explained-with-features-and-use-cases-4f966b910774?source=rss----3b93934e4b3d---4</link>
            <guid isPermaLink="false">https://medium.com/p/4f966b910774</guid>
            <category><![CDATA[interfaces]]></category>
            <category><![CDATA[ci]]></category>
            <category><![CDATA[storage-containers]]></category>
            <category><![CDATA[cloud-native]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Velotio Technologies]]></dc:creator>
            <pubDate>Wed, 18 Sep 2019 13:29:33 GMT</pubDate>
            <atom:updated>2020-07-04T14:49:39.178Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UcxkXEKLjbd0ZAx-9Q7krw.png" /></figure><p>Kubernetes Volume plugins have been a great way for the third-party storage providers to support a block or file storage system by extending the Kubernetes volume interface and are “In-Tree” in nature.</p><p>In this post, we will dig into Kubernetes <a href="https://kubernetes.io/blog/2019/01/15/container-storage-interface-ga/">Container Storage Interface</a>. We will use Hostpath CSI Driver locally on a single node bare metal cluster, to get the conceptual understanding of the CSI workflow in provisioning the Persistent Volume and its lifecycle. Also, a cool feature of snapshotting the volume and recover it back is explained.</p><h3>Introduction</h3><p>CSI is a standard for exposing storage systems in an arbitrary block and file storage to <a href="https://www.velotio.com/engineering-blog/automated-containerization-and-migration-of-on-premise-applications-to-cloud-platforms">containerized</a> workloads on Container Orchestrations like Kubernetes, Mesos, and Cloud Foundry. It becomes very extensible for third-party storage provider to expose their new storage systems using CSI, without actually touching the Kubernetes code. Single independent implementation of CSI Driver by a storage provider will work on any orchestrator.</p><p>This new plugin mechanism has been one of the most powerful features of Kubernetes. It enables the storage vendors to:</p><ol><li>Automatically create storage when required.</li><li>Make storage available to containers wherever they’re scheduled.</li><li>Automatically delete the storage when no longer needed.</li></ol><p>This decoupling helps the vendors to maintain the independent release and feature cycles and focus on the API implementation without actually worrying about the backward incompatibility and to support their plugin just as easy as deploying a few pods.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/711/1*RuYBVRJGfu1M0SC81Auzog.png" /><figcaption>Image Source: <a href="https://weekly-geekly.github.io/articles/424211/index.html">Weekly Geekly</a></figcaption></figure><h3>Why CSI?</h3><p>Prior to CSI, k8s volume plugins have to be “In-tree”, compiled and shipped with core kubernetes binaries. This means, it will require the storage providers to check-in their into the core k8s codebase if they wish to add the support for new storage system.</p><p>A plugin-based solution, flex-volume, tried to address this issue by exposing the exec based API for external plugins. Although it also tried to work on the similar notion of being detached with k8s binary, there were several major problems with that approach. Firstly, it needed the root access to the host and master file system to deploy the driver files.</p><p>Secondly, it comes with the huge baggage of prerequisites and OS dependencies which are assumed to be available on the host. CSI implicitly solves all these issues by being containerized and using the k8s storage primitives.</p><p>CSI has evolved as the one-stop solution addressing all the above issues which enables storage plugins to be out-of-tree and deployed via standard k8s primitives, such as PVC, PV and StorageClasses.</p><p>The main aim of introducing CSI is to establish a standard mechanism of exposing any type of storage system under-the-hood for all the container orchestrators.</p><h3>Deploy the Driver Plugin</h3><p>The CSI Driver comprises of a few main components which are various side cars and also the implementation of the CSI Services by the vendor, which will be understood by the Cos. The CSI Services will be described later in the blog. Let’s try out deploying hostpath CSI Driver.</p><h3>Prerequisites:</h3><ul><li><a href="https://www.velotio.com/engineering-blog/the-ultimate-guide-to-disaster-recovery-for-your-kubernetes-clusters">Kubernetes cluster</a> (not Minikube or Microk8s): Running version 1.13 or later</li><li>Access to the terminal with <strong>Kubectl</strong> installed</li></ul><h3>Deploying HostPath Driver Plugin</h3><ul><li>Clone the<a href="https://github.com/kubernetes-csi/csi-driver-host-path/"> repo</a> of HostPath Driver Plugin locally or just copy the <strong>deploy</strong> and <strong>example</strong> folder from the root path</li><li>Checkout the master branch (if not)</li><li>The hostpath driver comprises of manifests for following side-cars: (in ./deploy/master/hostpath/)<br>- csi-hostpath-attacher.yaml<br>- csi-hostpath-provisioner.yaml<br>- csi-hostpath-snapshotter.yaml<br>- csi-hostpath-plugin.yaml: <br> It will deploy 2 containers, one is <strong>node-driver-registrar</strong> and a <strong>hospath-plugin</strong></li><li>The driver also includes separate <em>Service</em> for each component and in the deployment file with statefulsets for the containers</li><li>It also deploys Cluster-role-bindings and RBAC rules for each component, maintained in a separate repo</li><li>Each Component (side-car) is managed in a separate repository</li><li>The<strong> /deploy/util/</strong> contains a shell script which handles the complete deployment process</li><li>After copying the folder or cloning the repo, just run:</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/304bb5619d7fc11b579bc415314435dc/href">https://medium.com/media/304bb5619d7fc11b579bc415314435dc/href</a></iframe><p>9. The output will be similar to:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2425176150af8514a2d602dd749dc15f/href">https://medium.com/media/2425176150af8514a2d602dd749dc15f/href</a></iframe><p>10. The driver is deployed, we can check:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7304d8b00e42f48a9b1b342603840671/href">https://medium.com/media/7304d8b00e42f48a9b1b342603840671/href</a></iframe><h3>CSI API-Resources</h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0938928a127692bc34ea90f83b81ace4/href">https://medium.com/media/0938928a127692bc34ea90f83b81ace4/href</a></iframe><p>There are resources from core <em>apigroups</em>, storage.k8s.io, and resources created by CRDs snapshot.storage.k8s.io and csi.storage.k8s.io.</p><h4>CSI SideCars</h4><p>K8s CSI containers are sidecars that simplify the development and deployment of the CSI Drivers on a k8s cluster. Different Drivers have some similar logic to trigger the appropriate operations against the “<em>CSI volume driver</em>” container and update the Kubernetes API as appropriate.</p><p>The common controller (common containers) has to be bundled with the provider-specific containers.</p><p>The official sig-k8s contributors maintain the following basic skeleton containers for any CSI Driver:</p><ul><li><a href="https://kubernetes-csi.github.io/docs/external-provisioner.html">external-provisioner</a></li><li><a href="https://kubernetes-csi.github.io/docs/external-attacher.html">external-attacher</a></li><li><a href="https://kubernetes-csi.github.io/docs/external-snapshotter.html">external-snapshotter</a></li><li><a href="https://kubernetes-csi.github.io/docs/node-driver-registrar.html">node-driver-registrar</a></li><li><a href="https://kubernetes-csi.github.io/docs/cluster-driver-registrar.html">cluster-driver-registrar</a></li><li><a href="https://kubernetes-csi.github.io/docs/livenessprobe.html">livenessprobe</a></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/711/1*bkMOGMuyCYXH8ZyR5Tngig.png" /></figure><p><strong>Note:</strong> In case of Hostpath driver, only ‘csi-hostpath-plugin’ container will be having the specific code. All the others are common<a href="https://kubernetes-csi.github.io/docs/sidecar-containers.html"> CSI sidecar containers</a>. These containers have a socket mounted in the socket-dir volume of type EmptyDir, which makes their communication possible using gRPC</p><ul><li><strong>External Provisioner:<br></strong>It is a sidecar container that watches Kubernetes <em>PersistentVolumeClaim</em> objects and triggers CSI <em>CreateVolume</em> and <em>DeleteVolume</em> operations against a driver endpoint.<br>The CSI external-attacher also supports the Snapshot <em>DataSource</em>. If a Snapshot CRD is specified as a data source on a PVC object, the sidecar container fetches the information about the snapshot by fetching the <em>SnapshotContent</em> object and populates the data source field indicating to the storage system that new volume should be populated using a specified snapshot.</li><li><strong>External Attacher :<br></strong>It is a sidecar container that watches Kubernetes <em>VolumeAttachment</em> objects and triggers CSI <em>ControllerPublish</em> and <em>ControllerUnpublish</em> operations against a driver endpoint</li><li><strong>Node-Driver Registrar:<br></strong>It is a sidecar container that registers the CSI driver with <em>kubelet</em>, and adds the drivers custom <em>NodeId</em> to a label on the Kubernetes Node API Object. The communication of this sidecar is handled by the ‘Identity-Service’ implemented by the driver. The CSI Driver is registered with the kubelet using its device — plugin mechanisms</li><li><strong>External Snapshotter:<br></strong>It is a sidecar container that watches the Kubernetes API server for <em>VolumeSnapshot</em> and <em>VolumeSnapshotContent</em> CRD objects.The creation of a new <em>VolumeSnapshot</em> object referencing a <em>SnapshotClass</em> CRD object corresponding to this driver causes the sidecar container to provision a new snapshot.</li><li>This sidecar listens to the service which indicates the successful creation of VolumeSnapshot, and immediately creates the VolumeSnapshotContent resource</li><li><strong>Cluster-driver Registrar:<br></strong>CSI driver is registered with the cluster by a sidecar container CSI cluster-driver-registrar creating a CSIDriver object. This CSIDriver enables the driver to customize the way of k8s interaction with it.</li></ul><h3>Developing a CSI Driver</h3><p>To start the implementation of CSIDriver, an application must implement the gRPC services described by the CSI Specification.</p><p>The minimum service a CSI application should implement are following:</p><ul><li><em>CSI Identity </em>service: Enables Kubernetes components and CSI containers to identify the driver</li><li><em>CSI Node</em> service: Required methods enable callers to make volume available at a specified path.</li></ul><p>All the required services may be implemented independently or in the same driver application. The CSI driver application should be containerised to make it easy to deploy on Kubernetes. Once the main specific logic of the driver is containerized, they can be attached to the sidecars and deployed, in node and/or controller mode.</p><h4>Capabilities</h4><p>CSI also have provisions to enable the custom CSI driver to support many additional features/services by using the “Capabilities”. It contains a list of all the features the driver supports.</p><p>Note: Refer the<a href="https://kubernetes-csi.github.io/docs/developing.html"> link</a> for detailed explanation for developing a CSI Driver.</p><h3>Try out provisioning the PV</h3><ol><li>We need to create a storage class with:</li></ol><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/78d42cb776b49f2de226980c6fad19ee/href">https://medium.com/media/78d42cb776b49f2de226980c6fad19ee/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1427d55cdb48f71df40b74e73f8ef1f7/href">https://medium.com/media/1427d55cdb48f71df40b74e73f8ef1f7/href</a></iframe><p>2. And also a sample pod is also required, so that it can be bounded with the PV created by the PVC from above step</p><p>The above files are found in ./exmples directory and can be deployed using create or apply kubectl commands</p><p>Validate the deployed components:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bb9b3168df68e48e5707480bbcb0d3f4/href">https://medium.com/media/bb9b3168df68e48e5707480bbcb0d3f4/href</a></iframe><p>3. The Pod to consume the PV</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/427c0a26645f7d162f653defd42ab30e/href">https://medium.com/media/427c0a26645f7d162f653defd42ab30e/href</a></iframe><p>The Pod should be bound to the PV from above step</p><p>The above files are found in ./exmples directory and can be deployed using create or apply kubectl commands</p><p>Validate the deployed components:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f874bbc32add49f1cf3587d25028b1fe/href">https://medium.com/media/f874bbc32add49f1cf3587d25028b1fe/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/55916389259f92f0111cc5a6829bea7c/href">https://medium.com/media/55916389259f92f0111cc5a6829bea7c/href</a></iframe><h4>Brief on how it works:</h4><ul><li><em>csi-provisioner </em>issues<a href="https://github.com/container-storage-interface/spec/blob/master/spec.md#createvolume"><em> </em>CreateVolumeRequest</a> call to the CSI socket, then <em>hostpath-plugin</em> calls <em>CreateVolume</em> and informs CSI about its creation</li><li><em>csi-provisioner creates PV and </em>updates <em>PVC</em> to be bound and the <em>VolumeAttachment</em> object is created by controller-manager</li><li><em>csi-attacher </em>which watches for VolumeAttachments submits<a href="https://github.com/container-storage-interface/spec/blob/master/spec.md#controllerpublishvolume"> ControllerPublishVolume</a> rpc call to <em>hostpath-plugin, then hostpaths-plugin </em>gets<a href="https://github.com/container-storage-interface/spec/blob/master/spec.md#controllerpublishvolume"> ControllerPublishVolume</a> and calls hostpath <em>AttachVolume</em> <em>csi-attacher update VolumeAttachment status</em></li><li>All this time <em>kubelet</em> waits for volume to be attached and submits<a href="https://github.com/container-storage-interface/spec/blob/master/spec.md#nodestagevolume"> NodeStageVolume</a><em> (format and mount to the node to the staging dir) to the csi-node.hostpath-plugin</em></li><li><em>csi-node.hostpath-plugin</em> gets<a href="https://github.com/container-storage-interface/spec/blob/master/spec.md#nodestagevolume"><em> </em>NodeStageVolume</a><em> </em>call and mounts to <em>`</em>/var/lib/kubelet/plugins/kubernetes.io/csi/pv/&lt;pv-name&gt;/globalmount`, then responses to kubelet</li><li><em>kubelet </em>calls<a href="https://github.com/container-storage-interface/spec/blob/master/spec.md#nodepublishvolume"> NodePublishVolume</a> (mount volume to the pod’s dir)</li><li><em>csi-node.hostpath-plugin performs</em><a href="https://github.com/container-storage-interface/spec/blob/master/spec.md#nodepublishvolume"><em> </em>NodePublishVolume</a> and mounts the volume to `/var/lib/kubelet/pods/&lt;pod-uuid&gt;/volumes/<a href="http://kubernetes.io/">kubernetes.io</a>~csi/&lt;pvc-name&gt;/mount`</li><li>Finally, kubelet starts container of the pod with the provisioned volume.</li></ul><h3>Let’s confirm the working of Hostpath CSI driver</h3><p>The Hostpath driver is configured to create new volumes in the hostpath container in the plugin daemonset under the ‘/tmp’ directory. This path persist as long as the DaemonSet pod is up and running.</p><p>If a file is written in the hostpath mounted volume in an application pod, should be seen in the hostpath cotainer.A file written in a properly mounted Hostpath volume inside an application should show up inside the Hostpath container.</p><ol><li>First, create a file from the application pod as shown:</li></ol><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ee80810b33939f01fae0858eb262ec5f/href">https://medium.com/media/ee80810b33939f01fae0858eb262ec5f/href</a></iframe><p>2. Next, Exec into the Hostpath container and verify that the file shows up there:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9579661320c418d4e2851749a887a9b5/href">https://medium.com/media/9579661320c418d4e2851749a887a9b5/href</a></iframe><p><strong>Note</strong>: A better way of the verification is to inspect the VolumeAttachment object created that represents the attached volume API object created that represents the attached volume</p><h3>Support for Snapshot</h3><p>Volume Snapshotting is introduced as an Alpha feature for the Kubernetes persistent volume in v1.12.</p><p>Being an alpha feature, ‘VolumeSnapshotDataSource’ feature gate needs to be enabled. This feature opens a pool of use cases of keeping the snapshot of data locally. The API objects used are VolumeSnapshot, VolumeSnapshotContent and VolumeSnapshotClass. It was developed with a similar notion and relationship of PV, PVC and StorageClass.</p><p>To create a snapshot, the VolumeSnapshot object needs to be created with the source as PVC and VolumeSnapshotClass</p><p>and the CSI-Snapshotter container will create a <em>VolumeSnaphsotContent.</em></p><h3>Let’s try out with an example:</h3><p>We can create a <em>volumesnapshot</em> object, which will create the <em>volumesnapshotcontent </em>object.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2144435c322c6435ecdf7a54f25ec95d/href">https://medium.com/media/2144435c322c6435ecdf7a54f25ec95d/href</a></iframe><p>The <em>volumesnapshotcontent </em>is created. The output will look like:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/13e55f02097a416b1f0eae0bf4f8676d/href">https://medium.com/media/13e55f02097a416b1f0eae0bf4f8676d/href</a></iframe><h3>Restore from the snapshot:</h3><p>The <strong><em>DataSource </em></strong>field in the PVC can accept the source of <em>kind: VolumeSnapsot </em>which will create a new PV from that volume snapshot when a Pod is bound to this PVC.</p><p>The new PV will be having the same data as of the PV from which the snapshot was taken and it can be attached to any other pod. The new pod having that PV, proves of the possible “Restore” and “Cloning” use cases.</p><p><strong>Tear Down CSI-Hostpath installation:</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c193a6fda2dcb27416e6bd0ab1f464eb/href">https://medium.com/media/c193a6fda2dcb27416e6bd0ab1f464eb/href</a></iframe><p>And we’re done here!</p><h3>Conclusion</h3><p>Since Kubernetes has started supporting the Raw Block Device type of Persistent Volume, hospath driver and any other driver may also support it, which will be explained in the next part of the blog. In this blog, we got deep understanding of the CSI, its components and services. The new features of CSI and the problems it solves. The CSI Hostpath driver was deeply in this blog to experiment and understand the provisioner, snapshotter flows for PV and VolumeSnapshots. Also, the PV snapshot, restore and cloning use cases were demonstrated.</p><p>*****************************************************************</p><p><em>This post was originally published on </em><a href="https://www.velotio.com/engineering-blog/kubernetes-csi-in-action-explained-with-features-and-use-cases"><strong><em>Velotio Blog</em></strong></a><strong><em>.</em></strong></p><p><a href="https://velotio.com/"><strong>Velotio Technologies</strong></a> is an outsourced software product development partner for technology startups and enterprises. We specialize in enterprise B2B and SaaS product development with a focus on artificial intelligence and machine learning, DevOps, and test engineering.</p><p><em>Interested in learning more about us? We would love to connect with you on our </em><a href="https://velotio.com/"><strong><em>Website</em></strong></a><em>, </em><a href="https://www.linkedin.com/company/velotio-technologies/"><strong><em>LinkedIn</em></strong></a><em> or </em><a href="https://twitter.com/velotiotech"><strong><em>Twitter</em></strong></a><em>.</em></p><p>*****************************************************************</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4f966b910774" width="1" height="1" alt=""><hr><p><a href="https://medium.com/velotio-perspectives/kubernetes-csi-in-action-explained-with-features-and-use-cases-4f966b910774">Kubernetes CSI in Action: Explained with Features and Use Cases</a> was originally published in <a href="https://medium.com/velotio-perspectives">Velotio Perspectives</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Write Jenkinsfile for Angular and .Net Based Applications]]></title>
            <link>https://medium.com/velotio-perspectives/how-to-write-jenkinsfile-for-angular-and-net-based-applications-1cbdbf56f906?source=rss----3b93934e4b3d---4</link>
            <guid isPermaLink="false">https://medium.com/p/1cbdbf56f906</guid>
            <category><![CDATA[jenkinsfile]]></category>
            <category><![CDATA[jenkins]]></category>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[angular]]></category>
            <category><![CDATA[dotnet]]></category>
            <dc:creator><![CDATA[Velotio Technologies]]></dc:creator>
            <pubDate>Thu, 12 Sep 2019 10:31:18 GMT</pubDate>
            <atom:updated>2020-07-04T10:39:15.178Z</atom:updated>
            <content:encoded><![CDATA[<h3>How to Write Jenkinsfile for Angular and .Net Based Applications</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kNiWuNXoZjGW5_jh8VWb9w.png" /></figure><p>If you landed here directly and want to know how to setup Jenkins master-slave architecture, please visit this post related to<a href="https://velotio.com/blog/2019/8/5/setup-jenkins-master-slave-architecture"> Setting-up the Jenkins Master-Slave Architecture</a>.</p><p>The source code that we are using here is also a continuation of the code that was written in this GitHub<a href="https://github.com/ismail0352/Packer-Terraform-Jenkins"> Packer-Terraform-Jenkins</a> repository.</p><h3>Creating Jenkinsfile</h3><p>We will create some Jenkinsfile to execute a job from our Jenkins master.</p><p>Here I will create two Jenkinsfile ideally, it is expected that your Jenkinsfile is present in source code repo but it can be passed directly in the job as well.</p><p>There are 2 ways of writing Jenkinsfile — Scripted and Declarative. You can find numerous points online giving their difference. We will be creating both of them to do a build so that we can get a hang of both of them.</p><h3>Jenkinsfile for Angular App (Scripted)</h3><p>As mentioned before we will be highlighting both formats of writing the Jenkinsfile. For the Angular app, we will be writing a scripted one but can be easily written in declarative format too.</p><p>We will be running this inside a docker container. Thus, the tests are also going to get executed in a headless manner.</p><p>Here is the Jenkinsfile for reference.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f8edf78ecc386826b78722306533412a/href">https://medium.com/media/f8edf78ecc386826b78722306533412a/href</a></iframe><p>Here we are trying to leverage Docker volume to keep updating our source code on bare metal and use docker container for the environments.</p><h3>Dissecting Node App’s Jenkinsfile</h3><ol><li>We are using CleanWs() to clear the workspace.</li><li>Next is the Main build in which we define our complete build process.</li></ol><ul><li>We are pulling the required images.</li><li>Highlighting the steps that we will be executing.</li></ul><p>3. <strong>Checkout SCM</strong>: Checking out our code from Git.</p><p>4. We are now starting the node container inside of which we will be running <strong><em>npm install</em></strong> and <strong><em>npm run lint</em></strong>.</p><p>5. <strong>Get test dependency</strong>: Here we are downloading chrome.json which will be used in the next step when starting the container.</p><p>6. Here we test our app. Specific changes for running the test are mentioned below.</p><p>7. <strong>Build</strong>: Finally, we build the app.</p><p>8.<strong> Deploy</strong>: Once CI is completed we need to start with CD. The CD itself can be a blog of itself but wanted to highlight what basic deployment would do.</p><ul><li>Here we are using Nginx container to host our application.</li><li>If the container does not exist it will create a container and use the “dist” folder for deployment.</li><li>If Nginx container exists, then it will ask for user input to recreate a container or not.</li><li>If you select not to create, don’t worry as we are using Nginx it will do a hot reload with new changes.</li></ul><p>The angular application used here was created using the standard generate command given by the CLI itself. Although the build and install give no trouble in a bare metal some tweaks are required for running test in a container.</p><p>In karma.conf.js update browsers withChromeHeadless.</p><p>Next in protractor.conf.js update browserName with chrome and add:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f440779b3be07c05bdc14f1a0f2cdc02/href">https://medium.com/media/f440779b3be07c05bdc14f1a0f2cdc02/href</a></iframe><p>That’s it! And We have our CI pipeline setup for Angular based application.</p><h3>Jenkinsfile for .Net App (Declarative)</h3><p>For a .Net application, we have to setup MSBuild and MSDeploy. In the blog post mentioned above, we have already setup MSBuild and we will shortly discuss how to setup MSDeploy.</p><p>To do the Windows deployment we have two options. Either setup MSBuild in Jenkins Global Tool Configuration or use the full path of MSBuild on the slave machine.</p><p>Passing the path is fairly simple and here we will discuss how to use global tool configuration in a Jenkinsfile.</p><p>First, get the path of MSBuild from your server. If it is not the latest version then the path is different and is available in Current directory otherwise always in &lt;version&gt; directory.</p><p>As we are using MSBuild 2017. Our MSBuild path is:</p><pre>C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\MSBuild\15.0\Bin</pre><p>Place this in /configureTools/ → MSBuild</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/903/1*-4eCLmWL0LTmGhjBs_A8ng.png" /></figure><p>Now you have your configuration ready to be used in Jenkinsfile.</p><p>Jenkinsfile to build and test the app is given below.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bc5bef9a7aad7da00fba9cef814e83da/href">https://medium.com/media/bc5bef9a7aad7da00fba9cef814e83da/href</a></iframe><p>As seen above the structure of Declarative syntax is almost same as that of Declarative. Depending upon which one you find easier to read you should opt the syntax.</p><h3>Dissecting Dotnet App’s Jenkinsfile</h3><ol><li>In this case too we are cleaning the workspace as the first step.</li><li><strong>Checkout</strong>: This is also the same as before.</li><li><strong>Nuget Restore</strong>: We are downloading dependent required packages for both PrimeService and PrimeService.Tests</li><li><strong>Build</strong>: Building the Dotnet app using MSBuild tool which we had configured earlier before writing the Jenkinsfile.</li><li><strong>UnitTest</strong>: Here we have used dotnet test although we could’ve used MSTest as well here just wanted to highlight how easy dotnet utility makes it. We can even use dotnet build for the build as well.</li><li><strong>Deploy</strong>: Deploying on the IIS server. Creation of IIS we are covering below.</li></ol><p>From the above-given examples, you get a hang of what Jenkinsfile looks like and how it can be used for creating jobs. Above file highlights basic job creation but it can be extended to everything that old-style job creation could do.</p><h3>Creating IIS Server</h3><p>Unlike our Angular application where we just had to get another image and we were good to go. Here we will have to Packer to create our IIS server. We will be automating the creation process and will be using it to host applications.</p><p>Here is a Powershell script for IIS for reference.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e2186f9aa80517d6e1de3c6b5ad73116/href">https://medium.com/media/e2186f9aa80517d6e1de3c6b5ad73116/href</a></iframe><p>We won’t be deploying any application on it as we have created a sample app for PrimeNumber. But in the real world, you might be deploying Web Based application and you will need IIS. We have covered here the basic idea of how to install IIS along with any dependency that might be required.</p><h3>Conclusion</h3><p>In this post, we have covered deploying Windows and Linux based applications using Jenkinsfile in both scripted and declarative format. If you have any doubts or suggestions please mention in the comment section.</p><p>Thanks for Reading! Till next time…!!</p><p>*****************************************************************</p><p><em>This post was originally published on </em><a href="https://www.velotio.com/engineering-blog/how-to-write-jenkinsfile-for-angular-and-net-based-applications"><strong><em>Velotio Blog</em></strong></a><strong><em>.</em></strong></p><p><a href="https://velotio.com/"><strong>Velotio Technologies</strong></a> is an outsourced software product development partner for technology startups and enterprises. We specialize in enterprise B2B and SaaS product development with a focus on artificial intelligence and machine learning, DevOps, and test engineering.</p><p><em>Interested in learning more about us? We would love to connect with you on our</em><a href="https://velotio.com/"><strong><em>Website</em></strong></a><em>, </em><a href="https://www.linkedin.com/company/velotio-technologies/"><strong><em>LinkedIn</em></strong></a><em> or </em><a href="https://twitter.com/velotiotech"><strong><em>Twitter</em></strong></a><em>.</em></p><p>*****************************************************************</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1cbdbf56f906" width="1" height="1" alt=""><hr><p><a href="https://medium.com/velotio-perspectives/how-to-write-jenkinsfile-for-angular-and-net-based-applications-1cbdbf56f906">How to Write Jenkinsfile for Angular and .Net Based Applications</a> was originally published in <a href="https://medium.com/velotio-perspectives">Velotio Perspectives</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using Packer and Terraform to Setup Jenkins Master-Slave Architecture]]></title>
            <link>https://medium.com/velotio-perspectives/using-packer-and-terraform-to-setup-jenkins-master-slave-architecture-7bcc4b014874?source=rss----3b93934e4b3d---4</link>
            <guid isPermaLink="false">https://medium.com/p/7bcc4b014874</guid>
            <category><![CDATA[linux-slave]]></category>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[master-slave-architecture]]></category>
            <category><![CDATA[packers]]></category>
            <category><![CDATA[jenkins]]></category>
            <dc:creator><![CDATA[Velotio Technologies]]></dc:creator>
            <pubDate>Fri, 06 Sep 2019 14:28:12 GMT</pubDate>
            <atom:updated>2020-07-04T14:56:33.643Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VNCIyfpuL7Barf8c3oDHDw.png" /></figure><p>Automation is everywhere and it is better to adopt it as soon as possible. Today in this blog post we are going to discuss creating the infrastructure. For this, we will be using AWS for hosting our deployment pipeline. Packer will be used to create AMI’s and terraform will be used for creating the master/slaves. We will be discussing different ways of connecting the slaves and will also run a sample application with the pipeline.</p><p>Please remember the intent of the blog is to accumulate all the different components together, this means some of the code which should be available in development code repo is also included here. Now that we have highlighted the required tools, 10000 ft view and intent of the blog. Let’s begin.</p><h3>Using Packer to Create AMI’s for Jenkins Master and Linux Slave</h3><p>Hashicorp has bestowed with some of the most amazing tools for simplifying our life. Packer is one of them. Packer can be used to create custom AMI from already available AMI’s. We just need to create a JSON file and pass installation script as part of creation and it will take care of developing the AMI for us. Install packer depending upon your requirement from the <a href="https://www.packer.io/downloads.html">packer downloads</a> page. For simplicity purpose, we will be using Linux machine for creating Jenkins Master and Linux Slave. JSON file for both of them will be same but can be separated if needed.</p><p><strong>Note</strong>: user-data passed from terraform will be different which will eventually differentiate their usage.</p><p>We are using Amazon Linux 2 — JSON file for the same.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c551b773912638c3510b375f367438ee/href">https://medium.com/media/c551b773912638c3510b375f367438ee/href</a></iframe><p>As you can see the file is pretty simple. The only thing of interest here is the install_amazon.bash script. In this blog post, we will deploy a Node-based application which is running inside a docker container. Content of the bash file is as follows:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/eecad308d618ca7a2f04d7402cc9b850/href">https://medium.com/media/eecad308d618ca7a2f04d7402cc9b850/href</a></iframe><p>Now there are a lot of things mentioned let’s check them out. As mentioned earlier we will be discussing different ways of connecting to a slave and for one of them, we need xmlstarlet. Rest of the things are packages that we might need in one way or the other.</p><p>Update ami_users with actual user value. This can be found on AWS console Under <strong><em>Support</em></strong> and inside of it <strong><em>Support Center.</em></strong></p><p>Validate what we have written is right or not by running <strong><em>packer validate amazon.json</em></strong>.</p><p>Once confirmed, build the packer image by running <strong><em>packer build amazon.json.</em></strong></p><p>After completion check your AWS console and you will find a new AMI created in “My AMI’s”.</p><p>It’s now time to start using terraform for creating the machines.</p><p><strong>Prerequisite</strong>:</p><ol><li>Please make sure you create a provider.tf file.</li></ol><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/08e17ab866e9321d01684b84d48019c3/href">https://medium.com/media/08e17ab866e9321d01684b84d48019c3/href</a></iframe><p>The <strong><em>‘credentials file’</em></strong> will contain <strong><em>aws_access_key_id</em></strong> and <strong><em>aws_secret_access_key.</em></strong></p><p>2. Keep SSH keys handy for server/slave machines. Here is a nice<a href="https://www.digitalocean.com/community/tutorials/how-to-set-up-ssh-keys--2"> article</a> highlighting how to create it or else create them beforehand on AWS console and reference it in the code.</p><p>3. VPC:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5ad8ff250e617771ca1d603bd140745b/href">https://medium.com/media/5ad8ff250e617771ca1d603bd140745b/href</a></iframe><h3>Creating Terraform Script for Spinning up Jenkins Master</h3><p>Creating Terraform Script for Spinning up Jenkins Master. Get terraform from<a href="https://www.terraform.io/downloads.html"> terraform download</a> page.</p><p>We will need to set up the Security Group before setting up the instance.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0cf0f01df76d767eaa67984c5265c9ff/href">https://medium.com/media/0cf0f01df76d767eaa67984c5265c9ff/href</a></iframe><p>Now that we have a custom AMI and security groups for ourselves let’s use them to create a terraform instance.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/53169be1c2a1f901af3509118e7ba97c/href">https://medium.com/media/53169be1c2a1f901af3509118e7ba97c/href</a></iframe><p>As mentioned before, we will be discussing multiple ways in which we can connect the slaves to Jenkins master. But it is already known that every time a new Jenkins comes up, it generates a unique password. Now there are two ways to deal with this, one is to wait for Jenkins to spin up and retrieve that password or just directly edit the admin password while creating Jenkins master. Here we will be discussing how to change the password when configuring Jenkins. (If you need the script to retrieve Jenkins password as soon as it gets created than comment and I will share that with you as well).</p><p>Below is the user data to install Jenkins master, configure its password and install required packages.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/31e4e20e3ab1ce1969c56ee39b7f6ac1/href">https://medium.com/media/31e4e20e3ab1ce1969c56ee39b7f6ac1/href</a></iframe><p>There is a lot of stuff that has been covered here. But the most tricky bit is changing Jenkins password. Here we are using a python script which uses <strong><em>brcypt</em></strong> to hash the plain text in Jenkins encryption format and <strong><em>xmlstarlet</em></strong> for replacing that password in the actual location. Also, we are using xmstarlet to edit the JNLP port for windows slave. Do remember initial username for Jenkins is admin.</p><p><strong>Command to run</strong>: Initialize terraform — <strong><em>terraform init , </em></strong>Check and apply — <strong><em>terraform plan</em></strong> -&gt; <strong><em>terraform apply</em></strong></p><p>After successfully running apply command go to AWS console and check for a new instance coming up. Hit the &lt;public IP&gt;:8080 and enter credentials as you had passed and you will have the Jenkins master for yourself ready to be used.</p><p><strong>Note</strong>: I will be providing the terraform script and permission list of IAM roles for the user at the end of the blog.</p><h3>Creating Terraform Script for Spinning up Linux Slave and connect it to master</h3><p>We won’t be creating a new image here rather use the same one that we used for Jenkins master.</p><p>VPC will be same and updated Security groups for slave are below:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8c1d72d274b0b37e475406013168e819/href">https://medium.com/media/8c1d72d274b0b37e475406013168e819/href</a></iframe><p>Now that we have the required security groups in place it is time to bring into light terraform script for Linux slave.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/43f4ec2727048d2475e43503d55ec6e5/href">https://medium.com/media/43f4ec2727048d2475e43503d55ec6e5/href</a></iframe><p>And now the final piece of code, which is user-data of slave machine.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fcf7bc20a09179618940f7b941209cbd/href">https://medium.com/media/fcf7bc20a09179618940f7b941209cbd/href</a></iframe><p>This will not only create a node on Jenkins master but also attach it.</p><p><strong>Command to run</strong>: Initialize Terraform — <strong><em>terraform init</em></strong>, Check and apply — <strong><em>terraform plan</em></strong> -&gt; <strong><em>terraform apply</em></strong></p><p>One drawback of this is, if by any chance slave gets disconnected or goes down, it will remain on Jenkins master as offline, also it will not manually attach itself to Jenkins master.</p><p>Some solutions for them are:</p><p>1. Create a cron job on the slave which will run user-data after a certain interval.</p><p>2. Use swarm plugin.</p><p>3. As we are on AWS, we can even use Amazon EC2 Plugin.</p><p>Maybe in a future blog, we will cover using both of these plugins as well.</p><h3>Using Packer to create AMI’s for Windows Slave</h3><p>Windows AMI will also be created using packer. All the pointers for Windows will remain as it were for Linux.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a580c644dc7f53baa172fae117ce6225/href">https://medium.com/media/a580c644dc7f53baa172fae117ce6225/href</a></iframe><p>Now when it comes to windows one should know that it does not behave the same way Linux does. For us to be able to communicate with this image an essential component required is WinRM. We set it up at the very beginning as part of user_data_file. Also, windows require user input for a lot of things and while automating it is not possible to provide it as it will break the flow of execution so we disable UAC and enable RDP so that we can connect to that machine from our local desktop for debugging if needed. And at last, we will execute install_windows.ps1 file which will set up our slave. Please note at the last we are calling two PowerShell scripts to generate random password every time a new machine is created. It is mandatory to have them or you will never be able to login into your machines.</p><p>There are multiple user-data in the above code, let’s understand them in their order of appearance.</p><p>SetUpWinRM.ps1:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d7277c9ab5fc3ee8642700a3f9644e44/href">https://medium.com/media/d7277c9ab5fc3ee8642700a3f9644e44/href</a></iframe><p>The content is pretty straightforward as it is just setting up WInRM. The only thing that matters here is the &lt;powershell&gt; and &lt;/powershell&gt;. They are mandatory as packer will not be able to understand what is the type of script. Next, we come across disable-uac.ps1 &amp; <strong><em>enable-rdp.ps1</em></strong>, and we have discussed their purpose before. The last user-data is the actual user-data that we need to install all the required packages in the AMI.</p><p>Chocolatey: a blessing in disguise — Installing required applications in windows by scripting is a real headache as you have to write a lot of stuff just to install a single application but luckily for us we have chocolatey. It works as a package manager for windows and helps us install applications as we are installing packages on Linux. <strong><em>install_windows.ps1 </em></strong>has installation step for chocolatey and how it can be used to install other applications on windows.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c8b6e3c764068e4b481a2831880a472e/href">https://medium.com/media/c8b6e3c764068e4b481a2831880a472e/href</a></iframe><p>See, such a small script and you can get all the components to run your Windows application in no time (Kidding… This script actually takes around 20 minutes to run :P)</p><p>Remaining user-data can be found <a href="https://github.com/velotio-tech/Packer-Terraform-Jenkins">here</a>.</p><p>Now that we have the image for ourselves let’s start with terraform script to make this machine a slave of your Jenkins master.</p><h3>Creating Terraform Script for Spinning up Windows Slave and Connect it to Master</h3><p>This time also we will first create the security groups and then create the slave machine from the same AMI that we developed above.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/433fe09c64c0fbf95fb46f9e513cf023/href">https://medium.com/media/433fe09c64c0fbf95fb46f9e513cf023/href</a></iframe><p>Once security groups are in place we move towards creating the terraform file for windows machine itself. Windows can’t connect to Jenkins master using SSH the method we used while connecting the Linux slave instead we have to use JNLP. A quick recap, when creating Jenkins master we used xmlstarlet to modify the JNLP port and also added rules in sg group to allow connection for JNLP. Also, we have opened the port for RDP so that if any issue occurs you can get in the machine and debug it.</p><p><strong>Terraform file</strong>:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ff8e617d3b868b6f03b6a7ef0025992a/href">https://medium.com/media/ff8e617d3b868b6f03b6a7ef0025992a/href</a></iframe><p>Finally, we reach the user-data for the terraform plan. It will download the required jar file, create a node on Jenkins and register itself as a slave.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/99f0bbb0d08a83382ed03260f59be7c0/href">https://medium.com/media/99f0bbb0d08a83382ed03260f59be7c0/href</a></iframe><p><strong>Command to run</strong>: Initialize Terraform — <strong><em>terraform init,</em></strong> Check and apply — <strong><em>terraform plan</em></strong> -&gt; <strong><em>terraform apply</em></strong></p><p>Same drawbacks are applicable here and the same solutions will work here as well.</p><p>Congratulations! You have a Jenkins master with Windows and Linux slave attached to it.</p><h3>IAM roles for reference</h3><h4>Jenkins Master:</h4><p><a href="https://github.com/ismail0352/Packer-Terraform-Jenkins/blob/master/iam/iam_role_jenkins_server.tf">https://github.com/ismail0352/Packer-Terraform-Jenkins/blob/master/iam/iam_role_jenkins_server.tf</a></p><h4>Linux Slave:</h4><p><a href="https://github.com/ismail0352/Packer-Terraform-Jenkins/blob/master/iam/iam_role_jenkins_worker_linux.tf">https://github.com/ismail0352/Packer-Terraform-Jenkins/blob/master/iam/iam_role_jenkins_worker_linux.tf</a></p><h4>Windows Slave:</h4><p><a href="https://github.com/ismail0352/Packer-Terraform-Jenkins/blob/master/iam/iam_role_jenkins_worker_windows.tf">https://github.com/ismail0352/Packer-Terraform-Jenkins/blob/master/iam/iam_role_jenkins_worker_windows.tf</a></p><h4>Bonus:</h4><p>If you want to associate IAM permissions to the user but cannot assign FULL ACCESS here is a curated list below for reference:</p><h4>Packer Policy:</h4><p><a href="https://github.com/ismail0352/Packer-Terraform-Jenkins/blob/master/user_policies/packer-policy.json">https://github.com/ismail0352/Packer-Terraform-Jenkins/blob/master/user_policies/packer-policy.json</a></p><h4>Terraform Policy:</h4><p><a href="https://github.com/ismail0352/Packer-Terraform-Jenkins/blob/master/user_policies/terraform-policy.json">https://github.com/ismail0352/Packer-Terraform-Jenkins/blob/master/user_policies/terraform-policy.json</a></p><h3>Conclusion</h3><p>This blog tries to highlight one of the ways in which we can use packer and terraform to create AMI’s which will serve as Jenkins master and slave. We not only covered their creation but also focused on how to associate security groups and checked some of the basic IAM roles that can be applied. Although we have covered almost all the possible scenarios but still depending on the use case, the required changes would be very less and this can serve as a boilerplate code when beginning to plan your infrastructure on the cloud.</p><p>*****************************************************************</p><p><em>This post was originally published on </em><a href="https://velotio.com/blog/2019/8/5/setup-jenkins-master-slave-architecture"><strong><em>Velotio Blog</em></strong></a><strong><em>.</em></strong></p><p><a href="https://velotio.com/"><strong>Velotio Technologies</strong></a> is an outsourced software product development partner for technology startups and enterprises. We specialize in enterprise B2B and SaaS product development with a focus on artificial intelligence and machine learning, DevOps, and test engineering.</p><p><em>Interested in learning more about us? We would love to connect with you on our</em><a href="https://velotio.com/"><strong><em>Website</em></strong></a><em>, </em><a href="https://www.linkedin.com/company/velotio-technologies/"><strong><em>LinkedIn</em></strong></a><em> or </em><a href="https://twitter.com/velotiotech"><strong><em>Twitter</em></strong></a><em>.</em></p><p>*****************************************************************</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7bcc4b014874" width="1" height="1" alt=""><hr><p><a href="https://medium.com/velotio-perspectives/using-packer-and-terraform-to-setup-jenkins-master-slave-architecture-7bcc4b014874">Using Packer and Terraform to Setup Jenkins Master-Slave Architecture</a> was originally published in <a href="https://medium.com/velotio-perspectives">Velotio Perspectives</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Ultimate Cheat Sheet on Splitting Dynamic Redux Reducers]]></title>
            <link>https://medium.com/velotio-perspectives/the-ultimate-cheat-sheet-on-splitting-dynamic-redux-reducers-322ca17d5350?source=rss----3b93934e4b3d---4</link>
            <guid isPermaLink="false">https://medium.com/p/322ca17d5350</guid>
            <category><![CDATA[modular-design]]></category>
            <category><![CDATA[reducer]]></category>
            <category><![CDATA[react]]></category>
            <category><![CDATA[redux]]></category>
            <category><![CDATA[code-splitting]]></category>
            <dc:creator><![CDATA[Velotio Technologies]]></dc:creator>
            <pubDate>Fri, 30 Aug 2019 10:06:50 GMT</pubDate>
            <atom:updated>2020-06-24T09:21:54.442Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UcxkXEKLjbd0ZAx-9Q7krw.png" /></figure><p>This post is specific to need for code-splitting in React/Redux projects. While exploring the possibility to optimize the application, the common problem occurs with reducers. This article specifically focuses on how do we split reducers to be able to deliver them in chunks.</p><p>What are the benefits of splitting reducers in chunks?</p><p>1) True code splitting is possible</p><p>2) Good architecture can be maintained by keeping page/component level reducers isolated from other parts of application minimizing the dependency on other parts of application.</p><h3>Why Do We Need to Split Reducers?</h3><h4>1. For fast page loads</h4><p>Splitting reducers will have and advantage of loading only required part of web application which in turn makes it very efficient in rendering time of main pages</p><h4>2. Organization of code</h4><p>Splitting reducers on page level or component level will give a better code organization instead of just putting all reducers at one place. Since reducer is loaded only when page/component is loaded will ensure that there are standalone pages which are not dependent on other parts of the application. That ensures seamless development since it will essentially avoid cross-references in reducers and throwing away complexities</p><h4>3. One page/component</h4><p>One reducer design pattern. Things are better written, read and understood when they are modular. With dynamic reducers, it becomes possible to achieve it.</p><h4>4. SEO</h4><p>SEO is a vast topic but it gets hit very hard if your website is having huge response times which happens in case if code is not split. With reducer level code-splitting, reducers can be code split on a split component level which will reduce the loading time of website thereby increasing SEO rankings.</p><h3>What Exists Today?</h3><p>A little googling around the topic shows us some options. Various ways have been discussed at<a href="https://stackoverflow.com/questions/32968016/how-to-dynamically-load-reducers-for-code-splitting-in-a-redux-application/33044701#33044701"> https://stackoverflow.com/questions/32968016/how-to-dynamically-load-reducers-for-code-splitting-in-a-redux-application/33044701#33044701</a> .</p><p>Dan Abramov’s answer is what we are following in this post and we will be writing a simple abstraction to have dynamic reducers but with more functionality.</p><p>A lot of solutions already exists, so why do we need to create our own? The answer is simple and straightforward:</p><p><strong>1) The ease of use</strong></p><p>Every library out there is little catchy is some way. Some have complex api’s while some have too much boilerplate codes. We will be targeting to be near react-redux api.</p><p><strong>2) Limitation to add reducers at top level only</strong></p><p>This is a very common problem that a lot of existing libraries have as of today. That’s what we will be targeting to solve in this post. This opens new doors for possibilities to do code splitting on component level.</p><p>A quick recap of redux facts:</p><p>1) Redux gives us the following methods:<br> — “getState”,<br> — “dispatch(action)”<br> — “subscribe(listener)”<br> — “replaceReducer(nextReducer)”</p><p>2) reducers are plain functions returning next state of the application</p><p>3) “replaceReducer” requires the entire root reducer.</p><p>What we are going to do?</p><p>We will be writing abstraction around “replaceReducer” to develop an api to allow us to inject a reducer at a given key dynamically.</p><p>A simple redux store definition goes like the following:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4b302283191af45333f2f7ea3887ab36/href">https://medium.com/media/4b302283191af45333f2f7ea3887ab36/href</a></iframe><p>Let’s simplify the store creation wrapper as:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d941d309861d9d1320b3489d7594304e/href">https://medium.com/media/d941d309861d9d1320b3489d7594304e/href</a></iframe><h3>What Does it Do?</h3><p>“dynamicActionGenerator” and “isValidReducer” are helper function to determine if given reducer is valid or not.</p><p>For e.g.</p><pre>isValidReducer(() =&gt; { return  ) // should return true</pre><pre>isValidReducer(1) //should return false</pre><pre>isValidReducer(true) //should return false</pre><pre>isValidReducer(“example”) //should return false</pre><p>This is an essential check to ensure all inputs to our abstraction layer over createStore should be valid reducers.</p><p>“createStore” takes initial Root reducer, initial state, and enhancers that will be applicable to created store.</p><p>In addition to that, we are maintaining, “asyncReducers” and “attachReducer” on store object.</p><p>“asyncReducers” keeps the mapping of dynamically added reducers.</p><p>“attachReducer” is partial in above implementation and we will see the complete implementation below. The basic use of “attachReducer” is to add reducer from any part of a web application.</p><p>Given that our store object now becomes like follows:</p><p>Store:</p><pre>- getState: Func</pre><pre>- dispatch(action): Func</pre><pre>- subscribe(listener): Func</pre><pre>- replaceReducer(RootReducer): Func</pre><pre>- attachReducer(reducer): Func</pre><pre>- asyncReducers: JSONObject</pre><p>Now here is an interesting problem, replaceReducer requires a final root reducer function. That means we will have to recreate the root reducers every time.<br>So we will create a dynamicRootReducer function itself to simply the process.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b7a6003eb6fbdd10637885097d9129e8/href">https://medium.com/media/b7a6003eb6fbdd10637885097d9129e8/href</a></iframe><p>So now our store object becomes as follows:<br>Store:</p><pre>- getState: Func</pre><pre>- dispatch(action) : Func</pre><pre>- subscribe(listener) : Func</pre><pre>- replaceReducer(RootReducer) : Func</pre><pre>- attachReducer(reducer) : Func</pre><p>What does dynamicRootReducer do?<br>1) Processes initial root reducer passed to it<br>2) Executes dynamic reducers to get next state.</p><p>So we now have an api exposed as : <br>store.attachReducer(“home”, (state = {}, action) =&gt; { return state }); // Will add a dynamic reducer after the store has been created</p><p>store.attachReducer(“home.grid”, (state={}, action) =&gt; { return state}); // Will add a dynamic reducer at a given nested key in store.</p><p>Final Implementation:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/15e82d24ede3671484d236780149a8df/href">https://medium.com/media/15e82d24ede3671484d236780149a8df/href</a></iframe><p>Working Example: <a href="https://stackblitz.com/edit/react-pvk4xw?file=Hello.js">https://stackblitz.com/edit/react-pvk4xw?file=Hello.js</a></p><p>Further implementations based on simplified code:</p><p>Based on it I have simplified implementations into two libraries:</p><ul><li>Redux-magic-reducer: <a href="https://github.com/saurabhnemade/redux-magic-reducer">https://github.com/saurabhnemade/redux-magic-reducer</a></li><li>A live example can be found at <a href="https://saurabhnemade.github.io/redux-magic-reducer/">https://saurabhnemade.github.io/redux-magic-reducer/</a></li><li>React-redux-magic-reducer: <a href="https://github.com/saurabhnemade/react-redux-magic-reducer">https://github.com/saurabhnemade/react-redux-magic-reducer</a></li><li>Example code can be found at <a href="https://github.com/saurabhnemade/react-redux-magic-reducer/tree/master/example/src">https://github.com/saurabhnemade/react-redux-magic-reducer/tree/master/example/src</a></li><li>A live example can be found at <a href="https://saurabhnemade.github.io/react-redux-magic-reducer/">https://saurabhnemade.github.io/react-redux-magic-reducer/</a></li></ul><h3>Conclusion</h3><p>In this way, we can achieve code splitting with reducers which is a very common problem in almost every react-redux application. With the above solution, you can do code splitting on a page level, component level and can also create reusable stateful components which use redux state. The simplified approach will reduce your application boilerplate. Moreover, common complex components like grid or even the whole pages like login can be exported and imported from one project to another making development faster than ever!</p><p>*****************************************************************</p><p><em>This post was originally published on </em><a href="https://www.velotio.com/engineering-blog/the-ultimate-cheat-sheet-on-splitting-dynamic-redux-reducers"><strong><em>Velotio Blog</em></strong></a><strong><em>.</em></strong></p><p><a href="https://velotio.com/"><strong>Velotio Technologies</strong></a> is an outsourced software product development partner for technology startups and enterprises. We specialize in enterprise B2B and SaaS product development with a focus on artificial intelligence and machine learning, DevOps, and test engineering.</p><p><em>Interested in learning more about us? We would love to connect with you on our</em><a href="https://velotio.com/"><strong><em>Website</em></strong></a><em>, </em><a href="https://www.linkedin.com/company/velotio-technologies/"><strong><em>LinkedIn</em></strong></a><em> or </em><a href="https://twitter.com/velotiotech"><strong><em>Twitter</em></strong></a><em>.</em></p><p>*****************************************************************</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=322ca17d5350" width="1" height="1" alt=""><hr><p><a href="https://medium.com/velotio-perspectives/the-ultimate-cheat-sheet-on-splitting-dynamic-redux-reducers-322ca17d5350">The Ultimate Cheat Sheet on Splitting Dynamic Redux Reducers</a> was originally published in <a href="https://medium.com/velotio-perspectives">Velotio Perspectives</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Step Towards Simplified Querying in NodeJS]]></title>
            <link>https://medium.com/velotio-perspectives/a-step-towards-simplified-querying-in-nodejs-8bfd9bb4097f?source=rss----3b93934e4b3d---4</link>
            <guid isPermaLink="false">https://medium.com/p/8bfd9bb4097f</guid>
            <category><![CDATA[nodejs]]></category>
            <category><![CDATA[sequalize]]></category>
            <category><![CDATA[knexjs]]></category>
            <category><![CDATA[objection-js]]></category>
            <category><![CDATA[simplified-querying]]></category>
            <dc:creator><![CDATA[Velotio Technologies]]></dc:creator>
            <pubDate>Mon, 26 Aug 2019 12:29:07 GMT</pubDate>
            <atom:updated>2020-06-28T09:35:58.697Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UcxkXEKLjbd0ZAx-9Q7krw.png" /></figure><h3>Introduction</h3><p>Recently, I came across a question on StackOverflow regarding querying the data on relationship table using <strong>sequalize</strong> and I went into flashback with the same situation and hence decided to write a blog over a better alternative <strong>Objection.js</strong>. When we choose ORM’s without looking into the use case we are tackling we usually end up with a mess.</p><p>The question on StackOverflow was about converting the below query into <strong>sequalize</strong> query.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/98a5c4ca66d82a842c0db03235390d2e/href">https://medium.com/media/98a5c4ca66d82a842c0db03235390d2e/href</a></iframe><p>(Pardon me for naming in the query, it was asked by novice programmer and I wanted to keep it as it is for purity sake).</p><p>Seems pretty straightforward right? So the solution is like below:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d6e52481cb0460ead10becd90d9c8f5c/href">https://medium.com/media/d6e52481cb0460ead10becd90d9c8f5c/href</a></iframe><p>If you look at this it’s much complex solution for simple querying and this grows with added relationships. And also for simple queries like this, the sequalize documentation is not sufficient. Now if you ask me how it can be done in a better way with Objection.js below is the same query in objection.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cfff59b5ce05c42de8fa59c044e76e81/href">https://medium.com/media/cfff59b5ce05c42de8fa59c044e76e81/href</a></iframe><blockquote>Note: It’s assumed that relationship is defined (in model classes) in both examples.</blockquote><p>Now you guys can see the difference this is just one example I came across there are others on the internet for better understanding. So are you guys ready for diving into Objection.js?</p><p>But before we dive in, I wanted to let you guys know whenever we check online for Node.js ORM, we always find some people saying <strong>“don’t use an ORM, just write plain SQL”</strong> and they are correct in their perception. If your app is small enough that you can write a bunch of query helper functions and carry out all the needed functionality, then don’t go with ORM approach, instead just use plain SQL.</p><p>But when your app has an ample amount of tables and relationships between them that need to be defined and multiple-joint queries need to done, there comes the power of ORM.</p><p>So when we search for the ORM’s (For relational DB) available in NodeJS arena we usually get the list below:</p><p>1. Sequelize (<a href="http://docs.sequelizejs.com/">http://docs.sequelizejs.com/</a>)</p><p>2. Objection.js (<a href="http://vincit.github.io/objection.js/">http://vincit.github.io/objection.js/</a>)</p><p>3. typeORM (<a href="https://typeorm.io/#/">https://typeorm.io/#/</a>)</p><p>There are others, I have just mentioned more popular ones.</p><p>Well, I have personally used both Sequelize and Objection.js as they are the most popular ORM available today. So if you are a person who is deciding on which ORM you should be using for your next project or <strong>got frustrated with the relationship query complexity of `Sequelize`</strong> then you have landed on the correct place.</p><p>I am going to be honest here, I am using Objection.js currently doesn’t make it <em>the facto</em> or best ORM for NodeJS. If you don’t love to write the SQL resembling queries and prefer the fully abstracted query syntax then I think `Sequelize` is the right option for you (though you might struggle with relationship queries as I did and land up with Objection.js later on) but if you want your queries to resemble the SQL one then you should read out this blog.</p><h3>What Makes Objection So Special?</h3><p>1. Objection under the hood uses KNEX.JS (<a href="https://knexjs.org/">https://knexjs.org/</a>) a powerful SQL query builder</p><p>2. Let’s you create models for tables with ES6 / ES7 classes and deﬁne the relationships between them</p><p>3. Make queries with async / await</p><p>4. Add validation to your models using JSON schema</p><p>5. Perform graph inserts and upserts</p><p>to name a few.</p><h3>The Learning Curve</h3><p>I have exclusively relied upon the documentation. The Knex.js and objection.js documentation are great and there are simple (One of them, I am going to use below for explanation) examples on the Objection GitHub. So if you have previously worked with any NodeJS ORM or you are a newbie, this will help you get started without any struggles.</p><p><strong>So let’s get started with some of the important topics while I explain to you the advantages over other ORM and usage along the way</strong>.</p><p>For setup (package installation, configuration, etc.) and full code, you can check out Github — <a href="https://github.com/Vincit/objection.js/tree/master/examples/express-es6">https://github.com/Vincit/objection.js/tree/master/examples/express-es6</a></p><h3>Creating and Managing DB Schema</h3><p>Migration is a good pattern to manage your changes in database schema. Objection.js uses knex.js migration for this purpose.</p><p>So what is <strong>Migration:</strong> Migrations are changes to a database’s schema specified within your ORM, so we will be defining the tables and columns of our database straight in JavaScript rather than using SQL.</p><p>One of the best features of Knex is its robust migration support. To create a new migration simply use the knex cli:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8422fc1948931fb9f48753ace4442801/href">https://medium.com/media/8422fc1948931fb9f48753ace4442801/href</a></iframe><p>After running this command you’ll notice that a new file is created within your migrations directory. This file will include a current timestamp as well as the name that you gave to your migration. The file will look like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5ba1e6a46470c7fa0a69bc8c6f59dbfd/href">https://medium.com/media/5ba1e6a46470c7fa0a69bc8c6f59dbfd/href</a></iframe><p>As you can notice the first is `exports.up`, which specifies the commands that should be run to make the database change that you’d like to make.e.g creating database tables, adding or removing a column from a table, changing indexes, etc.</p><p>The second function within your migration file is `exports.down`. This functions goal is to do the opposite of what exports.up did. If `exports.up` created a table, then `exports.down` will drop that table. The reason to include `exports.down` is so that you can quickly undo a migration should you need to.</p><p>For example:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b9df02957679ddb017d7e362601f3c07/href">https://medium.com/media/b9df02957679ddb017d7e362601f3c07/href</a></iframe><p>It’s that simple to create the migration. Now you can run your migration like below.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/87e22b018512b9ad64553500071525ab/href">https://medium.com/media/87e22b018512b9ad64553500071525ab/href</a></iframe><p>You can also pass the ` — env` flag or set `NODE_ENV` to select an alternative environment:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f57a543a0f13aec257d9a1e3892c7651/href">https://medium.com/media/f57a543a0f13aec257d9a1e3892c7651/href</a></iframe><p>To rollback the last batch of migrations:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/58014a60ded309fe1b6209f1f8b24020/href">https://medium.com/media/58014a60ded309fe1b6209f1f8b24020/href</a></iframe><h3>Models</h3><p>Models are wrappers around the database tables, they help to encapsulate the business logic within those tables.</p><p>Objection.js allows creating a model using ES classes.</p><p>Before diving into the example you guys need to clear your thoughts regarding model little bit as Objection.js Model does not create any table in DB. Yes! the only thing Models are used for are adding the validations and relationship mapping.</p><p>For example:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/53375634035f4027a4f1da6baa7c475e/href">https://medium.com/media/53375634035f4027a4f1da6baa7c475e/href</a></iframe><ul><li>Now let’s break it down, that static getter `tableName` return the table name.</li><li>We also have a second static getter method that defines the validations of each field and this is an optional thing to do. We can specify the required properties, type of the field i.e. number, string, object, etc and other validations as you can see in the example.</li><li>Third static getter function we see is `relationMappings` which defines this models relationship to other models. In this case, the key of the outside object `pets` is how we will refer to the child class. The join property in addition to the relation type defines how the models are related to one another. The from and to properties of the join object define the database columns through which the models are associated. The modelClass passed to the relation mappings is the class of the related model.</li></ul><p>So here `Person` has `HasManyRelation` with `Animal` model class and join is performed on persons `id` column and Animals `ownerId` column. So one person can have multiple pets.</p><h3>Queries</h3><p>Let’s start with simple SELECT queries:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b0326f40eb5fb457cf967401eb828f00/href">https://medium.com/media/b0326f40eb5fb457cf967401eb828f00/href</a></iframe><p>Can be done like:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/905790fe0b1e6974df6f916eeb7dc57c/href">https://medium.com/media/905790fe0b1e6974df6f916eeb7dc57c/href</a></iframe><p>Little advanced or should I say typical select query:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8056c11e6bf4a53ee974aa82bf7db167/href">https://medium.com/media/8056c11e6bf4a53ee974aa82bf7db167/href</a></iframe><p>Can be done like:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f731975b72b7905e6abc95307e5ef94d/href">https://medium.com/media/f731975b72b7905e6abc95307e5ef94d/href</a></iframe><p>So we can look how much objection queries resemble to the actual SQL queries so it’s always easy to transform SQL query easily into Objection.js one which is quite difficult with other ORMs.</p><h3>INSERT Queries:</h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/567d81c96fac6ed8393982abe1c21fc7/href">https://medium.com/media/567d81c96fac6ed8393982abe1c21fc7/href</a></iframe><p>Can be done like:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6cdcea5fc6fc2c40c8b8ba8a9090c04d/href">https://medium.com/media/6cdcea5fc6fc2c40c8b8ba8a9090c04d/href</a></iframe><h3>UPDATE Queries:</h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c3a914b2b88ee1c8fd0f41665b3e07c1/href">https://medium.com/media/c3a914b2b88ee1c8fd0f41665b3e07c1/href</a></iframe><p>Can be done like:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3dfacb101acd3a2c947dc5d0c08dcd9f/href">https://medium.com/media/3dfacb101acd3a2c947dc5d0c08dcd9f/href</a></iframe><h3>DELETE Queries:</h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4654d17a080528fe78d7d047e2c8dd86/href">https://medium.com/media/4654d17a080528fe78d7d047e2c8dd86/href</a></iframe><p>Can be done like:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/35db0d272c4550c35de6b0c7468878d0/href">https://medium.com/media/35db0d272c4550c35de6b0c7468878d0/href</a></iframe><h3>Relationship Queries:</h3><p>Suppose we want to fetch all the pets of Person whose first name is Ben.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7fc8865a72561f2f93bf6aa316ad8afa/href">https://medium.com/media/7fc8865a72561f2f93bf6aa316ad8afa/href</a></iframe><p>Now suppose you want to insert person along with his pets. In this case we can use the graph queries.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/00e93421a596d4198b5c1a187acd05f9/href">https://medium.com/media/00e93421a596d4198b5c1a187acd05f9/href</a></iframe><p>So here we can see the power of Objection queries and if try to compare these queries with other ORM queries you will find out the difference yourself which is better.</p><h3>Plugin Availability</h3><p><strong>objection-password </strong>(<a href="https://www.npmjs.com/package/objection-password">https://www.npmjs.com/package/objection-password</a>): This plugin automatically adds automatic password hashing to your Objection.js models. This makes it super-easy to secure passwords and other sensitive data.</p><p><strong>objection-graphql </strong>(<a href="https://www.npmjs.com/package/objection-graphql">https://www.npmjs.com/package/objection-graphql</a>): Automatic GraphQL API generator for objection.js models.</p><h3>Verdict</h3><p>I am having a fun time working with Objection and Knex currently! If you ask me to choose between <strong>sequalize</strong> and <strong>objection.js</strong> I would definitely go with objection.js to avoid all the relationship queries pain. It’s worth noting that Objection.js is unlike your other ORM’s, it’s just a wrapper over the KNEX.js query builder so its like using query builder with additional features.</p><p>*****************************************************************</p><p><em>This post was originally published on </em><a href="https://www.velotio.com/engineering-blog/simplified-querying-in-nodejs"><strong><em>Velotio Blog</em></strong></a><strong><em>.</em></strong></p><p><a href="https://velotio.com/"><strong>Velotio Technologies</strong></a> is an outsourced software product development partner for technology startups and enterprises. We specialize in enterprise B2B and SaaS product development with a focus on artificial intelligence and machine learning, DevOps, and test engineering.</p><p><em>Interested in learning more about us? We would love to connect with you on our</em><a href="https://velotio.com/"><strong><em>Website</em></strong></a><em>, </em><a href="https://www.linkedin.com/company/velotio-technologies/"><strong><em>LinkedIn</em></strong></a><em> or </em><a href="https://twitter.com/velotiotech"><strong><em>Twitter</em></strong></a><em>.</em></p><p>*****************************************************************</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8bfd9bb4097f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/velotio-perspectives/a-step-towards-simplified-querying-in-nodejs-8bfd9bb4097f">A Step Towards Simplified Querying in NodeJS</a> was originally published in <a href="https://medium.com/velotio-perspectives">Velotio Perspectives</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>