<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Hakan Eröztekin on Medium]]></title>
        <description><![CDATA[Stories by Hakan Eröztekin on Medium]]></description>
        <link>https://medium.com/@hakaneroztekin?source=rss-222c185243f1------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*TXNvE2J_QRykl0SS_aCEpg.png</url>
            <title>Stories by Hakan Eröztekin on Medium</title>
            <link>https://medium.com/@hakaneroztekin?source=rss-222c185243f1------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 13:04:52 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@hakaneroztekin/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[kubernetes in a nutshell]]></title>
            <link>https://hakaneroztekin.medium.com/kubernetes-in-a-nutshell-f2916a138f59?source=rss-222c185243f1------2</link>
            <guid isPermaLink="false">https://medium.com/p/f2916a138f59</guid>
            <category><![CDATA[kubectl]]></category>
            <category><![CDATA[kubernetes-cluster]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Hakan Eröztekin]]></dc:creator>
            <pubDate>Sun, 31 Jul 2022 11:52:35 GMT</pubDate>
            <atom:updated>2022-07-31T11:52:35.248Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*koN_QjtJPosdhMOEuu5Yuw.jpeg" /></figure><p>Kubernetes is one of the core components of modern software architecture. Each of our services has multiple instances which can scale up and down regardless of the time of the day. In a standard production environment, there can be hundreds to billions of those instances. Kubernetes comes to the rescue to manage them efficiently.</p><p>In this article, we are going to take a look at almost every aspect of Kubernetes. You will find it beneficial whether you’re just starting with Kubernetes or an experienced senior refreshing knowledge.</p><p>There are demo sections in the article. It is highly recommended to do them to get hands-on practice. You can find the demo code on this <a href="https://github.com/hakaneroztekin/kubernetes-manifest-samples">repository</a>.</p><p>Let’s start.</p><h3>table of contents</h3><pre>· <a href="#e894">what is kubernetes</a><br>· <a href="#b896">birds-eye-view of kubernetes</a><br>  ∘ <a href="#5fa8">kubernetes architecture</a><br>  ∘ <a href="#8f77">the control plane</a><br>  ∘ <a href="#2613">the worker node</a><br>  ∘ <a href="#fd57">kubernetes objects</a><br>  ∘ <a href="#51fa">pods</a><br>  ∘ <a href="#582f">deployments</a><br>  ∘ <a href="#2d83">services</a><br>  ∘ <a href="#7421">ingress</a><br>· <a href="#f2bc">installing kubernetes</a><br>· <a href="#1045">pods</a><br>  ∘ <a href="#b011">demo time: deploying a pod</a><br>· <a href="#a484">namespaces</a><br>· <a href="#e9eb">deployments</a><br>  ∘ <a href="#d3f6">rolling updates</a><br>  ∘ <a href="#dc43">rollbacks</a><br>  ∘ <a href="#2acd">demo time: scaling, rollouts, and rollbacks with deployments</a><br>· <a href="#ea4c">services</a><br>  ∘ <a href="#4251">endpoints</a><br>  ∘ <a href="#b098">service types</a><br>  ∘ <a href="#c285">demo time: make an app accessible from the browser with services</a><br>· <a href="#3bd9">ingress</a><br>  ∘ <a href="#576c">demo time: host-based and path-based routing with ingress</a><br>· <a href="#5db2">configmaps and secrets</a><br>  ∘ <a href="#462a">configmaps</a><br>  ∘ <a href="#1a20">secrets</a><br>  ∘ <a href="#ad7e">demo time: configmaps and secrets</a><br>· <a href="#c869">statefulsets</a><br>· <a href="#160d">conclusion</a></pre><h3>what is kubernetes</h3><p>Kubernetes is an application orchestrator. It deploys and manages applications.</p><p>It is mostly used for cloud-native microservice applications. Cloud-native is a term that means it supports features clouds need, such as scaling, rollout, and rollbacks.</p><p>Microservices is a pattern where you divide your application into small and independent logical parts, such as user service, invoice service, and product service.</p><p>So Kubernetes helps us to manage our microservice applications and provides automated and powerful features such as scaling, rollout, and rollbacks.</p><p>Other interesting bits about Kubernetes,</p><ul><li>It was developed by Google and donated to Cloud Native Computing Foundation (CNCF).</li><li>It’s an open-source project written in Go.</li></ul><pre>» go back to <a href="#3d61">table of contents</a></pre><h3>birds-eye-view of kubernetes</h3><p>We will take a high-level look at Kubernetes; the architecture and objects.</p><h4>kubernetes architecture</h4><p>A Kubernetes cluster is a set of nodes. There are two types of nodes; control plane nodes and worker nodes. A node can be a physical server, virtual machine, or cloud instance.</p><ul><li>control-plane nodes are the brain of the cluster; they manage workload.</li><li>worker nodes are for executing the workload given by the control plane.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-XmwRluJsBiAChK6RmZYqQ.png" /><figcaption>the kubernetes architecture (<a href="https://gist.github.com/hakaneroztekin/2fb5e5d0220eb3ac547e70875ff1f7c4#file-kube-architecture-sh">source</a>)</figcaption></figure><p>The architecture looks a bit complex but looking at the process of running an application in Kubernetes makes it easier to understand.</p><p>At a high level, we specify our app requirements such as which image to run, which port to expose, and how many replicas (application instances) we need. We write these down in a YAML file, which is called a manifest file. We then give it to Kubernetes and it handles the rest of the work.</p><p>Here are the steps of the process,</p><ol><li>we define our requirements in a manifest (YAML) file.</li><li>execute a terminal command to send our manifest file to kubernetes.</li><li>kubernetes checks our requirements and finds an appropriate worker node.</li><li>the worker node runs the application as specified in our file.</li><li>a periodical check is done to ensure the running application features match what we specify in our file.</li></ol><p>Before going any further, if you’re new to Kubernetes and these steps look quite abstract, don’t stress yourself out. We have hands-on demo sections which will make it easier to understand for you. Just try to grasp the idea this time, then you can come back to this section after doing the practices.</p><p>While keeping the steps in mind, let’s take a look at what happens at a low level.</p><h4><strong>the control plane</strong></h4><ul><li>the API server is the center of all communication between all components. When you execute a command in the terminal to deploy our pod, it sends a POST request to the API server.</li><li>cluster store is a stateful part of the cluster. It stores the configuration and state of the cluster. For example, it keeps the information that we want to have 5 replicas.</li><li>controllers are smart objects of Kubernetes that provide cloud-native features. You will usually deploy your application using a controller.</li><li>scheduler listens to the API server for new work and finds an appropriate node for the work. If it can’t find a healthy node, the work remains in a pending state.</li></ul><p>So, when we execute a terminal command to deploy our application, behind the scenes, we POST the file to the API server. The cluster store persists our requirement. Scheduler finds a healthy node to execute the work.</p><h4>the worker node</h4><p>Our applications run in worker nodes. They listen for the new work, execute it and report it back.</p><ul><li>kubelet listens to the API server for new work, executes it and reports status back to the control plane.</li><li>container runtime does container-related tasks such as pulling images and running the container. Kubernetes removed Docker, however, it just got rid of unnecessary parts of Docker, it’s still using the core component, containerd.</li><li>kube-proxy is used for networking inside a cluster. It handles IP assignments, routing, and load-balancing on the pod network.</li></ul><p>So, when the worker node gets new work, it executes it and reports it back. The controller periodically checks what we want (defined in our file, called desired state) matches what we have (actual state). If they don’t match the control plane controllers take the necessary action to match them.</p><h4>kubernetes objects</h4><p>These are default Kubernetes objects. We will take a deep dive into each of them throughout the article.</p><h4>pods</h4><ul><li>pods are the smallest unit of Kubernetes.</li><li>usually, a single pod runs a single container. There are scenarios where you run multiple containers in a pod.</li><li>pods are not responsible for running applications, they just provide an environment.</li><li>pods are immutable.</li></ul><h4>deployments</h4><ul><li>you can deploy pods by themselves but controllers provide pods with rich features, such as scaling, and self-healing.</li><li>some of the controllers are Deployments, DaemonSets, and StatefulSets.</li></ul><h4><strong>services</strong></h4><ul><li>pods come and go so accessing them directly with IP is not a stable approach. services provide stable DNS.</li><li>services monitor healthy pods and route the requests to them in a load-balancing fashion.</li></ul><h4>ingress</h4><ul><li>ingress allows us to redirect requests in a more intelligent way. we can route requests based on certain conditions.</li></ul><pre>» go back to <a href="#3d61">table of contents</a></pre><h3>installing kubernetes</h3><p>If you’re using Mac or Windows you can get a good Kubernetes environment with <a href="https://www.docker.com/products/docker-desktop/">Docker Desktop</a>. You can also use cloud environments. Since the installation details can change often, we won’t discuss them here.</p><pre>» go back to <a href="#3d61">table of contents</a></pre><h3>pods</h3><p>Pods are the smallest unit of Kubernetes. They wrap the containers and provide additional features. We can add labels to group pods, add health probes to find out healthy pods, and provide resource limits.</p><p>Pods have hundreds of fields that you don’t need to memorize. With the following commands, you can access the details of each feature.</p><pre>kubectl explain pods</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XD6nlM8v2K64KvpSW_zTMA.png" /></figure><p>You can check the details of a specific field with this command,</p><pre>kubectl explain pods.spec.volumes</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vK_RJVTRowln1jjVWoZR5w.png" /></figure><p>Let’s take a look at a simple pod.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J-er6JtD2b4sbLfRs9-tvQ.png" /><figcaption>basic pod manifest (<a href="https://gist.github.com/hakaneroztekin/685857bf3acb6306d896e0e28ef8c628">source code</a>)</figcaption></figure><ul><li>apiVersion defines the schema version.</li><li>kind specifies the object type.</li><li>metadata specifies the pod in a unique way which helps to group pods by these labels.</li><li>spec is for defining containers.</li></ul><pre>» go back to <a href="#3d61">table of contents</a><br>» skip the demo, jump to <a href="#a484">namespaces</a></pre><h4>demo time: deploying a pod</h4><p>In this demo, we’ll deploy a simple pod, check logs and then delete it.</p><p>Clone the clone to your computer if you haven’t before,</p><pre>git clone <a href="https://github.com/hakaneroztekin/kubernetes-manifest-samples.git">https://github.com/hakaneroztekin/kubernetes-manifest-samples.git</a></pre><p>Check the running pods,</p><pre>kubectl get pods </pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1022/1*Gzf26Zril4ZNgWn0lcOIPA.png" /></figure><p>We’re going to deploy the pod manifest we’ve seen before,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CVTWxKPJBVfExSrhBDQGmg.png" /><figcaption>pod manifest (<a href="https://gist.github.com/hakaneroztekin/685857bf3acb6306d896e0e28ef8c628">source code</a>)</figcaption></figure><p>This app is a basic Spring app with a single controller. It returns a message when we hit localhost:8080.</p><p>Deploy the pod declaratively,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/770/1*hJS2Bi05r6FWyhA-sfT2ig.png" /></figure><p>Check the pod,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZpDimcZDg4gEXWFJxarNFA.png" /></figure><p>Check the logs,</p><pre>kubectl logs basic-spring-app</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xzAhIHmFrkkLKPgUAeyYnA.png" /></figure><p>Note that, we can’t access our app in our browser atlocalhost:8080. We’re going to make it accessible in the demo section of services.</p><p>Voila. We’ve successfully deployed a pod.</p><p>Delete the pod,</p><pre>kubectl delete pods basic-spring-app</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-kxkF7zXbzy2iX3rxFUatQ.png" /></figure><p>Congratulations. We’ve specified a simple pod, and deployed it on a cluster. Now, what if the node running our pod fails? Our pod is lost. To prevent this we need self-healing. Deployment controllers provide this and other nice features. We’ll discuss them in a moment.</p><pre>» go back to <a href="#3d61">table of contents</a></pre><h3>namespaces</h3><p>Namespaces allow us to virtually split our cluster into sub-groups. We can specify different resource limits and users for each namespace. It is important to note that they don’t provide strong isolation.</p><pre>» go back to <a href="#3d61">table of contents</a></pre><h3>deployments</h3><p>Deployments give superpowers to our applications such as self-healing, scaling, and rollbacks.</p><p>There are multiple types of controllers. We will discuss deployment and ReplicaSet in this chapter, and StatefulSet in a later chapter.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*N59ArQuukkIvkrxf1eGppQ.png" /></figure><p>The deployment manifest file consists of multiple subgroups. As you may have noticed,</p><ul><li>containers part (the last part) is similar to a Dockerfile.</li><li>template part is similar to a Pod manifest file.</li></ul><p>Take a look at the following image,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AnwONrePiHSb7kusyoU58w.png" /></figure><p>The container part is related to a container. Pod wraps container. ReplicaSet, wrapping pod, adds scaling, and self-healing features. Deployment wraps ReplicaSet and adds rollout and rollback strategy.</p><h4>rolling updates</h4><p>Rolling update (rollout) means a new deployment. We update our deployment manifest file, post it to Kubernetes and it handles the rest. Depending on our rollout strategy, it creates new pods and kills old ones.</p><p>We can manage our strategy by adjusting these fields,</p><ul><li>maxUnavailable defines the limit of how many pods can be unavailable at a time. If our replica count is 5 and maxUnavailable is 2, at least 3 replicas can be running at any time.</li><li>maxSurge defines the limit of how many additional pods can be running at a time. If our replica count is 5 and maxSurge is 1, at most 6 pods can be running at any time.</li></ul><p>We can increase these numbers for faster but less-safe deployment, and keep them low for slower but safer deployment.</p><h4>rollbacks</h4><p>Rollback is reveting deployment to an older version. Imagine there’s a bug with the new deployment, in this case, you’d want to get back to the older version.</p><p>Kubernetes creates a new replicaset with each deployment. When you use a newer version to deploy, a second replicaset is created. The old one will still remain in the system. Thus, when we need to rollback, Kubernetes just uses the old replicaset.</p><p>We will see these features in practice in the demo section.</p><pre>» go back to <a href="#3d61">table of contents</a><br>» skip the demo, jump to <a href="#ea4c">services</a></pre><h4>demo time: scaling, rollouts, and rollbacks with deployments</h4><p>In this demo, we’re going to deploy our pod with deployment controller, scale it up and do a rollback.</p><p>Clone the clone to your computer if you haven’t before,</p><pre>git clone <a href="https://github.com/hakaneroztekin/kubernetes-manifest-samples.git">https://github.com/hakaneroztekin/kubernetes-manifest-samples.git</a></pre><p>Here’s our deployment manifest. It’s the same we’ve seen in the deployments chapter.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*N59ArQuukkIvkrxf1eGppQ.png" /></figure><p>Check active deployments,</p><pre>kubectl get deployments</pre><p>Deploy our app,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Vz860O9QIzl5kzBUlqqHRg.png" /></figure><p>Check the pods. Add --watch flag to automatically refresh the response when pod statuses change.</p><pre>kubectl get pods --watch</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nfmrdjvOGAMbiqH-eGH8cQ.png" /></figure><p>Update the spec.replicasin the manifest to 8 and apply the manifest again.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4dlvgj6FLBxIjhPNBveKCg.png" /></figure><p>Get the deployment status,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6WRAVP2_-l2HkG_JkkWUAQ.png" /></figure><p>We’ve successfully scaled up our application. Now, we’re going to practice rollbacks.</p><p>Let’s update our application version. Update the image tag to 2.0. The spec section of our deployment file should look like this,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-5vyXYj5eH43uU49QebbIg.png" /></figure><p>You can <a href="https://gist.github.com/hakaneroztekin/a67ab0e2849dd24e7b5bd7cc51ec3821">click here</a> to see the full manifest file. Now, deploy the manifest again. We’ll use the record=true flag to save this command in the change list. We’ll see it in action in a moment.</p><pre>kubectl apply -f deployment.yml --record=true</pre><p>Check the ReplicaSets,</p><pre>kubectl get replicasets</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FsFzolU3QdVLvlNZCzKm2A.png" /></figure><p>As you can see, there are 2 replicasets, the latter is the newer and active. Even though the old one is not used by any deployment, it still remains in the system. Now, we will do a rollback to that old replicasets. In other words, rollback making deployment with an older replicaset.</p><p>Get the rollout history,</p><pre>kubectl rollout history deployment basic-spring-app-deployment</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zXdDhP9K2LHsOvhKtxCHrA.png" /></figure><p>Now, we can see the effect of the record flag. It saves our command in the change-cause.</p><p>Let’s rollback to the first revision.</p><pre>kubectl rollout undo deployment basic-spring-app-deployment --to-revision=1</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iUUFs-OgDAvKFWaCWX3cxw.png" /></figure><p>Let’s list the replicasets again</p><pre>kubectl get replicasets</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Kb1nX3__7AOUuHQk6MGLOQ.png" /></figure><p>We’ve successfully done a rollback!</p><p>There’s one question left. Our app is exporting the port 8080 though we can’t access it when we hit localhost:8080. This is exactly what we’re going to do in the demo of services section.</p><pre>» go back to <a href="#3d61">table of contents</a></pre><h3>services</h3><p>Services provide stable networking. As you’ve seen in the deployments chapter, we can scale up and add new pods or rollout to kill old pods while creating new ones. Since each pod has its own IP, there’s no guarantee that we can access the app using the same IP. In addition, the pod network works inside a cluster so we can’t access it from outside, for e.g. from our browser.</p><p>Services have a fixed IP, DNS, and port. They watch the current status of pods and redirect the traffic to healthy ones. Services find related pods using label selectors. They also provide basic load-balancing out of the box.</p><h4>endpoints</h4><p>An endpoint is a Kubernetes object. When we create a service, Kubernetes finds healthy pods matching the service label selector. Then those pods are added to the endpoint list. This list is periodically updated.</p><p>When traffic hits the service, the service uses the endpoint list to route traffic.</p><p>The newer versions of Kubernetes have the EndpointSlices object which is a more capable version of the Endpoints object.</p><h4>service types</h4><ul><li>ClusterIP provides a network that is only accessible inside a cluster. Each pod can resolve a service by its DNS name using the cluster’s DNS service and access it.</li><li>NodePort provides a network that is accessible outside the cluster. NodePort services are built on top of ClusterIP services by exposing a port outside the cluster. This port is called NodePort.</li><li>LoadBalancer integrates with 3rd party cloud providers to create a service that’s accessible on the internet.</li></ul><p>Let’s create a NodePort service.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/806/1*Cw_h3pYsYtAyPlBa4AeVGA.png" /></figure><p>This service matches the pods with the label env:development. This is just for demo purposes, you can consider adding an app name in your labels for your clusters too.</p><p>Pods in the cluster can access this service using port 8080. Outside access is made available at the port 30056. Both are mapped to the port 8080 of the container.</p><pre>» go back to <a href="#3d61">table of contents</a><br>» skip the demo, jump to <a href="#3bd9">ingress</a></pre><h4>demo time: make an app accessible from the browser with services</h4><p>Let’s deploy a service.</p><p>Clone the clone to your computer if you haven’t before,</p><pre>git clone <a href="https://github.com/hakaneroztekin/kubernetes-manifest-samples.git">https://github.com/hakaneroztekin/kubernetes-manifest-samples.git</a></pre><p>Here’s our manifest,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/806/1*Cw_h3pYsYtAyPlBa4AeVGA.png" /></figure><p>Post it to Kubernetes,</p><pre>kubectl apply -f service.yml</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/950/1*qTAfETWPLNMCnazJoAeN3A.png" /></figure><p>List the services,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*x7zrdueHza3-phylu6GxxA.png" /></figure><p>Now, hit localhost:30056in your browser.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/484/1*T0MEAjJpGY7hlvDb8atyuQ.png" /></figure><p>Congratz. We’ve made our accessible from the outside world :)</p><p>Let’s list the endpoints,</p><pre>kubectl get endpoints</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wYt7vFPXJ5PVyOPDcWYEmg.png" /></figure><p>Each of them is a healthy pod. Once we make a request to access, the service checks for the endpoints and picks one pod to route our request in a load-balancing fashion.</p><pre>» go back to <a href="#3d61">table of contents</a></pre><h3>ingress</h3><p>Ingress is a more capable version of a service. We need a service for each of our applications when we use a LoadBalancer service. For example, if we have 8 apps, then we need 8 services. This is not cost-efficient. With Ingress, we expose multiple services with a single LoadBalancer service.</p><p>So how does Ingress know which service to route traffic to? How does it know we want to access user service, not invoice service? Ingress uses host-based and path-based routing for that.</p><p>Ingress and service meshes have similarities. Service meshes are an alternative to service-to-service communication. We won’t dive deep into service meshes but if you have them you may not need an Ingress.</p><pre>» go back to <a href="#3d61">table of contents</a><br>» skip the demo, jump to <a href="#5db2">configmaps and secrets</a></pre><h4>demo time: host-based and path-based routing with ingress</h4><p>In this demo, we will use a host-based ingress which will route,</p><ul><li>order.localhost to order-service</li><li>user.localhost to user-service</li></ul><p>Then we will do a path-based routing which will route,</p><ul><li>localhost/order to order-service</li><li>localhost/user to user-service</li></ul><p>Clone the clone to your computer if you haven’t before,</p><pre>git clone <a href="https://github.com/hakaneroztekin/kubernetes-manifest-samples.git">https://github.com/hakaneroztekin/kubernetes-manifest-samples.git</a></pre><p>Go to the ingress folder.</p><p>We will deploy two pods and two services,</p><ul><li>user-app-pod</li><li>order-app-pod</li><li>user-service</li><li>order-service</li></ul><p>Services are ClusterIP services that map ports 8080 to 8080. They are not accessible outside since they’re not NodePort services. Ingress will be exposed.</p><p>Here’s our host-based ingress manifest,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*H7h_zGpc4kycXjAOLkSEtQ.png" /></figure><p>We are defining two rules. Ingress redirects request to order.localhost at / path to order-service. It does similar work for the user service.</p><p>Deploy the ingress,</p><pre>kubectl apply -f ingress-host-based.yml</pre><p>Check it,</p><pre>kubectl get ingresses</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fOY9UfuDseo69rv68ddmGA.png" /></figure><p>Hit user.localhost on your browser,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/468/1*wT6e_y6TtNRdMXZWZsEvcw.png" /></figure><p>Great. What about order.localhost?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/471/1*Z5wyc5Dr3pHaGLZURaHzew.png" /></figure><p>That easy. Now, let’s do the path-based routing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JbvdRb6vp87A1_R0zYQrAg.png" /></figure><p>We’ve updated our rules with path-based routing. It redirects to related services by the path.</p><p>Deploy it,</p><pre>kubectl apply -f ingress-path-based.yml</pre><p>Hit your browser at localhost/user</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/467/1*nj3kWa9AGcc19vszut18Ug.png" /></figure><p>Great. Now for localhost/order</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/464/1*drTbnEgn3MXjfFZZI-Mrdw.png" /></figure><p>Delete ingresses,</p><pre>kubectl delete ingress ingress-host-based</pre><pre>kubectl delete ingress ingress-path-based</pre><p>That’s it. We’ve successfully applied Ingress intelligence. As you’ve seen, it is similar to services but it has the capability of conditional routing.</p><pre>» go back to <a href="#3d61">table of contents</a></pre><h3>configmaps and secrets</h3><h4>configmaps</h4><p>ConfigMaps are useful to decouple application and configuration. We can have a generic image for each of our environments (e.g. dev, stage, and prod) and customize it with configmaps.</p><p>For example, we can inject the configuration of a database, environment variables, and so on.</p><p>Let’s take a look at a simple configmap.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/932/1*2Mwv6HWW2VApyBeKF8extg.png" /></figure><p>The manifest defines a ConfigMap with two entries.</p><p>There are three ways to inject a configmap into our pods;</p><ul><li>As environment variables</li><li>As startup commands</li><li>As files in volumes</li></ul><p>The ConfigMaps with the first two options will be static. Once the app starts, you can’t change them afterward. The most flexible option is injecting a ConfigMap with a volume.</p><p>We’re going to use it in the demo section. We will see that updates to the ConfigMap will be reflected in the pod in a minute or two.</p><h4>secrets</h4><p>Secrets are the same as ConfigMaps but they are for credentials such as usernames, passwords, and tokens.</p><p>Secrets in Kubernetes are not secure by nature. The applications have to see the decoded version of the Secrets, so they are not encoded in the network. Further, they are only encoded with the basic base64 encoder. So it is very easy to decode them. To secure Secrets, it is recommended to use EncryptionConfiguration or 3rd party tools such as Hashicorp’s Vault.</p><p>Let’s take a look at a basic secret,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/662/1*QWa5jZwLVtJS-hBxnuBv6g.png" /></figure><p>The secret manifest defines two fields. As you can see the structure is the same as of ConfigMap.</p><p>Similar to ConfigMaps, we can inject secrets to pods with volume mapping. We’ll do this in the demo section.</p><pre>» go back to <a href="#3d61">table of contents</a><br>» skip the demo, jump to <a href="#c869">statefulsets</a></pre><h4>demo time: configmaps and secrets</h4><p>In the first part of the demo, we’ll deploy a ConfigMap in our cluster, inject it into a pod, and update it without killing the pod. In the second part, we will deploy a Secret, inject it into a pod, and also decode it easily.</p><p>Clone the clone to your computer if you haven’t before,</p><pre>git clone <a href="https://github.com/hakaneroztekin/kubernetes-manifest-samples.git">https://github.com/hakaneroztekin/kubernetes-manifest-samples.git</a></pre><p>This is the ConfigMap we’ll deploy in our cluster,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/932/1*2Mwv6HWW2VApyBeKF8extg.png" /></figure><p>Deploy the configmap,</p><pre>kubectl apply -f basic-configmap.yml</pre><p>List the configmaps,</p><pre>kubectl get configmaps</pre><p>Describe our configmap,</p><pre>kubectl describe configmap my-configmap</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*POZSgrtTxivNXJmYQQIb3g.png" /></figure><p>The result doesn’t look very tidy but we can see our two entries.</p><p>Here’s our pod manifest. Notice the volume mapping.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/842/1*2TZnxqIUvc6fgZiLybFozw.png" /></figure><p>Deploy the pod,</p><pre>kubectl apply -f configmap/pod-configmap.yml</pre><p>Let’s check if our pod can access the ConfigMap data.</p><pre>kubectl exec my-app -- ls /etc/name</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/968/1*TE5N8AM5QxthSc3tuvcG2Q.png" /></figure><p>Great. What about the data inside them?</p><pre>kubectl exec my-app — cat /etc/name/env<br>kubectl exec my-app — cat /etc/name/welcomeMessage</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9YzmuPAqB9PF1GW86Pb4gg.png" /></figure><p>Great. Let’s update our ConfigMap manifest, and deploy it.</p><pre>kubectl apply -f configmap/basic-configmap.yml</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OPRy9MT0KMib9iBmEY_v6A.png" /></figure><p>Run the same commands as before to see the contents of our ConfiMap data. The changes will be reflected in a pod in a minute or two.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TaeReZs34vVI5X4l3xQMzg.png" /></figure><p>Awesome. Without changing the pod at all, we dynamically changed the configmap. That’s it for ConfigMaps. Let’s proceed with Secrets.</p><p>Here is our Secret,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/662/1*QWa5jZwLVtJS-hBxnuBv6g.png" /></figure><p>Let’s deploy the Secret,</p><pre>kubectl apply -f basic-secret.yml</pre><p>List the secrets,</p><pre>kubectl get secrets</pre><p>Describe our secret,</p><pre>kubectl describe secret my-secret</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/932/1*1bcJL1HJ4dDcgvL179GZQg.png" /></figure><p>We can see our two keys but their details are hidden. Well, so far.</p><p>Let’s deploy the following pod manifest.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1004/1*iGOh4ye_9cjk4kGgDwlc2g.png" /></figure><pre>kubectl apply -f configmap/pod-secret.yml</pre><p>List the secret fields,</p><pre>kubectl exec basic-app -- ls /etc/my-secret-vol</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vPspt7n_Ilr_Yx3OEY8iVw.png" /></figure><p>What’s inside them?</p><pre>kubectl exec basic-app — cat /etc/my-secret-vol/username<br>kubectl exec basic-app — cat /etc/my-secret-vol/password</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jmPPKHZ4hu5qTqV48AE7CQ.png" /></figure><p>As you can see, Secrets are not really secure. Therefore, it is advised to use EncryptionConfiguration or 3rd party tools such as Hashicorp’s Vault.</p><p>Great, in this demo we practiced ConfigMaps and Secrets.</p><pre>» go back to <a href="#3d61">table of contents</a></pre><h3>statefulsets</h3><p>StatefulSets are for stateful applications. Examples of stateful applications are databases or applications that keep user sessions to be used for future sessions.</p><p>Let’s take a look at the statefulset architecture,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HuH-GVf63zqkuw8B0s1n5A.png" /><figcaption>Statefulset Architecture</figcaption></figure><p>The first pod has access to read/write replica of mysql; the others only have access to the read replica.</p><p>Pods of Statefulsets are predictable. Statefulsets guarantee mapping of Pod names, DNS hostnames and volume mappings. This means, the first created pod always has the name my-pod-1 and access to mysql r/w replica. The next one will have the name my-pod-2 and access to mysql read replica, and so on. Any consumer service can directly access to my-pod-1 and it is guaranteed that it will have write privilege.</p><pre>» go back to <a href="#3d61">table of contents</a></pre><h3>conclusion</h3><p>Kubernetes is one of the core tools of modern architecture. It has various terminologies like pods, replicasets, and rollbacks. It manages our applications with respect to the manifest files.</p><p>I would like to thank Nigel Poulton for his amazing book, <a href="https://www.amazon.com/Kubernetes-Book-Version-November-2018-ebook/dp/B072TS9ZQZ">The Kubernetes Book</a>. Some of the information and examples in this article are adapted from the book. It is highly recommended if you want to learn more about almost all aspects of Kubernetes.</p><p>If you would like to follow me, check out my <a href="https://medium.com/me/stories/public">Medium profile</a>, and <a href="https://www.linkedin.com/in/hakaneroztekin/">LinkedIn</a>, I share posts when I publish new articles. Let me know whether you liked the article and what I can change to make it better. Thanks for reading.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f2916a138f59" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[docker in a nutshell]]></title>
            <link>https://hakaneroztekin.medium.com/docker-in-a-nutshell-29cc2a938534?source=rss-222c185243f1------2</link>
            <guid isPermaLink="false">https://medium.com/p/29cc2a938534</guid>
            <category><![CDATA[docker-compose]]></category>
            <category><![CDATA[dockerfiles]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[docker-image]]></category>
            <category><![CDATA[docker]]></category>
            <dc:creator><![CDATA[Hakan Eröztekin]]></dc:creator>
            <pubDate>Sun, 19 Jun 2022 10:23:11 GMT</pubDate>
            <atom:updated>2022-07-31T11:55:37.252Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*H3kmDaKwxFingeTcuiYbvQ.jpeg" /><figcaption>original photo by <a href="https://unsplash.com/photos/MvNfMRw3I5k">Juan Pablo Daniel</a></figcaption></figure><p>Welcome to the one-stop shop for (almost) all aspects of Docker.</p><p>In this article, we’re going to look at the features of Docker in sufficient detail. We will cover a lot of topics that are listed right after this section. It will be useful whether you’re refreshing your knowledge or just starting to learn Docker.</p><p>Throughout the article, we will build a project. In the end, we will have built a project that uses docker compose and has multiple containers — one with our custom image, a custom network, and a volume. Don’t worry if these terms sound odd, they are, though they are simpler than they look and we’ll cover them one by one.</p><p>You can quickly copy the commands in the images by clicking the source code text below them. The complete project code is available on <a href="https://github.com/hakaneroztekin/java-spring-with-docker">Github</a>. You can check similar articles I’ve written before on my <a href="https://medium.com/me/stories/public">Medium profile</a>. You can also contact me on <a href="https://www.linkedin.com/in/hakaneroztekin/">LinkedIn</a>.</p><p>Feel free to drop a comment and let me know whether you liked it or not, your questions, and if there’s an incorrect/missing part in the article.</p><p>Alright, let’s start!</p><h4>table of contents</h4><pre>∘ <a href="#67c6">why docker</a><br>∘ <a href="#4d2c">docker terms</a><br>∘ <a href="#02c3">docker images</a><br>∘ <a href="#7e47">docker containers</a><br>  ∘ <a href="#2fa3">demo time: pull and inspect a docker image then run it</a><br>∘ <a href="#8c30">containerizing an app</a><br>∘ <a href="#af54">dockerfile</a><br>  ∘ <a href="#93b1">demo time: nginx Dockerfile with custom homepage</a><br>  ∘ <a href="#738a">demo time: run a spring app as a container</a><br>∘ <a href="#f39f">docker compose</a><br>∘ <a href="#e2db">docker networking</a><br>  ∘ <a href="#e5f6">demo time: docker networking</a><br>∘ <a href="#77b5">docker volumes</a><br>  ∘ <a href="#6792">demo time: docker volumes</a><br>∘ <a href="#b1bf">credits</a><br>∘ <a href="#2fd2">what’s next</a></pre><p>Two notes before we proceed,</p><ul><li>Feel free to click the titles here, I provide links to navigate at the end of each section to eliminate the hassle of scrolling back and forth to find the related section.</li><li>We’re not going to cover docker stacks and security right now. Docker stacks help us to deploy our applications in multiple hosts (clusters). We can achieve the same thing with Kubernetes which we’ll discuss in the upcoming article. However, we can also talk about them if you want to — just let me know.</li></ul><h4>why docker</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-3XQo2yZ7FMrVg3QbJXElg.png" /><figcaption>virtual machines vs containers (<a href="https://gist.github.com/hakaneroztekin/656a2b3192092f58cc56514c7aca221a">source code</a>)</figcaption></figure><ul><li>in the old old days, there were physical servers. they could only run one application at a time which was a huge waste of resources. they were occupying a lot of physical space and were hard to scale.</li><li>virtual machines (VMs) made it possible to run multiple applications. but they were wasting resources too since each guest OS was using some of the resources and it was still too slow to boot an application.</li><li>containers, such as docker, made efficient use of resources and it’s way faster. it uses fewer resources because containers share OS resources, it’s faster because it converts our applications and all the dependencies into small packets.</li></ul><p>By the way, you see <em>Hypervisor</em> in such graphics a lot. This fancy term means an OS tailored for VMs.</p><p>Both VMs and containers solve the “but it works on my machine” problem. Once you containerize your app you can run it practically anywhere.</p><pre>» go back to <a href="#f547">table of contents</a></pre><h4>docker terms</h4><p>Let’s take a look at the terms you hear a lot when using Docker.</p><ul><li><strong>docker container</strong> is all your app needs to run. it includes the application code, all the dependencies, and a part of the OS.</li><li><strong>docker image</strong> is all your app needs to run, very similar to the container. it’s like a stopped container. we’ll talk more about this relationship in the next section.</li><li><strong>docker desktop</strong> is software to use Docker on your Mac or Windows. it installs two tools; the docker engine and the docker client.</li><li><strong>docker engine</strong> handles docker container lifecycle, network, and OS operations.</li><li><strong>docker client</strong> is a gateway (a portal) that connects your terminal to the docker engine so you can use your terminal to manage your docker containers.</li><li><strong>dockerfile</strong> describes how to build an image.</li><li><strong>docker compose</strong> describes how Docker should build and deploy an app.</li><li><strong>docker volume</strong> is a shared folder between the host(s) and the container(s).</li></ul><p>Don’t worry if you don’t get the terms yet. We’ll talk through them in the article.</p><pre>» go back to <a href="#f547">table of contents</a></pre><h4>docker images</h4><p>A Docker image contains everything required for an application to run. It includes application code, dependencies, and a part of the OS. Images are similar to classes in OOP whereas containers are similar to objects. A container is a running instance of an image.</p><p>If you have an application’s Docker image, the only other thing you need to run that application is a computer running Docker.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5Gmfj2RmQCoRRoTwIN4qVA.png" /><figcaption>image vs container (<a href="https://gist.github.com/hakaneroztekin/2c503881e294a8374e848878259e71a9">source code</a>)</figcaption></figure><pre>» go back to <a href="#f547">table of contents</a></pre><h4>docker containers</h4><p>A container is a runtime instance of an image. Similar to the OOP <em>class </em>and <em>object</em> analogy, you can instantiate multiple containers from the same image.</p><p>Let’s run a ubuntu container.</p><p>The command template is shown below,</p><pre>docker container run image-name application-name</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r-MJPmHGFXx_WVoEYCIJIg.png" /><figcaption>docker container run (<a href="https://gist.github.com/hakaneroztekin/19d9182d049211aa73ec58687dc5a062">source code</a>)</figcaption></figure><p>We use -it flag for the interactive terminal. With this, we can run commands inside the container. /bin/bash is the main application to run. Thus, when we exit the terminal, the container will terminate.</p><p>Containers are designed to be immutable. Don’t update them by, for example, logging in to it and updating the configuration.</p><p>The data is stored in the Docker host’s filesystem which won’t survive in case of failure. So, you should use <em>volumes</em> to store data. We will talk more about volumes in the article.</p><pre>» go back to <a href="#f547">table of contents</a><br>» skip the demo, jump to <a href="#8c30">containerizing an app</a></pre><h4>demo time: pull and inspect a docker image then run it</h4><p>In this demo, we will,</p><ul><li>check out the details of Docker in your system</li><li>pull an image</li><li>inspect the image</li><li>run it as a container</li></ul><p>If you don’t already have Docker installed, you can install <a href="https://www.docker.com/products/docker-desktop/">Docker Desktop</a>. Start with it and use the command to ensure it’s up and running.</p><pre>docker version</pre><p>An example output,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*V4o16ADuXPp9Fwj5Onh5XQ.png" /><figcaption>docker version (<a href="https://gist.github.com/hakaneroztekin/e6837406a7b682e0102caaa5eac030c6">source code</a>)</figcaption></figure><p>As you can see, we have a client that connects our terminal commands to the Docker engine, and the engine that does the actual work.</p><p>We’re going to pull the alpine image,</p><pre>docker image pull nginx</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FLUpOlIirjG1GYqnic-Ipg.png" /><figcaption>docker image pull (<a href="https://gist.github.com/hakaneroztekin/11cf21d22a9351f6b3d818b6571175c3">source code</a>)</figcaption></figure><ul><li>if we don’t specify a version (for e.g docker pull nginx:1.22.0), it uses the latest image.</li><li>it pulls the image from the public Docker repository, i.e. <a href="https://hub.docker.com/">Docker Hub</a>.</li></ul><p>List the images in our host with the following command,</p><pre>docker image ls</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HYrU69fb9tSe8K9endbaRw.png" /><figcaption>docker image ls (<a href="https://gist.github.com/hakaneroztekin/11cf21d22a9351f6b3d818b6571175c3">source code</a>)</figcaption></figure><p>Inspect the image,</p><pre>docker image inspect nginx</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1004/1*x8yUdQd1W_aWjzrZSVwIBw.png" /><figcaption>docker image inspect (<a href="https://gist.github.com/hakaneroztekin/bbcbcf9431628981dfcbebbad2d999ec">source code</a>)</figcaption></figure><p>The trimmed output shows that the image consists of 6 layers. These layers are generated during the image creation via Dockerfile. They can be the OS (e.g. nginx uses debian), dependencies, and source code. Docker caches the layers so if you’re going to build an image that has some layers in the cache, cached layers will be used to speed up the process.</p><p>Let’s run it!</p><pre>docker container run --name my-container -p 3000:80 -d nginx</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yIBam5Ll6PcOdi2UkYek8g.png" /><figcaption>docker container run (<a href="https://gist.github.com/hakaneroztekin/e3d0993ba481ce8c500b872a64b900e0">source code</a>)</figcaption></figure><p>Let’s examine what we do here,</p><ul><li>we run the container in the background with -d flag. it’s not attached to our terminal so we can use our terminal normally.</li><li>we give a custom name to our container with name. it’s useful to identify the container.</li><li>by default, nginx container exposes the port 80. here, we connect the host port 3000 to the container port 80 to make it accessible outside the host.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1002/1*V_oaKU-U9ElCxE-qXEGs2g.png" /><figcaption>docker host and container port mapping (<a href="https://gist.github.com/hakaneroztekin/927d772329ce188f41f66d41f972a59c">source code</a>)</figcaption></figure><p>Open up a new tab in the browser and hit localhost:3000 or 127.0.0.1:3000 to access the container.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/990/1*Qa5j95qV8pz7dBsl9oD6vQ.png" /><figcaption>page of localhost:3000</figcaption></figure><p>Voila! We can access our container via browser. nginx shows its default page. Let’s update it!</p><p>We can execute commands in our containers with exec commands. We will use bash command to connect to the container terminal.</p><pre>docker exec -it my-container bash</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/942/1*NQyI82ghDh3-_BrLobbyNA.png" /><figcaption>docker exec (<a href="https://gist.github.com/hakaneroztekin/33831775a3dfd8eb8c4012a924cf3713">source code</a>)</figcaption></figure><p>nginx keeps the static content at usr/share/nginx/html/index.html. Update it with a welcome message. You can copy the command from the <a href="https://gist.github.com/hakaneroztekin/d53c76149a48124d55465e03b9cbc33c">source code</a>. Then type exit to leave the container terminal.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Tto2VTGHsVSYDSWG8CLR8A.png" /><figcaption>update nginx homepage (<a href="https://gist.github.com/hakaneroztekin/d53c76149a48124d55465e03b9cbc33c">source code</a>)</figcaption></figure><p>Refresh the page to see the results.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/990/1*o6NBzMu1Qt0LJEsciEIBhQ.png" /><figcaption>Updated homepage</figcaption></figure><p>If the page has not changed, try the incognito mode.</p><p>Check our container details,</p><pre>docker container ls</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GQA4ii_1Pg_VutN9QvtTBg.png" /><figcaption>docker container ls (<a href="https://gist.github.com/hakaneroztekin/34db12c2a37d59631b21c9d184ece9ba">source code</a>)</figcaption></figure><p>Notice the name and the port mapping.</p><p>To remove the image, we should first remove the containers which use the image.</p><pre>docker container rm my-container</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cXItTCTZQcOvUhFhRQN_YA.png" /><figcaption>docker container rm (<a href="https://gist.github.com/hakaneroztekin/f3965443745521e95319b7237feccfd3">source code</a>)</figcaption></figure><p>Use the following command to remove the image,</p><pre>docker image rmi nginx</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/824/1*8FNqKX1kd9tC1h3huLFfnA.png" /><figcaption>docker image rmi (<a href="https://gist.github.com/hakaneroztekin/25b6ceb3d130ea1ecc8986cf11ebde09">source code</a>)</figcaption></figure><p>We can also verify the image is deleted by listing the images via docker image ls command.</p><p>Well done, we’ve pulled an image, run it, accessed it in the browser, and even hacked into it and updated the contents. Updating the container like this is not recommended since containers are immutable by design. Also, if we need to show our custom page, how do we automatize it so that our custom container shows that homepage by default?</p><p>The answer is Dockerfile, which is our next stop.</p><pre>» go back to <a href="#f547">table of contents</a></pre><h4>containerizing an app</h4><p>Containerizing an app is making your application run in a container. For that, we should create Dockerfile which describes how to build an image for our app, then build the image and run it as a container.</p><p>We will take a look at Dockerfile, then in the demo section, we will provide a custom homepage for our nginx container.</p><pre>» go back to <a href="#f547">table of contents</a></pre><h4>dockerfile</h4><p>A Dockerfile is the starting point for creating a container image — it describes an application and tells Docker how to build it into an image.</p><p>The directory containing the application and dependencies is referred to as the <em>build context</em>.</p><p>Let’s look at the contents of the Dockerfile.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NrFvkby-iPm9A0DtPm21xA.png" /><figcaption>Dockerfile (<a href="https://gist.github.com/hakaneroztekin/dcf3f19b819eedf7e7b10eb043683a53">source code</a>)</figcaption></figure><p>Every Dockerfile starts with a base image. In our Dockerfile, we use ubuntu. Our app has a custom label and environment variable, and it prints directory contents when started. Every container needs the main app to run, so it’s essential to use CMD or ENTRYPOINT .</p><p>For the sake of simplicity, assume that we have table-of-contents.txt file in the src folder.</p><p>Let’s build our image,</p><pre>docker build -t my-awesome-app .</pre><p>We tag our image as my-awesome-appand use the current directory (.) as the build context (where our Dockerfile is located).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5wE2AhmQrNIDI7y5wDrZ-w.png" /><figcaption>docker build (<a href="https://gist.github.com/hakaneroztekin/48e4a15d0dd1f2848dea36c255c2efe9">source code</a>)</figcaption></figure><p>Notice the [3/3] notation. This means our image has 3 layers. One for the base image, one for COPY command, and one for WORKDIRcommand. Not every Dockerfile command creates a layer, some add metadata, like LABEL.</p><p>Inspect our image,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6uW6xsiDddQ3w3OV0aRLYg.png" /><figcaption>docker image inspect (<a href="https://gist.github.com/hakaneroztekin/6038631d11cf69f4f63942ca2bd8b349">source code</a>)</figcaption></figure><p>The trimmed output has a PUBLISHING_AT environment label, ARTICLE_NAMElabel, and three layers.</p><p>Let’s run our image.</p><pre>docker container run my-awesome-app</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sVhsPN7Uts2I23J_olsueQ.png" /><figcaption>docker container run (<a href="https://gist.github.com/hakaneroztekin/cbb4eaecfc1514c1cafa3a2c50f395ce">source code</a>)</figcaption></figure><p>We can list running containers with the following command,</p><pre>docker container ls</pre><p>However, since our application‘s main command (ls -l) is executed, and our container has terminated.</p><p>We can list all the containers with the following command</p><pre>docker container ls -l</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gMrS-FEG9cDmp7ZiFoHAhA.png" /><figcaption>docker container ls -l (<a href="https://gist.github.com/hakaneroztekin/e0c900bbba37bcdba2839e927ba0df47">source code</a>)</figcaption></figure><p>Congratulations. We’ve built our custom image and run it as a container.</p><pre>» go back to <a href="#f547">table of contents</a><br>» skip the demos, jump to <a href="#f39f">docker compose</a></pre><h4>demo time: nginx Dockerfile with a custom homepage</h4><p>Let&#39;s containerize our nginx app. Remember what we’ve done in the demo before,</p><ul><li>use nginx image</li><li>update contents</li><li>run the container</li></ul><p>We’ve had run the container before updating the contents but normally we want to see our custom page even from the first-page load. So we will update it before running the container.</p><p>So that’s exactly what we’re going to do with Dockerfile. We use nginx image and copy the contents.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kqUxhtBYeNf2ZkGoVBdRrg.png" /><figcaption>Dockerfile for nginx (<a href="https://gist.github.com/hakaneroztekin/d53c76149a48124d55465e03b9cbc33c">source code</a>)</figcaption></figure><p>As you see, we did not use CMDor ENTRYPOINT .This is because we’re going to use CMDand ENTRYPOINT provided by the nginx image. You can check those with docker image inspect nginx command.</p><p>Build the image,</p><pre>docker image build -t my-app .</pre><p>Run it,</p><pre>docker container run --name my-app -p 3000:80 my-app</pre><p>The result,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/990/1*-rZS7pvhUsSCi3Q-E--nQw.png" /><figcaption>result of localhost:3000</figcaption></figure><p>Great. We have a custom webpage ready to use with our container.</p><pre>» go back to <a href="#f547">table of contents</a><br>» skip the next demo, jump to <a href="#f39f">docker compose</a></pre><h4>demo time: run a spring app as a container</h4><p>Let’s get our hands dirty with an example closer to the real world. We’re going to build an image for our Spring application and run it as a container. We are going to add more Docker features like volumes, network, and compose to our example throughout the article. But first, let’s containerize a simple Spring app.</p><p>Don’t worry if you’re not familiar with Spring or Java. Following along with the article will provide you with enough information. Otherwise, practicing while reading would be the best option.</p><p><strong>create a simple Spring app</strong></p><p>You can initialize a Spring app via <a href="https://start.spring.io/">Spring Initializr</a>. Create an app named spring-app and create a simple controller.</p><p>You can also get the code from Github and switch to the branch demo/1-basic-spring-app</p><pre>git clone <a href="https://github.com/hakaneroztekin/java-spring-with-docker.git">https://github.com/hakaneroztekin/java-spring-with-docker.git</a></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/914/1*YwSV6Z7LDYcPSMyuIDFoxw.png" /><figcaption>Example controller (<a href="https://gist.github.com/hakaneroztekin/bd4cc181930566e7bb8ea30d6c2f75f5">source code</a>)</figcaption></figure><p>Run maven clean package for your application. This will generate a jar file at /target/spring-app.jarthat we’ll use to run our application.</p><p>Run the following command in your terminal,</p><pre>java -jar target/spring-app.jar</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ETJVVkTFYa59rV_GbG2K3g.png" /><figcaption>Our app is started</figcaption></figure><p>We are going to take a similar step for our Dockerfile.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MHjyky1o-qgzJgJqZGR2Yg.png" /><figcaption>Dockerfile for Spring app (<a href="https://gist.github.com/hakaneroztekin/f0b016bf09282901213660e4fa4b4aff">source code</a>)</figcaption></figure><p>Build our image,</p><pre>docker image build -t spring-app .</pre><p>And run it,</p><pre>docker container run -d -p 5000:8080 --name my-spring-app spring-app<br></pre><p>Wait a few seconds for it to boot up, and voila :)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/990/1*RCvxYXberyvcZOHJE2KmEg.png" /><figcaption>Result of localhost:5000</figcaption></figure><p>Congratulations!</p><p>As you’ve seen, we need to generate our jar manually for our Dockerfile to work. We can also automate this with the Dockerfile.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tFUYX7kY-fNTCz4_XEBq0A.png" /><figcaption>multi-stage build (<a href="https://gist.github.com/hakaneroztekin/02ce630e42c1f8f93977e04a87ed3ec1">source code</a>)</figcaption></figure><p>This looks cool but also frightening. It is simpler than it looks. This is called a <em>multi-stage build</em>. Multi-stage builds have multiple FROM commands (one for each stage) and we can pass content from earlier stages. It is a good practice since it reduces eventual image size.</p><p>In the Build stage, we generate a spring-app.jar file, the same as our maven clean package command. In the Package stage, we use the jar file as we did before.</p><p>You can try it out by getting the code from Github if you haven’t already and switching to the branch demo/2-multi-stage-dockerfile</p><pre>git clone <a href="https://github.com/hakaneroztekin/java-spring-with-docker.git">https://github.com/hakaneroztekin/java-spring-with-docker.git</a></pre><p>Are you tired of writing port and name each time running a container? We’ll get rid of them with Docker compose. We can also do much more with it, so let’s get into it.</p><pre>» go back to <a href="#f547">table of contents</a></pre><h4>docker compose</h4><p>Docker compose is a tool to deploy and manage multiple related containers at the same time. It works in a single cluster (i.e. Docker host). For production builds, multi-cluster is a better approach, so Docker Stacks is the tool for that. Alternatively, Kubernetes is one of the most popular tools for that, which we’re going to cover in the next article.</p><p>Docker compose is managed with a file docker-compose.yml. It has four main fields,</p><ul><li>version defines Docker compose version</li><li>services defines containers and their configuration</li><li>networks defines networks</li><li>volumes defines volumes</li></ul><p>Let’s see an example.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dy40SgbpBylsNvxdjlMetQ.png" /><figcaption>docker-compose.yml (<a href="https://gist.github.com/hakaneroztekin/8576cf3acfcae666a3396e0d1c518f30">source code</a>)</figcaption></figure><p>This docker compose file,</p><ul><li>Uses compose file version 3.7</li><li>Defines one service (container) named backend-service.</li><li>backend-service uses the current directory which includes the Dockerfile, and maps the host port 4000 to container port8080.</li></ul><p>The equivalent command would be,</p><pre>docker image build -t spring-app . &amp;&amp; docker container run --name backend-service -p 4000:8080 spring-app</pre><p>The command gets more complex when there are multiple services (containers) that have port mappings, volumes, and networks.</p><p>You can try it out by getting the code from Github if you haven’t already and switching to the branch demo/3-docker-compose-basic</p><pre>git clone <a href="https://github.com/hakaneroztekin/java-spring-with-docker.git">https://github.com/hakaneroztekin/java-spring-with-docker.git</a></pre><p>Boot up the application with the following command,</p><pre>docker compose up</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PTp79pQux8FJvWtyiC6ZaA.png" /><figcaption>docker compose up (<a href="https://gist.github.com/hakaneroztekin/0368091c30057975f5538bb2b2f01cd0">source code</a>)</figcaption></figure><p>You can access the app at localhost:4000 or 127.0.0.1:4000.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/990/1*_2KVYFPN4F0wzTVHqsJNKw.png" /><figcaption>result page</figcaption></figure><p>Check the running compose projects,</p><pre>docker compose ls</pre><p>Stop the compose,</p><pre>docker compose stop</pre><p>Stopping the compose keeps the containers and networks. If you want to delete them,</p><pre>docker compose down</pre><p>Delete the compose project</p><pre>docker compose rm</pre><p>Great, now that we know about Docker Compose, we should talk about networks and volumes and then use them in our Docker compose in the demo.</p><pre>» go back to <a href="#f547">table of contents</a></pre><h4>docker networking</h4><p>Docker containers need to communicate with each other, this is where Docker networking comes into play. Fortunately, it is simple and easy to use.</p><p>There are three natively supported network drivers. It is possible to plug in 3rd party network drivers too.</p><ul><li>bridge enables communication only in a single cluster (host)</li><li>overlay enables the communication between multiple clusters</li><li>macvlan used for integrating external networks (such as non-containerized running on physical networks or VLANs)</li></ul><p>We won’t discuss the details of overlay and macvlan since their a little too much detail and we‘re going to talk about multi-cluster networking in the upcoming <a href="https://hakaneroztekin.medium.com/kubernetes-in-a-nutshell-f2916a138f59">kubernetes in a nutshell</a> article.</p><p>We’re going to create a network, attach two containers and make containers communicate.</p><p>Create a bridge network,</p><pre>docker network create -d bridge my-bridge-network</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OmdPawCSDVefIj9y5FxxTg.png" /><figcaption>creating a bridge network (<a href="https://gist.github.com/hakaneroztekin/d1852c32bf5a8e0c00d5a2851e8c76b2">source code</a>)</figcaption></figure><p>Attach a container to the network,</p><pre>docker container run -d --network my-bridge-network <br>                        --name my-app alpine sleep 10m</pre><p>Inspect the network,</p><pre>docker network inspect my-bridge-network</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kN3e4a2Zb7ibFISMCSnLCw.png" /><figcaption>docker network inspect (<a href="https://gist.github.com/hakaneroztekin/f3082de6a9f67c8d9a16f4cb291cb545">source code</a>)</figcaption></figure><p>Run a second container,</p><pre>docker container run -it --network my-bridge-network <br>                         --name my-second-app <br>                         alpine sh</pre><p>Ping the first container by name,</p><pre>ping my-app</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Au1Vo2-SxhAeXos6icBtHw.png" /><figcaption>accessing to other containers</figcaption></figure><p>Voila. We’ve created a network and made communication between containers in the network.</p><pre>» go back to <a href="#f547">table of contents</a><br>» skip the demo, jump to <a href="#77b5">docker volumes</a></pre><h4>demo time: docker networking</h4><p>So far we’ve created a docker-compose file that has a single service and we know we get rid of long commands by docker compose. So let’s create two services and network through Docker Compose.</p><p>That was our docker compose file from the previous demo,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dy40SgbpBylsNvxdjlMetQ.png" /><figcaption>docker-compose.yml</figcaption></figure><p>We will add a network with an arbitrary name and a redis service.</p><p>You can get the code if you haven’t already and switch to the branch demo/4-docker-compose-with-network</p><pre>git clone <a href="https://github.com/hakaneroztekin/java-spring-with-docker.git">https://github.com/hakaneroztekin/java-spring-with-docker.git</a></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/886/1*H1s5xanfChaI8xwG0DA36w.png" /><figcaption>docker compose with a network (<a href="https://gist.github.com/hakaneroztekin/8c8ae13af0e7a41496846f7c6c1d0a56">source code</a>)</figcaption></figure><ul><li>We define our network below networks field.</li><li>Docker compose will start spring-service after booting up the redis-service. Note that this doesn’t guarantee that redis-servicewill be ready before spring-service.</li></ul><p>That’s it for our docker compose file.</p><p>Now we should use redis in our project.</p><p>Add redis dependency to pom.xml,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CqIPx1EaNYCNPpODJK2z_w.png" /><figcaption>pom.xml (<a href="https://gist.github.com/hakaneroztekin/f1bfa532fb145923a381891198b6f128">source code</a>)</figcaption></figure><p>Add a model,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YisWEzjCSCvrAllNCYxxWA.png" /><figcaption>ClickCount model (<a href="https://gist.github.com/hakaneroztekin/60511c1cf74f0639f2ea98c45eedd4d8">source code</a>)</figcaption></figure><p>Create a repository,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mlJSBGZnZtySLfSqzpjW0w.png" /><figcaption>ClickCountRepository (<a href="https://gist.github.com/hakaneroztekin/da005e3ce9d6f8cfbe8cfaf2927530c8">source code</a>)</figcaption></figure><p>If we were using redis installed on our computer, we’d access it via localhost, but now backend-service will access it via redis-service. So we will update the hostname in the application.properties (or application.yaml).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*inVDbJ-XH-8XaOxUdjC9xA.png" /><figcaption>application.properties</figcaption></figure><p>Lastly, we will update our controller. The best practice is to move the logic to service but for the demonstration purposes, we will keep it simple.</p><p>Our app will store click count on redis. So when the page refreshes it will get the click count if it exists it will be returned, otherwise, a new record will be saved. Also, it will be incremented each time the page is refreshed.</p><p>The main controller method,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Agxo8rSuJC8fIAu2b_UWIA.png" /><figcaption>printClickCount method (<a href="https://gist.github.com/hakaneroztekin/9d58a4c56df02c93ed78db91b9493559">source code</a>)</figcaption></figure><p>To keep the article simple we won’t cover the implementation details of getClickCountInDatabase() and incrementClickCountInDatabase(). Respectively, they get an existing key or save a new one with zero count, and increment the click count of the key in the database. If you want to see the implementation of the methods, click <a href="https://gist.github.com/hakaneroztekin/9d58a4c56df02c93ed78db91b9493559">here</a>.</p><p>Run the app,</p><pre>docker compose up</pre><p>When you run the project and hit localhost:4000, you will see the output,</p><pre>Hello, this page is visited 1 times.</pre><p>With the console output,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FYQjtYwWyyjR1MungM_xRQ.png" /></figure><p>When you refresh the page, the output should be,</p><pre>Hello, this page is visited 2 times.</pre><p>With the console output,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NHOCV_tg01eSR0U1bbQsuw.png" /></figure><p>Great. We’ve created two containers, one with our custom Dockerfile, made established a network between them. We’ve done this with the help of Docker compose so that we don’t have to write long commands.</p><p>There’s only one thing left to do.</p><p>If you stop the project with the following command,</p><pre>docker compose stop</pre><p>The containers created by Docker Compose will remain in the host. So our click count data will exist in the redis container.</p><p>However, we can’t rely on this since in modern applications the containers come and go whether with code changes, adding up more containers, or unexpected errors. Thus, it’s not a good practice to store data in the container.</p><p>When you use the following command, compose will remove the containers and networks,</p><pre>docker compose down</pre><p>And when you run the compose project, you will see the click count data from before doesn’t exist anymore. Solution? Docker volumes, which is our next stop. We’ll make the data persistent in the upcoming and last demo.</p><pre>» go back to <a href="#f547">table of contents</a></pre><h4>docker volumes</h4><p>There are two ways of persisting data in Docker.</p><ul><li>container’s storage is used for non-persistent (temporary) data. the data gets removed if you delete the container.</li><li>volumes are used for persistent (permanent) data. the data remains if you delete the container.</li></ul><p>There are three points to note about volumes,</p><ul><li>Volume is a shared directory between container(s) and host(s).</li><li>Multiple containers from different hosts can share the same volume.</li><li>Volumes can be connected to external storage systems.</li></ul><p>Create a volume,</p><pre>docker volume create my-volume</pre><p>List the volumes,</p><pre>docker volume ls</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/934/1*8sA7CHe1aC0Pwc8gUAtcCA.png" /><figcaption>docker volume ls</figcaption></figure><p>Remove a volume,</p><pre>docker volume rm my-volume</pre><p>Let’s see volumes at work. By default, redis image stores the data in /data folder. Let’s attach a volume to a redis container.</p><pre>docker container run -it --name my-redis --mount source=my-redis-volume,target=/data redis</pre><p>Open another terminal, and connect to redis-cli and save a key.</p><pre>docker exec -it my-redis redis-cli</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wYFNWAz1FWvxLXAkIvLStw.png" /><figcaption>saving to redis</figcaption></figure><p>Remove the container,</p><pre>docker container rm my-redis</pre><p>Start a new container with the same volume,</p><pre>docker container run -it --name my-redis2 — mount source=my-redis-volume,target=/data redis</pre><p>Connect to the terminal to get the key,</p><pre>docker exec -it my-redis2 redis-cli</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UqCUfpvAL9VCHrOdB1VB3g.png" /><figcaption>reading from redis</figcaption></figure><p>Congrats. We’ve created and attached a volume to a container, saved new data, and read it using volume.</p><p>So far so good. In the demo, we’re going to use volumes in Docker compose.</p><pre>» go back to <a href="#f547">table of contents</a><br>» skip the demo, jump to <a href="#b1bf">credits</a></pre><h4>demo time: docker volumes</h4><p>We attached a volume to redis container’s /datadirectory and made our data persistent. Let’s do the same thing with Docker compose. All we need to do is define a volume and use it in the container configuration.</p><p>This is the shortest and the final demo but if you want to see the working code, get the code if you haven’t already and switch to the branch demo/5-docker-compose-with-volume</p><pre>git clone <a href="https://github.com/hakaneroztekin/java-spring-with-docker.git">https://github.com/hakaneroztekin/java-spring-with-docker.git</a></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OyDjJUB-cmynCg5a5WdfgA.png" /><figcaption>docker-compose.yml (<a href="https://gist.github.com/hakaneroztekin/983a868d3cb8340a5e0cd98b3e984f9e">source code</a>)</figcaption></figure><p>That’s it! Now our click count is persistent. If you remove the containers with docker compose down and create new ones with docker compose up and hit localhost:4000 you will see the click count data from before is still alive.</p><pre>» go back to <a href="#f547">table of contents</a></pre><h4>credits</h4><p><em>Special thanks to Nigel Poulton. His book </em><a href="https://www.amazon.com/Docker-Deep-Dive-Nigel-Poulton/dp/1521822808"><em>Docker Deep Dive</em></a><em> is an amazing head-start to the Docker world. Some of the information and examples in this article are adapted from the book.</em></p><h4>what’s next</h4><p>That’s it for the Docker. You can follow me on <a href="https://medium.com/me/stories/public">Medium</a>, and jump next to <a href="https://hakaneroztekin.medium.com/kubernetes-in-a-nutshell-f2916a138f59">kubernetes in a nutshell</a>. You can also follow me on <a href="https://www.linkedin.com/in/hakaneroztekin/">LinkedIn</a>, I share posts when I publish new articles. Let me know whether you liked the article and what I can change to make it better. Thanks for reading.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=29cc2a938534" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to manage indexing with dynamic time-series data in Elasticsearch?]]></title>
            <link>https://medium.com/trendyol-tech/how-to-manage-indexing-with-dynamic-time-series-data-in-elasticsearch-40077ab4ad10?source=rss-222c185243f1------2</link>
            <guid isPermaLink="false">https://medium.com/p/40077ab4ad10</guid>
            <category><![CDATA[gitlab]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[kibana]]></category>
            <category><![CDATA[grafana]]></category>
            <category><![CDATA[elasticsearch]]></category>
            <dc:creator><![CDATA[Hakan Eröztekin]]></dc:creator>
            <pubDate>Fri, 07 Jan 2022 08:53:35 GMT</pubDate>
            <atom:updated>2022-01-18T12:25:28.788Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fA4DS9GTWZGlZ3KKzhXczQ.png" /></figure><p>Elasticsearch is a well-known database for searching a large amount of data. It performs blazingly well with search queries, such as full-text search and term-based queries, compared to regular databases. However, maintenance can be a nightmare if you don’t handle dynamic time-series data carefully.</p><p>Why?</p><p>First off, let’s talk about what we mean by dynamic time-series data. Simply put, if you update your data at any moment after you created it, it’s dynamic data.</p><p>As in other databases, you will eventually want to have a mechanism to delete old data since the resources are finite. This can be problematic if you have dynamic data. If you use multiple indexes, data duplication can cause you headaches. If you use a single index, you have to manually set up a way to delete old documents.</p><p>In this article, with my colleague <a href="https://tr.linkedin.com/in/f%C4%B1rat-fero%C4%9Flu-11458818b">Fırat Feroğlu</a>, we are going to talk about the problem dynamic data caused us in production, how we solved it and, automated deletion of the old documents without any duplication or performance problems.</p><pre><strong>Table of Contents</strong><br>∘ <a href="#2147">Our Story</a><br>∘ <a href="#9e61">The problem of Elastic Index Lifecycle Management</a><br>∘ <a href="#cda7">The permanent solution: Delete by query</a><br>∘ <a href="#a602">Automated document deletion with CI/CD Integration</a><br>∘ <a href="#3e28">Performance metrics</a></pre><h4>Our Story</h4><p>To better explain the problem, let’s first talk about why and how we use Elasticsearch.</p><p>As a Delivery Core team, one of our responsibilities is to provide a way to visualize how many cargoes are on the way, delivered, canceled, or else. It also needs to be grouped by cargo firms. So that it is possible to know how many cargoes of FedEx are delivered each day and so.</p><p>With the help of these dashboards, the warehouse team adjusts cargo firms’ capacities. For example, if one cargo firm is not performing well, we don’t overload it with new cargoes, instead, we reduce its capacity and direct new cargoes to other cargo firms. Thus we manage cargo deliveries more efficiently.</p><p>Every day more than a million cargo is created, transferred, or delivered. To be able to visualize them quickly, we use Elasticsearch.</p><p>Below you can see a simplified cargo document we store in Elasticsearch.</p><pre>{</pre><pre>“shipmentId”: 123,</pre><pre>“status”: “DELIVERED”</pre><pre>“cargoFirm”: “UPS”,</pre><pre>}</pre><p>We create Kibana dashboards using these documents, as simplified below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VYvvH65h9-aGDwyGAPUF5Q.png" /></figure><p>As you can see, we group all the cargo documents by their statuses and cargo firms.</p><p>Notice that the status of the document is being updated. We update the status as TRANSFERRED when it is shipped, or CANCELLED when it is canceled by the user.</p><p>It’s also important to mention that we use SQL database and feed Elasticsearch when cargoes are updated. So Elasticsearch is not the main database.</p><p>Alright, now that we’ve covered our need and usage of Elasticsearch, we can talk about how deleting old documents has caused us trouble.</p><h4>The problem of Elastic Index Lifecycle Management</h4><p>Index Lifecycle Management (ILM) is a feature of Elasticsearch that can manage index lifecycles. You can automate the creation, archiving, and deletion of indexes.</p><p>You can set document, size, or age limits for indexes. For example, once there are more than 10 million documents or they occupy at least 1 GB or they’re more than 30 days old, the index changes phase by becoming read-only. At the same time, a new active (read/write) index is created.</p><p>Our need was to keep the last 90 days of data so we’ve set up ILM to change phase after that period. And when that happened, that was the point of no return for us. There were two main problems:</p><ul><li>The new index did not have much data. So for the dashboards we still needed to query both indexes. Therefore, we’ve had to keep the old index until the new index has 3 months of data. Even though we’ve only needed the last 3 months, we started to store up to 6 months of data.</li><li>A bigger issue was that data duplication occurred. Same cargo is both counted as Transferred and Delivered.</li></ul><p>Figuring out how duplication occurred puzzled us a bit. Here is how. Say we have a single index A which stores our cargo which has status Transferred. Then a new index is created, say B, and our cargo is delivered to the customer; we update it as Delivered. Since the active index is B, instead of updating the document at A, we put “Delivered” document to B. So we now have Transferred documents at A, Delivered documents at B. Thus, it’s counted as both Transferred and Delivered.</p><p>For a quick fix, we could’ve simply updated the dashboard queries but there was no way to query data in Kibana like “search this cargo, and if there are multiple documents, ignore the old and get the only with the latest status”.</p><p>We solved it by the following steps,</p><ul><li>feeding the Elasticsearch with the last 3 months’ data</li><li>update dashboards to query on only the latest index</li><li>disabled the ILM</li></ul><p>This solved our problem but we needed a permanent solution. Our requirement was to keep the last 3 months of data. It should be handled without regular monitoring and manual intervention. Our research started here.</p><p>Unfortunately, there was not much information on this issue. The first thing we’ve come across was Elastic Curator. It has extended features on index management. Yet, it could not delete data inside an index.</p><h4><strong>The permanent solution: Delete by query</strong></h4><p>When we came across this query, we’ve got excited and were questioning the potential performance problems of deleting more than a million documents in a single query. Fortunately, it turned out a good solution.</p><p>Here is how we use it.</p><p>Say we have an index named my-index. If we want to get the number of documents older than 90 days we run the query below.</p><pre>GET my-index/_count</pre><pre>{</pre><pre>  &quot;query&quot;: {</pre><pre>   &quot;range&quot;: {</pre><pre>    &quot;@timestamp&quot;: {</pre><pre>     &quot;lt&quot;: &quot;now-90d/d&quot;</pre><pre>    }</pre><pre>   }</pre><pre> }</pre><pre>}</pre><p>The response,</p><pre>{</pre><pre> &quot;count&quot; : 1260000,</pre><pre> &quot;_shards&quot; : {</pre><pre>  &quot;total&quot; : 5,</pre><pre>  &quot;successful&quot; : 5,</pre><pre>  &quot;skipped&quot; : 0,</pre><pre>  &quot;failed&quot; : 0</pre><pre> }</pre><pre>}</pre><p>Let’s delete the old documents with delete by query.</p><pre>POST my-index/_delete_by_query?conflicts=proceed&amp;slices=5&amp;refresh</pre><pre>{</pre><pre> &quot;query&quot;: {</pre><pre>  &quot;range&quot;: {</pre><pre>   &quot;@timestamp&quot;: {</pre><pre>    &quot;lt&quot;: &quot;now-90d/d&quot;</pre><pre>   }</pre><pre>  }</pre><pre> }</pre><pre>}</pre><p>We use three parameters conflicts, slices, and refresh.</p><ul><li><strong>conflicts:</strong> A conflict can occur if the same document is attempted to be updated and deleted at the same time. Default is “abort” which cancels the execution of the query. Proceed allows us to continue execution without deleting that document.</li><li><strong>slice:</strong> splits the job into tasks. You can manually set it to the number of shards or you can leave Elasticsearch to handle that.</li><li><strong>refresh:</strong> refreshes shard caches thus preventing searching the deleted data immediately after executing the query.</li></ul><p>In addition, there is wait_for_completion parameter. We don’t use it but it is useful if you want to run the delete query asynchronously.</p><p>The simplified response,</p><pre>{</pre><pre> &quot;took&quot; : 72246,</pre><pre> &quot;timed_out&quot; : false,</pre><pre> &quot;total&quot; : 1260000,</pre><pre> &quot;deleted&quot; : 1260000</pre><pre>}</pre><p>It took 72.2 seconds to delete 1.26 million documents. timed_out shows if the request is timed out. Even if your query gets a timeout, Elasticsearch will keep deleting the documents.</p><p>For more on request and response parameters, you can check out the official <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-delete-by-query.html">documentation</a>.</p><h4>Automated document deletion with CI/CD Integration</h4><p>So far we can successfully delete the documents. But we don’t want to run the query manually. That’s why we schedule it using Gitlab. You can also write a scheduler service. Since using a Gitlab scheduler is quite effortless, we’ve preferred that. You can also adjust the code below to your CI/CD tool.</p><p>We create a script that consists of the queries we’ve discussed. We made it reusable for other indexes, even other Elasticsearch instances, with the use of parameters.</p><p>We print variables, a number of old documents, then delete them and print the result.</p><pre>#!/usr/bin/env bash</pre><pre>set -e</pre><pre>echo &quot;ELASTICSEARCH URL: $ELASTICSEARCH_URL&quot;</pre><pre>echo &quot;INDEX: $ELASTICSEARCH_INDEX&quot;</pre><pre>echo &quot;OLDER THAN: $DELETE_DAYS_OLDER_THAN&quot;</pre><pre>echo &quot;SLICES: $SLICES&quot;</pre><pre>echo &quot;Matching document count:&quot;</pre><pre>curl -s -X GET &quot;$ELASTICSEARCH_URL/$ELASTICSEARCH_INDEX/_count&quot; -H &#39;Content-Type: application/json&#39; -d &#39;</pre><pre>{</pre><pre> &quot;query&quot;: {</pre><pre>  &quot;range&quot;: {</pre><pre>   &quot;@timestamp&quot;: {</pre><pre>    &quot;lte&quot;: &quot;now-&#39;${DELETE_DAYS_OLDER_THAN}&#39;d/d&quot;</pre><pre>   }</pre><pre>  }</pre><pre> }</pre><pre>}</pre><pre>&#39;</pre><pre>echo &quot;Running delete query:&quot;</pre><pre>curl -s -X POST &quot;$ELASTICSEARCH_URL/$ELASTICSEARCH_INDEX/_delete_by_query?pretty&amp;conflicts=proceed&amp;slices=$SLICES&amp;refresh&quot; -H &#39;Content-Type: application/json&#39; -d &#39;</pre><pre>{</pre><pre> &quot;query&quot;: {</pre><pre>  &quot;range&quot;: {</pre><pre>   &quot;@timestamp&quot;: {</pre><pre>    &quot;lte&quot;: &quot;now-&#39;${DELETE_DAYS_OLDER_THAN}&#39;d/d&quot;</pre><pre>   }</pre><pre>  }</pre><pre> }</pre><pre>}</pre><pre>&#39;</pre><pre>echo &quot;Script completed.&quot;</pre><p>Here’s the pipeline definition,</p><pre>stages:</pre><pre>- delete</pre><pre>delete:</pre><pre> stage: delete</pre><pre> image:</pre><pre> name: path-to-image/centos:centos7</pre><pre> only:</pre><pre>  - schedules</pre><pre> variables:</pre><pre>  ELASTICSEARCH_URL: http://localhost:9200</pre><pre>  ELASTICSEARCH_INDEX: test</pre><pre>  DELETE_DAYS_OLDER_THAN: 365</pre><pre>  SLICES: 1</pre><pre> script:</pre><pre>  - bash delete.sh</pre><p>It has a single stage, named “delete”. It takes parameters from Gitlab Scheduler, otherwise uses the defaults and runs the script on centos image.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3aTmFGKnnFOX5bjAo_uYog.png" /><figcaption>Our pipeline</figcaption></figure><p>In the Gitlab Scheduler page, we set the cron period as 16:00 and define the variables.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/952/1*4RQQCShTFuyqAl0pK7AeIw.png" /><figcaption>Scheduler configuration</figcaption></figure><p>Thanks to this simple structure, we can adjust day, index, and Elasticsearch instance without any code. We can also define multiple jobs for multiple indexes.</p><h4>Performance metrics</h4><p>The main concern we’ve had with this solution is whether deleting more than a million documents at a time causes a performance hit. We could not find any resources on the web. Fortunately, it went better than expected and we’re happy to provide the metrics.</p><p>Our scheduled job starts around 3:05 PM. You can see the effect in the metrics below. We use the monitoring pages of Kibana and Grafana for the metrics.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/911/1*9jCSvY5fPwy24Xjl1F-LKg.png" /><figcaption>Approximately 2 million documents are deleted</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/913/1*gN9MrFLvmRtSvkssYc_8RA.png" /><figcaption>Since we search for document count in our script, the search rate has peaked for a short time</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/909/1*PGPzGIA2KsAXeZEwVm4LDQ.png" /><figcaption>Search latency is less than the usual values</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/909/1*2_iuehYYOsArmXPgKO6c5g.png" /><figcaption>Indexing latency is not effected</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/903/1*lOnDiQlNbdC2rx6182nc_g.png" /><figcaption>Load average has increased but not too much</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/904/1*IJrfR2sdA7ZZUMgTj5h-HA.png" /><figcaption>CPU usage has increased but not to an alarming level</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/902/1*JosM8x826C8RkcZI5fWBqg.png" /><figcaption>There is no significant change in JVM memory usage</figcaption></figure><p>Throughout the article, we’ve talked about potential indexing problems with dynamic data, how to solve it by using delete by query, automation with CI/CD, and performance effects of the solution.</p><p>Special thanks to <a href="https://tr.linkedin.com/in/ahmet-dikici-530229182">Ahmet Dikici</a> and <a href="https://tr.linkedin.com/in/bilal-caliskan-980490115">Bilal Çalışkan</a> from the DevOps team and our team for collaboration and support at all stages of the process.</p><p>Thank you for reading.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=40077ab4ad10" width="1" height="1" alt=""><hr><p><a href="https://medium.com/trendyol-tech/how-to-manage-indexing-with-dynamic-time-series-data-in-elasticsearch-40077ab4ad10">How to manage indexing with dynamic time-series data in Elasticsearch?</a> was originally published in <a href="https://medium.com/trendyol-tech">Trendyol Tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Alert and Monitoring with Grafana]]></title>
            <link>https://medium.com/trendyol-tech/alert-and-monitoring-with-grafana-b659c433bb51?source=rss-222c185243f1------2</link>
            <guid isPermaLink="false">https://medium.com/p/b659c433bb51</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[alerting]]></category>
            <category><![CDATA[grafana]]></category>
            <category><![CDATA[monitoring]]></category>
            <category><![CDATA[observability]]></category>
            <dc:creator><![CDATA[Hakan Eröztekin]]></dc:creator>
            <pubDate>Thu, 24 Dec 2020 13:53:09 GMT</pubDate>
            <atom:updated>2021-08-02T23:53:47.460Z</atom:updated>
            <content:encoded><![CDATA[<p>Trendyol is one of the most popular e-commerce websites in Turkey. At any given time, there are a vast number of customers surfing on the website and apps. This means that hundreds of thousands of requests are generated for our services every second.</p><p>To ensure high availability and seamless customer experience, we should closely monitor our services and interfere when there’s a problem. This is where alert and monitoring tools come into play. Thanks to these tools, we can analyze the status of our services and get a notification in case of any failure.</p><p>Today, we’ll take a glance at some of the most popular monitoring tools, their features, and our use cases, and we’ll discuss Grafana in depth. After reading the article, various monitoring tools will be introduced to you, and you will learn what Grafana can and can’t do for you with examples. This will help you decide whether or not you should use Grafana.</p><pre><strong>Topics<br></strong>1. <a href="#dce5">Alert and Monitoring tools</a><br>  1.1 <a href="#e39b">Kibana</a><br>  1.2 <a href="#a3e4">New Relic</a><br>  1.3 <a href="#a31c">PRTG</a><br>  1.4 <a href="#a085">Prometheus</a><br>  1.5 <a href="#eff4">Delivery Alerts</a><br>  1.6 <a href="#3985">Grafana</a><br>2. <a href="#7185">Grafana</a><br>  2.1 <a href="#9064">Monitoring with Grafana</a><br>  2.2 <a href="#a1d0">Alerting with Grafana</a><br>  2.3 <a href="#4413">Additional features</a><br>  2.4 <a href="#e687">Lack of features</a><br>3. <a href="#3b07">Conclusion</a></pre><h3>1. Alert &amp; Monitoring tools</h3><p>When you place an order at Trendyol, we designate a cargo company such as Aras Kargo, Yurtiçi Kargo for it. Thereon, we inform their services about your order. Until your order is delivered to you, you can track your order: whether it’s shipped or arrived at the cargo office.</p><p>This process requires the integration of multiple microservices. Technically, we have a custom microservice for each cargo company. Each of them listens to multiple RabbitMQ queues. So, we need to monitor the queues’ statuses, RabbitMQ server status, microservices’ pod statuses and resource utilization for this process.</p><p>For this and other similar needs, we use Kibana, Prometheus, New Relic, PRTG, and Grafana. Each one serves a different purpose in our team. We’ll briefly discuss what these tools are, what are their pros and cons, and our use cases.</p><h3><strong>1.1 Kibana</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hCUdDwLcKvMeFzBD5XhihQ.png" /></figure><blockquote>Kibana is a great UI tool for Elasticsearch. Elasticsearch is an excellent database if your primary use case is searching for data.</blockquote><ul><li>Using Kibana, you can create tables, graphs, and dashboards for your data.</li><li>Kibana is the best monitoring tool for Elasticsearch. However, it is just Elasticsearch oriented. So, you can’t integrate it with other data sources such as MSSQL, Couchbase, and RabbitMQ as you do with Grafana.</li></ul><h4><strong>How we use</strong></h4><p>Each of your orders is shipped to you in one or several packages (we call them “shipments”, or “gönderi” in Turkish). We use Kibana to visualize how many shipments we have had in the recent weeks and also by cargo company. Since there is a vast number of shipments, the use of Elasticsearch enables us to query data quickly, and Kibana is the best tool to visualize Elasticsearch data.</p><h3><strong>1.2 New Relic</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ewr92cztQQEYv-RA.jpg" /></figure><blockquote>New Relic is a pioneering monitoring &amp; alert tool by which you can access a large amount of data about your services.</blockquote><ul><li>You can monitor response time, throughput, errors, CPU and RAM usage of each Kubernetes pod and much more. You can analyze your endpoints’ performance only by using Transactions feature.</li><li>You can add alerts to your services and integrate them with other services. For example, when the error count goes above 5% for 3 minutes, you can get a message on your Slack with the error graph.</li></ul><h4>How we use</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*e-23o_OK7GrB6d0qfaBPAw.png" /><figcaption>Error percentage alert from New Relic on Slack</figcaption></figure><p>We use New Relic for almost all of our services. We also pay attention to adding alerts to our services. During a regular day, New Relic notifies us via Slack messages concerning whether there’s a high error rate or slow response time for each of our services.</p><h3><strong>1.3 PRTG</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*5DtuPzZ5Vg9tyH3l.png" /></figure><blockquote>PRTG is a network-oriented monitoring tool.</blockquote><ul><li>You can monitor a wide variety of metrics with PRTG. Some examples are as follows; uptime, load, CPU/RAM and disk usage, database requests, and endpoints’ status.</li></ul><h4>How we use</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OJOzisf1VK2BG7Rw45qD0w.png" /><figcaption>Free memory alert from PRTG on Slack</figcaption></figure><p>We use PRTG to monitor cargo company services (to assure you can track your shipment on the app), and resource usage.</p><h3><strong>1.4 Prometheus</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*XAB8UPG91I1FFTPs.png" /></figure><blockquote>Prometheus is a tool to analyze your data on other data sources (such as RabbitMQ and Kubernetes).</blockquote><ul><li>When you integrate your Prometheus with other services, you get ready for using functions. For example, you can see your total message count on RabbitMQ in near real-time.</li><li>Prometheus doesn’t have a great UI. Fortunately, you can integrate it with Grafana and get to use the functions on Grafana.</li></ul><h4>How we use</h4><p>We use Prometheus as a query provider for Grafana. Since Grafana integrates well with Prometheus, we can access all Prometheus functions on Grafana.</p><h3><strong>1.5 Delivery Alerts</strong></h3><blockquote>This is a scheduler project we’ve developed on Java 11 to define our custom alerts which are not provided by other tools.</blockquote><ul><li>This project is listed to point out that using a tool is not always the best solution. You can develop your project for your alerting requirements.</li><li>Each day, the scheduler runs custom queries on Elasticsearch and produces reports and sends them on a custom Slack channel.</li><li>We can also define specific alerts for the MSSQL database since Grafana SQL-like queries are not that great in terms of development and ease of use.</li><li>We also integrate some of our other projects with Slack, so in some cases, we send a Slack message when a custom exception is thrown.</li></ul><h3><strong>1.6 Grafana</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*sPr5zN8-3hikR012.png" /></figure><blockquote>Grafana is a popular tool to analyze and visualize our data and alert.</blockquote><ul><li>It’s an open source freemium tool that has approximately 40k stars on <a href="https://github.com/grafana/grafana">Github</a>.</li><li>It combines with <a href="https://grafana.com/grafana/plugins?type=datasource">a wide variety of data sources</a>, including PostgreSQL, Elasticsearch, MongoDB, Github, and even Google Sheets.</li><li>It is used by Booking, Stack Overflow, eBay, Red Hat and <a href="https://grafana.com/case-study/">many more</a>.</li></ul><h4>How we use</h4><p>We monitor tens of metrics and many defined alerts on Grafana. We monitor &amp; alert on RabbitMQ server status, resource utilization and queue metrics, Kubernetes pod restarts, MSSQL database status, and some SQL queries.</p><p>If you are wondering which tool is the best, there’s no single tool that fits well for every use case. You may need more than one. You can monitor application throughputs with New Relic, pod restarts with Grafana, 3rd party services’ endpoints with PRTG.</p><p>Without further ado, let’s dive deep with Grafana.</p><h3>2. Grafana</h3><p>First we’ll take a look at a sample dashboard. And then, we’ll talk about configuring it from zero and after then adding alerts to it, which will be published to your desired Slack channel. Along the way we’ll see what Grafana can and can’t do.</p><p>For the examples, we will mainly focus on RabbitMQ. If your technology choice to monitor is not RabbitMQ, remember that you can employ the same principles for your tool.</p><p>Here’s our sample for the RabbitMQ dashboard.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nQ3yAURs80CYoxvwj6iVBA.gif" /></figure><p>Each panel given above displays a metric we monitor. In near real-time, we can monitor RabbitMQ server status, channel and consumer counts, total message count, memory usage, and free disk space.</p><p>As you can see, there are 2 types of panels in this dashboard. The ones at the top display a single value (“Up”, 6763, 3.3 Mil, etc.), and they are called singlestat panels. They display a single value in a series of data. The ones below are graph panels.</p><p>There’s a significant difference between these panel types that you should know. If you want to add alerts, you have to use graph panels. So, even though they’re providing good visuals and history data, you will need them even though you don’t need their existence. This is a lack of feature for Grafana.</p><h3>2.1 Monitoring with Grafana</h3><h4>Configuring a dashboard</h4><blockquote>How do we configure such a dashboard from zero?</blockquote><p>It may look a bit complex at firstü but notice that a dashboard is just a collection of panels. Fist we will configure a single panel and the rest is the repetition of this process with minor changes.</p><p>Let’s start with the configuration of a panel.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R4RCtdSF77wLl6MSnAhRYQ.png" /><figcaption>Server Status Panel</figcaption></figure><p>Grafana provides highly customizable panels as well as high-quality visuals.</p><p>To configure your metric from zero, you’ll need to;</p><ul><li><strong>Get data</strong>: For this we will add a data source</li><li><strong>Configure query</strong>: Pick a function that provides data for your metric</li><li><strong>Customize panel</strong>: To make our metric more expressive</li></ul><h4>Get data</h4><p>To get your data, you should define a data source. This is a one time process. So after adding a data source you can use it in all your panels without any extra effort.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*g2kHshYU55nyetVd7chnZg.gif" /><figcaption>Grafana supports various data sources</figcaption></figure><p>To add a datasource, head over to the data sources page. Select your data source from the list and enter the required information. Grafana supports various data sources and each of them has different options. You can check the <a href="https://grafana.com/docs/grafana/latest/datasources/">data sources</a> page to find information for the data source you’d like to use.</p><p>If you can’t find your datasource on the page above, the plugin may be missing. To solve this, head over to the <a href="https://grafana.com/grafana/plugins?utm_source=grafana_plugin_list">plugins page</a> and install the related one to your Grafana.</p><p>We’ll use Prometheus for our RabbitMQ metrics because,</p><ul><li>Prometheus integrates well with RabbitMQ.</li><li>It provides a vast amount of ready-to-use queries.</li></ul><p>Prometheus has its own UI, but it plays a better role as a backend for Grafana since Grafana provides better UI and UX.</p><h4>Configure query</h4><p>Now, define your query for metric. To monitor the server status, we use the rabbitmq_up query. Grafana refreshes the panel automatically, so you don’t need to do it.</p><p>Here’s the result.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sIzsWY49NBX8aVb6X4hqSA.gif" /><figcaption>Adding a query to our panel</figcaption></figure><h4>Customize panel</h4><p>What we see is the current value, which is 1. So, the server is up. 1 or 0 is understandable, but we should make it more expressive.</p><p>Click the “Show options” button at the top right to see the available customizations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ko2bAITZP4miNqIHCyojpA.gif" /><figcaption>Customizations for our panel</figcaption></figure><p>Two great customizations you need in similar panels are value mappings and thresholds. You can map certain values and define colors for value ranges. That’s all for our server status panel.</p><p>From now on, we can add the rest of the panels just by changing the query field.</p><p>Queries we use for the panels,</p><ul><li>Server status: rabbitmq_up</li><li>Channel count: rabbitmq_channelsTotal</li><li>Node up stats:rabbitmq_running</li><li>Consumer count:rabbitmq_consumersTotal</li><li>Free disk space:rabbitmq_node_disk_free</li><li>Total messages: sum(rabbitmq_queue_messages)</li></ul><p>You can also play around with queries with functions and even use regex to generate your custom queries.</p><ul><li>Kubernetes pod restart counts: sum(kube_pod_container_status_restarts_total{namespace=&quot;your-name-space&quot;, pod=~&quot;your-pod-name&quot;}</li><li>RabbitMQ memory usage: 100 * (rabbitmq_node_mem_used / rabbitmq_node_mem_limit)</li><li>Top 5 defer queues by message count: topk(5, rabbitmq_queue_messages{queue=~&quot;.*defer&quot;})</li></ul><p>So far, so good. We have a server status panel. It displays a green colored “Up” and red colored “Down” messages.</p><h3>2.2 Alerting with Grafana</h3><blockquote>We want to get a notification when the server is down. How do we do it?</blockquote><p>In my opinion, alerting is the most important feature of Grafana. The ability to check metrics whenever you want is great, but you should not need to constantly check metrics to find out if something is wrong. Instead, Grafana should tell you something is wrong and you can check out the metrics then.</p><p>Adding an alert from zero is a two step process:</p><ul><li>Define a notification channel</li><li>Add an alert for your panel</li></ul><p>Let’s see.</p><h4>Define a notification channel</h4><p>You can get notifications via email, Discord, Slack, LINE, and <a href="https://grafana.com/docs/grafana/latest/alerting/notifications/#list-of-supported-notifiers">so on</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*L3ZrZzjOelcNL8C0KzXArw.gif" /><figcaption>Adding a notification channel</figcaption></figure><p>We like Slack in Trendyol. So, we’ve used Slack as the notification channel. When adding a channel, you use your Slack Webhook URL and your channel name. You should write your Slack channel name (say “delivery-alerts”) to the recipient section.</p><p>The worth-mentioning two options are to send reminders and include images. If you’re new to Grafana, these can be a bit detailed. You can skip them at the beginning but may need them in the future.</p><h4>Send reminders</h4><blockquote>A feature to periodically send alert messages to remind you that there’s something wrong.</blockquote><p>Say you set the period to 3 minutes and you get an alert indicating that a server is down alert at 3:00 PM. You will get a “Server is down” alert again at 3:03 PM, 03:06 PM and so on; until the server is up.</p><p>This feature is useful for less critical alarms so in that case, you can set period to 6 hours.</p><p>Beware that you are defining reminders per notification channel, not per alert. Thus, you will get reminders for <strong>every alerts</strong> you add to this notification channel.</p><p>Say that you have 8 alerts defined for your Slack channel (“delivery-alerts”) but you only want to get a reminder for a specific alert (e.g. “message count alert”). In this case, you should add another notification channel for your Slack channel.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*MTgrwh7FONSNHWXmwvCRsw.png" /><figcaption>Define two Grafana notification channels for your Slack channel</figcaption></figure><p>The two configurations are the same except for the send reminders option. When selecting an alert channel, we can pick delivery-alerts-with-reminder if we want reminders.</p><h4>Include images</h4><blockquote>Adding images to your alerts is possible, but not that possible</blockquote><p>You have the option to show images of your panels on your alerts. This is a helpful feature, but it doesn’t work just by enabling it. This rather simple feature requires defining an <a href="https://grafana.com/docs/grafana/latest/alerting/notifications/#external-image-store">image storage provider</a> and a lot of configuration work. Grafana definitely needs improvement here.</p><h4>Add an alert to your panels</h4><blockquote>How do we add our “If server is down, send an alert to Slack”?</blockquote><p>Remember that we can’t add alerts to singlestat panels, so we should use a graph panel to add alerts.</p><p>To change our panel type, head over to the options and pick the one you want. Notice that the “Alerts” tab becomes visible when we use a Graph panel.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cQqegvOIM3QoQ9JM5DNihQ.gif" /><figcaption>Adding an alert to a panel</figcaption></figure><p>Here are the basics of alert customization;</p><ul><li><strong>Conditions</strong>: Add a condition(s) so that when they are evaluated as TRUE, your metric will be alerting.</li><li><strong>Notifications</strong>: Select the notification channel (a Slack channel in our case) which your alert message should be sent.</li><li><strong>Message</strong>: To add some extra information to your alert message.</li><li><strong>Rule</strong>: “Evaluate every” is the frequency your alert condition be checked. “For” is the waiting duration before sending an alert, 0m means disabled.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/771/1*5wE8pT45j2N1qItU11Bpnw.png" /><figcaption>Server Status Alert</figcaption></figure><p>This is the exact alert configuration we use in our Server Status panel. Every 5 seconds, it checks if server status is 0, and when so, it sends an “Server is down” alert to the delivery-alerts Slack channel.</p><p>In the send to section, if we want periodical reminder messages, we can select the delivery-alerts-with-reminder option that we’ve discussed at the Send reminders feature.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/588/1*SH77wUX5eMeurbGu1R7Wxg.png" /><figcaption>Condition functions</figcaption></figure><p>Let’s explore the conditions section.</p><ul><li>In the query section, we define the metric to which we want to add an alert. We also specify a range.</li><li>Query section is structured as query(metricName, start, end) andquery(A, 1m, now) means that it will work on metric A, and will check the range from 1 minute ago to now. A is the name assigned to the rabbitmq_up metric we’ve added.</li><li>last function picks the latest value in the range. Since our query is defined from 1 minute ago to now, it’ll just fetch the current server status.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/966/1*wLJtUoYwtDmdf6Ogb3qrLA.png" /><figcaption>Our alert on Slack</figcaption></figure><p>That’s all for adding an alert. Now, we have a two panels; one singlestat and one graph. We’ll get message on our Slack channel when the server is down. From this point on, you can repeat the same process with minor tweaks (such as changing your query and value mappings) to create your unique dashboards and add them alerts.</p><h3>2.3 Additional features</h3><p>Let’s briefly talk about the two of the beneficial features of Grafana.</p><h4>Ready-to-use dashboards</h4><p>There are more than 3000 <a href="https://grafana.com/grafana/dashboards">dashboards</a> that can be easily imported to your Grafana. When you find a dashboard you like, you just copy the id (for e.g. 2343) and paste it in the import section in your Grafana. With little-to-no configuration, your dashboard will be ready. This one is a quite powerful feature of Grafana.</p><h4>Grafana API</h4><p>Grafana has an <a href="https://grafana.com/docs/grafana/latest/http_api/">HTTP API</a> with countless endpoints. Considering that each panel and dashboard is a JSON object, this means that there’s a countless way to customize and manage your Grafana. For example, you can execute CRUD operations on your dashboards using <a href="https://grafana.com/docs/grafana/latest/http_api/dashboard/">Dashboard API</a>.</p><p>It’s also worth mentioning that Grafana is evolving pretty quickly. They’re <a href="https://grafana.com/categories/release/">releasing</a> new features and bug fixed actively. So, it’s definitely worth hopping on the train.</p><h3>2.4 Lack of features</h3><p>Lastly, we’ll talk about some disadvantages and lack of features you can encounter when using Grafana. We’ll also talk about which version to use.</p><p><strong>A bug on an alerting method:</strong> Earlier versions of Grafana has a bug on diffand percent_diff functions. Say that you want to define an alarm for your 20% increase in message count in the last 30 minutes. Our condition will be as follows;percent_diffof query(A, 30m, now) is above 20. It will work quite well with the 20% <strong>increase</strong>, but you will still get the alert for 20% <strong>decrease </strong>which can result in pretty annoying unwanted alerts. Fortunately this <a href="https://github.com/grafana/grafana/issues/10129">bug</a> is fixed.</p><p><strong>Security issue</strong>: Grafana has had a security issue in the older versions. Thus it’s <a href="https://grafana.com/categories/release/">advised</a> to use 7.0.2 or higher.</p><p><strong>Not customizable messages</strong>: there are very limited options for customizing your alert message. You can’t use styling or variables in the message section. You can’t add buttons or something else, even though Slack is supporting these.</p><p><strong>No scheduling</strong>: You can’t turn off alerts. You can manually pause them one by one, but can’t schedule or temporarily disable them. You will get alerts after midnight or in Sunday morning.</p><p><strong>No warning state</strong>: Alerting is a boolean state. There’s no warning alert you can send to your Slack. It’s either alerting (red) or not (green). There’s a pending state too, but it’s not the same as a warning state.</p><p><strong>Lack of endpoint monitoring &amp; alert</strong>: You can’t simply check the status of your URL endpoints with Grafana. There’s a <a href="https://worldping.raintank.io/pricing/">plugin</a> for that, but it’s limited to 3 endpoints. Otherwise, you’ll need to pay $100 monthly. Thats why we use PRTG tool for this need.</p><h4>Conclusion</h4><p>Being one of the most popular monitoring tools, Grafana is extremely beneficial tool with great visuals. It helped us to monitor and manage the alerts of various technologies (varying from RabbitMQ to Kubernetes, MSSQL, and Elasticsearch) in one place. It’s easy to learn and use. It is no surprise a lot of great companies use it. Even though it’s evolving quickly, it also lacks some basic features we’ve discussed. Overall, Grafana is a very powerful tool that you should consider using in your company.</p><p>Happy coding!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b659c433bb51" width="1" height="1" alt=""><hr><p><a href="https://medium.com/trendyol-tech/alert-and-monitoring-with-grafana-b659c433bb51">Alert and Monitoring with Grafana</a> was originally published in <a href="https://medium.com/trendyol-tech">Trendyol Tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Java 11 in a Nutshell]]></title>
            <link>https://medium.com/analytics-vidhya/java-11-in-a-nutshell-f3f452bf2951?source=rss-222c185243f1------2</link>
            <guid isPermaLink="false">https://medium.com/p/f3f452bf2951</guid>
            <category><![CDATA[programming-languages]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[java]]></category>
            <category><![CDATA[coding]]></category>
            <dc:creator><![CDATA[Hakan Eröztekin]]></dc:creator>
            <pubDate>Sun, 11 Oct 2020 08:25:51 GMT</pubDate>
            <atom:updated>2021-01-05T09:25:03.376Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ol4QDMB5QYilOliYweRpAA.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@blakewisz?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Blake Wisz</a> on <a href="https://unsplash.com/s/photos/java?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>Java is one of the world’s most widely used and in-demand programming languages. Since it’s arrival in 1995, Java kept improving over time. With its frequent release cycle in the recent years, it requires a bit more effort to keep up with the new releases.</p><p>We will start with a small Q&amp;A section to clear out the questions and then discuss the Java 11 changes with “in a nutshell” approach. In approximately 15 minutes, you will get familiar with what’s changed with Java 11. There is also a <a href="https://drive.google.com/file/d/1PpNXu4U05uvaT3eRhyI3dhF6Ukza4cl2/view?usp=sharing">slide version</a> of the article as well.</p><p>Let’s begin.</p><h4>First of all, why are there so many releases?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/334/1*tTcgZFRDChW4EeLfHislGA.png" /></figure><p>Oracle used to release a new version every 2 to 3 years. However, it took great effort for companies to adopt too many changes at once.</p><p>Oracle then adopted continuous release approach and announced 6 months release period. Since Java 9, Oracle releases a new Java version with relatively minor changes every 6 months.</p><h4>What is LTS?</h4><p>A long term version is tend to be an industry standart for several years. Even if new releases come after it, it still gets updates. For example, Java 8 is released in 2014, still getting updates in 2020 and will get extended support until 2030.</p><h4>So which version should I upgrade to?</h4><p>Oracle strongly recommends switching to Java 11 or a latest version. The section below from Wikipedia is important,</p><blockquote>Oracle (and others) “highly recommend that you uninstall older versions of Java”, because of serious risks due to unresolved security issues. Since Java 9 (and 10) is no longer supported, Oracle advises its users to “immediately transition” to Java 11 (Java 15 is also a non-LTS option) - <a href="https://en.wikipedia.org/wiki/Java_(software_platform)">wiki</a></blockquote><p>Java 11 is the latest LTS, until the release of Java 17 in September 2021. It’s backwards compatible with Java 8 (we’ll talk about the details). Also it brings noteworthy features and improvements in addition to Java 8.</p><h4>Is Java 11 free to use?</h4><p>In short, yes.</p><p>The confusion comes from licensing. There are two types of licenses. OpenJDK is free to use, while Oracle JDK provides long-term support for a price. The binaries of these are almost identical so you can use OpenJDK with ease. If you’re interested <a href="https://www.oracle.com/java/technologies/javase/jdk-11-relnote.html#Diffs">here</a> is the complete list of differences for Java 11.</p><p>Let’s continue with what Java 11 brings to the table.</p><h3>Overview of New Features</h3><ol><li>HTTP API</li><li>Launching Single-File Java Program</li><li>New Library Methods for Strings, Collections and Files</li><li>Performance and Security Improvements</li><li>Other Enhancements</li><li>Removed and Deprecated Features</li></ol><h3>1. HTTP API</h3><blockquote>HTTP API is a Java library to execute HTTP requests.</blockquote><h4>motivation</h4><p>To replace HttpURLConnection which was introduced with Java 1.1 in 1997, and is difficult to use and maintain.</p><h4><strong>features</strong></h4><ul><li>Supports HTTP/1.1 and HTTP/2 protocols</li><li>Supports common HTTP methods like GET, POST, PUT, DELETE</li><li>Can handle synchronous and asynchronous communication</li></ul><h4>how to use</h4><ol><li>Create, initialize and configure HttpClient</li><li>Create and initialize HttpRequest</li><li>Pass HttpRequest to HttpClient, execute the request and retrieve HttpResponse</li><li>Handle HttpResponsefor your purposes</li></ol><h4>example</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a7d065341267e6907bb690e1263ec984/href">https://medium.com/media/a7d065341267e6907bb690e1263ec984/href</a></iframe><h3>2. Launching Single-File Java Program</h3><blockquote>Java 11 supports execution of single file Java program with a single command.</blockquote><h4>details</h4><ul><li>Enables beginner friendly Java development and high speed prototyping.</li><li>It has to be a single-file program only (with one or more classes).</li><li>We can’t use external dependencies.</li></ul><h4>how to use</h4><p>Let’s assume our file name is MyProgram.java, to execute it, we only need to use java MyProgram.java.</p><h3>3. New Library Methods for Strings, Collections and Files</h3><blockquote>Java 11 added some useful methods for the mentioned libraries for convenience.</blockquote><h4>strings api</h4><ul><li><strong>Repeat a string</strong>: Repeat a string n times with String.repeat(Integer). Example: &quot; &quot;.repeat(10)</li><li><strong>Check empty/whitespace</strong>: Check if string is empty or whitespace only. String.isBlank(). It’ll throw NullPointerExceptionif the string is blank.</li><li><strong>Remove whitespaces</strong>: Remove leading String.stripLeading(), trailing String.stripTrailing() or both whitespaces String.strip()</li><li><strong>Process multiline string</strong>: Convert multiline /n string to stream of lines with String.lines()</li></ul><h4>collections api</h4><ul><li><strong>Convert collection to an array</strong>: This process made easier with Java 11. For example, converting a list of students to string array would be performed with students.toArray(String[]::new)</li></ul><h4>optional and predicate.</h4><ul><li><strong>Negating a predicate </strong>: We can filter out the elements not having a certain condition with Predicate.not(myCustomPredicate).</li></ul><h4>files api</h4><ul><li><strong>Easier file read/write</strong>: Thanks to Files.readString(path) and Files.writeString(path, charSequence, options) we can achieve this operations.</li></ul><h3>4. Performance and Security Improvements</h3><h4>performance Improvements</h4><ul><li><strong>G1 Garbage Collector Improvements</strong>: G1 was introduced in Java 7 and became the default GC in Java 9. With Java 11, G1 is improved for high performance. Java 11 improves pause times (client waiting for application response caused by GC) by up to 60% (<a href="https://mail.openjdk.java.net/pipermail/hotspot-gc-use/2018-June/002759.html">source</a>).</li><li><strong>ZGC</strong>: An experimental but scalable and low-latency GC. It guarantees maximum pause time of 10 ms. ZGC is great for applications memory consuming applications that handle a lot of data.</li><li><strong>Epsilon</strong>: An experimental but low-overhead GC. It’s a good fit for short living applications (executing a single command then closing).</li></ul><h4>security Improvements</h4><ul><li>The recent Transport Layer Security (TLS) improved HTTPS performance and security. Java 11 supports the TLS 1.3 protocol.</li></ul><h3>5. Other Enhancements</h3><ul><li><strong>Using </strong><strong>var in lambdas</strong>: We&#39;ve met var with Java 10, now we can use it in lambdas. Advantage of this feature is now we can annotate lambda variables (eg. @NotNull)</li><li><strong>Adoption of Unicode 10</strong>: Java 11 uses Unicode 10, an upgrade from Java 10 with Unicode 8. That includes 16,018 new characters and 10 new scripts.</li><li><strong>Nested access control</strong>: Inner class can access all private variables and methods of Outer class (eg. Outer.name).</li><li><strong>Java Flight Recorder</strong>: JFR is a profiling and diagnostics (resource usage etc.) tool. We can also use Java Mission Control to visualize data provided by JFR.</li></ul><h3>6. Removed and Deprecated Features</h3><h4>removed features</h4><p>Java 11 is backwards compatible with Java 8. So you can swiftly change from Java 8 to 11. If you use some of the packages below, all you need to do is add the replacement dependency to your project.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4e8ca9b23fc6b17f1bcfdf93b1adb734/href">https://medium.com/media/4e8ca9b23fc6b17f1bcfdf93b1adb734/href</a></iframe><p>Also, Applets and Java Web Start are completely removed without a replacement. If you’re looking for Java desktop application development, check out JavaFX.</p><h4>deprecated features</h4><ul><li><strong>Nashorn JavaScript Engine:</strong> Since it’s hard to maintain compatibility with rapid changing of ECMAScript, it’s now deprecated.</li></ul><h4>conclusion</h4><p>Java 11 is the latest LTS release and introduced beneficial features and improvements on top of previous releases. A new HTTP client, new library methods for strings, collections, predicates and files are introduced. Further, garbage collector performances are optimized, along with two new garbage collectors, ZGC and Epsilon.</p><p>If you’re into learning all the aspects of Java in a nutshell, feel free to check out my <a href="https://medium.com/@hakaneroztekin/java-in-a-nutshell-a-quick-review-guide-of-important-topics-of-java-962cbf62b761">Java in a Nutshell</a> article.</p><p>Happy coding!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f3f452bf2951" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/java-11-in-a-nutshell-f3f452bf2951">Java 11 in a Nutshell</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Java in a Nutshell]]></title>
            <link>https://hakaneroztekin.medium.com/java-in-a-nutshell-a-quick-review-guide-of-important-topics-of-java-962cbf62b761?source=rss-222c185243f1------2</link>
            <guid isPermaLink="false">https://medium.com/p/962cbf62b761</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[java]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[coding]]></category>
            <dc:creator><![CDATA[Hakan Eröztekin]]></dc:creator>
            <pubDate>Sun, 18 Aug 2019 10:55:47 GMT</pubDate>
            <atom:updated>2021-01-02T14:06:00.580Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*38onBwLqzG1KMdueRuYetA.jpeg" /></figure><p>Hello, welcome to this ‘panaromic’ Java guide. You can find easily digestable information on almost all the topics of Java.</p><p>When I first started my software engineering career, I was looking for an all-in-one Java guide but I couldn’t have found it. Then I prepared one which benefited me a lot.</p><p>A quick update, I’ve just published changes of <a href="https://medium.com/@hakaneroztekin/java-11-in-a-nutshell-f3f452bf2951">Java 11 in a Nutshell</a>. Check it out if you’re interested.</p><p>If you’re <strong>new to software development</strong> or looking for a resource to <strong>quickly review topics for interview prep</strong>, you’re in the right place.</p><p>Let’s go!</p><h3>Topics We Will Cover</h3><ul><li>Properties of Java</li><li>Variable Types</li><li>What are JVM, JRE and JDK?</li><li>Object Creation Stages</li><li>Constructor types</li><li>Data Types</li><li>Access Modifiers</li><li>Passing By Value</li><li>Inheritance</li><li>Overloading</li><li>Overriding</li><li>Encapsulation</li><li>Abstraction</li><li>Interface</li><li>Abstract class</li><li>The Relation between Overriding, Upcasting, Overloading, Runtime Polymorphism and Interfaces?</li><li>Association</li><li>Aggregation</li><li>Composition</li><li>I/O</li><li>Serialization &amp; Deserialization</li><li>Exceptions (Try/Catch/Finally)</li><li>Inner Class</li><li>Java 8 Features (Functional Interface, Lambda Expressions, Default methods, Streams, Optional, Collections, DateTime API)</li></ul><p>You can find references at the end of the guide. Please let me know if the guide has any incorrect/missing information. Feedbacks are always welcomed.</p><h3>Properties of Java</h3><ul><li><strong>Object oriented:</strong> Java is a true object oriented programming language. Except of the primitive data types, everything is a object.</li><li><strong>Platform independent:</strong> Java is compiled into a platform independent byte code (unlike C and C++). The byte code is interpreted by the Java Virtual Machine (JVM) on whichever platform it’s being run on.</li><li><strong>Simple:</strong> Java is a high-level programming language and is closer to the human language. Thus it’s easier to learn.</li><li><strong>Interpreted: </strong>Java byte code is translated (by JVM) on the fly to native machine instructions.</li><li><strong>High performance:</strong> With Just-In-Time (JIT) compilers, Java enables high performance.</li></ul><h3>Variable Types</h3><ul><li><strong>Local variables: </strong>Variables defined inside methods.</li><li><strong>Instance variables:</strong> Variables within a class, outside any method.</li><li><strong>Class variables:</strong> Static variables within a class, outside any method.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/dd1f90ae9f30c7e96e95c1d74deca059/href">https://medium.com/media/dd1f90ae9f30c7e96e95c1d74deca059/href</a></iframe><h3>What are JVM, JRE and JDK?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/632/1*h6urEX-WtyMRSd6e7kz13g.jpeg" /></figure><ol><li>When we write Java code, java compiler (javac) converts it to byte code. Javac is included in Java Development Kit (JDK).</li><li>The byte code is then converted to machine-specific instructions via Java Virtual Machine (JVM).</li><li>Machine-specific instructions are then run on the computer by Java Runtime Environment (JRE).</li></ol><ul><li><strong>JVM = Execution of a Java program</strong>. Because Java is a platform-independent language, it needs a virtual machine environment (interpreter) to run, which is provided by JVM. So, JVM is used to run Java code. Whenever we run a Java program either using JRE or JDK, it goes to JVM to be executed.</li><li><strong>JRE = JVM + Library Classes</strong>. JRE provides environment to run Java programs on our machines.</li><li><strong>JDK = JRE + Development Tools</strong>. JDK is used to develop and run Java code. It is used by developers.</li></ul><h3>Object Creation Stages</h3><pre>Apple myApple = new Apple();</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/602/1*3zodmxTSAu6TkI2iA3qkVw.png" /></figure><ul><li><strong>Declaration:</strong> A variable is declared with an object type and a variable name. Apple myApple;</li><li><strong>Instantiation: </strong>‘new’ keyword is used to create object. = new</li><li><strong>Initialization:</strong> The call to a constructor. Apple();</li></ul><h3>Constructor Types</h3><ul><li><strong>Default constructor:</strong> A constructor without any argument. Also known as no argument constructor.</li><li><strong>Parametrized constructor: </strong>A constructor with parameters.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3df4e78b3ec3c652192242506f9a2a8e/href">https://medium.com/media/3df4e78b3ec3c652192242506f9a2a8e/href</a></iframe><h3>Data Types</h3><ul><li><strong>Primitive data types: </strong>There are 8 of them; boolean, char, short, byte, int, long, float, double.</li><li><strong>Non-primitive data types:</strong> The rest of the data types, i.e. reference/object data types.</li></ul><h3>Access Modifiers</h3><ul><li>default is visible to the package.</li><li>private is visible to the class only.</li><li>public is visible to the world.</li><li>protected<strong> </strong>is visible to the package and sub-classes.</li></ul><p>If no access modifier is denoted, defaultis used as a default.</p><p>Non-access modifiers are static, final, abstract, synchronized and volatile<strong>.</strong></p><h3>Passing By Value</h3><p>Java is a pass-by-value language. Referencing <a href="https://stackoverflow.com/questions/40480/is-java-pass-by-reference-or-pass-by-value">here</a>, check the below example.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e37672bd9b6be6dc95b88c8706f4622c/href">https://medium.com/media/e37672bd9b6be6dc95b88c8706f4622c/href</a></iframe><h3>Inheritance</h3><ul><li><strong>Definition: </strong>Inheritance is a mechanism where one class acquires all the properties of another class.</li><li><strong>Benefits:</strong> Inheritance promotes <strong>code reusability </strong>and <strong>enables method overriding </strong>so that run-timepolymorphism can be achieved. Run-time polymorphism is briefly, “call to a method is resolved at runtime” (whether it’s childs method or parents method).</li><li><strong>Extra details:</strong> private methods, variables and constructors are not inherited while static field/methods are inherited (public static variables aren’t a good practice). Inheritance represents <strong>IS-A relationship</strong>, also known as parent-child relationship. One class can only extend only one other class. Multiple inheritance is not possible in Java (<a href="https://stackoverflow.com/questions/20615391/multiple-inheritance-in-java-since-all-classes-extend-from-object-class">a response</a> about all classes inheriting from Object, it’s briefly chain of inheritances). Reason for that is, say class A and B have talk() method and C inherits both. What happens when we call c.talk() would be unresolved.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a117ee7ce1058e6c691c6719b4bfc2f5/href">https://medium.com/media/a117ee7ce1058e6c691c6719b4bfc2f5/href</a></iframe><h3>Overloading</h3><ul><li><strong>Definition:</strong> If a class has more than one methods with the same name but with different type or number of arguments, it’s called overloading.</li><li><strong>Benefits:</strong> It provides cleaner code. Also, it enables to call a similar method with different parameters.</li><li><strong>Extra details:</strong> We can’t only change the return type of a method as this may cause confusion. Ex. sum() returns int, but overloading only with return type as double will cause confusion on which method to call.</li></ul><h3>Overriding</h3><ul><li><strong>Definition: </strong>If a subclass provides a specific implementation of a method that is already provided by its parent class, it’s called overriding.</li><li><strong>Overriding rules:</strong> The methods must have the same name and the same signature. Also, the two classes must have an IS-A relationship (i.e inheritance).</li><li><strong>Overriding benefits</strong>: It is useful to achieve runtime polymorphism and to implement interface methods.</li><li><strong>Extra details:</strong> We can’t override static methods because overriding is for instance methods (not statics, not field variables), statics are <a href="https://stackoverflow.com/questions/10291949/are-static-methods-inherited-in-java">hidden</a> if the subclass has the same signature. We can override overloaded methods. @Override signature is not compulsory but it is a good practice to use as it provides additional checks. We can change the scope of overridden method in the subclass to a broader type like from protected to public but not vice versa.</li></ul><h3>Encapsulation</h3><ul><li><strong>Definition:</strong> Encapsulation is <strong>data hiding</strong>. Methods or variables of a class will be hidden from other classes, and can be accessed only through the methods of their current class.</li><li><strong>Benefits:</strong> We prevent direct access to instance &amp; class variables thus the class will have more control. We can make the class read-only or write-only (by not providing getter or setter). Also, encapsulated classes are easy to (unit) test.</li><li><strong>Encapsulation</strong> is one of the four fundamental OOP concepts. The other three are inheritance, polymorphism, and abstraction.</li><li><strong>Encapsulation rules:</strong> Declare the variables of a class as private. Provide public setter and getter methods to modify and view the variables’ values.</li></ul><h3>Abstraction</h3><ul><li><strong>Definition: </strong>Abstraction is hiding details and showing only the functionality.</li><li><strong>Another definition: </strong>Abstraction is providing user-level details while hiding the implementation details.</li><li><strong>Example: </strong>A good analogy is a coffee machine. As a coffee user all you want is to click a button or two to get your favorite fresh coffee. You don’t want to know how coarse to grind the coffee, the ideal water temperature and what’s the ideal coffee to water ratio. All you want is to get your coffee with your buttons. All the details are hidden and you only see the functionality with buttons, this is the abstraction.</li><li><strong>Example: </strong>A technical example can be user clicking “Order” button, without caring about what happens in the background when he clicks.</li><li><strong>Benefits:</strong> It helps to reduce programming complexity and effort.</li></ul><p>There are two ways to achieve abstraction; <strong>interface </strong>(100% abstraction) and <strong>abstract class</strong> (0–100% abstraction).</p><h4>Interface</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/516/1*vetPFmWvs_V-AlnfxBROcw.jpeg" /><figcaption>Reasons to use interface</figcaption></figure><ul><li><strong>Interface is a blueprint of a class</strong>. It has static constants and abstract methods.</li><li>Interface represents <strong>IS-A</strong> relationship.</li><li>Since all methods are abstract, none of the methods can have a body.</li><li>Just like the abstract class, it cannot be instantiated.</li><li>Since Java 8, interfaces can have <strong>default and static methods</strong>.</li><li>Since Java 9, interfaces can have <strong>private methods</strong>.</li></ul><pre>interface myInterface { <br>// declare constant fields <br>// declare methods (they are abstract by default)<br>}</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/590/1*X6St0jbqvH2mWHDgnJkffw.jpeg" /></figure><ul><li>Multiple inheritance is not supported by Java because of ambiguity but it’s possible when the parents are interfaces.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/724/1*1QmBq_X9ZODdt4vC551M-g.jpeg" /><figcaption>Multiple Inheritance is possible through Interfaces</figcaption></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8fb3cb6c1e07755b87440ac92103056f/href">https://medium.com/media/8fb3cb6c1e07755b87440ac92103056f/href</a></iframe><h4>Abstract class</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*VFDss9TRkd7XNDVk0EBriw.jpeg" /><figcaption>Abstract Class Rules</figcaption></figure><p>A class which is declared as abstract is known as an <strong>abstract class</strong>. It can have abstract and non-abstract methods. It needs to be extended and its method implemented. It cannot be instantiated.</p><pre>abstract class Bike{ // abstract class<br>     abstract void use(); // no method body<br>} </pre><pre>class Bianchi extends Bike{ <br>     void use(){ System.out.println(&quot;Bike is on the use.&quot;); }<br> <br>     public static void main(String args[]){ <br>          Bike obj = new Bianchi(); <br>          obj.use(); <br>     } <br>}</pre><h3>The Big Picture: What’s the relation between Overriding, Upcasting, Overloading, Runtime Polymorphism and Interfaces?</h3><h4>Relation Group 1: Overloading &amp; Compile Time Polymorphism</h4><p>It’s <strong>compile time polymorphism</strong> when a call to an <strong>overloaded </strong>method is resolved at compile time. Also known as <strong>static binding</strong> and <strong>early binding</strong>.</p><h4>Relation Group 2: Overriding &amp; Runtime Polymorphism</h4><p>It’s <strong>run time polymorphism </strong>when a call to an overridden method is resolved at runtime. Also known as <strong>dynamic binding</strong> and <strong>late binding</strong>.</p><ul><li>StaticType x = new DynamicType(); is the format we define Java objects. StaticType is determined at compile time, DynamicType is determined at runtime. For the example Fruit f = new Apple() the type of f is Fruit at compile time, Apple at runtime. If apple overrides color() method, it’s a call to the parent’s color() method at compile time, but then changed as childs’ color() method at runtime (runtime polymorphism).</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/521/1*E5Nf7pesb_7fNt2WfZRzXg.png" /><figcaption>Upcasting</figcaption></figure><pre>class A {}<br>class B extends A {}</pre><pre>//Upcasting<br>A a = new B();</pre><p><strong>Upcasting </strong>is a cast from a derived class to a more general base class.</p><pre>Dog dog = new Dog();<br>Animal animal = (Animal) dog;</pre><p>If the reference variable of Parent class refers to the object of Child class, it is known as upcasting.</p><h4>Extra Details</h4><p><strong>Downcasting </strong>is a cast from a base class to a more specific class.</p><pre>Animal animal = new Dog();<br>Dog dog = (Dog) animal;</pre><ul><li>Java permits an object of a subclass type to be treated as an object of any superclass type. This is called upcasting.</li><li>Upcasting is done automatically, while downcasting must be manually done by the developer.</li><li>Upcasting and downcasting are <strong>NOT</strong> like casting primitives from one to the other (<a href="http://www.cs.utexas.edu/~cannata/cs345/Class%20Notes/14%20Java%20Upcasting%20Downcasting.htm">a good resource</a>).</li></ul><h3>Association</h3><ul><li>Association refers to the relationship between multiple objects.</li><li><strong>Aggregation</strong> and <strong>composition </strong>are two types of association.</li></ul><h3>Aggregation</h3><ul><li>If a class have an entity reference, it is known as aggregation.</li><li>Aggregation represents <strong>HAS-A</strong> relationship.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/17ea21bdd535935e6b8f967dfb9d289c/href">https://medium.com/media/17ea21bdd535935e6b8f967dfb9d289c/href</a></iframe><ul><li><strong>Aggregation benefits: </strong>Promotes code reusability. In the above example, we just simply put use of Address class just with a line of code.</li></ul><h3>Composition</h3><ul><li><strong>Definition: </strong>Composition is the strong type of association.</li><li><strong>Composition rule: </strong>An association is said to composition if an object owns another object and another object cannot exist without the owner object.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/225/1*MCWMUrpM1wUljMBpus7FvQ.jpeg" /></figure><ul><li><strong>Example:</strong> Think about lego homes. A lego home can’t exist without single lego(s), but legos can exist without a lego home.</li></ul><h3>I/O</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*axwdC_Yo0ukc9v3Oy2lRXw.png" /><figcaption>A stream</figcaption></figure><p><strong>Stream is a flow of data from an input (source) to an output (destination).</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*dTbWMDyIIpRWxO7SgC3msQ.jpeg" /><figcaption>Java I/O Streams</figcaption></figure><ul><li><strong>InputStream:</strong> <em>Read data </em>from a source.</li><li><strong>OutputStream:</strong> <em>Write data</em> to a destination.</li><li><strong>Extra details:</strong> stdin and stdout in C++, System.in and System.out in Java are examples of (standart) streams. byte streams are 8-bit, character streams are 16-bit unicode format used to perform I/O operations.</li></ul><h4>Serialization &amp; Deserialization</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*LgFqoXfI63pLjv3p-5BP8g.jpeg" /></figure><ul><li><strong>Definition:</strong> Serialization is the <strong>conversion </strong>of an object to a <strong>series of bytes</strong>. The byte stream can then be deserialized — converted into a replica of the original object (<a href="https://stackoverflow.com/questions/447898/what-is-object-serialization">source</a>).</li><li><strong>Benefit:</strong> The object can be easily saved to persistent storage or streamed across a communication link.</li><li><strong>Extra details: </strong>In Java, the serialization mechanism is built into the platform, but you need to implement the <em>Serializable</em> interface to make an object serializable.</li></ul><h3>Exceptions</h3><p>An exception (or exceptional event) is a problem that arises during the execution of a program. When an <strong>Exception</strong> occurs the normal flow of the program is disrupted and the program/Application terminates abnormally, which is not recommended, therefore, these exceptions are to be handled.</p><p>Some of these exceptions are caused by user error, others by programmer error, and others by physical resources that have failed in some manner.</p><ul><li><strong>Checked exceptions:</strong> A checked exception is an exception that is checked (notified) by the compiler at compilation-times, also known as <strong>compile time exceptions</strong>. These exceptions cannot simply be ignored, the programmer should take care of (handle) these exceptions.</li><li><strong>Unchecked exceptions: </strong>An unchecked exception is an exception that occurs at the time of execution, also known as <strong>runtime exceptions</strong>. These include programming bugs, such as logic errors or improper use of an API. Runtime exceptions are ignored at the time of compilation.</li></ul><h4>Try/Catch and Finally</h4><ul><li>With the help of try/catch block we are able to catch exceptions.</li><li>A try/catch block is placed around the code that might generate an exception.</li><li>Code within a try/catch block is referred to as protected code.</li></ul><pre>try {<br>   // Protected code<br>} catch (ExceptionName e1) {<br>   // Catch block<br>}</pre><ul><li><strong>Definition: Finally block</strong> always executes, irrespective of occurrence of an Exception.</li><li><strong>Rule: </strong>It’s placed after try or catch block.</li><li><strong>Benefit: </strong>Using a finally block allows you to run any cleanup-type statements that you want to execute, no matter what happens in the protected code.</li></ul><pre>try{<br>   // Protected code<br>}catch(IOException | SQLException e){//multiple exception syntax<br>   // Catch block<br>}finally{<br>   // The finally block always executes.<br>}</pre><h3>Inner Class</h3><ul><li><strong>Definition: </strong>Inner class is <strong>a class inside a class</strong>.</li><li>Inner classes can be defined inside another class (nested inner class) or another classes’ method (method local inner class).</li></ul><pre>class OuterClass {<br>   class NestedInnerClass {<br>   }<br>}</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/920/1*rpTHOBxhcnhSfyg3OkW7zA.png" /><figcaption>Nested Class Types</figcaption></figure><ul><li><strong>Anonymous inner class</strong> is a <strong>nameless inner class</strong>. An inner class declared without a class name is known as an <strong>anonymous inner class</strong>.</li><li>In case of anonymous inner classes, we declare and instantiate them at the same time. Generally, they are used whenever you need to override the method of a class or an interface.</li></ul><pre>AnonymousInnerClass anonInner = new AnonymousInnerClass() {<br>   public void my_method() {<br>      // ...<br>   }   <br>};</pre><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d913a1815588ab39c0b993fe8903b4e5/href">https://medium.com/media/d913a1815588ab39c0b993fe8903b4e5/href</a></iframe><ul><li><strong>Nested Inner classes </strong>vs<strong> Anonymous Inner Classes</strong>: If we’re going to use a method only once, there’s no need to write it as a seperate class. Also we can prevent inner class to be used elsewhere. On the other hand, we can use nested inner class if we’re going to use it in other places (<a href="https://stackoverflow.com/questions/714602/anonymous-vs-named-inner-classes-best-practices">source</a>).</li></ul><h3>Java 8 Features</h3><h4>Functional Interface</h4><ul><li>Functional interfaces are also called <em>Single Abstract Method interfaces (SAM Interfaces)</em>. As name suggest, they <strong>permit exactly one abstract method</strong> inside them. Java 8 comes with @FunctionalInterface annotation to ensure the restrictions of functional interfaces are applied. Also, since default methods are not abstract you’re <em>free to add </em>default<em> methods</em> to your functional interface as many as you like. With default methods we can define default implementation of a method in an interface, so classes that extend the interface can directly use the method without providing any extra implementation information. Details are provided in the Default methods section.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/916801122f9e4ad74ce646d19505f28f/href">https://medium.com/media/916801122f9e4ad74ce646d19505f28f/href</a></iframe><h4>Lambda Expressions</h4><ul><li><strong>Definition:</strong> A lambda expression (or function) is just an anonymous function, i.e., a function with no name and without being bounded to an identifier.</li><li><strong>Syntax:</strong> (parameters) -&gt; expression, (parameters) -&gt; { statements; }, () -&gt; expression</li><li><strong>Benefits:</strong> Lambdas provide more brief and readable code. It eliminates the need of creating anonymous classes. The comparison will be provided after the examples.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*GtnVgMDXOlkCHLmf.jpeg" /></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a0b062f211b4ecd6f9dcd287357a7043/href">https://medium.com/media/a0b062f211b4ecd6f9dcd287357a7043/href</a></iframe><ul><li>Anonymous class vs Lambda expression: Lambdas eliminate the need of writing anonymous classes. Below is the comparison of the codes(<a href="https://stackoverflow.com/questions/33492203/what-are-the-advantages-of-using-lambda-expressions-in-java-8">source</a>):</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cdd6d7e31590ed9822e96c3c24a36e99/href">https://medium.com/media/cdd6d7e31590ed9822e96c3c24a36e99/href</a></iframe><h4><strong>Default methods</strong></h4><p>Java 8 allows you to add non-abstract methods in interfaces. These methods must be declared default methods. Default methods were introduces in java 8 to enable the functionality of lambda expression. Default methods enable you to add new functionality to the interfaces of your libraries and ensure binary compatibility with code written for older versions of those interfaces.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ef1e50acf0f943769eebf4ac5206eacb/href">https://medium.com/media/ef1e50acf0f943769eebf4ac5206eacb/href</a></iframe><p>If a class willingly wants to customize the behavior of move() method then it can provide it’s own custom implementation and override the method.</p><h4><strong>Streams API</strong></h4><p>A stream is a sequence of objects that supports various methods which can be pipelined to produce the desired result.</p><p><strong>Intermediate Operations</strong></p><ol><li><strong>map: </strong>The map method is used to <strong>map the items</strong> in the collection <strong>to other objects</strong> according to the Function passed as argument.</li><li><strong>filter:</strong> The <strong>filter </strong>method is used to select elements as per the Predicate passed as argument.</li><li><strong>sorted:</strong> The sorted method is used to <strong>sort </strong>the stream.</li></ol><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/eddfc4666cb7cc1af487b74e2870850e/href">https://medium.com/media/eddfc4666cb7cc1af487b74e2870850e/href</a></iframe><p><strong>Terminal Operations</strong></p><ol><li><strong>collect:</strong> The collect method is used to return the result of the intermediate operations performed on the stream.</li><li><strong>forEach:</strong> The forEach method is used to iterate through every element of the stream.</li><li><strong>reduce:</strong> The reduce method is used to reduce the elements of a stream to a single value.<br>The reduce method takes a BinaryOperator as a parameter.</li></ol><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4263c3e678bfaa534801ad6561d78c55/href">https://medium.com/media/4263c3e678bfaa534801ad6561d78c55/href</a></iframe><h4>Optional</h4><ul><li>To prevent NullPointerExceptionand too many null checks, Optional class is introduced.</li><li>Optional class has multiple methods to manage null’s. empty() method creates an empty optional. ofNullable(myVariable) method returns non-empty optional if a value (myVariale) is present, empty optional if the value is absent. of() converts a variable to an optional, does not accept null values.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ac4806b65ca74aef3a8f254a2b23d63b/href">https://medium.com/media/ac4806b65ca74aef3a8f254a2b23d63b/href</a></iframe><h4>Collections</h4><p>A Java collection framework provides an architecture to store and manipulate a group of objects.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/854/1*5zgVjvXRlfMGBE-zQKmthQ.png" /><figcaption>Java Collection Hierarchy (java.util)</figcaption></figure><p><strong>Iterable</strong></p><ul><li>The Iterable interface is the root interface for all the collection classes.</li><li>Contains only one abstract method: Iterator&lt;T&gt; iterator()</li></ul><p><strong>Collection</strong></p><ul><li>The Collection interface extends the Iterable interface and therefore all the subclasses of Collection interface also implement the Iterable interface.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qgSOMOFGISeaLYDBi4ovng.png" /><figcaption>Methods of Collection interface</figcaption></figure><p><strong>List</strong></p><ul><li>List is ordered.</li><li>We can store duplicates.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c658f2e96fb6bf9c3b65cc0da98e547f/href">https://medium.com/media/c658f2e96fb6bf9c3b65cc0da98e547f/href</a></iframe><p><strong>ArrayList</strong></p><ul><li>ArrayList is ordered and non-synchronized.</li><li>Uses a dynamic array to store duplicate elements and different data types.</li><li>Elements can be randomly accessed.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1a56a5c97aee016d9f41c84c1820db0a/href">https://medium.com/media/1a56a5c97aee016d9f41c84c1820db0a/href</a></iframe><p><strong>LinkedList</strong></p><ul><li>LinkedList is ordered and non-synchronized.</li><li>It uses a doubly linked list.</li><li>Data manipulation is fast (no shifting required).</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/49d6442c51ad1c3328026c149721fbc0/href">https://medium.com/media/49d6442c51ad1c3328026c149721fbc0/href</a></iframe><p><strong>Vector</strong></p><ul><li>Vector uses a dynamic array to store the data elements.</li><li>It is similar to ArrayList. However, It is synchronized and contains many methods that are not the part of Collection framework.</li></ul><p><strong>Stack</strong></p><ul><li>The stack is the subclass of Vector.</li><li>It implements the LIFO (last-in-first-out) data structure, stack.</li><li>The stack contains all of the methods of Vector class and also provides its methods like push(), pop() and peek() which defines its properties.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4f374ad31fca38a4899fab23bc0e705b/href">https://medium.com/media/4f374ad31fca38a4899fab23bc0e705b/href</a></iframe><p><strong>Queue</strong></p><ul><li>Queue interface maintains the FIFO (first-in-first-out) order.</li><li>It is similar to an ordered list.</li><li>The implementations are PriorityQueue, Deque, and ArrayDeque.</li></ul><p><strong>PriorityQueue</strong></p><ul><li>Processes elements with respect to their priorities</li><li>PriorityQueue doesn’t allow null values to be stored in the queue.</li><li>PriorityQueue iterator does not traverse in any particular order, for ordered purpose we should use poll() method (<a href="https://stackoverflow.com/questions/37459143/how-to-set-priority-in-priority-queue">details</a>).</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/69f37b9ee482af60746da72b41fa7da0/href">https://medium.com/media/69f37b9ee482af60746da72b41fa7da0/href</a></iframe><p><strong>ArrayDeque</strong></p><ul><li>Implements the Deque(double ended queue) interface.</li><li>Unlike queue, we can add or delete the elements from both ends.</li><li>ArrayDeque is faster than ArrayList and Stack and has no capacity restrictions.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cdcfbf5f8fe2d9c92aa54f0d0431c55e/href">https://medium.com/media/cdcfbf5f8fe2d9c92aa54f0d0431c55e/href</a></iframe><p><strong>Set</strong></p><ul><li>Unordered set of elements, duplicates not allowed</li><li>We can store at most one null value in Set</li><li>Set is implemented by HashSet, LinkedHashSet, and TreeSet.</li></ul><p><strong>HashSet</strong></p><ul><li>Uses hash table for storage</li><li>Hashing is used to store elements</li><li>Each element is unique</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/75067c936ab23abcedafe84d50b8fd18/href">https://medium.com/media/75067c936ab23abcedafe84d50b8fd18/href</a></iframe><p><strong>LinkedHashSet</strong></p><ul><li>Represents LinkedList implementation of Set Interface.</li><li>Insertion order is maintained and null elements are permitted.</li><li>Each element is unique.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bed64bc349efcbf1ef323511d4008382/href">https://medium.com/media/bed64bc349efcbf1ef323511d4008382/href</a></iframe><p><strong>SortedSet Interface</strong></p><ul><li>Elements are sorted in the increasing (ascending) order.</li><li>Can be instantiated as SortedSet&lt;datatype&gt; set = new TreeSet();</li></ul><p><strong>TreeSet</strong></p><ul><li>Java TreeSet class implements the Set interface that uses a tree for storage.</li><li>Like HashSet, TreeSet also contains unique elements. However, the access and retrieval time of TreeSet is quite fast.</li><li>The elements in TreeSet stored in ascending order.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8ff1d3a58d4f8db0da521adaee5db1eb/href">https://medium.com/media/8ff1d3a58d4f8db0da521adaee5db1eb/href</a></iframe><h4><strong>DateTime API changes</strong></h4><p>A big part of developer community has been complaining about Date and Calendar classes. Reasons were many such as hard to understand, hard to use and not flexible. Date class has even become obsolete and java docs suggest to use Calendar class instead of Date class. And on top of all, Date comparison is buggy and I have also faced such issue in past.</p><p>Date class has even become obsolete. The new classes intended to replace Date class are LocalDate, LocalTime and LocalDateTime.</p><ul><li>LocalDate class represents a date. There is no representation of a time or time-zone.</li><li>LocalTime class represents a time. There is no representation of a date or time-zone.</li><li>LocalDateTime class represents a date-time. There is no representation of a time-zone.</li><li>For representing the specific timestamp ant any moment, the class needs to be used is Instant. The Instant class represents an instant in time to an accuracy of nanoseconds.</li><li>Duration class is a whole new concept brought first time in java language. It represents the time difference between two time stamps.</li></ul><h3>Source</h3><ul><li><a href="https://www.tutorialspoint.com/java/java_tutorial.pdf">https://www.tutorialspoint.com/java/java_tutorial.pdf</a></li><li><a href="http://www.tutorialspoint.com/java/java_quick_guide.htm">http://www.tutorialspoint.com/java/java_quick_guide.htm</a></li><li><a href="https://www.geeksforgeeks.org/differences-jdk-jre-jvm/">https://www.geeksforgeeks.org/differences-jdk-jre-jvm/</a></li><li><a href="https://www.tutorialspoint.com/java/java_encapsulation.htm">https://www.tutorialspoint.com/java/java_encapsulation.htm</a></li><li><a href="https://www.guru99.com/java-data-abstraction.html">https://www.guru99.com/java-data-abstraction.html</a></li><li><a href="https://www.javatpoint.com/aggregation-in-java">https://www.javatpoint.com/aggregation-in-java</a></li><li><a href="https://www.tutorialspoint.com/Association-Composition-and-Aggregation-in-Java">https://www.tutorialspoint.com/Association-Composition-and-Aggregation-in-Java</a></li><li><a href="https://www.javatpoint.com/inheritance-in-java">https://www.javatpoint.com/inheritance-in-java</a></li><li><a href="https://www.javatpoint.com/runtime-polymorphism-in-java">https://www.javatpoint.com/runtime-polymorphism-in-java</a></li><li><a href="http://www.cs.utexas.edu/~cannata/cs345/Class%20Notes/14%20Java%20Upcasting%20Downcasting.htm">http://www.cs.utexas.edu/~cannata/cs345/Class%20Notes/14%20Java%20Upcasting%20Downcasting.htm</a></li><li><a href="http://mindprod.com/jgloss/upcasting.html">http://mindprod.com/jgloss/upcasting.html</a></li><li><a href="http://mindprod.com/jgloss/downcasting.html">http://mindprod.com/jgloss/downcasting.html</a></li><li><a href="https://www.javatpoint.com/corejava-interview-questions">https://www.javatpoint.com/corejava-interview-questions</a></li><li><a href="https://www.tutorialspoint.com/java/java_files_io.htm">https://www.tutorialspoint.com/java/java_files_io.htm</a></li><li><a href="https://stackoverflow.com/questions/447898/what-is-object-serialization">https://stackoverflow.com/questions/447898/what-is-object-serialization</a></li><li><a href="https://www.tutorialspoint.com/java/java_exceptions.htm">https://www.tutorialspoint.com/java/java_exceptions.htm</a></li><li><a href="https://www.tutorialspoint.com/Flow-control-in-a-try-catch-finally-in-Java">https://www.tutorialspoint.com/Flow-control-in-a-try-catch-finally-in-Java</a></li><li><a href="https://www.geeksforgeeks.org/inner-class-java/">https://www.geeksforgeeks.org/inner-class-java/</a></li><li><a href="https://www.tutorialspoint.com/java/java_innerclasses.htm">https://www.tutorialspoint.com/java/java_innerclasses.htm</a></li><li><a href="https://www.javatpoint.com/interface-in-java">https://www.javatpoint.com/interface-in-java</a></li><li><a href="https://www.javatpoint.com/abstract-class-in-java">https://www.javatpoint.com/abstract-class-in-java</a></li><li><a href="https://www.geeksforgeeks.org/stream-in-java/">https://www.geeksforgeeks.org/stream-in-java/</a></li><li><a href="https://kodedu.com/2014/10/java-8-optional-yeniligi/">https://kodedu.com/2014/10/java-8-optional-yeniligi/</a></li><li><a href="https://www.edureka.co/blog/java-collections/#framework">https://www.edureka.co/blog/java-collections/#framework</a></li><li><a href="https://www.javatpoint.com/collections-in-java">https://www.javatpoint.com/collections-in-java</a></li><li><a href="https://howtodoinjava.com/java-8-tutorial/#date-time">https://howtodoinjava.com/java-8-tutorial/</a></li><li><a href="https://howtodoinjava.com/java8/date-and-time-api-changes-in-java-8-lambda/">https://howtodoinjava.com/java8/date-and-time-api-changes-in-java-8-lambda/</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=962cbf62b761" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>