<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Igor Kliushnikov on Medium]]></title>
        <description><![CDATA[Stories by Igor Kliushnikov on Medium]]></description>
        <link>https://medium.com/@kliushnikov?source=rss-af9c27f30570------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*qcjDJk1w4lrkOPZU-m2RxQ.jpeg</url>
            <title>Stories by Igor Kliushnikov on Medium</title>
            <link>https://medium.com/@kliushnikov?source=rss-af9c27f30570------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 03:20:46 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@kliushnikov/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Temporary Storage for Kubernetes Pods]]></title>
            <link>https://itnext.io/temporary-storage-for-kubernetes-pods-f8330ad8db88?source=rss-af9c27f30570------2</link>
            <guid isPermaLink="false">https://medium.com/p/f8330ad8db88</guid>
            <category><![CDATA[persistent-volume]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[storage]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[containers]]></category>
            <dc:creator><![CDATA[Igor Kliushnikov]]></dc:creator>
            <pubDate>Mon, 11 Jul 2022 21:11:08 GMT</pubDate>
            <atom:updated>2022-07-11T21:11:08.769Z</atom:updated>
            <content:encoded><![CDATA[<h4><strong><em>Or emptyDir vs. container File System</em></strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*zZi1DSgvvyLmc706s9R6Rg.jpeg" /></figure><p>Kubernetes applications might need some temporary storage that could be discarded after a container is stopped/removed. Applications might use FS to store intermediate results, checkpoint a process, download temporary files, cache results of requests, and others. For this purpose you can use Kubernetes emptyDir or containers local storage.</p><h4>Containers Storage</h4><p>Once your container runs, you already have a file system shipped together with the container image. This file system is consistent and read-only: every time you start a new container, you get the same files in the exact locations. However, being inside a container is not necessarily force you to keep anything untouched: you can make changes to existing files or create new ones. It is possible, as containers are working on top of the <a href="https://en.wikipedia.org/wiki/UnionFS">layered file system</a> with <a href="https://en.wikipedia.org/wiki/Copy-on-write">copy-on-write (COW)</a>.</p><p>Any changes in a running container are usually lost after the container restarts. You can read more about container FS and storage-drivers that implement it in the official <a href="https://docs.docker.com/storage/storagedriver/">docker/storage</a> documentation.</p><p>All the above is also valid for Kubernetes Pods that wrap regular containers. If you need a bit of temporary storage, you do not need to do anything: just start the pod and do whatever you want.</p><p>While the container’s temporary storage could be a fit for a subset of use-case, it also has the following attributes:</p><ul><li><strong>containers writable layer is bound to the containers’ lifecycle</strong> — once the container is gone, the writable layer follows it and gets destroyed</li><li>getting anything from this writable layer could be tricky for other processes</li><li>to create a usable file system from container layers, a storage driver has to do extra work, which has its performance penalties</li></ul><p>So here comes the …</p><h4>Kubernetes EmptyDir</h4><p>First of all, emptyDir is one of the volume types. An emptyDir volume is first created when a Pod is assigned to a node, and initially, it is … (you won’t believe it!) empty.</p><p>This volume type can address some of the limitations listed above:</p><ul><li>emptyDir can be shared between all containers inside the pod</li><li><strong>The volume follows the Pod lifecycle</strong> — it is created together with the pod and removed only after the Pod is removed.</li><li>If a container crashes (an error, out of memory, or whatever reason), it is removed with its temporary writable storage layer. If Pod backs the container, kubelet might recreate a new container on the same node, but the container FS will be the same as in the container image without all the temporary changes you made there. But container crashing does NOT remove the pod from the node, so the data in an emptyDir volume is safe across container restarts. This allows, for example, checkpointing a long computation for recovery from unexpected failures.</li><li>emptyDir is a Linux-native FS that is faster and does not have penalties of container storage-driver.</li></ul><p>Additionally, emptyDir allows you to define the maximum size and use memory as a storage medium. Imagine all of the containers write to the host file system without limits, so the only rogue container can consume the whole disk space and cause issues to all the neighbors. Honestly, sizeLimit does not entirely eliminate such problems, but it can give at least some control over space usage.</p><p>Here is a copy of the api-reference for mentioned before emptyDir parameters:</p><blockquote><em>medium<br>What type of storage medium should back this directory. The default is &quot;&quot; which means to use the node&#39;s default medium. Must be an empty string (default) or Memory. More info: </em><a href="https://kubernetes.io/docs/concepts/storage/volumes#emptydir"><em>https://kubernetes.io/docs/concepts/storage/volumes#emptydir</em></a></blockquote><blockquote><em>sizeLimit<br>Total amount of local storage required for this EmptyDir volume. The size limit is also applicable for memory medium. The maximum usage on memory medium EmptyDir would be the minimum value between the SizeLimit specified here and the sum of memory limits of all containers in a pod. The default is nil which means that the limit is undefined. More info: </em><a href="http://kubernetes.io/docs/user-guide/volumes#emptydir"><em>http://kubernetes.io/docs/user-guide/volumes#emptydir</em></a></blockquote><h4>How to choose?</h4><p>In my opinion, you have to go with emptyDir if:</p><ul><li>you need to share a part of file-system between containers inside the same pod</li><li>applications work a lot with temporary storage, and performance is crucial</li><li>you need to checkpoint your process in a way that it can survive unexpected container crashes</li><li>you need to control how much disk space a single pod can use</li></ul><p>In all the other cases, just do nothing and stick with the container-native writable layer.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f8330ad8db88" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/temporary-storage-for-kubernetes-pods-f8330ad8db88">Temporary Storage for Kubernetes Pods</a> was originally published in <a href="https://itnext.io/">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Stateful Applications in Kubernetes]]></title>
            <link>https://itnext.io/stateful-applications-in-kubernetes-808a60bc109?source=rss-af9c27f30570------2</link>
            <guid isPermaLink="false">https://medium.com/p/808a60bc109</guid>
            <category><![CDATA[stateful-applications]]></category>
            <category><![CDATA[pods]]></category>
            <category><![CDATA[persistent-volume]]></category>
            <category><![CDATA[statefulsets]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Igor Kliushnikov]]></dc:creator>
            <pubDate>Thu, 05 Nov 2020 20:37:00 GMT</pubDate>
            <atom:updated>2020-11-05T20:37:00.437Z</atom:updated>
            <content:encoded><![CDATA[<p>Kubernetes is well-known for managing stateless services, while it is not limited to them. The following article gathers my notes about running Stateful Applications in Kubernetes. If you find any inaccuracy, want to share your experience or want to propose a change — do not hesitate to contact me ;)</p><p>Writing up everything about stateful application in a single place is too ambitious goal. Therefore here you can find short statements with reference to pages with more details about each particular statement.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iyML8NGyguGVJHBO6zl72A.jpeg" /><figcaption><a href="https://ghibli.fandom.com/wiki/Susuwatari">Susuwatari </a>carry their assets © <a href="http://www.ghibli.jp/gallery/chihiro014.jpg">ghibli</a></figcaption></figure><p>Stateful application — is the one, which uses local file system to preserve own data. There are two ways to run such applications in Kubernetes:</p><ul><li>StatefulSets — Kubernetes object, which manages set of pods and provides guarantees about the ordering and uniqueness of these pods.</li><li><a href="https://kubernetes.io/docs/tasks/run-application/run-single-instance-stateful-application/">Single-Instance stateful Application</a> — Deployment object with a single replica and attached volume. Personally, I can not imagine any valid use-case which makes this approach more valuable than StatefulSet with a single replica.</li></ul><p>I think, to run Stateful Applications in Kubernetes I must:</p><ul><li>understand what are StatefulSets and how to use them;</li><li>how persistence layer is represented in Kubernetes;</li><li>to properly handle application state, I have to understand pod lifecycle.</li></ul><h4>1. StatefulSets</h4><p>1.1. <a href="https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/"><em>StatefulSets </em></a>provides to each pod in it two stable unique identities:</p><ul><li><em>Network Identity</em>: regardless of amount of restarts, pod is assigned with the same DNS name. Note, IP addresses might be still different, so consumers should depend on DNS name (or watch for changes and update internal cache)</li><li><em>Storage Identity</em>: the same Network Identity always receives the same instance of Storage, regardless of which node it’s rescheduled on.</li></ul><p>1.2. Each StatefulSet should have associated Kubernetes Service instance to manage its network-identities. If StatefulSet does not need Service with internal or external IP, it can use <a href="https://kubernetes.io/docs/concepts/services-networking/service/#headless-services">Headless Service</a></p><p>Each pod receives DNS name according to the pattern: {statefulset_name}-{0..N-1}.{service_name}.{namespace_name}.svc.cluster.local. For example, cassandra-0.cassandra-headless.dev-namespace.svc.cluster.local.</p><p>1.3. StatefulSets are built around Safety First principal — in each controversial/questionable situation Kubernetes team chooses data safety. For example, by default, when StatefulSet deleted or scaled, associated Volumes are not deleted.</p><p>1.4. As StatefulSets works with data, we should be careful while stopping pod instances: give enough time to persist data from memory to disk, complete started operation, etc. There still might be valid reasons to perform<a href="https://kubernetes.io/docs/tasks/run-application/force-delete-stateful-set-pod/"> force deletion</a>, for example, when <a href="https://medium.com/tailwinds-navigator/kubernetes-tip-how-statefulsets-behave-differently-than-deployments-when-node-fails-d29e36bca7d5">Kubernetes Node fails</a>.</p><p>1.5. <a href="https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#deployment-and-scaling-guarantees">Deployment and upgrades</a></p><p>By default, StatefulSet rolls out sequentially: created in order from 0 to N-1; terminated from N-1 to 0. It could be changed to Parallel mode, but mostly not needed</p><p>An interesting property, deployment parameter is <a href="https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#partitions">Partitions</a>: it enables canary releases from the box.</p><p>1.6. I do recommend to go through these fantastic tutorials:</p><ul><li><a href="https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set/">StatefulSet Basics</a>, which is focused on basic operations</li><li><a href="https://kubernetes.io/docs/tutorials/stateful-application/zookeeper/">Running ZooKeeper</a>, which considers a real-world scenario and shows more advanced topics, for example, pod-affinity and disruption-budget.</li></ul><h4>2. Persistent Volumes</h4><p>2.1. Main objects:</p><ul><li><a href="https://kubernetes.io/docs/concepts/storage/persistent-volumes/">PersistentVolume</a> (PV) — a piece of storage in the cluster with reference to physical data location</li><li><a href="https://kubernetes.io/docs/concepts/storage/persistent-volumes/#persistentvolumeclaims">PersistentVolumeClaim </a>(PVC) — a request for storage by a user. Often used to map StatefulSets to PVs</li><li><a href="https://kubernetes.io/docs/concepts/storage/storage-classes/">StorageClass</a> defines parameters of PV: provisioner (AzureDisk, AWSElasticBlockStore, etc), reclaim policy, allows resizing, etc</li></ul><blockquote>It is similar to a Pod. Pods consume node resources and PVCs consume PV resources. Pods can request specific levels of resources (CPU and Memory). Claims can request specific size and access modes</blockquote><p>2.2. Volume provisioning:</p><ul><li><em>Static</em>: administrator pre-creates Volumes and PVs, then PVC consumes existing PV.</li><li><em>Dynamic</em>: cluster tries to provision a volume when PVC is created.</li></ul><p>When no <em>StorageClass </em>is defined — cluster uses DefaultStorageClass</p><p>2.3. <a href="https://kubernetes.io/docs/concepts/storage/persistent-volumes/#reclaiming">Reclaiming</a></p><p>There are following reclaim policies:</p><ul><li><em>Retain</em>: if PVC is deleted, PV stays, but is not available for other users (because there might be data!). Operator should make a decision what to do with volume: assign it to a new pod as is, cleanup it before assignment or just delete.</li><li><em>Delete</em>: removes both the <em>PersistentVolume </em>object from Kubernetes, as well as the associated storage asset in the external infrastructure (not all providers support it)</li></ul><p>2.4. Volume plugins:</p><ul><li><em>in-tree</em>: source-code is inside the main Kubernetes git-repository</li><li><em>out-of-tree</em>: plugins, that enable storage vendors to create custom storage plugins without adding their plugin source code to the Kubernetes repository: <a href="https://github.com/container-storage-interface/spec/blob/master/spec.md">Container Storage Interface</a> (CSI) and FlexVolume</li></ul><p>Likely, CSI is Kubernetes future, but not everyone supports it yet (for example, at the moment Amazon EBS <a href="https://docs.aws.amazon.com/eks/latest/userguide/ebs-csi.html">CSI driver is in beta</a>, Azure has <a href="https://docs.microsoft.com/en-us/azure/aks/azure-disk-csi">CSI drivers only in preview</a>). Also, CSI Volumes could be <a href="https://kubernetes.io/docs/concepts/storage/volume-pvc-datasource/">Cloned </a>at PVC creation time.</p><p>2.5. <a href="https://kubernetes.io/docs/concepts/storage/volume-snapshots">Volume Snapshots</a> (beta in 1.17)</p><ul><li>Like <em>PersistentVolume </em>and <em>PersistentVolumeClaim </em>— <em>VolumeSnapshotContent </em>and <em>VolumeSnapshot</em></li><li>could be scheduled</li></ul><h4>3. Pod lifecycle</h4><p>3.1. Very often, inside StatefulSet we run more sophisticated applications, that require warm-up, leader-election, data processing, etc. As an outcome, pods in StatefulSet might have special requirements on how to start/stop, on when to receive traffic, and others.</p><p>I think, once you have to use stateful-sets, you have to attentively read <a href="https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/">Pod Lifecycle doc</a>, which explains all possible states and transitions between them</p><p>3.2. <a href="https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#container-probes">Probes</a>:</p><ul><li>startupProbe indicates when pod is ready to receive other probes. The probe is executed only at starting time, while the next two are executed during the entire pod lifetime</li><li>livenessProbe indicates whether the container is running. When probe fails — pod is restarted.</li><li>readinessProbe indicates whether the pod should receive Ingress traffic. When probe fails — pod is removed from Endpoints (e.g. iptables)</li></ul><p>3.3. <a href="https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks">Hooks</a></p><ul><li>PostStart the hook is executed immediately after a container is created (might be even before container entrypoint!)</li><li>PreStop the hook is called immediately before a container is terminated. Could be useful to prepare pod for termination: stop receiving new requests, dump in-memory data to disk, etc.</li></ul><p>3.4. <a href="https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-termination">Pod termination</a></p><p>When API server received a command to remove a Pod it:</p><ul><li>updates <em>Endpoints</em>, which leads to iptables, Ingress, and DNS changes</li><li>sends command to <em>kubelet </em>to stop the pod, which invokes <em>preStop </em>hook and waits <em>terminationGracePeriodSeconds </em>seconds (30 seconds by default)</li></ul><p>Very detailed explanation of graceful pod shutdown: <a href="https://learnk8s.io/graceful-shutdown.">https://learnk8s.io/graceful-shutdown.</a> It gives nice visualizations and at the end proposes an interesting pattern to work with long-running pods: during a new version release, instead of updating the same deployment, create a new one (I do not want to repeat article content, if you are curious — go and check it out).</p><p>3.5. <a href="https://kubernetes.io/docs/concepts/workloads/pods/disruptions/">Disruptions </a>could be:</p><ul><li><em>voluntary</em>: delete deployment or pod, scale, etc</li><li><em>involuntary</em>: failed VM, hardware failure, <a href="https://kubernetes.io/docs/tasks/administer-cluster/out-of-resource/">OOM</a></li></ul><p>To reduce disruptions:</p><ul><li>use resource quotas</li><li>replicate applications</li><li>use<a href="https://kubernetes.io/docs/tasks/run-application/configure-pdb/"> disruption budgets</a></li></ul><p>With Disruption Budgets you can:</p><ul><li>define minimum-available or maximum-unavailable amount of instances;</li><li>prevent any re-scheduling unless disruption-budget is removed</li></ul><h4>4. Miscellaneous</h4><p>The following topics do not belong to any group above, but still worth mentioning</p><p>4.1. Initially, StatefulSets were named <a href="https://github.com/kubernetes/community/blob/master/contributors/design-proposals/apps/stateful-apps.md#history">PetSets</a></p><p>4.2.</p><blockquote>Create a Service before its corresponding backend workloads</blockquote><ul><li>k8s provides env-vars with service host/port to each container at creation time, if a pod is created before service — no env-vars</li><li>service creation leads to a new DNS record. If pod queries DNS before service is created — non-existing result is cached for a while</li></ul><p>4.3. imagePullPolicy: Always does not literally pull the image every time. It queries Container Registry every time and pulls image only when (a) image is not in local VM cache or (b) cached locally image digest is different from what is in Container Registry.</p><p>4.4.</p><blockquote>Volumes that support ownership management are modified to be owned and writable by the GID specified in fsGroup</blockquote><p>Thus, instead of writing custom change-ownership scripts, you might use securityContext.fsGroup</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=808a60bc109" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/stateful-applications-in-kubernetes-808a60bc109">Stateful Applications in Kubernetes</a> was originally published in <a href="https://itnext.io/">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Starter-kit for Elasticsearch operations]]></title>
            <link>https://kliushnikov.medium.com/starter-kit-for-elasticsearch-operations-d0f9d1427e24?source=rss-af9c27f30570------2</link>
            <guid isPermaLink="false">https://medium.com/p/d0f9d1427e24</guid>
            <category><![CDATA[logs]]></category>
            <category><![CDATA[elk]]></category>
            <category><![CDATA[elasticsearch]]></category>
            <category><![CDATA[operations]]></category>
            <dc:creator><![CDATA[Igor Kliushnikov]]></dc:creator>
            <pubDate>Sun, 20 Sep 2020 13:41:19 GMT</pubDate>
            <atom:updated>2020-09-28T08:22:02.498Z</atom:updated>
            <content:encoded><![CDATA[<p>Elasticsearch is that kind of product, which is very easy to start, but then you’re getting one hundred and one questions about it on the go. Personally, I spent weeks jumping over documentation (it’s excellent, but it’s massive), querying API, reading blogs — everything just to understand what the hell is going on. Cluster unexpectedly (for me) becomes red, or just rejects new data, or one node going down once a day.</p><p>The following article is my side-notes after a year of running Elasticsearch cluster to store a mid-size distributed project <strong>logs</strong>.</p><p>The article <strong>does not </strong>teach how to install Elasticsearch cluster. Official <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html">installation </a>and <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started-install.html">getting-started</a> posts are pretty good (there is also <a href="https://www.elastic.co/elastic-cloud-kubernetes">kubernetes operator</a>).</p><p>The article expects that you have already Elasticsearch cluster running and now you have a question about it. Here you can find:</p><ul><li>high-level explanation of Elasticsearch concepts;</li><li>a few clues how to size nodes;</li><li>references to other posts and parts of official documentation, which should be useful in particular context;</li><li>proposed troubleshooting guide.</li></ul><p>I wish, I had something like this when I started. Thus, I hope to make someone’s life a bit easier.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*l-TEHGS-aSVZxHTc5m1xsA.jpeg" /><figcaption>©<a href="https://www.loc.gov/pictures/item/2011661555/"> Library of congress</a></figcaption></figure><h3>A step back. What is Elasticsearch?</h3><p>Elasticsearch is distributed search engine. Originally it was built for a full-text search with miscellaneous tokenizers and text-analyzers. Later, turned out that it also can do a good job in storing and querying logs.</p><p>Important logs feature:</p><ul><li>In most cases customer is interested in text (log message) and full-text search on top of it;</li><li>Often, logs have attached attributes, to express context of logged event, that are important to be able to filter. For example, event-origin (like host name).</li><li>Mostly, logs are stored for a fixed amount of days and then can be discarded.</li><li>Amount of logs might vary and likely would not fit to a single machine</li></ul><p>Elasticsearch can satisfy all of these requirements and it’s very easy to start with (but not that easy to maintain later).</p><h3>Indices and shards</h3><p>All the data in Elasticsearch is organized in <strong><em>indices</em></strong>. Index is a namespace, which consists of real unit with data — <strong><em>shards</em></strong>. Every shard is an instance of <a href="https://en.wikipedia.org/wiki/Apache_Lucene">Lucene</a>.</p><ul><li>every index has at least one primary shard and 0..N replicas;</li><li>a primary shard can technically contain up to Integer.MAX_VALUE-128 documents;</li><li>data structure — <a href="https://www.elastic.co/guide/en/elasticsearch/guide/current/inverted-index.html">Inverted Index</a>.</li></ul><p>The number of primary shards in an index is fixed at the time that an index is created, but the number of replica shards can be changed at any time.</p><h3>Elasticsearch cluster</h3><p>Elasticsearch can run as a single instance or in a cluster mode. If Elasticsearch instances form a cluster, they might have different roles:</p><ul><li><a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-node.html#master-node">master</a>: a node that has node.master configuration property set to true (it’s default value), which makes it eligible to be <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-discovery.html">elected as the <strong>master</strong> node</a>, which controls the cluster.</li><li><a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-node.html#data-node">data</a> (hot and warm): a node that has node.dataconfiguration property set to true (it’s default value too). Data nodes hold data and perform data related operations such as CRUD, search, and aggregations.</li><li><a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-node.html#node-ingest-node">ingest</a>: a node that has node.ingest configuration property set to true (it’s default value again). Ingest nodes are able to apply an <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/pipeline.html">ingest pipeline</a> to a document in order to transform and enrich the document before indexing.</li></ul><p>By default, Elasticsearch instance is Jack of all trades: it can be <em>master</em>, <em>data</em>, and <em>ingest</em> at the same time.</p><p>Each Elasticsearch instance has a <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-discovery.html">discovery module</a>, which is responsible for finding nodes to form a cluster, electing a master, send state updates. If you have an issue with forming a cluster from individual nodes — discovery module configuration should be one of your first stops.</p><h4>Elasticsearch Cluster State</h4><p>Maybe, the most important part of Elasticsearch cluster is <strong>Cluster State</strong>:</p><ul><li>stored in memory of every node;</li><li>only master node is allowed to update it;</li><li>includes cluster-level settings, nodes, indices (settings, mappings), shards</li><li>state updates are serializable. when <em>master</em> gets a change request, it sends changeset to followers and waits for acknowledges; only when configured minimum of acknowledges is received, change is applied and sent to all the nodes.</li></ul><p>State API supports multiple parameters: <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-state.html">official documentation</a>.</p><h4>Elasticsearch Cluster Health</h4><p>Cluster Health can be queried with GET _cluster/health, where:</p><ul><li><em>Green</em> means everything is good (cluster is fully functional)</li><li><em>Yellow</em>: all data is available but some replicas are not yet allocated (cluster is still fully functional)</li><li><em>Red</em>: some data is not available for whatever reason (cluster is partially functional, but continues serving search requests from the available shards)</li></ul><p>Health API also supports multiple parameters: <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-health.html">official documentation</a>.</p><p>Very simplified, you can think about cluster health in the following way:</p><ol><li>cluster health depends on amount of healthy indices;</li><li>index health depends on shards allocation status;</li><li>shard allocation depends on available resources of your ES cluster;</li><li>shard allocation happens when (a) index is created (b) when ES tries to rebalance shards (for example, when node leaves or joins the cluster).</li></ol><p>Often, if you suspect that your cluster is <em>not healthy</em> you have to find a shard, which fails to be allocated.</p><h3>Elasticsearch resource consumption</h3><p>Obviously, you have to monitor used disk space. Once ES has no disk space left — it rejects any new data. To keep size at more or less same level, old indices could be daily/weekly/monthly deleted from the cluster, which is responsibility of <a href="https://www.elastic.co/guide/en/elasticsearch/client/curator/current/index.html">Elasticsearch Curator</a> (separate application) or <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-settings.html">Index Lifecycle Management</a> (part of Elasticsearch if you have a commercial version).</p><p>But, maybe the most important thing to keep in mind: <strong>Elasticsearch State is kept in memory of each ES instance</strong>. This means, the bigger the state — more memory should be dedicated to each ES instance. If the state is too big — instances might crush with <em>OutOfMemory </em>exceptions (very common, if you are running Elastic in docker containers with set memory limits).</p><p>The size of Elasticsearch State is very influenced by the way the data is stored. Here are a few quotes from <a href="https://www.elastic.co/blog/how-many-shards-should-i-have-in-my-elasticsearch-cluster">ES blog</a> about sizing shards:</p><blockquote>Small shards result in small segments, which increases overhead. Aim to keep the average shard size between at least a few GB and a few tens of GB. For use-cases with time-based data, it is common to see shards between 20GB and 40GB in size.</blockquote><p>plus:</p><blockquote>As the overhead per shard depends on the segment count and size, forcing smaller segments to merge into larger ones through a <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-forcemerge.html">forcemerge</a> operation can reduce overhead and improve query performance. This should ideally be done once no more data is written to the index. Be aware that this is an expensive operation that should ideally be performed during off-peak hours.</blockquote><p>and finally:</p><blockquote>A good rule-of-thumb is to ensure you keep the number of shards per node below 20 per GB heap</blockquote><p>Each ingest-data request also use a subset of memory. Therefore, if the cluster has very little room after storing the entire state file, it can also go out of memory because of too many concurrent requests.</p><p>More details about sizing the heap:</p><ul><li><a href="https://www.elastic.co/blog/a-heap-of-trouble">A heap of Trouble</a></li><li><a href="https://www.elastic.co/guide/en/elasticsearch/guide/2.x/heap-sizing.html">Sizing the heap</a></li><li><a href="https://www.elastic.co/blog/why-am-i-seeing-bulk-rejections-in-my-elasticsearch-cluster">Why bulk requests are rejected</a></li><li><a href="https://www.elastic.co/guide/en/cloud/current/ec-metrics-memory-pressure.html">How memory pressure affect ES performance</a></li></ul><h3>Elasticsearch curator</h3><p>If you use open-source version of Elasticsearch without any license, <a href="https://www.elastic.co/guide/en/elasticsearch/client/curator/current/index.html">Elasticsearch Curator</a> is the easiest way to deal with daily ES routine: delete obsolete indices, optimize existing, etc.</p><p>There is no official Elasticsearch curator docker image yet:</p><blockquote>For now, your best bet is to build one yourself from the Dockerfile in the github repository. You could publish it yourself, or just host it in your local Docker repository. There are a few more steps before Curator will be published with its own Docker image, so I apologize for its absence.</blockquote><p>© from <a href="https://discuss.elastic.co/t/curator-docker-image/87507">Elastic discussion forum</a>.</p><p>Curator is often used at least to delete obsolete indices, but also it could be used to <a href="https://www.elastic.co/guide/en/elasticsearch/client/curator/current/close.html">close</a> and <a href="https://www.elastic.co/guide/en/elasticsearch/client/curator/current/forcemerge.html">forcemerge</a> indices, make <a href="https://www.elastic.co/guide/en/elasticsearch/client/curator/current/snapshot.html">snapshots</a>, etc.</p><p>For example, here is Curator config to delete indices older than 14 days with index name pattern logs-%Y-%m-%d:</p><pre>actions:</pre><pre>  1:</pre><pre>    action: delete_indices</pre><pre>    description: &quot;Clean up ES by deleting old indices&quot;</pre><pre>    options:</pre><pre>      timeout_override:</pre><pre>      continue_if_exception: False</pre><pre>      disable_action: False</pre><pre>      ignore_empty_list: True</pre><pre>    filters:</pre><pre>    - filtertype: age</pre><pre>      source: name</pre><pre>      direction: older</pre><pre>      timestring: &#39;%Y-%m-%d&#39;</pre><pre>      unit: days</pre><pre>      unit_count: 14</pre><pre>      field:</pre><pre>      stats_result:</pre><pre>      epoch:</pre><pre>      exclude: False</pre><h3>Elasticsearch troubleshooting</h3><p>Most of the issues with Elasticsearch can be investigated through two sources:</p><ul><li>ES instance logs (written to <em>stdout </em>by default)</li><li>ES REST API (Kibana Dev Tools, or connect to any ES instance)</li></ul><p>A lot of GET API support query parameters, for example:</p><ul><li>?v to include headers;</li><li>?s=column_name to sort by a column.</li></ul><p>Full list of common query option you can find at <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/common-options.html">official docs</a>. Also, filtering by index names supports a kind of <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/date-math-index-names.html">math</a>, which might be useful as during ingestion time, also during querying API.</p><p>Useful APIs:</p><p>- GET /_cat/health?v to get cluster health state<br>- GET /_nodes To get cluster nodes information (also has heap/cpu usage stats)<br>- GET /_nodes/stats <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-nodes-stats.html">Stats details explained</a><br>- GET /_cluster/settings?include_defaults=true<br>- GET /_cat/thread_pool?v&amp;s=name returns amount of current thread pools (for example, might help to identify how many bulk requests each node process at the moment)<br>- GET /_cat/indices?v to list all the indices (includes index health, size, number of documents)<br>- GET /_cat/shards?v&amp;s=index&amp;s=state&amp;human to list all the shards (for example, can help to find unallocated shards)<br>- GET /_template gets all the index templates<br>- GET /{index_name} to get index detailed description<br>- GET /{index_name}/_recovery?human to get details about shard stages and statuses of these stages (if index state is not healthy, the api gives more clues where to dive in further).<br>- GET /{index_name}/_mapping to get mapping of exact index (might be useful if some data is rejected by ES because of mapping conflicts)<br>- GET /_cluster/allocation/explain?pretty with request body {“index&quot;: “logs-2020-09-18&quot;,&quot;shard”:3, “primary”:false}— to garner more information about shard allocation issues. More details at <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-allocation-explain.html">allocation explained</a> article.<br>- POST /_cluster/reroute?retry_failed=true forces allocation retry. Elasticsearch tries to allocate a shard several times and if all tries fails — the shard stays in failed state. For example, at some point of time, Elasticsearch did not have enough disk space; some time later, one you cleaned up disk or added more nodes, you can retry shard allocation.</p><p>The most common troubleshooting scenario:</p><ul><li>check cluster health GET /_cat/health?v</li><li>check nodes load GET /_nodes</li><li>get list of indexes with state of each index GET /_cat/indices?v</li><li>get list of shard with allocation state: GET /_cat/shards?v&amp;s=index&amp;s=state&amp;human</li><li>based on the responses, identify the issue.</li></ul><p>If API is not available or any instance is disconnected from the cluster — check instance logs.</p><h3>Elasticsearch further reading</h3><ul><li><a href="https://www.datadoghq.com/blog/elasticsearch-performance-scaling-problems/">five most common problems</a> explained by Datadog.</li><li><a href="https://www.elastic.co/videos/pro-tips-from-elastic-support-engineers">Short video tips</a> for ES operations team</li><li>How to setup <a href="https://www.datadoghq.com/blog/monitor-elasticsearch-performance-metrics/">monitoring of Elasticsearch</a></li><li><a href="https://www.elastic.co/guide/en/cloud/current/ec-monitoring.html">Keeping the cluster Healthy</a></li><li>Anatomy of Elasticsearch Cluster: <a href="https://blog.insightdatascience.com/anatomy-of-an-elasticsearch-cluster-part-i-7ac9a13b05db">part 1</a>, <a href="https://blog.insightdatascience.com/anatomy-of-an-elasticsearch-cluster-part-ii-6db4e821b571">part 2</a>, <a href="https://blog.insightdatascience.com/anatomy-of-an-elasticsearch-cluster-part-iii-8bb6ac84488d">part 3</a></li></ul><h3>Questions?</h3><p>If you have further questions, you are welcome to contact me in <a href="https://twitter.com/kliushnikov">twitter</a> or here in comments ;)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d0f9d1427e24" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introducing Joseki: the open-source product to improve cloud infrastructure security]]></title>
            <link>https://kliushnikov.medium.com/introducing-joseki-the-open-source-product-to-improve-cloud-infrastructure-security-fb71d58f41bd?source=rss-af9c27f30570------2</link>
            <guid isPermaLink="false">https://medium.com/p/fb71d58f41bd</guid>
            <category><![CDATA[security]]></category>
            <category><![CDATA[devsecops]]></category>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Igor Kliushnikov]]></dc:creator>
            <pubDate>Tue, 01 Sep 2020 20:10:09 GMT</pubDate>
            <atom:updated>2020-09-01T20:10:09.089Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/330/1*zEGlrf_FG_pVkjUQnNyz8Q.jpeg" /><figcaption>Credits: By78 © <a href="https://en.wikipedia.org/w/index.php?curid=23020551">wikipedia</a></figcaption></figure><blockquote><em>In Go, Joseki is a known sequence of moves which leads to a balanced position</em>.</blockquote><p>Today I am excited to present you <a href="https://github.com/deepnetworkgmbh/joseki">Joseki</a>: the open-source product, which empowers engineering teams to keep their Cloud and Kubernetes infrastructure closer to known best-practices.</p><p>At <a href="https://github.com/deepnetworkgmbh/">Deepnetwork, </a>we work with Clouds and Kubernetes for a while, and we know by our own experience that maintaining configuration in good-enough shape might be tiresome. The amount of services and workloads types only grows, and especially for a small team, it might be a huge burden to keep in mind all the possible settings and tweaks.</p><p>Thankfully, there are open-sourced tools on the market that can help to make a product safer. They often are summoned to enforce known best-practices to docker images, Kubernetes, and cloud infrastructure at large. However, the approach that relies on many tools comes up with its own set of problems:</p><ul><li>a lot of products cover just a single aspect of security management;</li><li>tools are disconnected and just figuring out how to use them together is a hassle;</li><li>often, they have no reporting capabilities and no historical overview.</li></ul><p><em>Joseki</em> is here to address these problems.</p><h3>What the heck Joseki is?</h3><p>Joseki is designed to <strong>audit the configuration of cloud systems</strong>. It differs from other tools by combining various scanners to <strong>target many object types at the same time</strong>, reducing the number of tools needed to be learned, installed, and maintained.</p><p>Audit results are seamlessly combined in the user interface, making it easier to consume and understand. The results are <strong>ranked based on severity,</strong> and each discovered issue is accompanied by a <strong>recommended action</strong>. Joseki also offers <strong>a historical view</strong> to monitor and compare the security of your system over time.</p><p>Joseki does not need agents to be installed into each corner of your system: it requires only temporary access to monitored resources to perform an audit.</p><p>At the moment, the product integrates:</p><ul><li>Container Image vulnerabilities scanner <a href="https://github.com/aquasecurity/trivy">trivy</a>.</li><li>Kubernetes objects validator <a href="https://github.com/FairwindsOps/polaris">polaris</a>.</li><li>Azure Cloud infrastructure auditor <a href="https://github.com/azsk/DevOpsKit">az-sk</a></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-UgFwH2GGCe1BreZefJ7tw.png" /></figure><h3>Features</h3><p>The main Joseki tasks are:</p><ul><li>Display the current security state of the system</li><li>Help to find weak parts in monitored infrastructure</li><li>Explain how to improve the configuration</li><li>Track changes over time</li></ul><p>To accomplish these, Joseki performs <strong>scheduled scans </strong>of monitored infrastructure: once a day it audits Cloud and Kubernetes components, normalizes results, and persists them in a database. Each check has associated <strong>remediation action.</strong></p><figure><img alt="AzSK check description" src="https://cdn-images-1.medium.com/max/1024/1*hCF9KWGzWswBoTb_00BEIA.png" /></figure><p>Based on audit results, Joseki builds a <strong>general infrastructure overview</strong>: how many checks failed/succeeded and what are their severity:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cGXGYU9Qy9tR0tdQZ1JDHg.png" /><figcaption>Single component summary view</figcaption></figure><p><strong>Historical audit results</strong> are available for review and comparison at any given point of time:</p><ul><li>How many failed or succeeded checks were today, a week ago, a month ago?</li><li>What changed in each Kubernetes cluster or Azure subscription since last week/month?</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zYKEo16dY7gvenyZUjpNdA.png" /><figcaption>Single component diff</figcaption></figure><p>During the Kubernetes audit, Joseki automatically discovers used docker images and <strong>scans them for known </strong><a href="https://en.wikipedia.org/wiki/Common_Vulnerabilities_and_Exposures"><strong>CVE</strong></a>s.</p><p>To nail down the exact problem or find only <em>your components</em>, you can group and filter the data at the portal based on a variety of attributes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AfFXZMlHIJZe0adygsVoCg.png" /></figure><p>And a final touch: Joseki <strong>supports role-based access and integrates with Azure Active Directory</strong> so that you can give your users different visibility scopes. For example, you might want to limit who can see the state of the production environment, but give everyone access to sandbox resources.</p><h3>Architecture overview</h3><p>Joseki consists of three main parts:</p><ul><li><em>frontend </em>— a web application, which interacts with end-user.</li><li><em>backend </em>— expose web-api for frontend and does the most of business logic: shaping audit data, historical view, reporting, configuration.</li><li><em>scanners </em>— a set of applications (one per audit/scan type), that once in a while perform audit/scan operation and uploads raw results to a Blob Storage. Each scanner job can be deployed to different locations: cloud or bare metal, VMs, Kubernetes, FaaS.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/717/1*XotQ0RUzdUqGl1BohxiCaQ.png" /></figure><p>The entire product can be installed into a single node (VM) with all its components, as long as it has access to targets to be scanned.</p><p>Individual scanners can be installed separately and scaled horizontally. For example, you can provision multiple instances of <em>trivy</em> scanner to increase the throughput of docker-images scanning, and Joseki would divide the work between these instances.</p><p>The product needs read-only access to audited targets (cloud-vendor and Kubernetes APIs). Each scanner has its configuration. They can be enabled or disabled based on needs.</p><p>When a new user starts working with Joseki, the application needs just to<em> work</em>, but once it works, it needs to be fully customizable and fully adaptable. Therefore, each service should be able to run with good-enough defaults but also be flexible to change these defaults by an experienced user.</p><h3>Roadmap</h3><p>The project is still in its early days, and we have many ideas on <a href="https://github.com/deepnetworkgmbh/joseki/blob/master/PRODUCTOVERVIEW.md#roadmap">our Roadmap</a>. Among them:</p><ul><li><strong>Multi-cloud support</strong>. Now, Joseki can scan only Azure infrastructure and supports only Azure deployment, but AWS and Google Cloud are on our radar.</li><li><strong>Reporting</strong>. Joseki is going to send misc reports to different communication channels. Do you want to receive an email into your mailbox weekly? Stay in touch: we are going to work on it soon.</li><li><strong>Issue attestation</strong>: assign targets to owners, suppress results, reminders, triaging, and taking actions.</li></ul><h3>Getting started</h3><p>You can deploy Joseki to your infrastructure using a <a href="https://github.com/deepnetworkgmbh/joseki/blob/master/DEPLOYMENT.md">deployment guide</a>.</p><p>If you need help or have questions — you are welcome to open <a href="https://github.com/deepnetworkgmbh/joseki/issues">github issues</a>, vote for existing, contact me, or anyone from the development team.</p><p>If you want to see Joseki in action, drop me a line and we can arrange a demo.</p><p>We would be glad to help you hit the ground running or just hear your feedback 😉</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fb71d58f41bd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Azure Active Directory: Application or Service Principal?]]></title>
            <link>https://kliushnikov.medium.com/azure-active-directory-application-or-service-principal-b5a5e14f2a23?source=rss-af9c27f30570------2</link>
            <guid isPermaLink="false">https://medium.com/p/b5a5e14f2a23</guid>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[active-directory]]></category>
            <category><![CDATA[azure-key-vault]]></category>
            <dc:creator><![CDATA[Igor Kliushnikov]]></dc:creator>
            <pubDate>Fri, 11 Oct 2019 21:22:24 GMT</pubDate>
            <atom:updated>2020-02-23T19:36:28.632Z</atom:updated>
            <content:encoded><![CDATA[<h3>Azure Active Directory: Service Principal or Application?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/602/1*7SPZhm2giC0rmUM4ox3I-g.png" /></figure><p>We work in the Azure cloud, and we try to keep our secrets secret. To accomplish that, we use <a href="https://azure.microsoft.com/en-in/services/key-vault/">Azure Key Vault</a>, which allows access to the internals only to pre-defined Azure Active Directory identities. Things sound very simple:</p><ul><li>associate your piece of code with Azure Active Directory Identity</li><li>setup Key Vault to allow this Identity to access secrets</li><li>done</li></ul><p>It turned out to be harder (at least without carefully reading the documentation). <a href="https://docs.microsoft.com/en-us/azure/templates/microsoft.keyvault/2018-02-14/vaults">ARM</a> and <a href="https://www.terraform.io/docs/providers/azurerm/r/key_vault.html">Terraform </a>templates for Key Vault says about some object_id . Ok, I can go to Azure Portal, find my application in app-registrations, and here is an ObjectId, right?</p><p>But when you want to connect your Application to Key Vault from the native library, it asks client-id and client-secret . So, maybe it’s application-id?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/554/1*EBl49u46ysLsqqxryUDwfA.png" /></figure><p>But why do some docs mention the service-principal identifier, but not the application itself? If I even find Service Principal, what should I use object-id or application-id?</p><p>If you made a mistake, Azure Key Vault would not say a single word against it. It would accept your request and maybe would show your application name in the Access-Policies list.</p><p>This topic beat me hard several times, so I want to summarize my understanding in a single post.</p><h4>Azure Active Directory</h4><p>Let’s start with simplified Azure Active Directory terminology.</p><p><em>Azure Active Directory</em> or AD is a cloud-based identity and access management service — it takes care of authentication and authorization of human-beings and software-based identities.</p><p>One instance of Azure AD associated with a single organization is named <em>Tenant</em>.</p><p>Each AD tenant might have 1 to N Azure <em>Subscriptions</em>. Azure Subscription — is a group of cloud services that are billed together.</p><p>Azure AD has associated <em>Identities </em>— things that can be authenticated within the AD. Identities might belong to <em>Users</em> (a human with an email address or phone number) or to <em>Applications</em> (a piece of software like my shiny website or database, which stores its data).</p><p>Very simplified, all the listed above terminology is thrown to the plate to do a single thing: prove that an <em>Identity </em>has the rights to do <em>an operation</em> against <em>a resource </em>(website, database, etc.). Or in other words, Azure AD helps to answer two questions:</p><ol><li>Can I trust the Identity? (authentication)</li><li>Is Identity allowed to read/update/delete a resource? (authorization)</li></ol><p>Any AD <em>Identity</em> might be <em>single-tenant</em> (has access to resources of a single organization) or <em>multi-tenant</em> (two or more organizations).</p><p>Managing Azure AD is possible in several ways:</p><ul><li>through Azure Portal</li><li><a href="https://docs.microsoft.com/en-us/cli/azure/ad">Azure CLI</a></li><li><a href="https://docs.microsoft.com/en-us/powershell/module/azuread">Powershell module</a></li><li>REST requests to <a href="https://docs.microsoft.com/en-us/azure/active-directory/develop/active-directory-graph-api">Azure Active Directory Graph API</a>. For example, using <a href="https://developer.microsoft.com/en-us/graph/graph-explorer">Graph Explorer</a>.</li></ul><h4>Application registration</h4><p>Application registration is a process of adding a new non-human Identity to AD. It leads to the creation of two objects in an Azure AD tenant:</p><ul><li>An application object</li><li>A service principal object</li></ul><h4>Application object</h4><p>An Azure AD Application is defined by its <strong>one and only application object. The object resides in the Azure AD tenant, where the application was registered</strong>, known as the application’s “home” tenant.</p><p>To query entity details with az cli:</p><p>az ad app list --display-name APP_NAME</p><p>or with PowerShell:</p><p>Get-AzureADApplication -Filter &quot;DisplayName eq &#39;APP_NAME&#39;&quot;</p><p>(<strong><em>EDITED 23.02.2020</em></strong><em>:</em> Thanks <a href="https://medium.com/u/321963066f59">Vladyslav Kurmaz</a> for pointing, Graph API was changed since the article was published and the following examples do not work)</p><p>o̶r̶ ̶w̶̶̶i̶̶̶t̶̶̶h̶̶̶ ̶̶̶G̶̶̶r̶̶̶a̶̶̶p̶̶̶h̶̶̶ ̶̶̶A̶̶̶P̶̶̶I̶̶̶:̶̶̶</p><p>G̶E̶T̶ ̶h̶t̶t̶p̶s̶:̶/̶/̶g̶r̶a̶p̶h̶.̶w̶i̶n̶d̶o̶w̶s̶.̶n̶e̶t̶/̶T̶E̶N̶A̶N̶T̶_̶N̶A̶M̶E̶/̶a̶p̶p̶l̶i̶c̶a̶t̶i̶o̶n̶s̶?̶$̶f̶i̶l̶t̶e̶r̶=̶d̶i̶s̶p̶l̶a̶y̶N̶a̶m̶e̶ ̶e̶q̶ ̶’̶A̶P̶P̶_̶N̶A̶M̶E̶’̶</p><h4>Service principal object</h4><p>To access resources that are secured by an Azure AD tenant (for example, components in an Azure Subscription), the entity must be represented by a security principal, which Azure names <em>Service Principal</em>. <strong>The service principal defines the access policy and permissions for the user/application in a single Azure AD tenant</strong>.</p><p>Service Principal inherits base properties of associated Application (for example, AppName and ApplicationId)</p><p>To create a new one with cli:</p><ol><li>az ad sp create-for-rbac --name APP_NAME - creates <strong>Application and Service Principal</strong> in the current tenant</li><li>az ad sp create --id APP_ID - creates <strong>only a Service Principal</strong> for the specified ApplicationId.</li></ol><p>To query entity details with az cli</p><p>az ad sp list --display-name APP_NAME</p><p>or with PowerShell:</p><p>Get-AzureADServicePrincipal -Filter &quot;DisplayName eq &#39;APP_NAME&#39;&quot;</p><p>(<strong><em>EDITED 23.02.2020</em></strong><em>:</em> Graph API was changed since the article was published)</p><p>o̶r̶ ̶w̶̶̶i̶̶̶t̶̶̶h̶̶̶ ̶̶̶G̶̶̶r̶̶̶a̶̶̶p̶̶̶h̶̶̶ ̶̶̶A̶̶̶P̶̶̶I̶̶̶:̶̶̶</p><p>G̶E̶T̶ ̶h̶t̶t̶p̶s̶:̶/̶/̶g̶r̶a̶p̶h̶.̶w̶i̶n̶d̶o̶w̶s̶.̶n̶e̶t̶/̶T̶E̶N̶A̶N̶T̶_̶N̶A̶M̶E̶/̶s̶e̶r̶v̶i̶c̶e̶P̶r̶i̶n̶c̶i̶p̶a̶l̶s̶?̶$̶f̶i̶l̶t̶e̶r̶=̶d̶i̶s̶p̶l̶a̶y̶N̶a̶m̶e̶ ̶e̶q̶ ̶’̶A̶P̶P̶_̶N̶A̶M̶E̶’̶</p><h4>Application and service principal relationship</h4><p>The <em>Application </em>object serves as the template from which common and default properties are derived for use in creating corresponding <em>Service Principal</em> objects.</p><p><strong>Consider the application object as the <em>global</em> representation of your application for use across all tenants and the service principal as the <em>local</em> representation for use in a specific tenant</strong>.</p><p>A <em>single-tenant</em> <em>Application</em> has the only one <em>Service Principal</em> (in its home tenant), created and permitted for use during application registration. A <em>multi-tenant</em> Web application/API has a <em>Service Principal</em> created in each <em>Tenant</em> where the application is allowed to access resources.</p><p>I can extrapolate <a href="https://stackoverflow.com/a/53109692/1952990">SO author suggested analogy</a>: Azure Tenant — is a <em>process</em>; AD Application is <em>class</em>; Service Principal — is a <em>singleton instance</em> of the class.</p><h4>An example?</h4><p>Let’s consider a sample configuration</p><ul><li>Active Directory 1 (AD1) and Active Directory 2 (AD2) are Active Directory instances (tenants), each has two subscriptions inside.</li><li>Application is a multi-tenant application registered in AD1.</li><li>SP1 and SP2 — Service Principals associated with the application.</li><li>In AD1 SP1 can access Blob Storage in Subscription1 and Website in Subscription2, but it has no access to the Database in Subscription2.</li><li>In AD2 SP2 can access virtual Machine from Subscription2, but not a Website in Subscription1.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/821/1*8hBPK_JiFmGlqClZxi7swA.png" /></figure><p>If you want to login with Azure CLI, you have to use Application identifier and password associated with one of Service Principals, for example: az login --service-principal --username APP_ID --password SP1_PASSWORD --tenant TENANT_ID.</p><h4>Azure Key Vault example</h4><p>Now, when we know a bit of theory, let’s get back to the original Key Vault problem.</p><p>There is an Azure Key Vault with secrets. You are a developer of an application, which needs access to the Key Vault.</p><p>Your app is associated 1–1 with Azure Active Directory Application, which might be provisioned in any Azure Active Directory Instance (Tenant). Let’s name this tenant <em>Hommy Tenant</em>.</p><p>Azure Key Vault is created in an exact Azure Subscription, which is a part of a concrete Active Directory Tenant. Let it be <em>Guest Tenant</em>.</p><p>To build a communication channel between the app and key-vault, you have to:</p><ul><li>create a multi-tenant application az ad app create --display-name FOO_APP --available-to-other-tenants true in <em>Hommy Tenant</em>.</li><li>create a Service Principal in <em>Guest Tenant </em>with applicaitonId from the previous clause: az ad sp create --id hommy_application_id. Created Service Principal has own password guest_sp_password, but the same applicationId.</li><li>allow <strong>Service Principal</strong> to read secrets from Key Vault in <em>Guest Tenant</em>: az keyvault set-policy --name guest-kv --resource-group guest-rg --object-id service_principal_object_id --secret-permissions get list</li><li>setup the application to use hommy_application_id and guest_sp_password to access guest-kv.</li></ul><h4>Is that all?</h4><p>Hopefully, that’s all you need to know not to misuse Azure Applications and Service Principals and their identifiers.</p><p>In the end, I strongly recommend to go and read the following documentation pages and two stack-overflow threads:</p><ol><li>Azure Active Directory <a href="https://docs.microsoft.com/en-us/azure/active-directory/fundamentals/active-directory-whatis">documentation</a>.</li><li>Application and Service Principal <a href="https://docs.microsoft.com/en-us/azure/active-directory/develop/app-objects-and-service-principals">documentation</a></li><li>Two SO threads that helped me a lot to understand the topic:</li></ol><ul><li><a href="https://stackoverflow.com/a/54067067/1952990">https://stackoverflow.com/a/54067067/1952990</a></li><li><a href="https://stackoverflow.com/a/53109692/1952990">https://stackoverflow.com/a/53109692/1952990</a> explains the application and service-principal concepts in other words.</li></ul><p>And just in case you need a handy list of AD terms, here is the official <a href="https://docs.microsoft.com/en-us/azure/active-directory/develop/developer-glossary">glossary</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b5a5e14f2a23" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Filtering docker images by pattern with wildcards]]></title>
            <link>https://kliushnikov.medium.com/filtering-docker-images-5eb5aee358df?source=rss-af9c27f30570------2</link>
            <guid isPermaLink="false">https://medium.com/p/5eb5aee358df</guid>
            <category><![CDATA[containers]]></category>
            <category><![CDATA[cli]]></category>
            <category><![CDATA[docker]]></category>
            <dc:creator><![CDATA[Igor Kliushnikov]]></dc:creator>
            <pubDate>Sun, 08 Sep 2019 19:58:47 GMT</pubDate>
            <atom:updated>2020-09-23T11:23:34.468Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1001/1*X2rc8jKr0jEyYggiqyfrOg.png" /><figcaption>Jack Sparrow knows how to find desired artifacts</figcaption></figure><p>Likely, if you work with docker containers on a regular basis, your local image cache might be full of more or less important possessions and finding the right set of images might be tricky.</p><p>To list docker images we can use docker image list command or a shorter version of it docker images. To filter the result of this output there is an argument--filter. A lot of details and examples are in the <a href="https://docs.docker.com/engine/reference/commandline/images/#filtering">official doc</a>: filter by a label, by date, by a <em>pattern, and others</em></p><p>The pattern option supports the wildcard character *, which is barely documented, only a few examples:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/922/1*NG0vuqdwv74IxPm7PRZqaw.png" /><figcaption>Official Docker filtering documentation</figcaption></figure><p>What is not documented — you can avoid --filter=reference keyword and just type docker images “example.com/*”:</p><pre>REPOSITORY       TAG       IMAGE ID         CREATED        SIZE<br>example.com/baz  latest    43773d1dba76    3 months ago    4.03MB<br>example.com/bar  latest    d8233ab899d4    3 months ago    1.2MB</pre><p>But what is not obvious for me and docs also do not say it: <strong>the wildcard character does not apply to </strong><strong>/ in repository name</strong>.</p><p>In our company case, we extensively use repository names following the pattern mycontainerregistry.azurecr.io/{moduleName}/{applicationName} and to filter all our custom images I have to use docker images “mycontainerregistry.azurecr.io/*/*, not just“mycontainer.azurecr.io/* or “mycontainerregistry.azurecr.io/**:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/875/1*gZsWyVzZy7RbXuD-gs2ToQ.png" /></figure><p>I spent <em>some</em> time figuring out why my simple wildcard notation was not working, so I hope I would save someone’s time ;)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5eb5aee358df" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>