<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Sunny Kr. on Medium]]></title>
        <description><![CDATA[Stories by Sunny Kr. on Medium]]></description>
        <link>https://medium.com/@sunnykrgupta?source=rss-f6c22db16b21------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*yvR0j2DXqnIFeJXr94NIKA.jpeg</url>
            <title>Stories by Sunny Kr. on Medium</title>
            <link>https://medium.com/@sunnykrgupta?source=rss-f6c22db16b21------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 18:08:50 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@sunnykrgupta/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Kubernetes HPA Autoscaling with Kafka metrics]]></title>
            <link>https://medium.com/google-cloud/kubernetes-hpa-autoscaling-with-kafka-metrics-88a671497f07?source=rss-f6c22db16b21------2</link>
            <guid isPermaLink="false">https://medium.com/p/88a671497f07</guid>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[container-orchestration]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[google-stackdriver]]></category>
            <dc:creator><![CDATA[Sunny Kr.]]></dc:creator>
            <pubDate>Wed, 26 Dec 2018 21:18:50 GMT</pubDate>
            <atom:updated>2018-12-29T14:25:42.009Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/595/1*POTo0rgHZ746YItPa8medw.png" /></figure><p>Autoscaling is natively supported on Kubernetes. Since 1.7 release, Kubernetes added a feature to scale your workload based on custom metrics. Prior release only supported scaling your apps based on CPU and memory.</p><p>Kubernetes 1.7 introduced “Aggregator Layer” which allows Kubernetes to be extended with additional APIs, beyond what is offered by the core Kubernetes APIs. This gives you the power to enable your own custom APIs.</p><p>In this guide, we are going to understand the basics of HPAs, custom, external metrics APIs working and scaling workloads based on external metrics scraped from Kafka Cluster.</p><p>The following are the steps you will complete in this guide:</p><ul><li>Step 1: Enable cluster monitoring for Stackdriver.</li><li>Step 2: Deploy a custom API server and register it to the aggregator layer.</li><li>Step 3: Deploy metrics exporter and write to Stackdriver.</li><li>Step 4: Deploy a sample application written in Golang to test the autoscaling.</li><li>Step 5: Write a custom metrics based HPA to scale application.</li></ul><p>We need to cover some concepts which are good to know before we move forward to the demonstration.</p><h3>How Kubernetes HPA works?</h3><p>The Horizontal Pod Autoscaler is implemented as a Kubernetes API resource and a controller. The resource determines the behaviour of the controller. The controller periodically adjusts the number of replicas in a replication controller or deployment.</p><p>The HorizontalPodAutoscaler normally fetches metrics from a series of aggregated APIs (metrics.k8s.io, custom.metrics.k8s.ioand external.metrics.k8s.io). The metrics.k8s.io API is usually provided by <strong>metrics-server</strong>, which needs to be launched separately. The HorizontalPodAutoscaler can also fetch metrics directly from Heapster.</p><p>Here, in this guide, we will deploy our HPA reading from external.metrics.k8s.io as our Kafka metrics will be exposed to that API.</p><h3>What is custom metrics?</h3><p>Kubernetes allows us to deploy your own metrics solutions. By default, <strong>metric-server</strong> and <strong>heapster</strong> act as core metrics backend.</p><p>Kubernetes has extended the support to allow custom APIs to expose other metrics provider. Few adapters are written by the third party to implement custom APIs which can be used to expose these metrics to Kubernetes resources such like HPA.</p><p>Current Implementations: <a href="https://github.com/Kubernetes/metrics/blob/master/IMPLEMENTATIONS.md">github.com/Kubernetes/IMPLEMENTATIONS.md</a></p><h3>How custom API server and HPA ties together?</h3><p>The custom API server that we deploy registers an API to Kubernetes and allows the HPA controller query custom metrics from that. API server that we are going to deploy here is Stackdriver adapter which can collect metrics from Stackdriver and send them to the HPA controller via REST queries.</p><p>Our custom API server will register two APIs to Kubernetes: custom.metrics.k8s.io and external.metrics.k8s.io.</p><p>We will be also deploying an application to write metrics (in this case Kafka metrics) to google stackdriver. The kind of metrics which we are going to write to Stackdriver will be exposed under external.metrics.k8s.io instead of custom.metrics.k8s.io.</p><h3>Prerequisites</h3><p>Ensure the following dependencies are already fulfilled:</p><ol><li>You have a Docker running. You know the rules of this game. ;)</li><li>You have a Kubernetes cluster (GKE) running on GCP.</li><li>You have kubectl CLI installed and configured to your GKE cluster.</li></ol><p>Let’s begin the demonstration.</p><h3>1. Enable cluster monitoring for Stackdriver</h3><p>GCP helper docs: <a href="https://cloud.google.com/kubernetes-engine/docs/how-to/monitoring#enabling_monitoring_for_an_existing_cluster">https://cloud.google.com/kubernetes-engine/docs/how-to/monitoring#enabling_monitoring_for_an_existing_cluster</a></p><p>Monitoring scope should be enabled on cluster nodes. It is enabled by default, <a href="https://cloud.google.com/kubernetes-engine/docs/how-to/monitoring#enabling_monitoring_for_an_existing_cluster">https://cloud.google.com/kubernetes-engine/docs/how-to/monitoring#enabling_monitoring_for_an_existing_cluster</a>so you need not do anything. If you have an older version, upgrade it to the latest version and then update your node version as well. The scope will enable write permission to stackdriver which is important for writing metrics.</p><pre>$ gcloud container clusters list<br>NAME                 LOCATION       MASTER_VERSION  MASTER_IP       MACHINE_TYPE   NODE_VERSION   NUM_NODES  STATUS<br>my-kube-cluster  us-central1-b  1.10.7-gke.11   xxx.xxx.xxx.xxx  n1-standard-1    1.10.7-gke.11    2          RUNNING</pre><pre>$ gcloud container clusters describe my-kube-cluster<br>............<br>..&lt;output&gt;..<br>............<br>oauthScopes:<br>    - <a href="https://www.googleapis.com/auth/compute">https://www.googleapis.com/auth/compute</a><br>    - <a href="https://www.googleapis.com/auth/devstorage.read_only">https://www.googleapis.com/auth/devstorage.read_only</a><br>    - <a href="https://www.googleapis.com/auth/service.management">https://www.googleapis.com/auth/service.management</a><br>    - <a href="https://www.googleapis.com/auth/servicecontrol">https://www.googleapis.com/auth/servicecontrol</a><br>    - <a href="https://www.googleapis.com/auth/logging.write">https://www.googleapis.com/auth/logging.write</a><br>    - <a href="https://www.googleapis.com/auth/monitoring">https://www.googleapis.com/auth/monitoring</a>   <em># enabled</em><br>............<br>..&lt;output&gt;..<br>............</pre><h3>2. Deploy a custom API server and register it to the aggregator layer.</h3><p>Before we deploy our custom server, let’s see what APIs are available in our k8s.</p><pre>## Type commands to see existing available APIs<br>$ kubectl api-versions</pre><pre>admissionregistration.k8s.io/v1beta1<br>apiextensions.k8s.io/v1beta1<br>apiregistration.k8s.io/v1<br>apiregistration.k8s.io/v1beta1<br>............<br>..&lt;other-APIs&gt;..<br>............<br>metrics.k8s.io/v1beta1    # this is our core metrics APIs<br>............<br>..&lt;other-APIs&gt;..<br>............</pre><p>Here, you can see we have metrics.k8s.io available but we need external.metrics.k8s.io to expose our stackdriver metrics and allow HPA to read it through.</p><p>We are going to use custom metrics stackdriver adapter to register our APIs.</p><pre># Use one of google user account to create a cluster role<br>$ kubectl create clusterrolebinding cluster-admin-binding \<br>    --clusterrole cluster-admin --user &quot;$(gcloud config get-value account)&quot;</pre><pre># We will deploy new resource model based APIs</pre><pre>$ kubectl apply -f <a href="https://raw.githubusercontent.com/GoogleCloudPlatform/k8s-stackdriver/master/custom-metrics-stackdriver-adapter/deploy/production/adapter_new_resource_model.yaml">https://raw.githubusercontent.com/GoogleCloudPlatform/k8s-stackdriver/master/custom-metrics-stackdriver-adapter/deploy/production/adapter_new_resource_model.yaml</a></pre><pre>namespace &quot;custom-metrics&quot; created<br>serviceaccount &quot;custom-metrics-stackdriver-adapter&quot; created<br>clusterrolebinding.rbac.authorization.k8s.io &quot;custom-metrics:system:auth-delegator&quot; created<br>rolebinding.rbac.authorization.k8s.io &quot;custom-metrics-auth-reader&quot; created<br>clusterrolebinding.rbac.authorization.k8s.io &quot;custom-metrics-resource-reader&quot; created<br>deployment.extensions &quot;custom-metrics-stackdriver-adapter&quot; created<br>service &quot;custom-metrics-stackdriver-adapter&quot; created<br>apiservice.apiregistration.k8s.io &quot;v1beta1.custom.metrics.k8s.io&quot; created<br>apiservice.apiregistration.k8s.io &quot;v1beta1.external.metrics.k8s.io&quot; created<br>clusterrole.rbac.authorization.k8s.io &quot;external-metrics-reader&quot; created<br>clusterrolebinding.rbac.authorization.k8s.io &quot;external-metrics-reader&quot; created</pre><p>Now, we have deployed our custom server and registered APIs to Aggregator Layer.</p><pre># Custom Metrics APIs verification</pre><pre>$  kubectl get all -n custom-metrics<br>NAME                                                 READY     STATUS    RESTARTS   AGE<br>custom-metrics-stackdriver-adapter-6c9bd9679-m2cnh   1/1       Running   0          2m</pre><pre># Type again to get available APIs<br>$ kubectl api-versions<br>............<br>..&lt;other-APIs&gt;..<br>............<br>metrics.k8s.io/v1beta1<br>.....<br>custom.metrics.k8s.io/v1beta1   # for custom Kubernetes metrics<br>external.metrics.k8s.io/v1beta1 # for external metrics<br>............<br>..&lt;other-APIs&gt;..<br>............</pre><p>Now, query our new APIs which just came online.</p><pre>$ kubectl get --raw &quot;/apis/external.metrics.k8s.io/v1beta1&quot; | jq<br>$ kubectl get --raw &quot;/apis/custom.metrics.k8s.io/v1beta1&quot; | jq</pre><p>Now you can see, both APIs are available are online with a lot of custom metrics are available for query. Next, we will be writing Kafka metrics to stackdriver and these APIs will help us read from it.</p><p>Resource in details : <a href="https://github.com/GoogleCloudPlatform/k8s-stackdriver/tree/master/custom-metrics-stackdriver-adapter">k8s-stackdriver/custom-metrics-stackdriver-adapter</a></p><h3>3. Deploy metrics exporter and write to stackdriver.</h3><p>We are using <a href="https://github.com/danielqsj/kafka_exporter">Kafka exporter</a> which will read my Kafka cluster and expose metrics in <a href="https://prometheus.io/docs/instrumenting/exposition_formats/">prometheus format</a> and we combine our deployment with this <a href="https://github.com/GoogleCloudPlatform/k8s-stackdriver/tree/master/prometheus-to-sd">k8s-stackdriver/prometheus-to-sd</a> sidecar container. Kafka-exporter will read the Kafka cluster and expose the metrics on particular web-URL and our sidecar container will read those and write to stackdriver. Easy enough, let’s do it.</p><pre><em># prometheus-to-sd-custom-metrics-kafka-exporter.yaml</em><br>apiVersion: extensions/v1beta1<br>kind: Deployment<br>metadata:<br>  name: custom-metrics-kafka-exporter<br>spec:<br>  replicas: 1<br>  selector:<br>    matchLabels:<br>      custom: metrics<br>  template:<br>    metadata:<br>      labels:<br>        custom: metrics<br>    spec:<br>      hostNetwork: true<br>      containers:<br>      - name: kafka-exporter<br>        image: danielqsj/kafka-exporter<br>        command:<br>        - kafka_exporter<br>        - &quot;--kafka.server=my-kafka-broker-1:9092&quot;<br>        - &quot;--kafka.server=my-kafka-broker-2:9092&quot;<br>        ports:<br>          - name: http-metrics<br>            containerPort: 9308<br>        readinessProbe:<br>          httpGet:<br>            path: /<br>            port: 9308<br>          initialDelaySeconds: 5<br>          timeoutSeconds: 5<br>      - name: prometheus-to-sd<br>        image: gcr.io/google-containers/prometheus-to-sd:v0.3.2<br>        ports:<br>          - name: profiler<br>            containerPort: 6060<br>        command:<br>          - /monitor<br>          - --stackdriver-prefix=custom.googleapis.com<br>          - --source=kafka-exporter:http://localhost:9308?whitelisted=kafka_brokers,kafka_topic_partitions,kafka_consumergroup_current_offset_sum,kafka_consumergroup_lag_sum<br>          - --pod-id=<strong>$(</strong>POD_NAME<strong>)</strong><br>          - --namespace-id=<strong>$(</strong>POD_NAMESPACE<strong>)</strong><br>        env:<br>          - name: POD_NAME<br>            valueFrom:<br>              fieldRef:<br>                fieldPath: metadata.name<br>          - name: POD_NAMESPACE<br>            valueFrom:<br>              fieldRef:<br>                fieldPath: metadata.namespace</pre><p>Deploy this to ship your metrics to stackdriver.</p><pre>kubectl apply -f prometheus-to-sd-custom-metrics-kafka-exporter.yaml</pre><p>Next step is exploring stackdriver and filter your external metrics. Available metrics : <a href="https://github.com/danielqsj/kafka_exporter#metrics">https://github.com/danielqsj/kafka_exporter#metrics</a></p><blockquote>For example search : kafka_brokers</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/933/0*vwiOn5-ph1c1G1Ln.png" /><figcaption>Stackdriver metrics explorer</figcaption></figure><p>You can also query registered APIs to read these external values.</p><pre>$ kubectl get --raw &quot;/apis/external.metrics.k8s.io/v1beta1/namespaces/default/custom.googleapis.com|kafka-exporter|kafka_brokers&quot; | jq</pre><pre>{<br>  &quot;kind&quot;: &quot;ExternalMetricValueList&quot;,<br>  &quot;apiVersion&quot;: &quot;external.metrics.k8s.io/v1beta1&quot;,<br>  &quot;metadata&quot;: {<br>    &quot;selfLink&quot;: &quot;/apis/external.metrics.k8s.io/v1beta1/namespaces/default/custom.googleapis.com%7Ckafka-exporter%7Ckafka_brokers&quot;<br>  },<br>  &quot;items&quot;: [<br>    {<br>      &quot;metricName&quot;: &quot;custom.googleapis.com|kafka-exporter|kafka_brokers&quot;,<br>      &quot;metricLabels&quot;: {<br>        &quot;resource.labels.cluster_name&quot;: &quot;my-kube-cluster&quot;,<br>        &quot;resource.labels.container_name&quot;: &quot;&quot;,<br>        &quot;resource.labels.instance_id&quot;: &quot;gke-my-kube-cluster-69201eb2-dvdg&quot;,<br>        &quot;resource.labels.namespace_id&quot;: &quot;default&quot;,<br>        &quot;resource.labels.pod_id&quot;: &quot;custom-metrics-kafka-exporter-56764bbbc9-p5xqb&quot;,<br>        &quot;resource.labels.zone&quot;: &quot;us-central1-b&quot;,<br>        &quot;resource.type&quot;: &quot;gke_container&quot;<br>      },<br>      &quot;timestamp&quot;: &quot;2018-12-25T12:56:46Z&quot;,<br>      &quot;value&quot;: &quot;2&quot;<br>    }<br>  ]<br>}</pre><h3>4. Deploy a sample application written in golang to test autoscaling.</h3><p>Till now, we went through all the sophisticated pieces of stuff which will help to expose our metrics to HPA. From now on, we will run some basic ops that we usually do in Kubernetes.</p><p>To simulate the autoscaling, I have deployed a sample application written in golang which will act as Kafka client ( producer and consumer ) for Kafka topics.</p><blockquote>Code for reference : <a href="https://github.com/sunnykrGupta/k8s-hpa-custom-autoscaling-kafka-metrics/tree/master/go-kafka">k8s-hpa-custom-autoscaling-kafka-metrics/go-kafka</a></blockquote><pre>$ kubectl get deploy -llang=golang<br>NAME                       DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE<br>consumer-kafka-go-client   1         1         1            1           25m<br>producer-kafka-go-client   1         1         1            1           30m</pre><pre><em># scaling the producer app to build a consumer lag on kafka topic</em><br>$ kubectl scale --replicas=2 deployment/producer-kafka-go-client</pre><p>I have scaled my producer so that it will push enough messages to build lag for my consumer clients and we will test our HPA on consumer deployment.</p><h3>5. Write a custom metrics based HPA to scale application.</h3><p>Now that I have scaled my Kafka producer, there should be a consumer lag build up in the Kafka topic. We have prepared HPA to read from external metrics coming from stackdriver through our newly registered APIs. All the hard work above is for this moment.</p><p>Below is my HPA manifest file, which is going to help scale our Kafka consumer :</p><pre># kafka-custom-metrics-hpa.yaml<br>apiVersion: autoscaling/v2beta1<br>kind: HorizontalPodAutoscaler<br>metadata:<br>  name: consumer-kafka-go-client<br>spec:<br>  minReplicas: 1<br>  maxReplicas: 5<br>  metrics:<br>  - type: External<br>    external:<br>      # which metrics to read from stackdriver<br>      metricName: custom.googleapis.com|kafka-exporter|kafka_consumergroup_lag_sum<br>      metricSelector:<br>        matchLabels:<br>          # define labels to target<br>          metric.labels.consumergroup: golang-consumer<br>      # scale +1 whenever it crosses multiples of mentioned value<br>      targetAverageValue: &quot;1000&quot;<br>  # define deployment to control<br>  scaleTargetRef:<br>    apiVersion: extensions/v1beta1<br>    kind: Deployment<br>    name: consumer-kafka-go-client</pre><pre>$ kubectl apply -f kafka-custom-metrics-hpa.yaml<br>horizontalpodautoscaler.autoscaling &quot;consumer-kafka-go-client&quot; configured</pre><p>Let’s query our kafka_consumer_lag to see what are the current stats.</p><pre>kubectl get --raw &quot;/apis/external.metrics.k8s.io/v1beta1/namespaces/default/custom.googleapis.com|kafka-exporter|kafka_consumergroup_lag_sum&quot; | jq</pre><pre>{<br>  &quot;kind&quot;: &quot;ExternalMetricValueList&quot;,<br>  &quot;apiVersion&quot;: &quot;external.metrics.k8s.io/v1beta1&quot;,<br>  &quot;metadata&quot;: {<br>    &quot;selfLink&quot;: &quot;/apis/external.metrics.k8s.io/v1beta1/namespaces/default/custom.googleapis.com%7Ckafka-exporter%7Ckafka_consumergroup_lag_sum&quot;<br>  },<br>  &quot;items&quot;: [<br>    {<br>      &quot;metricName&quot;: &quot;custom.googleapis.com|kafka-exporter|kafka_consumergroup_lag_sum&quot;,<br>      &quot;metricLabels&quot;: {<br>        &quot;metric.labels.consumergroup&quot;: &quot;golang-consumer&quot;,<br>        &quot;metric.labels.topic&quot;: &quot;custom-topic&quot;,<br>        &quot;resource.labels.container_name&quot;: &quot;&quot;,<br>        &quot;resource.labels.namespace_id&quot;: &quot;default&quot;,<br>        &quot;resource.labels.pod_id&quot;: &quot;custom-metrics-kafka-exporter-547c7f4d5c-tqtw8&quot;,<br>        &quot;resource.labels.zone&quot;: &quot;us-central1-b&quot;,<br>        &quot;resource.type&quot;: &quot;gke_container&quot;<br>      },<br>      &quot;timestamp&quot;: &quot;2018-12-25T18:46:54Z&quot;,<br>      &quot;value&quot;: &quot;732&quot;<br>    }<br>  ]<br>}</pre><p>After polling some stats for a while, I saw auto scale trigger after some time. Yey!! In this case, it scaled the app to 4 replicas based on the numbers found on stackdriver.</p><pre>$ kubectl get hpa<br>NAME                       REFERENCE                             TARGETS        MINPODS   MAXPODS   REPLICAS   AGE<br>consumer-kafka-go-client   Deployment/consumer-kafka-go-client   732/1k (avg)   1         5         1          4m</pre><pre>$ kubectl get hpa<br>NAME                       REFERENCE                             TARGETS         MINPODS   MAXPODS   REPLICAS   AGE<br>consumer-kafka-go-client   Deployment/consumer-kafka-go-client   4032/1k (avg)   1         5         1          12m</pre><pre>$ kubectl get hpa<br>NAME                       REFERENCE                             TARGETS         MINPODS   MAXPODS   REPLICAS   AGE<br>consumer-kafka-go-client   Deployment/consumer-kafka-go-client   1008/1k (avg)   1         5         4          13m</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/827/0*js1y0U5R2cSgX4EI.png" /><figcaption>Kafka consumer lags metrics</figcaption></figure><p>Custom autoscaling is a very useful feature when you wanted to scale your workloads based on the complexity of your works deployed in your production. It could be disk size, networking, Loadbalancers connections etc.</p><p>Hope you enjoyed this guide. Give thumbs up and ask questions in the comments.</p><p><strong>Github: </strong><a href="https://github.com/sunnykrGupta/k8s-hpa-custom-autoscaling-kafka-metrics"><strong>https://github.com/sunnykrGupta/k8s-hpa-custom-autoscaling-kafka-metrics</strong></a></p><p>Thank you <a href="https://medium.com/u/be723eddb0e6">Arvind Jha</a> for edits and suggestions.</p><h4>Resources :</h4><ul><li><a href="https://github.com/GoogleCloudPlatform/k8s-stackdriver/tree/master/prometheus-to-sd">prometheus-to-sd</a></li><li><a href="https://github.com/danielqsj/kafka_exporter">kafka_exporter</a></li><li><a href="https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale">run-application/horizontal-pod-autoscale</a></li><li><a href="https://github.com/GoogleCloudPlatform/k8s-stackdriver/tree/master/custom-metrics-stackdriver-adapter">custom-metrics-stackdriver-adapter</a></li><li><a href="https://cloud.google.com/kubernetes-engine/docs/tutorials/external-metrics-autoscaling">https://cloud.google.com/kubernetes-engine/docs/tutorials/external-metrics-autoscaling</a></li><li><a href="https://kubernetes.io/docs/concepts/extend-Kubernetes/api-extension/apiserver-aggregation/">https://Kubernetes.io/docs/concepts/extend-Kubernetes/api-extension/apiserver-aggregation/</a></li><li><a href="https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/#autoscaling-on-metrics-not-related-to-Kubernetes-objects">autoscaling-on-metrics-not-related-to-Kubernetes-objects</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=88a671497f07" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/kubernetes-hpa-autoscaling-with-kafka-metrics-88a671497f07">Kubernetes HPA Autoscaling with Kafka metrics</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Sharded Mongodb in Kubernetes StatefulSets on GKE]]></title>
            <link>https://medium.com/google-cloud/sharded-mongodb-in-kubernetes-statefulsets-on-gke-ba08c7c0c0b0?source=rss-f6c22db16b21------2</link>
            <guid isPermaLink="false">https://medium.com/p/ba08c7c0c0b0</guid>
            <category><![CDATA[google-container-engine]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[containers]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[mongodb]]></category>
            <dc:creator><![CDATA[Sunny Kr.]]></dc:creator>
            <pubDate>Tue, 02 Jan 2018 09:04:40 GMT</pubDate>
            <atom:updated>2018-05-08T18:30:52.764Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Originally published.</em></p><p>Daemon blog : <a href="https://sunnykrgupta.github.io/sharded-mongodb-in-kubernetes-statefulsets-on-gke.html">https://sunnykrgupta.github.io/sharded-mongodb-in-kubernetes-statefulsets-on-gke.html</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/0*u_M0HV4Zf1gUUlbI.png" /></figure><p>This blog is going to demonstrate the setup of Sharded MongoDB Cluster on Google Kubernetes Engine. We will use kubernetes StatefulSets feature to deploy mongodb containers.</p><p>We need to cover some concepts before we move on to demonstration.</p><h3><a href="https://kubernetes.io/docs/concepts/workloads/controllers/statefulset">StatefulSets</a></h3><p>A StatefulSets is like a Deployment which manages Pods and guarantees about the ordering and uniqueness of these Pods. It maintains a sticky identity for each of their Pods. It helps in deployment of application that needs persistency, unique network identifiers (DNS, Hostnames etc) and are meants for stateful application. If a pod gets terminated or deleted, a volume data will remain intact if managed by persistentvolumes.</p><h3><a href="https://kubernetes.io/docs/concepts/storage/storage-classes/#gce">StorageClass</a></h3><p>StorageClass helps in administration to describe the “classes” of storage offered by Kubernetes. Each StorageClass has different provisioner (GCEPersistentDisk, AWSElasticBlockStore, AzureDisk etc) that determines what volume plugin is used for provisioning storage.</p><h3><a href="https://kubernetes.io/docs/concepts/storage/persistent-volumes/">PersistentVolume</a></h3><p>A PersistentVolume (PV) is a piece of storage in the cluster that has been provisioned by an administrator. PVs are resources available to be used by any Pod. Any Pod can claim these volumes by mean of PersistentVolumeClaims (PVC) and released eventually when claim is deleted.</p><h3><a href="https://kubernetes.io/docs/concepts/services-networking/service/#headless-services">Headless Services</a></h3><p>Headless Services are used to configure DNS of pods having same selectors defined by services. It is not generally used for load-balancing purpose. Each headless services configured with label selectors helps in defining unique network identifiers for pods running in statefulset.</p><p>Lets begins the demonstration. Please switch to your terminal and follow the instructions.</p><blockquote>Note : This setup is compatible with &lt;= mongo 3.2.</blockquote><h3>1. Prerequisites</h3><p>Ensure the following dependencies are already fulfilled on your host Linux system:</p><ol><li>GCP’s cloud client command line tool <a href="https://cloud.google.com/sdk/docs/quickstarts">gcloud</a></li><li>gcloud authentication to a project to manage container engine.</li><li>Install the Kubernetes command tool (“kubectl”),</li><li>Configure kubernetes authentication credentials.</li></ol><h3>2. Create namespace, storageclass, Google compute Disk and persistentvolumes.</h3><p>Our Mongodb Setup will be as follows :</p><ul><li>1x Config Server (k8s deployment type: “StatefulSet”)</li><li>2x Shards with each Shard being a Replica Set containing 1x replicas (k8s deployment type: “StatefulSet”)</li><li>2x Mongos Routers (k8s deployment type: “Deployment”)</li></ul><p>We will create a kubernetes namespace and will deploy all our above resources in our defined namespaces. We will define disk that will be used by our statefulset containers. Disk will be mounted on pods running our mongodb server by means of APIs defined in StorageClass and PersistentVolume.</p><h4>2.1 Create Namespace</h4><p>Create a file as namespace.yaml and replace NAMESPACE_ID with your handle name or any other name. I will create a namespace with daemonsl.</p><pre>#namespace.yaml<br>apiVersion: v1<br>kind: Namespace<br>metadata:<br>  name: NAMESPACE_ID<br></pre><pre>#To apply resources to kubernetes, run<br>sed -e &quot;s/NAMESPACE_ID/daemonsl/g&quot; namespace.yaml &gt; tmp-namespace.yaml<br>kubectl apply -f tmp-namespace.yaml</pre><pre>#To verify namespaces<br>kubectl get ns</pre><h4>2.2 Create StorageClass</h4><p>Create a file as gce-ssd-storageclass.yaml. We are defining our storageclass name as <strong>fast</strong> and using GCE persistent disk as our provisioner with type: pd-ssd to allow SSD disk type allocation to requester (ie, statefulset container here).</p><pre>#gce-ssd-storageclass.yaml<br>kind: StorageClass<br>apiVersion: storage.k8s.io/v1<br>metadata:<br>  name: fast<br>provisioner: kubernetes.io/gce-pd<br>parameters:<br>  type: pd-ssd</pre><pre>#To apply resources to kubernetes, run<br>kubectl apply -f gce-ssd-storageclass.yaml</pre><pre>#To verify storageclass<br>kubectl get sc</pre><h4>2.3 Create GCE SSD Disks</h4><p>We will create some disk to be used by mongodb statefulset container. We are ordering two 10GBdisk and one 5GB disk of type SSD.</p><pre>#For MainDB servers<br>gcloud compute disks create --size 10GB --type pd-ssd pd-ssd-disk-k8s-mongodb-daemonsl-10g-1<br>gcloud compute disks create --size 10GB --type pd-ssd pd-ssd-disk-k8s-mongodb-daemonsl-10g-2</pre><pre>#For Config servers<br>gcloud compute disks create --size 5GB --type pd-ssd pd-ssd-disk-k8s-mongodb-daemonsl-5g-1</pre><h4>2.4 Create PersistentVolume</h4><p>Create a file as ext4-gce-ssd-persistentvolume.yaml. We are defining our PersistentVolume storage capacity 10GB to be bounded by maindb pod and 5GB to be bounded by configdb pod.</p><pre>#ext4-gce-ssd-persistentvolume.yaml<br>apiVersion: &quot;v1&quot;<br>kind: &quot;PersistentVolume&quot;<br>metadata:<br>  name: data-volume-k8s-mongodb-daemonsl-SIZEg-INSTANCE<br>spec:<br>  capacity:<br>      storage: SIZEGi<br>  accessModes:<br>    - ReadWriteOnce<br>  persistentVolumeReclaimPolicy: Retain<br>  storageClassName: fast<br>  gcePersistentDisk:<br>    fsType: ext4<br>    pdName: pd-ssd-disk-k8s-mongodb-daemonsl-SIZEg-INSTANCE</pre><p>Use above template, modify and apply in following order :</p><pre>#Replace &#39;SIZE&#39; with 10 and &#39;INSTANCE&#39; with 1,<br><strong># Ex: data-volume-k8s-mongodb-daemonsl-10g-1, storage: 10Gi,<br></strong>sed -e &quot;s/INSTANCE/1/g; s/SIZE/10/g&quot; ext4-gce-ssd-persistentvolume.yaml &gt; tmp-ext4-gce-ssd-persistentvolume.yaml<br><strong>kubectl apply -f tmp-ext4-gce-ssd-persistentvolume.yaml</strong></pre><pre><strong>#Replace &#39;SIZE&#39; with 10 and &#39;INSTANCE&#39; with 2<br></strong>sed -e &quot;s/INSTANCE/2/g; s/SIZE/10/g&quot; ext4-gce-ssd-persistentvolume.yaml &gt; tmp-ext4-gce-ssd-persistentvolume.yaml<br><strong>kubectl apply -f tmp-ext4-gce-ssd-persistentvolume.yaml</strong></pre><pre><strong>#Replace &#39;SIZE&#39; with 5 and &#39;INSTANCE&#39; with 1<br></strong>sed -e &quot;s/INSTANCE/1/g; s/SIZE/5/g&quot; ext4-gce-ssd-persistentvolume.yaml &gt; tmp-ext4-gce-ssd-persistentvolume.yaml<br><strong>kubectl apply -f tmp-ext4-gce-ssd-persistentvolume.yaml</strong></pre><pre>#To verify PersistentVolume creation,<br>kubectl get pv</pre><p>Now we have <strong>storageclass, namespace, disks</strong> and <strong>persistentvolume</strong> ready as resources for statefulset container.</p><h3>3. StatefulSet containers and Mongos Deployment.</h3><h4>3.1 Statefulset ConfigDB</h4><p>Create a file as mongodb-configdb-service-stateful.yaml and copy the following template. Replace NAMESPACE_ID with daemonsl, or whatever name you have defined and DB_DISK with 5Gi.</p><p>We have created a headless service with clusterIP None with selector as role: mongodb-configdb listening to port 27019. We have defined our statefulset definition with mongodb arguments and volumeClaimTemplates. Here, VolumeClaimTemplates is requesting storageclass fast with storage capacity 5GB. This volumeClaimTemplates register this requests to storageclass and storageclass fulfill this requests by PersistentVolume (PV) and register claim in PersistentVolumeClaims (PVC).</p><pre>#mongodb-configdb-service-stateful.yaml<br>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: mongodb-configdb-headless-service<br>  namespace: NAMESPACE_ID<br>  labels:<br>    name: mongodb-configdb<br>spec:<br>  ports:<br>  - port: 27019<br>    targetPort: 27019<br>  clusterIP: None<br>  selector:<br>    role: mongodb-configdb<br>---<br>apiVersion: apps/v1beta2  #change this version based on master version<br>kind: StatefulSet<br>metadata:<br>  name: mongodb-configdb<br>  namespace: NAMESPACE_ID<br>spec:<br>  selector:<br>    matchLabels:<br>      role: mongodb-configdb # has to match .spec.template.metadata.labels<br>  serviceName: mongodb-configdb-headless-service<br>  replicas: 1<br>  template:<br>    metadata:<br>      labels:<br>        role: mongodb-configdb<br>        tier: configdb<br>    spec:<br>      affinity:<br>        podAntiAffinity:<br>          preferredDuringSchedulingIgnoredDuringExecution:<br>          - weight: 100<br>            podAffinityTerm:<br>              labelSelector:<br>                matchExpressions:<br>                - key: tier<br>                  operator: In<br>                  values:<br>                  - configdb<br>              topologyKey: kubernetes.io/hostname<br>      terminationGracePeriodSeconds: 10<br>      containers:<br>        - name: mongodb-configdb-container<br>          image: mongo<br>          command:<br>            - &quot;mongod&quot;<br>            - &quot;--port&quot;<br>            - &quot;27019&quot;<br>            - &quot;--dbpath&quot;<br>            - &quot;/mongo-disk&quot;<br>            - &quot;--bind_ip&quot;<br>            - &quot;0.0.0.0&quot;<br>            - &quot;--configsvr&quot;<br>          resources:<br>            requests:<br>              cpu: 50m<br>              memory: 100Mi<br>          ports:<br>            - containerPort: 27019<br>          volumeMounts:<br>            - name: mongodb-configdb-persistent-storage-claim<br>              mountPath: /mongo-disk<br>  volumeClaimTemplates:<br>  - metadata:<br>      name: mongodb-configdb-persistent-storage-claim<br>      annotations:<br>        volume.beta.kubernetes.io/storage-class: &quot;fast&quot;<br>    spec:<br>      accessModes: [ &quot;ReadWriteOnce&quot; ]<br>      resources:<br>        requests:<br>          storage: DB_DISK</pre><p>Now, apply resources via kubectl~</p><pre><strong>sed -e &quot;s/NAMESPACE_ID/daemonsl/g; s/DB_DISK/5Gi/g&quot; </strong> mongodb-configdb-service-stateful.yaml &gt; tmp-mongodb-configdb-service-stateful.yaml<br><strong>kubectl apply -f tmp-mongodb-configdb-service-stateful.yaml</strong></pre><h4>3.2 Statefulset mainDB</h4><p>Create a file as mongodb-maindb-service-stateful.yaml and copy the following template. Replace NAMESPACE_ID with daemonsl, or whatever name you have defined, DB_DISK with 10Giand shardX &amp; ShardX to 1 and then 2 and apply template two times to create two different statefulsets configuration. After deploying in kubernetes, we will have two statefulsets running with name as mongodb-shard1 and mongodb-shard2</p><p>Here again, We have created headless service and VolumeClaimTemplates which is requesting storageclass fast with storage capacity 10GB.</p><pre>#mongodb-maindb-service-stateful.yaml<br>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: mongodb-shardX-headless-service<br>  namespace: NAMESPACE_ID<br>  labels:<br>    name: mongodb-shardX<br>spec:<br>  ports:<br>  - port: 27017<br>    targetPort: 27017<br>  clusterIP: None<br>  selector:<br>    role: mongodb-shardX<br>---<br>apiVersion: apps/v1beta2<br>kind: StatefulSet<br>metadata:<br>  name: mongodb-shardX<br>  namespace: NAMESPACE_ID<br>spec:<br>  selector:<br>    matchLabels:<br>      role: mongodb-shardX # has to match .spec.template.metadata.labels<br>  serviceName: mongodb-shardX-headless-service<br>  replicas: 1<br>  template:<br>    metadata:<br>      labels:<br>        role: mongodb-shardX<br>        tier: maindb<br>        replicaset: ShardX<br>    spec:<br>      affinity:<br>        podAntiAffinity:<br>          preferredDuringSchedulingIgnoredDuringExecution:<br>          - weight: 100<br>            podAffinityTerm:<br>              labelSelector:<br>                matchExpressions:<br>                - key: replicaset<br>                  operator: In<br>                  values:<br>                  - ShardX<br>              topologyKey: kubernetes.io/hostname<br>      terminationGracePeriodSeconds: 10<br>      containers:<br>        - name: mongodb-shardX-container<br>          image: mongo<br>          command:<br>            - &quot;mongod&quot;<br>            - &quot;--port&quot;<br>            - &quot;27017&quot;<br>            - &quot;--bind_ip&quot;<br>            - &quot;0.0.0.0&quot;<br>            - &quot;--replSet&quot;<br>            - &quot;ShardX&quot;<br>            - &quot;--dbpath&quot;<br>            - &quot;/mongo-disk&quot;<br>          resources:<br>            requests:<br>              cpu: 50m<br>              memory: 100Mi<br>          ports:<br>            - containerPort: 27017<br>          volumeMounts:<br>            - name: mongo-shardX-persistent-storage-claim<br>              mountPath: /mongo-disk<br>  volumeClaimTemplates:<br>  - metadata:<br>      name: mongo-shardX-persistent-storage-claim<br>      annotations:<br>        volume.beta.kubernetes.io/storage-class: &quot;fast&quot;<br>    spec:<br>      accessModes: [ &quot;ReadWriteOnce&quot; ]<br>      resources:<br>        requests:<br>          storage: DB_DISK</pre><p>Apply resources via kubectl~</p><pre>#replace &#39;shardX&#39; &amp; &#39;ShardX&#39; with shard1 &amp; Shard1.<br><strong>sed -e &quot;s/shardX/shard1/g; s/ShardX/Shard1/g; s/NAMESPACE_ID/daemonsl/g; s/DB_DISK/10Gi/g&quot;</strong> mongodb-maindb-service-stateful.yaml &gt; tmp-mongodb-maindb-service-stateful.yaml<br><strong>kubectl apply -f tmp-mongodb-maindb-service-stateful.yaml</strong></pre><pre>#replace &#39;shardX&#39; &amp; &#39;ShardX&#39; with shard2 &amp; Shard2. <br><strong>sed -e &quot;s/shardX/shard2/g; s/ShardX/Shard2/g; s/NAMESPACE_ID/daemonsl/g; s/DB_DISK/10Gi/g&quot;</strong> mongodb-maindb-service-stateful.yaml &gt; tmp-mongodb-maindb-service-stateful.yaml<br><strong>kubectl apply -f tmp-mongodb-maindb-service-stateful.yaml</strong></pre><pre>#run command to see Pods &amp; Services spinning up<br>kubectl get svc,po --namespace=daemonsl</pre><p>Till here, we have accomplished statefulsets container running along with headless services and mounted a SSD volume that fulfills Pods requirement.</p><pre>kubectl get persistentvolumes</pre><pre># Get persistent volume claims<br>kubectl get persistentvolumeclaims --namespace=daemonsl</pre><h4>3.3 Mongos Deployment</h4><p>We have configdb and maindb pods up and running. We will spin up mongos server to establish a sharding cluster. Replace NAMESPACE_ID with daemonsl, or whatever name you have defined.</p><p>We have configured config server information in mongos using --configdb flag with unique network identifiers of configdb pod. DNS of statefulset pods goes by convention &lt;POD_NAME&gt;.&lt;SERVICE_NAME&gt;.&lt;NAMESPACE&gt;.svc.&lt;CLUSTER_DOMAIN&gt;.</p><p><strong>Reference : </strong><a href="https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#stable-network-id">https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#stable-network-id</a></p><pre>apiVersion: apps/v1beta1<br>kind: Deployment<br>metadata:<br>  name: mongos<br>  <strong>namespace</strong>: NAMESPACE_ID<br>spec:<br>  replicas: 2<br>  template:<br>    metadata:<br>      labels:<br>        role: mongos<br>        tier: routers<br>    spec:<br>      affinity:<br>        podAntiAffinity:<br>          preferredDuringSchedulingIgnoredDuringExecution:<br>          - weight: 100<br>            podAffinityTerm:<br>              labelSelector:<br>                matchExpressions:<br>                - key: tier<br>                  operator: In<br>                  values:<br>                  - routers<br>              topologyKey: kubernetes.io/hostname<br>      terminationGracePeriodSeconds: 10<br>      containers:<br>        - name: mongos-container<br>          image: mongo<br>          command:<br>            - &quot;mongos&quot;<br>            - &quot;--port&quot;<br>            - &quot;27017&quot;<br>            - &quot;--bind_ip&quot;<br>            - &quot;0.0.0.0&quot;<br>            - &quot;--configdb&quot;<br>            - &quot;mongodb-configdb-0.mongodb-configdb-headless-service.daemonsl.svc.cluster.local:27019&quot;<br>          resources:<br>            requests:<br>              cpu: 50m<br>              memory: 100Mi<br>          ports:<br>            - containerPort: 27017</pre><p>Apply resources via kubectl~</p><pre><strong>sed -e &quot;s/NAMESPACE_ID/daemonsl/g&quot;</strong> mongodb-mongos-deployment-service.yaml &gt; tmp-mongodb-mongos-deployment-service.yaml<br><strong>kubectl apply -f tmp-mongodb-mongos-deployment-service.yaml</strong></pre><h3>4. Configure Sharding</h3><p>Now, we have mongos, configdb and maindb up and running. We need to create Replicaset in MainDB servers that we are intending to make shard. We will run rs.initiate() command to make PRIMARY replica. Since we are going with one replica member in each shard. We will run initiate command in each of the maindb pod.</p><pre><strong>echo</strong> &quot;Replicaset Init mongodb-shard1-0 &quot;<br><strong>kubectl exec</strong> --namespace=daemonsl mongodb-shard1-0 -c mongodb-shard1-container -- mongo --port 27017 --eval &quot;rs.initiate({_id: \&quot;Shard1\&quot;, version: 1, <strong>members: [ {_id: 0, host: \&quot;mongodb-shard1-0.mongodb-shard1-headless-service.daemonsl.svc.cluster.local:27017\&quot;} ]</strong> });&quot;<br></pre><pre><strong>echo</strong> &quot;Replicaset Init mongodb-shard2-0 &quot;  <br><strong>kubectl exec</strong> --namespace=daemonsl mongodb-shard2-0 -c mongodb-shard2-container -- mongo --port 27017 --eval &quot;rs.initiate({_id: <strong>\&quot;</strong>Shard2<strong>\&quot;</strong>, version: 1,<strong> members: [ {_id: 0, host: \&quot;mongodb-shard2-0.mongodb-shard2-headless-service.daemonsl.svc.cluster.local:27017\&quot;} ]</strong> });&quot;</pre><p>Above lines, will make both pods PRIMARY of their respective replicaset. You can even go into container to verify replicaset status by running rs.status() command.</p><p>We are proceeding now to add shards to mongos server. We will run below command in any of the mongos pod. Mongos server are stateless application, they save the configuration in configdb server which we have made stateful application by declaring them under statefulset container.</p><pre><strong>echo</strong> &quot;Adding Shard 1 : Shard1 &quot;<br><strong>kubectl</strong> <strong>exec</strong> <strong>--namespace</strong>=<strong>daemonsl</strong> $(<strong>kubectl</strong> <strong>get</strong> <strong>pod</strong> <strong>-l</strong> &quot;tier=routers&quot; <strong>-o</strong> <strong>jsonpath</strong>=&#39;{.items[0].metadata.name}&#39; <strong>--namespace</strong>=<strong>daemonsl</strong> ) <strong>-c</strong> <strong>mongos-container</strong> <strong>--</strong> <strong>mongo</strong> <strong>--port</strong> <strong>27017</strong> <strong>--eval</strong> &quot;sh.addShard(\&quot;Shard1/mongodb-shard1-0.mongodb-shard1-headless-service.daemonsl.svc.cluster.local:27017\&quot;);&quot;</pre><pre><strong>echo</strong> &quot;Adding Shard 2 : Shard2 &quot;<br><strong>kubectl</strong> <strong>exec</strong> <strong>--namespace</strong>=<strong>daemonsl</strong> $(<strong>kubectl</strong> <strong>get</strong> <strong>pod</strong> <strong>-l</strong> &quot;tier=routers&quot; <strong>-o</strong> <strong>jsonpath</strong>=&#39;{.items[0].metadata.name}&#39; <strong>--namespace</strong>=<strong>daemonsl</strong> ) <strong>-c</strong> <strong>mongos-container</strong> <strong>--</strong> <strong>mongo</strong> <strong>--port</strong> <strong>27017</strong> <strong>--eval</strong> &quot;sh.addShard(\&quot;Shard2/mongodb-shard2-0.mongodb-shard2-headless-service.daemonsl.svc.cluster.local:27017\&quot;);&quot;</pre><p>Now, we can get into one of the mongos container to verify the sharding status of cluster. All the above steps can be automated to make any number of shards within your cluster and thus concepts are very trivial to support stateful application powered by GKE.</p><h4>Test Sharding</h4><p>To test that the sharded cluster is working properly, connect to the container running the first “mongos” router, then use the Mongo Shell to authenticate, enable sharding on a specific database &amp; collection, add some test data to this collection and then view the status of the Sharded cluster and collection:</p><pre>$ kubectl exec -it <strong>$(</strong>kubectl get pod -l &quot;tier=routers&quot; -o jsonpath=&#39;{.items[0].metadata.name}&#39;<strong>)</strong> -c mongos-container bash<br>$ mongo<br>&gt; sh.enableSharding(&quot;&lt;Database_name&gt;&quot;);<br>&gt; sh.status();<br>&gt; use admin<br>&gt; db.admin.runCommand(&quot;getShardMap&quot;)</pre><h4>Tearing &amp; Cleaning Down the Kubernetes Environment</h4><p><strong>Important:</strong> This step is required to ensure you aren’t continuously charged by Google Cloud for an environment you no longer need.</p><p>Run the following script to undeploy the MongoDB Services &amp; StatefulSets/Deployments plus related Kubernetes resources, followed by the removal of the GCE disks. This script is available in repository.</p><pre>$ sh teardown.sh   <em>#To delete all resources provisioned above</em></pre><h3>Factors Addressed in this Demonstration</h3><ul><li>Deployment of a MongoDB on the Google Kubernetes Engine</li><li>Use of Kubernetes StatefulSets and PersistentVolumeClaims to ensure data is not lost when containers are recycled</li><li>Proper configuration of a MongoDB Sharded Cluster for Scalability with each Shard being a Replica Set for full resiliency</li><li>Controlling Anti-Affinity for Mongod Replicas to avoid a Single Point of Failure</li></ul><p><strong>Github reference</strong> : <a href="https://github.com/sunnykrGupta/gke-mongodb-shards">https://github.com/sunnykrGupta/gke-mongodb-shards</a></p><p><strong>Credit :</strong> This blog is based on workdone by <a href="https://twitter.com/TheDonester">Paul Done</a></p><h4>Must read below resources in order :</h4><ul><li><a href="https://kubernetes.io/docs/concepts/workloads/controllers/statefulset">https://kubernetes.io/docs/concepts/workloads/controllers/statefulset</a></li><li><a href="https://kubernetes.io/docs/concepts/services-networking/service/#headless-services">https://kubernetes.io/docs/concepts/services-networking/service/#headless-services</a></li><li><a href="https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#stable-network-id">https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#stable-network-id</a></li><li><a href="https://kubernetes.io/docs/concepts/storage/storage-classes/#gce">https://kubernetes.io/docs/concepts/storage/storage-classes/#gce</a></li><li><a href="https://kubernetes.io/docs/concepts/storage/persistent-volumes/">https://kubernetes.io/docs/concepts/storage/persistent-volumes/</a></li><li><a href="http://blog.kubernetes.io/2017/03/dynamic-provisioning-and-storage-classes-kubernetes.html">http://blog.kubernetes.io/2017/03/dynamic-provisioning-and-storage-classes-kubernetes.html</a></li><li><a href="http://blog.kubernetes.io/2017/03/advanced-scheduling-in-kubernetes.html">http://blog.kubernetes.io/2017/03/advanced-scheduling-in-kubernetes.html</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ba08c7c0c0b0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/sharded-mongodb-in-kubernetes-statefulsets-on-gke-ba08c7c0c0b0">Sharded Mongodb in Kubernetes StatefulSets on GKE</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Patch and Update Table — BigQuery Part-III]]></title>
            <link>https://medium.com/google-cloud/patch-and-update-table-bigquery-part-iii-7f2547021926?source=rss-f6c22db16b21------2</link>
            <guid isPermaLink="false">https://medium.com/p/7f2547021926</guid>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[bigquery]]></category>
            <dc:creator><![CDATA[Sunny Kr.]]></dc:creator>
            <pubDate>Sat, 04 Nov 2017 14:25:35 GMT</pubDate>
            <atom:updated>2022-01-30T22:25:39.508Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Originally published.</em></p><p>Daemon blog : <a href="https://sunnykrgupta.github.io/patch-and-update-table-bigquery-part-iii.html">https://sunnykrgupta.github.io/patch-and-update-table-bigquery-part-iii.html</a></p><p><strong>Other posts in the series:</strong></p><ul><li><a href="https://medium.com/google-cloud/export-load-job-with-mongodb-bigquery-part-i-64a00eb5266b">Export &amp; Load Job with MongoDB — BigQuery Part-I | by Sunny Gupta | Google Cloud — Community | Medium</a></li><li><a href="https://medium.com/google-cloud/streaming-with-redis-bigquery-part-ii-763cdec9a9ec">Streaming with Redis — BigQuery Part-II | by Sunny Gupta | Google Cloud — Community | Medium</a></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/440/0*4dorQigbt7af0Fjm.png" /></figure><p>This post is 3rd part of 3-post series. In the earlier post, we understood fundamentals of BigQuery Load Jobs <a href="https://sunnykrgupta.github.io/export-load-job-with-mongodb-bigquery-part-i.html"><strong>Export &amp; Load Job with MongoDB — BigQuery Part-I</strong></a><strong> </strong>and<strong> </strong>the streaming in BigQuery <a href="https://sunnykrgupta.github.io/streaming-with-redis-bigquery-part-ii.html"><strong>Streaming with Redis — BigQuery Part-II</strong></a>.</p><p>In this post, we are going to learn patching and updating table schemas.</p><h3>Update/Patch Table in BigQuery</h3><p>Table updation is important when you have data ready inside the table and suddenly you have a requirement of adding more fields in the table to do analysis.</p><p>Here, we are going to add a field and see the changes in the table after update operation.</p><h4>1. Add field into schema</h4><p>We are going to use same table <strong>StreamTable</strong> which we used earlier for streaming and introducing one more field as <strong>UniqueSocialNumber</strong> with datatype <strong>INTEGER</strong>. Lets add these changes to <strong>schema.py</strong> which will be used by our main program written in later steps.</p><pre>$ cat schema.py</pre><pre><em>#Add field schema</em><br>TableObject = {<br>    &quot;tableReference&quot;: {<br>      &quot;projectId&quot;: &quot;mimetic-slate&quot;,<br>      &quot;tableId&quot;: &quot;StreamTable&quot;,<br>      &quot;datasetId&quot;: &quot;BQ_Dataset&quot;,<br>    },</pre><pre>    &quot;schema&quot;: {<br>      &quot;fields&quot;: [<br>          {<br>              &quot;name&quot;: &quot;username&quot;,<br>              &quot;type&quot;: &quot;STRING&quot;,<br>              &quot;mode&quot;: &quot;NULLABLE&quot;<br>          },<br>          {<br>              &quot;name&quot;: &quot;name&quot;,<br>              &quot;type&quot;: &quot;STRING&quot;,<br>              &quot;mode&quot;: &quot;NULLABLE&quot;<br>          },<br>          {<br>              &quot;name&quot;: &quot;birthdate&quot;,<br>              &quot;type&quot;: &quot;STRING&quot;,<br>              &quot;mode&quot;: &quot;NULLABLE&quot;<br>          },<br>          {<br>              &quot;name&quot;: &quot;sex&quot;,<br>              &quot;type&quot;: &quot;STRING&quot;,<br>              &quot;mode&quot;: &quot;NULLABLE&quot;<br>          },<br>          {<br>              &quot;name&quot;: &quot;address&quot;,<br>              &quot;type&quot;: &quot;STRING&quot;,<br>              &quot;mode&quot;: &quot;NULLABLE&quot;<br>          },<br>          {<br>              &quot;name&quot;: &quot;mail&quot;,<br>              &quot;type&quot;: &quot;STRING&quot;,<br>              &quot;mode&quot;: &quot;NULLABLE&quot;<br>          },<br>          {<br>              &quot;name&quot;: &quot;UniqueSocialNumber&quot;,   <em>#New Field</em><br>              &quot;type&quot;: &quot;INTEGER&quot;,<br>              &quot;mode&quot;: &quot;NULLABLE&quot;<br>          }<br>      ],<br>  },<br>}</pre><h4>2. Patch/Update API in BigQuery</h4><p>We have schema.py script ready and below is our main program <strong>tablePatch.py</strong>that will execute the table patch API call to bigquery.</p><p>We have two methods available in BigQuery to make updates in table.</p><ul><li><strong>tables.patch</strong> - This method only replaces fields that are provided in the resources</li><li><strong>tables.update</strong> - This method replaces the entire table resource.</li></ul><blockquote><strong>Patch Example :</strong></blockquote><pre>$ cat tablePatch.py</pre><pre><em>#!/usr/bin/env python</em></pre><pre><em>#https://developers.google.com/api-client-library/python/</em><br><strong>from</strong> <strong>googleapiclient</strong> <strong>import</strong> discovery<br><strong>from</strong> <strong>oauth2client.client</strong> <strong>import</strong> GoogleCredentials<br><strong>from</strong> <strong>schema</strong> <strong>import</strong> TableObject<br></pre><pre><em># [START Table Creater ]</em><br><strong>def</strong> PatchTable(bigquery):<br>    tables = bigquery.tables()</pre><pre>    tableStatusObject =tables.patch(                         <br>          projectId=TableObject[&#39;tableReference&#39;][&#39;projectId&#39;],\<br>          datasetId=TableObject[&#39;tableReference&#39;][&#39;datasetId&#39;],\<br>          tableId=TableObject[&#39;tableReference&#39;][&#39;tableId&#39;], \<br>          body=TableObject).execute()<br>    <strong>print</strong> &quot;<strong>\n\n</strong>Table Patched&quot;<br><em># [END]</em></pre><pre><strong>def</strong> main():<br>    <em>#to get credentials from my laptop</em><br>    credentials = GoogleCredentials.get_application_default()<br>    <em># Construct the service object for interacting with the BigQuery API.</em><br>    bigquery = discovery.build(&#39;bigquery&#39;, &#39;v2&#39;, credentials=credentials)<br>    PatchTable(bigquery)</pre><pre><em>&#39;&#39;&#39;</em><br><em>https://cloud.google.com/bigquery/docs/tables#update-schema</em><br><em>&#39;&#39;&#39;</em></pre><pre><strong>if</strong> __name__ == &#39;__main__&#39;:<br>    main()<br>    <strong>print</strong> &quot;BQ Table Patch !!&quot;</pre><blockquote>Update Example :</blockquote><pre>#instead of calling patch(), we call update() to apply updates<br>tableStatusObject =tables.update(                         <br>          projectId=TableObject[&#39;tableReference&#39;][&#39;projectId&#39;],\<br>          datasetId=TableObject[&#39;tableReference&#39;][&#39;datasetId&#39;],\<br>          tableId=TableObject[&#39;tableReference&#39;][&#39;tableId&#39;], \<br>          body=TableObject).execute()</pre><p>After applying changes we can verify changes in schema of bigquery table.</p><p>Visit UI :<a href="https://bigquery.cloud.google.com/">https://bigquery.cloud.google.com</a></p><p>You can also verify table schema by running <strong>bq</strong> CLI commands</p><pre>$ bq show BQ_Dataset.StreamTable<br>Table mimetic-slate:BQ_Dataset.StreamTable</pre><pre>   Last modified                Schema               Total Rows   Total Bytes   Expiration   Time Partitioning   Labels  <br> ----------------- -------------------------------- ------------ ------------- ------------ ------------------- --------<br>  04 Nov 15:48:43   |-username: string              451          50837                                                  <br>                    |-name:string                                                                                      <br>                    |-birthdate:string                                                                                 <br>                    |-sex:string                                                                                       <br>                    |-address:string                                                                                   <br>                    |-mail:string                                                                                      <br>                    |-UniqueSocialNumber: integer</pre><p><strong>Reference </strong>:</p><ul><li><strong>Table Patch and Update :</strong> <a href="https://developers.google.com/resources/api-libraries/documentation/bigquery/v2/python/latest/bigquery_v2.tables.html">https://developers.google.com/resources/api-libraries/documentation/bigquery/v2/python/latest/bigquery_v2.tables.html</a></li></ul><h4>3. Preview of data in bigquery table</h4><p>Click on table preview to see new field. It will show <strong>null</strong> with old records. You can start streaming data which contains this newly added field to be written in table going forward.</p><p><strong>Table Preview</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hxQMjePUvTMIB8Dw.png" /></figure><p><strong>Github reference</strong> : <a href="https://github.com/sunnykrGupta/Bigquery-series">https://github.com/sunnykrGupta/Bigquery-series</a></p><h3>Conclusion</h3><p>This was introduction to updation in BigQuery table, thus concludes our 3-post series.</p><p>From this series, we covered basic features of Google BigQuery serverless product. There are other similar product available on other clouds like AWS redshift, Azure SQL Data Warehouse which serve infinite computing resources like BigQuery.</p><p>Please comment your thoughts/feedback about this series.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7f2547021926" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/patch-and-update-table-bigquery-part-iii-7f2547021926">Patch and Update Table — BigQuery Part-III</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Streaming with Redis — BigQuery Part-II]]></title>
            <link>https://medium.com/google-cloud/streaming-with-redis-bigquery-part-ii-763cdec9a9ec?source=rss-f6c22db16b21------2</link>
            <guid isPermaLink="false">https://medium.com/p/763cdec9a9ec</guid>
            <category><![CDATA[streaming]]></category>
            <category><![CDATA[bigquery]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[redis]]></category>
            <dc:creator><![CDATA[Sunny Kr.]]></dc:creator>
            <pubDate>Thu, 02 Nov 2017 06:09:04 GMT</pubDate>
            <atom:updated>2022-01-30T22:24:22.173Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Originally published. More words added in some topic that needs explaining.</em></p><p>Daemon blog : <a href="https://sunnykrgupta.github.io/streaming-with-redis-bigquery-part-ii.html">https://sunnykrgupta.github.io/streaming-with-redis-bigquery-part-ii.html</a></p><p><strong>Other posts in the series:</strong></p><ul><li><a href="https://medium.com/google-cloud/export-load-job-with-mongodb-bigquery-part-i-64a00eb5266b">Export &amp; Load Job with MongoDB — BigQuery Part-I | by Sunny Gupta | Google Cloud — Community | Medium</a></li><li><a href="https://medium.com/google-cloud/patch-and-update-table-bigquery-part-iii-7f2547021926">Patch and Update Table — BigQuery Part-III | by Sunny Gupta | Google Cloud — Community | Medium</a></li></ul><p>This is 2nd part of 3-post series. In the earlier post, we understood the fundamentals of BigQuery Load Jobs <a href="https://sunnykrgupta.github.io/export-load-job-with-mongodb-bigquery-part-i.html"><strong>Export &amp; Load Job with MongoDB — BigQuery Part-I</strong></a>.</p><p>In this post, we are going to dive into Streaming feature of BigQuery.</p><h3><a href="https://cloud.google.com/bigquery/streaming-data-into-bigquery">Streaming</a></h3><p><strong>Why ?</strong> — Streaming helps in pushing our data into BigQuery (short for BQ) and helps in making data available for query without delay of running load jobs.</p><blockquote>There are some trade-offs to choose Streaming. A few are belows :</blockquote><ul><li>We need to follow a few quotas like http body size, maximum rows / request etc while making streaming API calls.</li><li>Written data in tables are not instantly available for copy or for export jobs in bigquery, it will take upto 90 minutes to be made available while load based tables are available instantly.</li><li>At the time of writing this post, charges incurred in streaming whereas load jobs were free.</li></ul><p>Keeping above in mind, we need to choose streaming vs load jobs in BigQuery.</p><p><strong>Quotas :</strong> <a href="https://cloud.google.com/bigquery/quotas#streaminginserts">https://cloud.google.com/bigquery/quotas#streaminginserts</a></p><h3>Streaming Data into BigQuery</h3><p>In this article, we are going to use a <a href="https://redis.io/">redis server</a> as a message broker to hold our data.</p><p>We are going to prepare data and the skeleton of data is going to be basic information of any person (username, name, birthdate, sex, address, email). As per this information, we need schema and table in bigquery to be created in advance before streaming. Post table creation, we are going to run streaming program to ingest our data in bulk which will be read from redis and same will be written to bigquery table in real time. We are going to use python as our programming language.</p><h4>1. Prepare data in Redis</h4><p>We are going to write a small python script to preapare data in <a href="https://redis.io/topics/data-types">redis List</a>.</p><p><strong>Redis Installation :</strong> <a href="https://redis.io/download">https://redis.io/download</a></p><pre><em>#!/usr/bin/env python</em></pre><pre><strong>import</strong> <strong>redis</strong><br><strong>import</strong> <strong>json</strong>, <strong>time</strong><br><em>#Faker : </em><a href="https://github.com/joke2k/faker"><em>https://github.com/joke2k/faker</em></a><br><strong>from</strong> <strong>faker</strong> <strong>import</strong> Faker</pre><pre>streamRedis = redis.Redis(host=&#39;127.0.0.1&#39;,<br>             port=&#39;6379&#39;,<br>            password=&#39;&#39;)</pre><pre>fake = Faker()</pre><pre><strong>def</strong> profile_generator():<br>    <strong>return</strong> fake.simple_profile(sex=None)</pre><pre><strong>def</strong> main():<br>    <strong>while</strong> True:<br>        streamRedis.lpush(&quot;redisList&quot;, json.dumps(profile_generator()) )<br>        <em>#sleep 200ms</em><br>        time.sleep(0.2)</pre><pre><strong>if</strong> __name__ == &quot;__main__&quot;:<br>    main()</pre><h4>2. Inspect data and prepare schema for Table</h4><p>Our data looks like :</p><pre>{<br>    &#39;username&#39;: u&#39;tarawade&#39;,<br>    &#39;name&#39;: u&#39;Jennifer Lewis&#39;,<br>    &#39;birthdate&#39;: &#39;2005-06-14&#39;,<br>    &#39;sex&#39;: &#39;F&#39;,<br>    &#39;address&#39;: u&#39;7134 Robinson Club Apt. 530\nPort Andreachester, GA 19011-6162&#39;,<br>    &#39;mail&#39;: u&#39;tmorgan@yahoo.com&#39;<br>}</pre><p>For the above formatted data, below schema is going to work</p><pre>[<br>    {<br>        &quot;name&quot;: &quot;username&quot;, &quot;type&quot;: &quot;STRING&quot;, &quot;mode&quot;: &quot;NULLABLE&quot;<br>    },<br>    {<br>        &quot;name&quot;: &quot;name&quot;, &quot;type&quot;: &quot;STRING&quot;, &quot;mode&quot;: &quot;NULLABLE&quot;<br>    },<br>    {<br>        &quot;name&quot;: &quot;birthdate&quot;, &quot;type&quot;: &quot;STRING&quot;, &quot;mode&quot;: &quot;NULLABLE&quot;<br>    },<br>    {<br>        &quot;name&quot;: &quot;sex&quot;, &quot;type&quot;: &quot;STRING&quot;, &quot;mode&quot;: &quot;NULLABLE&quot;<br>    },<br>    {<br>        &quot;name&quot;: &quot;address&quot;, &quot;type&quot;: &quot;STRING&quot;, &quot;mode&quot;: &quot;NULLABLE&quot;<br>    },<br>    {<br>        &quot;name&quot;: &quot;mail&quot;, &quot;type&quot;: &quot;STRING&quot;, &quot;mode&quot;: &quot;NULLABLE&quot;<br>    }<br>]</pre><h4>3. Create a table in BigQuery</h4><p>We are going to create two python file, ie, <strong>createConfig.py</strong> that will keep schema configuration and <strong>tableCreate.py</strong> that will execute the table creation API call to bigquery. We are going to use <strong>Google Application Default Credentials</strong> to authorize our python application to talk to bigquery APIs.</p><pre>$ cat createConfig.py</pre><pre>TableObject = {<br>    &quot;tableReference&quot;: {<br>      &quot;projectId&quot;: &quot;mimetic-slate&quot;,<br>      &quot;tableId&quot;: &quot;StreamTable&quot;,<br>      &quot;datasetId&quot;: &quot;BQ_Dataset&quot;,<br>    },</pre><pre>    &quot;schema&quot;: {<br>      &quot;fields&quot;: [<br>          {<br>              &quot;name&quot;: &quot;username&quot;,<br>              &quot;type&quot;: &quot;STRING&quot;,<br>              &quot;mode&quot;: &quot;NULLABLE&quot;<br>          },<br>          {<br>              &quot;name&quot;: &quot;name&quot;,<br>              &quot;type&quot;: &quot;STRING&quot;,<br>              &quot;mode&quot;: &quot;NULLABLE&quot;<br>          },<br>          {<br>              &quot;name&quot;: &quot;birthdate&quot;,<br>              &quot;type&quot;: &quot;STRING&quot;,<br>              &quot;mode&quot;: &quot;NULLABLE&quot;<br>          },<br>          {<br>              &quot;name&quot;: &quot;sex&quot;,<br>              &quot;type&quot;: &quot;STRING&quot;,<br>              &quot;mode&quot;: &quot;NULLABLE&quot;<br>          },<br>          {<br>              &quot;name&quot;: &quot;address&quot;,<br>              &quot;type&quot;: &quot;STRING&quot;,<br>              &quot;mode&quot;: &quot;NULLABLE&quot;<br>          },<br>          {<br>              &quot;name&quot;: &quot;mail&quot;,<br>              &quot;type&quot;: &quot;STRING&quot;,<br>              &quot;mode&quot;: &quot;NULLABLE&quot;<br>          }<br>      ],<br>  },<br>}</pre><p>We are going to use <a href="https://developers.google.com/api-client-library/python/"><strong>google-api-python-client</strong></a> library for interacting to our bigquery APIs.</p><pre><em>#!/usr/bin/env python</em></pre><pre><strong>from</strong> <strong>googleapiclient</strong> <strong>import</strong> discovery<br><strong>from</strong> <strong>oauth2client.client</strong> <strong>import</strong> GoogleCredentials<br><strong>from</strong> <strong>tableCreate</strong> <strong>import</strong> TableObject<br></pre><pre><strong>def</strong> createTable(bigquery):<br>    tables = bigquery.tables()</pre><pre>    <em>#insert utility make call to BQ API with payload \</em><br>    <em>#(TableObject) contains schema and table-name information</em><br>    tableStatusObject = tables.insert( projectId=&#39;mimetic-slate&#39;, \<br>     datasetId=&#39;BQ_Dataset&#39;, body=TableObject).execute()<br><em># [END]</em></pre><pre><strong>def</strong> main():<br>    <em>#to get credentials from my laptop</em><br>    credentials = GoogleCredentials.get_application_default()<br>    <em># Construct the service object for interacting with the BigQuery API.</em><br>    bigquery = discovery.build(&#39;bigquery&#39;, &#39;v2&#39;, credentials=credentials)<br>    createTable(bigquery)</pre><pre><strong>if</strong> __name__ == &#39;__main__&#39;:<br>    main()<br>    <strong>print</strong> &quot;BQ Table Creator !!&quot;</pre><p>Run the above program to create table with name <strong>StreamTable</strong> in bigquery dataset <strong>BQ_Dataset</strong>. Make sure you have created dataset already.</p><p>You can verify the table created by visiting bigquery UI. Visit : <a href="https://bigquery.cloud.google.com/">https://bigquery.cloud.google.com</a></p><p><strong>Reference </strong>:</p><ul><li><strong>Table Creation : </strong><a href="https://developers.google.com/resources/api-libraries/documentation/bigquery/v2/python/latest/bigquery_v2.tables.html">https://developers.google.com/resources/api-libraries/documentation/bigquery/v2/python/latest/bigquery_v2.tables.html</a></li><li><strong>Google Application Default Credentials : </strong><a href="https://developers.google.com/identity/protocols/application-default-credentials">https://developers.google.com/identity/protocols/application-default-credentials</a></li><li><strong>Build the service object :</strong> <a href="https://developers.google.com/api-client-library/python/start/get_started#build-the-service-object">https://developers.google.com/api-client-library/python/start/get_started#build-the-service-object</a></li></ul><h4>4. Streaming into Bigquery</h4><p>Now, we have table created and data queued into redis list, We are ready to stream right away by running a python script, lets call this script <strong>bq-streamer.py</strong></p><pre><em>#!/usr/bin/env python</em></pre><pre><strong>from</strong> <strong>googleapiclient</strong> <strong>import</strong> discovery<br><strong>from</strong> <strong>oauth2client.client</strong> <strong>import</strong> GoogleCredentials</pre><pre><strong>import</strong> <strong>json</strong>, <strong>time</strong>, <strong>copy</strong><br><strong>import</strong> <strong>redis</strong><br></pre><pre>batchCount = 100</pre><pre>redisStream = redis.Redis(host=&#39;127.0.0.1&#39;,<br>             port=&#39;6379&#39;,<br>            password=&#39;&#39;)</pre><pre>streamObject = {<br>    &quot;rows&quot;: [<br>      <em>#{ &quot;json&quot;: {# Represents a single JSON object. } }</em><br>    ],<br> }</pre><pre><em>#[START Streaming batcher]</em><br><strong>def</strong> streamBuilder():<br>    <em>#Every API needs a refresh copy of dict</em><br>    newStreamObject = copy.deepcopy(streamObject)</pre><pre>    currentCounter = 0<br>    <strong>while</strong> currentCounter &lt; batchCount:<br>        packet = redisStream.brpop(&quot;redisList&quot;,  timeout=0)[1]<br>        newStreamObject[&quot;rows&quot;].append({&quot;json&quot;:json.loads(packet)})<br>        currentCounter +=  1</pre><pre>    <strong>return</strong> newStreamObject<br><em>#[END]</em><br></pre><pre><em># [START Streaming Utility]</em><br><strong>def</strong> streamUtils(bigquery):<br>    tabledata = bigquery.tabledata()</pre><pre>    <em>#Run infinitely</em><br>    <strong>while</strong> True:<br>        streamBuildBatch = streamBuilder()</pre><pre>        <em>#BQ API to insert bulk data into table</em><br>        insertStatusObject= tabledata.insertAll( \<br>                   projectId=&#39;mimetic-slate&#39;, \<br>                   datasetId=&#39;BQ_Dataset&#39;, tableId=&#39;StreamTable&#39;, \<br>                   body=streamBuildBatch).execute()<br></pre><pre><em># [ MAIN]</em><br><strong>def</strong> main():<br>    credentials = GoogleCredentials.get_application_default()<br>    <em># Construct the service object for interacting with the BigQuery API.</em><br>    bigquery = discovery.build(&#39;bigquery&#39;, &#39;v2&#39;, credentials=credentials)</pre><pre>    <em>#Stream utility</em><br>    streamUtils(bigquery)<br><em># [END]</em><br></pre><pre><strong>if</strong> __name__ == &#39;__main__&#39;:<br>    main()</pre><p>Above program is going to read redis running on 127.0.0.1:6379 from list name <strong>redisList</strong> and build a dict object <strong>streamObject</strong> that is accepted by bq streaming API. We are calling <strong>insertAll</strong> utility to submit our streaming request to bigquery API.</p><p><strong>insertAll</strong> takes <strong>projectId, datasetId, tableId</strong> as an argument and <strong>body</strong> which contains your data to be streamed.</p><p>Script has been configured to <a href="https://redis.io/commands/brpop">pop</a> 100 entries from redis list and prepare it to be pushed into table. Run <strong>bq-streamer.py</strong> script to start streaming data into bigquery table.</p><p><strong>Reference</strong> :</p><ul><li><strong>Streaming insertAll :</strong> <a href="https://developers.google.com/resources/api-libraries/documentation/bigquery/v2/python/latest/bigquery_v2.tabledata.html">https://developers.google.com/resources/api-libraries/documentation/bigquery/v2/python/latest/bigquery_v2.tabledata.html</a></li></ul><h4>5. Verify the data in BigQuery Table</h4><p>After running streaming, you will start seeing something similar as shown below when you click table info. Clicking on preview will not show you any streamed data, it will take a while to appear but it will be in buffer to be available for query instantly.</p><p><strong>Buffer Statistics</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/393/1*wo2MLplXNvJb6xvVxcxo1A.png" /></figure><p><strong>Table Preview</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/786/1*uFtCcXhX0tyQ3WWVszCNUw.png" /></figure><h3>SQL Query in BQ Table</h3><p>We are going to run a simple query to show the output that shows your streamed data.</p><pre>SELECT * FROM [mimetic-slate:BQ_Dataset.StreamTable]</pre><p><strong>Query Result</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/932/1*2WUi4e0x0cSkK1G4Y26n3A.png" /></figure><p><strong>Github reference</strong> : <a href="https://github.com/sunnykrGupta/Bigquery-series">https://github.com/sunnykrGupta/Bigquery-series</a></p><h3>Conclusion</h3><p>That’s all from this <strong>series Part-II</strong>. Hope you will get basic understanding of Streaming in BigQuery from this post. Streaming is helpful in cases when you want your data to be instantly available for query, helps in scenario where you have a requirement of building real time analysis.</p><p>I would appreciate feedback via comments. In next blog which is part of this series, I will be covering <a href="https://medium.com/google-cloud/patch-and-update-table-bigquery-part-iii-7f2547021926"><strong>Patching and Updating table schema in BigQuery</strong></a> which is important when you want to add or update fields of table.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=763cdec9a9ec" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/streaming-with-redis-bigquery-part-ii-763cdec9a9ec">Streaming with Redis — BigQuery Part-II</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Export & Load Job with MongoDB — BigQuery Part-I]]></title>
            <link>https://medium.com/google-cloud/export-load-job-with-mongodb-bigquery-part-i-64a00eb5266b?source=rss-f6c22db16b21------2</link>
            <guid isPermaLink="false">https://medium.com/p/64a00eb5266b</guid>
            <category><![CDATA[google-cloud-storage]]></category>
            <category><![CDATA[mongodb]]></category>
            <category><![CDATA[google-big-query]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <dc:creator><![CDATA[Sunny Kr.]]></dc:creator>
            <pubDate>Sun, 15 Oct 2017 20:58:22 GMT</pubDate>
            <atom:updated>2022-01-30T22:27:14.884Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*QWhrrXANazlixPrT9I8aLw.jpeg" /></figure><p>This is 3 blog series and is intended for audience who wanted to get into fundamentals of BigQuery (<strong>BQ</strong> in short) and related jobs needed to get your data inside BigQuery system.</p><p><strong>Other posts in the series:</strong></p><ul><li><a href="https://medium.com/google-cloud/streaming-with-redis-bigquery-part-ii-763cdec9a9ec">Streaming with Redis — BigQuery Part-II | by Sunny Gupta | Google Cloud — Community | Medium</a></li><li><a href="https://medium.com/google-cloud/patch-and-update-table-bigquery-part-iii-7f2547021926">Patch and Update Table — BigQuery Part-III | by Sunny Gupta | Google Cloud — Community | Medium</a></li></ul><h3><a href="https://cloud.google.com/bigquery/what-is-bigquery">Google BigQuery</a></h3><p><strong>What ?</strong> — BigQuery is Google’s fully managed, petabyte scale, low cost analytics data warehouse.</p><p><strong>Why ?</strong> — BigQuery is NoOps — there is no infrastructure to manage and you don’t need a database administrator — so you can focus on analyzing data to find meaningful insights, use familiar SQL, and take advantage of our pay-as-you-go model.</p><p><strong>How ?</strong> — Signup to <a href="https://bigquery.cloud.google.com/">Google Cloud platform — GCP</a> using your google account, start loading your data and leverage the power of this NoOps system.</p><h3>Terminology in BigQuery</h3><p><strong>Dataset</strong></p><p>A dataset is contained within a specific project. Datasets enable you to organize and control access to your tables. A table must belong to a dataset, so you need to create at least one dataset before loading data into BigQuery.</p><p><strong>Tables</strong></p><p>A BigQuery table contains individual records organized in rows, and a data type assigned to each column (also called a field).</p><p><strong>Schema</strong></p><p>Each Every table is defined by a schema that describes field names, types, and other information. You can specify the schema of a table during the initial table creation request, or you can create a table without a schema and declare the schema in the query or load job that first populates the table. If you need to change the schema later, you can update the schema.</p><h3>Loading Data into BigQuery</h3><p>In this article, we are going to use a mongoDB server to export our data and going to import into BQ. There are several other ways to import data into BQ.</p><h4>1. Export data from MongoDB</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/526/1*oi-OTO8tHMQASPOVKs4mwA.jpeg" /></figure><p>In this example, i have a database in mongoDB server with name restaurantdb with collection name restaurantCollection. We are going to export using mongoexportbinary available with mongodb server tools.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/838/0*pPORsDUmWl5zE0c0.png" /></figure><pre>$ mongoexport -d restaurantdb -c restaurantCollection -o restaurant.json</pre><p>Once export is done, we can see content of restaurant.json file</p><pre>$ head -n 1 restaurant.json</pre><pre>{ &quot;_id&quot; : { &quot;$oid&quot; : &quot;55f14312c7447c3da7051b26&quot; }, \<br>    &quot;URL&quot; : &quot;http://www.just-eat.co.uk/restaurants-cn-chinese-cardiff/menu&quot;, \<br>    &quot;address&quot; : &quot;228 City Road&quot;, &quot;name&quot; : &quot;.CN Chinese&quot;, \<br>    &quot;outcode&quot; : &quot;CF24&quot;, &quot;postcode&quot; : &quot;3JH&quot;, &quot;type_of_food&quot; : &quot;Chinese&quot; }</pre><pre>//pretty json<br>{<br>    &quot;name&quot;: &quot;.CN Chinese&quot;,<br>    &quot;URL&quot;: &quot;http://www.just-eat.co.uk/restaurants-cn-chinese-cardiff/menu&quot;,<br>    &quot;outcode&quot;: &quot;CF24&quot;,<br>    &quot;postcode&quot;: &quot;3JH&quot;,<br>    &quot;address&quot;: &quot;228 City Road&quot;,<br>    &quot;_id&quot;: {<br>        &quot;$oid&quot;: &quot;55f14312c7447c3da7051b26&quot;<br>    },<br>    &quot;type_of_food&quot;: &quot;Chinese&quot;<br>}</pre><h4>2. Prepare schema for Table</h4><p>Now we have our data ready in json format to be imported into BQ table. We need schema to design in order to import these records. Schema is skeleton of each field with datatype and the field not described in schema will not be imported. We have given all fields as <strong>NULLABLE</strong> ie, if field didn’t came in any records BQ will define null value.</p><pre>$ cat restaurantSchema.json<br>[<br>    {<br>        &quot;name&quot;: &quot;name&quot;,<br>        &quot;type&quot;: &quot;STRING&quot;,<br>        &quot;mode&quot;: &quot;NULLABLE&quot;<br>    },<br>    {<br>        &quot;name&quot;: &quot;URL&quot;,<br>        &quot;type&quot;: &quot;STRING&quot;,<br>        &quot;mode&quot;: &quot;NULLABLE&quot;<br>    },<br>    {<br>        &quot;name&quot;: &quot;address&quot;,<br>        &quot;type&quot;: &quot;STRING&quot;,<br>        &quot;mode&quot;: &quot;NULLABLE&quot;<br>    },<br>    {<br>        &quot;name&quot;: &quot;outcode&quot;,<br>        &quot;type&quot;: &quot;STRING&quot;,<br>        &quot;mode&quot;: &quot;NULLABLE&quot;<br>    },<br>    {<br>        &quot;name&quot;: &quot;postcode&quot;,<br>        &quot;type&quot;: &quot;STRING&quot;,<br>        &quot;mode&quot;: &quot;NULLABLE&quot;<br>    },<br>    {<br>        &quot;name&quot;: &quot;type_of_food&quot;,<br>        &quot;type&quot;: &quot;STRING&quot;,<br>        &quot;mode&quot;: &quot;NULLABLE&quot;<br>    }<br>]</pre><p>Reference for schema : <a href="https://cloud.google.com/bigquery/docs/schemas">https://cloud.google.com/bigquery/docs/schemas</a></p><h4>3. Google Cloud SDK Installation</h4><p><strong>Installation : </strong><a href="https://cloud.google.com/sdk/">https://cloud.google.com/sdk/</a></p><p>Once installation is done, run following command to verify account setup.</p><pre>$ gcloud auth list</pre><pre>$ gcloud config list</pre><h4>4. Create BigQuery Dataset</h4><p>Go to your BigQuery in google console. <a href="https://bigquery.cloud.google.com/">https://bigquery.cloud.google.com</a></p><p>Follow below instruction to create dataset.</p><blockquote><strong>Step 1:</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/670/0*bSDm5cZW6vojNn_f.png" /></figure><blockquote><strong>Step 2:</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/487/0*IGIz60SGVC0veOYS.png" /></figure><blockquote>To create dataset through bq command line interface</blockquote><pre>$ bq mk -d --data_location=US   BQ_Dataset</pre><pre>// Verify your dataset creation<br>$ bq ls</pre><pre>    datasetId<br> ----------------<br>  BQ_Dataset</pre><p><strong>Read :</strong> <a href="https://cloud.google.com/bigquery/docs/datasets#create-dataset">https://cloud.google.com/bigquery/docs/datasets#create-dataset</a></p><h4>5. Load data into BigQuery</h4><p>Now, my directory consists two files ie, data and schema.</p><pre>$ tree<br>.<br>├── restaurant.json<br>└── restaurantSchema.json</pre><pre>0 directories, 2 files</pre><p>Run command to load data into BQ. Once you submit load job, it will take seconds to minute depends on size of data you are importing into BQ table.</p><pre>Ex: bq load --project_id=&lt;PROJECT-ID&gt; --source_format=NEWLINE_DELIMITED_JSON \<br>    mydataset.mytable ./myfile.json ./myschema.json</pre><pre>$ bq load --project_id=mimetic-slate-179915   \<br>    --source_format=NEWLINE_DELIMITED_JSON --max_bad_records 10 \<br>    BQ_Dataset.Restaurant ./restaurant.json  /restaurantSchema.json</pre><p>--max_bad_records 10 are additional flags to allow 10 bad records while importing your job, exceeding this value will result in import failure.</p><p><strong>Import through Cloud Storage</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*rDKuC_Itm6d5iKBJ4c9nXw.png" /></figure><blockquote>Another methods to import the data through Cloud Storage, this method is lot faster compared to above one.</blockquote><p><strong>Read : </strong><a href="https://cloud.google.com/storage/docs/creating-buckets">https://cloud.google.com/storage/docs/creating-buckets</a></p><pre>//to create Cloud storage bucket for this example.<br>$ gsutil mb  gs://bq-storage</pre><pre>//to verify bucket creation<br>$ gsutil ls<br>gs://bq-storage/</pre><p>Now , we will upload our data restaurant.json to storage in bucket gs://bq-storage/</p><pre>//Run command to upload<br>$ gsutil cp restaurant.json gs://bq-storage/restaurant.json</pre><p>Now, we can use storage path of object to import into BQ tables.</p><pre>$ bq load --project_id=mimetic-slate-179915  \<br>    --source_format=NEWLINE_DELIMITED_JSON --max_bad_records 10  \<br>    BQ_Dataset.Restaurant \<br>    gs://bq-storage/restaurant.json ./restaurantSchema.json</pre><p>After bq load finished, run following command to verify the table creation.</p><pre>$ bq show BQ_Dataset.Restaurant</pre><blockquote><strong>Output will be similar to this :</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/723/0*rnykG2OHvGomSwPG.png" /></figure><p>You can also verify in bigQuery UI after hitting refresh. Visit to table and click <em>preview</em>. You will start seeing records in table.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/969/0*-KUP1oO3lMv5wftr.png" /></figure><p><strong>More into bq CLI :</strong> <a href="https://cloud.google.com/storage/docs/creating-buckets">https://cloud.google.com/bigquery/bq-command-line-tool</a></p><p><strong>More into gsutil CLI :</strong> <a href="https://cloud.google.com/storage/docs/quickstart-gsutil">https://cloud.google.com/storage/docs/quickstart-gsutil</a></p><h3>SQL Query in BQ Table</h3><p>We are going to run a simple query to show the output.</p><pre>SELECT<br>  name,<br>  address<br>FROM<br>  [mimetic-slate-179915:BQ_Dataset.Restaurant]<br>WHERE<br>  type_of_food = &#39;Thai&#39;<br>GROUP BY<br>  name, address</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*9Q72jvVKPm8DbreX.png" /></figure><h3>Conclusion</h3><ul><li>BigQuery is a query service that allows you to run SQL-like queries against multiple terabytes of data in a matter of seconds. The technology is one of the Google’s core technologies, like MapReduce and Bigtable, and has been used by Google internally for various analytic tasks since 2006.</li><li>While MapReduce is suitable for long-running batch processes such as data mining, BigQuery is the best choice for ad hoc OLAP/BI queries that require results as fast as possible.</li><li>Wildcard can also be applied into Bigquery tables to expand your computations to multiple tables.</li><li>BigQuery is the cloud-powered massively parallel query database that provides extremely high full-scan query performance and cost effectiveness compared to traditional data warehouse solutions and appliances</li></ul><blockquote><strong>Next from here:</strong> Play around with more <a href="https://cloud.google.com/bigquery/docs/how-to">functionality available in BigQuery</a> and dive into it for more computation hungry jobs.</blockquote><p>That’s all from this <strong>series Part-I</strong>. Hope you get basic understanding of import jobs, storage and basic outline of BigQuery from this page. I have seen power of BigQuery in my workplace to crunch 100–120 TB of data and getting results in minute or two, its really incredibly awesome. I would appreciate a feedback via comments available below and claps on medium.</p><p>In next blog which is part of this series, we will be understanding more about <a href="https://medium.com/google-cloud/streaming-with-redis-bigquery-part-ii-763cdec9a9ec"><strong><em>Streaming feature available in Bigquery</em></strong></a> to push data in BQ tables in real-time to make it available for instant query on changing dataset.</p><p><strong>Github reference : </strong><a href="https://github.com/sunnykrGupta/Bigquery-series"><strong>https://github.com/sunnykrGupta/Bigquery-series</strong></a></p><p><strong>Daemon Blog : </strong><a href="https://sunnykrgupta.github.io/export-load-job-with-mongodb-bigquery-part-i.html">https://sunnykrgupta.github.io/export-load-job-with-mongodb-bigquery-part-i.html</a></p><p><a href="https://sunnykrgupta.github.io/export-load-job-with-mongodb-bigquery-part-i.html">Daemon Blog - Export &amp; Load Job with MongoDB - BigQuery Part-I</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=64a00eb5266b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/export-load-job-with-mongodb-bigquery-part-i-64a00eb5266b">Export &amp; Load Job with MongoDB — BigQuery Part-I</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Installation of VNC server on Ubuntu]]></title>
            <link>https://medium.com/hackernoon/installation-of-vnc-server-on-ubuntu-1cf035370bd3?source=rss-f6c22db16b21------2</link>
            <guid isPermaLink="false">https://medium.com/p/1cf035370bd3</guid>
            <category><![CDATA[linux]]></category>
            <category><![CDATA[installation]]></category>
            <category><![CDATA[desktop-environments]]></category>
            <category><![CDATA[ubuntu]]></category>
            <category><![CDATA[vnc-server-on-ubuntu]]></category>
            <dc:creator><![CDATA[Sunny Kr.]]></dc:creator>
            <pubDate>Mon, 04 Sep 2017 18:32:40 GMT</pubDate>
            <atom:updated>2019-04-03T15:59:25.395Z</atom:updated>
            <content:encoded><![CDATA[<blockquote><a href="https://sunnykrgupta.github.io/installation-of-vnc-server-on-ubuntu.html">Originally Published on sunnykrgupta.github.io</a></blockquote><p>This blog is intended for people who wanted to install GUI or desktop environment on linux <a href="https://en.wikipedia.org/wiki/Server_(computing)">servers</a> running on <a href="https://en.wikipedia.org/wiki/Cloud_computing">cloud</a> and connect.</p><p>We are going to use VNC (Virtual Network Computing) protocol for accessing our remote desktop server.</p><h4>What is VNC ?</h4><p><a href="https://en.wikipedia.org/wiki/Virtual_Network_Computing">Virtual network computing</a>, or VNC, is a graphical desktop sharing system that allows you to control one computer remotely from another. A VNC server transfers keyboard and mouse events, and displays the remote host’s screen via a network connection, which allows you to operate a full <a href="https://en.wikipedia.org/wiki/Desktop_environment">desktop environment</a>.</p><p>Basically ubuntu server and ubuntu cloud editions does not contains GUI, which needs to be installed before installing VNC server. Please note that server and cloud editions are carefully designed to utilize less hardware resources ( minimal environment ), installing GUI might leads to high hardware utilization.</p><h4>Why I needed desktop environment in remote server ?</h4><p>Just to explain a use case, let me tell you how I ended up using VNC in first place. I was working on a problem which relates with cloud latency testing. My Friend, <a href="https://www.linkedin.com/in/neekneeraj/">Neeraj</a> <em>(whose work revolves around core JS research &amp; development)</em> developed a javascript code that makes an cross origin <a href="https://sunnykrgupta.github.io/(https://en.wikipedia.org/wiki/Web_API)">HTTP API</a> call to a <a href="https://en.wikipedia.org/wiki/Cloud_load_balancing">loadbalancer</a> near to the geographical location of browser and response will be delivered from loadbalancer in geographic proximity. To test this setup, executing JS code and to use <a href="https://developer.mozilla.org/en/docs/Tools/Browser_Console">developer console</a> to see what’s happening under the network layer, we were in need of a browser engine in different geographical location. I could have used some online paid or free service to get browser rented, services like <a href="https://www.browserstack.com/">browserstack</a> or other alternatives but that has free minutes based trial restrictions.</p><h3>Install a Desktop and VNC Server on Ubtunu 14.04</h3><h4>Step 1 — Install Ubuntu desktop</h4><p>Start installing below <em>gnome packages</em> which helps VNC to load properly . These packages are required for all editions including <em>ubuntu desktop</em> .</p><pre>$ sudo apt-get install --no-install-recommends ubuntu-desktop gnome-panel gnome-settings-daemon metacity nautilus gnome-terminal gnome-core</pre><h4>Step 2 — Install vnc4server package.</h4><pre>$ sudo apt-get install vnc4server</pre><h4>Step 3 — Make configuration changes in vncserver</h4><p>Open /usr/bin/vncserver file and edit as follows . Before editing, make a backup copy.</p><pre>$ sudo cp /usr/bin/vncserver /usr/bin/vncserver.bkp</pre><pre>$ sudo vim /usr/bin/vncserver</pre><pre><em>#Find this line &quot;# exec /etc/X11/xinit/xinitrcnn&quot;.</em><br><em>#and add these lines like below</em></pre><pre>    &quot;# exec /etc/X11/xinit/xinitrcnn&quot;.<br>       &quot;gnome-panel &amp;n&quot;.<br>       &quot;gnome-settings-daemon &amp;n&quot;.<br>       &quot;metacity &amp;n&quot;.<br>       &quot;nautilus &amp;n&quot;.<br>       &quot;gnome-terminal &amp;n&quot;.</pre><h4>Step 4 — Start your vncserver</h4><p>Now type the command vncserver to start VNC session. you will be prompted for creating new vnc password.</p><pre>$ vncserver<br>You will require a password to access your desktops through VNC Clients.<br>Password:******<br>Verify:******</pre><pre>xauth: file /root/.Xauthority does not exist<br>New &#39;ubuntu-desktop:1 (root)&#39; desktop is ubuntu-desktop:1</pre><pre>Starting applications specified in /root/.vnc/xstartup<br>Log file is /root/.vnc/ubuntu-desktop:1.log</pre><h4>Step 5 — To check VNC server has started, follow</h4><pre>$ netstat -tulpn</pre><pre>Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name<br>tcp        0      0 0.0.0.0:6001            0.0.0.0:*               LISTEN      28372/Xvnc4<br>tcp6       0      0 :::5901                 :::*                    LISTEN      28372/Xvnc4</pre><blockquote>VNC server is running and listening on <strong>5901 port</strong>. Make sure your firewall allows <strong>inbound</strong> TCP connection to this port.</blockquote><h4>Step 6 — Configure your Firewall</h4><p>If <strong>firewall</strong> is active, you need to open ports for inbound communication. If no firewall is enabled, you can skip this section.</p><pre><em>#allow SSH</em><br>$ sudo ufw allow OpenSSH</pre><pre><em>#allowing single port 5901 port</em><br>$ sudo ufw allow 5901/tcp</pre><pre><em>#To allow series of port 5901 - 5910, follow</em><br>$ sudo ufw allow 5901:5910/tcp</pre><pre><em>#To check firewall rules</em><br>$ sudo  ufw status verbose</pre><pre>Status: active<br>Logging: on (low)<br>Default: deny (incoming), allow (outgoing), disabled (routed)<br>New profiles: skip</pre><pre>To                         Action      From<br>--                         ------      ----<br>22/tcp (OpenSSH)           ALLOW IN    Anywhere<br>5901:5910/tcp              ALLOW IN    Anywhere<br>22/tcp (OpenSSH (v6))      ALLOW IN    Anywhere (v6)<br>5901:5910/tcp (v6)         ALLOW IN    Anywhere (v6)WW</pre><blockquote><a href="https://www.digitalocean.com/community/tutorials/how-to-set-up-a-firewall-with-ufw-on-ubuntu-14-04"><strong>Good reads on configuring UFW firewall</strong></a></blockquote><h4>Step 7 — Connect to VNC Server</h4><p>Use any remote desktop connect client that allow VNC protocol. Use <a href="https://en.wikipedia.org/wiki/IP_address">IP address</a> of server along with port where VNC server is listening.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/596/1*EBrbQBlml2ziyVR2OQTo-A.png" /><figcaption>Connect -Remote Desktop Viewer</figcaption></figure><p>Once connected to your VNC server, you will see screen of remote server where you installed desktop GUI.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/743/1*Iq9_DYdunxqif7Rd1HadiQ.png" /><figcaption>Launch Firefox from Terminal</figcaption></figure><p>Browser screen running on remote server.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*31nIviJgshvUXZ_pNGZ2aA.png" /><figcaption>Google UK</figcaption></figure><p>That’s it, your VNC server is working.</p><blockquote>Here I created my linux server in london, UK. I opened firefox through terminal to reach out to URL google.com. It opened google.co.uk domain based on regional search engine. You can do lot of other stuffs on VNC protocol to get things done from remote location.</blockquote><p><strong>Daemon Blog : </strong><a href="https://sunnykrgupta.github.io/installation-of-vnc-server-on-ubuntu.html">https://sunnykrgupta.github.io/installation-of-vnc-server-on-ubuntu.html</a></p><p><a href="https://sunnykrgupta.github.io/installation-of-vnc-server-on-ubuntu.html">Daemon Blog - Installation of VNC server on Ubuntu</a></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fupscri.be%2Fdde502%3Fas_embed%3Dtrue&amp;dntp=1&amp;url=https%3A%2F%2Fupscri.be%2Fdde502%2F&amp;image=https%3A%2F%2Fucarecdn.com%2F57662ddc-da7c-407b-afcb-cb45184b2705%2F&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=upscri" width="800" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/d25dd15e8727342107fbe0b6d67b030e/href">https://medium.com/media/d25dd15e8727342107fbe0b6d67b030e/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1cf035370bd3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/hackernoon/installation-of-vnc-server-on-ubuntu-1cf035370bd3">Installation of VNC server on Ubuntu</a> was originally published in <a href="https://medium.com/hackernoon">HackerNoon.com</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Managing fleet on Kubernetes]]></title>
            <link>https://medium.com/hackernoon/managing-fleet-on-kubernetes-8cac6483b64?source=rss-f6c22db16b21------2</link>
            <guid isPermaLink="false">https://medium.com/p/8cac6483b64</guid>
            <category><![CDATA[container-orchestration]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[google-container-engine]]></category>
            <dc:creator><![CDATA[Sunny Kr.]]></dc:creator>
            <pubDate>Sun, 13 Aug 2017 18:19:57 GMT</pubDate>
            <atom:updated>2018-04-05T04:43:22.088Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>Couple of months ago, we were tackling challenges with scalability of system and were in pursuit of finding right orchestration tools which can help in scaling systems quickly. This draft is outline of things we have tried and learned along the way, most of things might sound familiar to you. A Quick glance of things we came across while building fleets on Kubernetes.</blockquote><p>We started exploring popular project managed by Google for orchestration management, <a href="https://kubernetes.io/">Kubernetes</a> for DevOps. Starting with two weeks of learning curves, we get our working staging system in <em>kubes</em> (kubernetes in short) and did small working setup to visualize the power of this orchestration framework.</p><h4>Microservices</h4><p>Microservice architectures have been trending because its architectural style aims to tackle the problems of managing modern application by decoupling software solutions into smaller functional services that are expected to fail.</p><p>This help in quick recovery from failure on smaller functional units in contrast to making recovery from big monolithic software systems. Microservices helps in making your release cycle faster even because you will be focusing on smaller changes in single app instead of pushing code changes in bigger software systems that has multiple dependencies.</p><h4>Containers</h4><p>Microservice architectures got a big tide in 2013 when Docker inc. released Docker technology. <strong>Docker containers </strong>gave<strong> </strong>perfect alternatives to virtual machines and drove software packaging methods in a more developer friendly way. Docker containers are comparatively smaller than virtual machines (VMs). It shares underlying host OS resources, we can spin up hundreds of these small units in order of milliseconds. Their smaller size helps in faster packaging, testing and even deployments because of its portable nature.</p><p>Docker’s container-based platform allows highly portable workloads. Docker containers can run on a developer’s local laptop, on physical or virtual machines in a data center, on cloud providers, or in a mixture of environments.</p><p>We started with <a href="https://cloud.google.com/container-engine/">Google Container Engine</a> (GCE) to get things work quickly. We started with a cluster with few <em>10’s of Nodes</em>, each Node with configuration <em>12 vCore</em> and <em>30 GB</em> in <a href="https://cloud.google.com/container-engine/docs/node-pools"><em>default pool</em></a><em> </em>to run stateless components.</p><p>Before going in depth, we need some <em>gears</em> (concepts/tools/theory) to onboard into container ship and sail out for cruise.</p><p>We are dividing gears that we need to know into two parts, i.e. first will be <strong>Docker</strong> and second will focused on <strong>Kubernetes</strong>.</p><h3>Part — I (Understanding Docker at Dock)</h3><h4>Stateless and stateful components</h4><p>In computing, a stateless protocol is a communication protocol in which no information is retained by either sender or receiver. The sender transmits a packet to the receiver and does not expect an acknowledgment of receipt. A UDP connection-oriented session is a stateless connection because neither systems maintains information about the session during its life.<br>In contrast, a protocol that requires keeping of the internal state on the server is known as a stateful protocol. A TCP connection-oriented session is a ‘stateful’ connection because both systems maintain information about the session itself during its life.</p><h4><a href="https://www.redhat.com/en/containers">Understanding containerization concept</a></h4><p>Container provides operating system-level virtualization through a virtual environment that has its own process and network space, instead of creating a full-fledged virtual machine <a href="https://en.wikipedia.org/wiki/Virtual_machine">(https://en.wikipedia.org/wiki/Virtual_machine)</a>. This enables the kernel of an operating system to allow the existence of multiple isolated user-space instances, instead of just one.</p><h4><a href="https://docs.docker.com/engine/userguide/eng-image/dockerfile_best-practices/">Writing good Dockerfile for modules</a></h4><p>Dockerfile is set of instructions used by Docker to build an image. Containers are created using docker images, which can be built either by executing commands manually or automatically through Dockerfile. Docker achieves this by creating safe, LXC (i.e. Linux Containers) based environments for applications called “<strong>docker containers</strong>”.</p><p>* Writing optimized Dockerfile, understanding order of commands. Each command that we run in Dockerfile is executed as a layer and subsequent command will be build on top of previous layer. Each layer is managed in cache by Docker tool. Docker manages cache itself to reuse layer of previously build Docker images to save time &amp; disk.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/989d992c8a34d793d5e4c480c1502275/href">https://medium.com/media/989d992c8a34d793d5e4c480c1502275/href</a></iframe><p><a href="https://docs.docker.com/engine/userguide/eng-image/dockerfile_best-practices/#each-container-should-have-only-one-concern"><strong>Running a single process inside a Docker container</strong></a><br> “one process per container” is frequently a good rule of thumb, it is not a hard and fast rule.</p><blockquote>Use your best judgment to keep containers as clean and modular as possible.</blockquote><p>Understanding remote <strong>Docker container registry</strong> for storing/pushing our locally built docker images, here we have used <strong>Google container registry</strong> (GCR) for docker image management.</p><blockquote>Pushing and Pulling Images to GCR (<a href="https://cloud.google.com/container-registry/docs/pushing-and-pulling">https://cloud.google.com/container-registry/docs/pushing-and-pulling</a>)<br> Push images to Docker Cloud (<a href="https://docs.docker.com/docker-cloud/builds/push-images/">https://docs.docker.com/docker-cloud/builds/push-images/</a>)</blockquote><h3>Part — II ( Understanding Kubernetes in Ocean )</h3><p><strong>Learning basics of kubernetes &amp; </strong><a href="https://kubernetes.io/docs/tutorials/kubernetes-basics/"><strong>work flow training</strong></a></p><blockquote>Kubernetes is an open-source platform for automating deployment, scaling, and operations of application containers across clusters of hosts, providing container-centric infrastructure — Kubernetes.io</blockquote><p><a href="https://kubernetes.io/docs/concepts/workloads/pods/pod/#what-is-a-pod"><strong>What are Pods</strong></a><strong>? How container runs inside a pod?</strong></p><p>* Pods are the atomic unit on the Kubernetes platform. A Pod is a Kubernetes abstraction that represents a group of one or more application containers (such as Nginx or redis), and some shared resources for those containers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o68H9ID0OxR7yc4qeaBokA.png" /><figcaption>Pod Overview : Images by Kubernetes.io</figcaption></figure><p><a href="https://kubernetes.io/docs/concepts/nodes/node/#what-is-a-node"><strong>What are Nodes</strong></a><strong>?</strong> <em>(also known as worker or minion)</em></p><p>* A Pod always runs inside a Node. A Node is a worker machine in Kubernetes and may be either a virtual or a physical machine, depending on the cluster. Node is controlled by Kubernetes Master. Kubernetes manages scheduling of pods in Nodes running in a cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/836/1*_DvKCivXyOXdtSvJiPp-RQ.png" /><figcaption>Node Overview : Images by Kubernetes.io</figcaption></figure><p><a href="https://kubernetes.io/docs/concepts/workloads/controllers/deployment/"><strong>What are Deployments</strong></a><strong>?</strong></p><p>* We use Deployments to create new resources, or replace existing ones by new ones by means of configuration defined. You can think of it as a supervisor of pods management.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ad442d6971237c09c599ecb58945406a/href">https://medium.com/media/ad442d6971237c09c599ecb58945406a/href</a></iframe><p><a href="https://kubernetes.io/docs/concepts/workloads/controllers/replicaset/"><strong>What is replication controller and Replica sets</strong></a><strong>?</strong></p><p>* A ReplicationController and Replica Sets ensures that a specified number of pod “replicas” are running at any one time. In other words, it makes sure that a pod or homogeneous set of pods are always up and available. If there are too many pods, it will kill some. If there are too few, it will start more.</p><blockquote>In above <em>yaml</em> file, you can see <strong>replicas</strong> keyword, this is being managed by replication utility.</blockquote><p><a href="https://kubernetes.io/docs/concepts/overview/components/"><strong>What is Kubernetes master</strong></a><strong>?</strong></p><p>* The controlling services in a Kubernetes cluster are called the master, or control plane, components. For example, master components are responsible for making global decisions about the cluster (e.g., scheduling), and detecting and responding to cluster events (e.g., starting up a new pod when a replication controller’s ‘replicas’ field is unsatisfied). Kubernetes provides a REST API supporting primarily CRUD operations on (mostly) persistent resources, which serve as the hub of its control plane.</p><p><a href="https://github.com/kubernetes/community/blob/master/contributors/design-proposals/architecture.md#architecture"><em>Kubernetes Ecosystem consists of mutiple components</em></a><em>.</em></p><p><a href="https://kubernetes.io/docs/concepts/services-networking/service/"><strong>What are services</strong></a><strong>?</strong></p><p>* A Kubernetes Service is an abstraction which defines a logical set of Pods and a policy by which to access them. The set of Pods targeted by a Service is (usually) determined by a Label Selector. Service keep on looking for pods which has specific labels assigned and keep tracks of those pods for request offloading.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/608/1*GdTo4cb2iujH-z2Koy4gxw.png" /><figcaption>Service Overview : Images by Kubernetes.io</figcaption></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/faa0d35331f7e4a0c436818db6492e7c/href">https://medium.com/media/faa0d35331f7e4a0c436818db6492e7c/href</a></iframe><p><a href="https://kubernetes.io/docs/user-guide/kubectl-cheatsheet/"><strong>How to debug or get cluster info from command-line</strong></a><strong>?</strong></p><p>* <strong><em>kubectl</em></strong> is a command line interface for running commands against Kubernetes clusters.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/49d15d3b9ae58cbcbf3c060f80b56f54/href">https://medium.com/media/49d15d3b9ae58cbcbf3c060f80b56f54/href</a></iframe><p><strong>How do we run containers in GCE ?</strong></p><p>We have number of <strong>deployments</strong> which manages scaling pods up/down depending upon processing that we need. <em>Pods</em> run containers inside <em>Node</em> available in a cluster. We need to follow proper versioning of modules to distinguish what is running inside your system and this helps in rollback releases in case of issues in production.</p><blockquote>How about services/APIs we need to expose ?</blockquote><p>- There comes kubes <strong><em>services</em></strong>. We have plenty of APIs we need to expose to outside world. To make it happen, we have couple of kube services exposed using tcp loadbalancer which has been assigned public IP. Internally, these services keeps on doing service discovery using <strong>label selector</strong> to find pods and attach it to this service, pods having same label will be targeted by a service. Its same concept of how we manage loadbalancer on cloud, attach VMs to a loadbalancer to offload incoming traffic.</p><blockquote>Resources running inside Kube ship knows each other very well. Each services/pods can communicate by names assigned to each. Instead of using IPs (private) assigned to each of them, you can use names given as FQDN. It’s a good practise to use names instead of IPs because of dynamic nature of network resource allocation since resources get destroyed and created again in a container lifecycle management. Kube-DNS maintains all list of IPs internally assigned and helps finding resources by names.</blockquote><p><a href="https://kubernetes.io/docs/tasks/configure-pod-container/assign-cpu-ram-container/"><strong>How to decide what resources you should allocate to your pods resources</strong>?</a></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7fac9d1cbb85882f62bb5e37f369f8d7/href">https://medium.com/media/7fac9d1cbb85882f62bb5e37f369f8d7/href</a></iframe><p>Each container has its own requirements of resources (ie, <em>CPU, RAM, disk, network </em>etc), there comes <strong>requests</strong> &amp; <strong>limits</strong> in kubes. This helps in keeping your nodes healthy. Many times due to bad limits or not defining limits, your pods can go crazy at utilization. They might eat any of the sources and can lead to node starvation which makes the Node unhealthy and it goes in <strong>[Not Ready]</strong> state due to resource exhaustion. We faced this multiple times at early stage but now we have fine tuned each pods’ resources based on its hunger behaviour.</p><p><strong>How to define Node resources in kubernetes cluster?</strong></p><p>Depends on container type (which is running inside a <em>pod</em>), you can define different <strong><em>Node pools</em></strong>. Suppose you have modules named ```<strong>Core.X</strong>, <strong>Core.Y</strong> and <strong>Core.Z</strong>``` , all of them needs ```<strong>2 core, 2 GB</strong>``` each to run, then you can have <em>Standard Node Pool</em> to run them. In this case, i will allocate following config for my Node pool.</p><p>- Name : Standard Pool<br> — Pool Size : 2<br> — Node Config: 4 Core, 4 GB<br> — Node Pool Resource : 8 Core, 8 GB<br> — <strong>Utilization</strong> : 6 Core, 6 GB (75 % used Core &amp; RAM)</p><p>Now, lets say i have high memory eater modules. let call them ```<strong>Mem.X, Mem.Y and Mem.Z</strong>``` , all of them needs ```<strong>0.5 core, 4 GB</strong>``` each to run, then you need <em>High memory Node Pool</em> to run them. In this case, i will allocate different config for my Node pool.</p><p>- Name : HighMem Pool<br>- Pool Size : 2<br>- Node Config : 1 Core, 8 GB<br>- Node Pool Resource : 2 Core, 16 GB<br>- <strong>Utilization</strong> : 1.5 Core, 12 GB (75 % used Core &amp; RAM)</p><blockquote>So, based on your Node pool type (<a href="https://cloud.google.com/container-engine/docs/node-pools">https://cloud.google.com/container-engine/docs/node-pools</a>), you can deploy your pods in different Node pools by using nodeSelector in kubes.</blockquote><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/dfd61453181d2d314b76dfbb65739e2d/href">https://medium.com/media/dfd61453181d2d314b76dfbb65739e2d/href</a></iframe><p><strong>How we monitor Kubernetes ?</strong></p><p>We can run custom monitoring setup to keep an eye on Nodes. You can run [heapster](<a href="https://github.com/kubernetes/heapster">https://github.com/kubernetes/heapster</a>), ie. responsible for compute resource usage analysis and monitoring of container clusters, hooked with [influxdb](<a href="https://github.com/influxdata/influxdb">https://github.com/influxdata/influxdb</a>) that consumes reporting pushed by heapster and can be visualized in [grafana](<a href="https://grafana.com/">https://grafana.com/</a>).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nvskytQPJNyNBethBoGm8w.png" /><figcaption>Monitoring in Grafana</figcaption></figure><p><strong>Note :</strong> Some configuration in GCE should be taken care, like <strong>autoupgrade kubernetes version</strong>. If you are running RabbitMQ, Redis or any other message queue as service that needs uptime, better you turn off autoupgrade because kubernetes new version release will schedule all your node for maintenance, however it rolles updates one by one but could affect your production system. Else, if you are fully stateless, you can keep default or skip this warning!</p><p><a href="https://cloud.google.com/container-engine/docs/node-auto-upgrade">https://cloud.google.com/container-engine/docs/node-auto-upgrade</a></p><blockquote>gcloud beta container node-pools update &lt;NODEPOOL&gt; — cluster &lt;CLUSTER&gt; — zone &lt;ZONE&gt; — no-enable-autoupgrade</blockquote><p>Pretty much all above understandings are based on what I learned in last six months of kubernetes running in production. Container management is easy to adapt and lot of new observation is yet to be discovered as we go along the way.</p><p>Looking at deployments today, Kubernetes is absolutely fantastic in Auto-pilot and doing self-healing jobs itself. We are running more than <strong>1000 pods</strong> in cluster together and processing <strong>10’s of Billions of API calls per month</strong> and are pushing more to handle.</p><blockquote><strong><em>Conclusion : </em></strong><em>Kubernetes has lifted lot of server management and helped in faster depployments &amp; scaling system. Adaptability is much quicker, most of security and other concerns is being managed by Google. Kubernetes aims to offer a better orchestration management system on top of clustered infrastrcuture. Development on Kubernetes has been happening at storm-speed, and the community of Kubernauts (</em><a href="https://kubernetes.io/community/"><em>https://kubernetes.io/community/</em></a><em>) has grown bigger.</em></blockquote><p><strong>Daemon Blog : </strong><a href="https://sunnykrgupta.github.io/managing-fleet-on-kubernetes.html">https://sunnykrgupta.github.io/managing-fleet-on-kubernetes.html</a></p><p><a href="https://sunnykrgupta.github.io/managing-fleet-on-kubernetes.html">Daemon Blog - Managing fleet on Kubernetes</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8cac6483b64" width="1" height="1" alt=""><hr><p><a href="https://medium.com/hackernoon/managing-fleet-on-kubernetes-8cac6483b64">Managing fleet on Kubernetes</a> was originally published in <a href="https://medium.com/hackernoon">HackerNoon.com</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>