<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Domas Jautakis on Medium]]></title>
        <description><![CDATA[Stories by Domas Jautakis on Medium]]></description>
        <link>https://medium.com/@domisj?source=rss-b00dd99bdcc------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*tt1OdtX_MGV93UM_1e3gNA.jpeg</url>
            <title>Stories by Domas Jautakis on Medium</title>
            <link>https://medium.com/@domisj?source=rss-b00dd99bdcc------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 11:27:23 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@domisj/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Cloudy in the clouds]]></title>
            <link>https://domisj.medium.com/cloudy-in-the-clouds-5018714da483?source=rss-b00dd99bdcc------2</link>
            <guid isPermaLink="false">https://medium.com/p/5018714da483</guid>
            <dc:creator><![CDATA[Domas Jautakis]]></dc:creator>
            <pubDate>Mon, 13 Jun 2022 08:46:37 GMT</pubDate>
            <atom:updated>2022-06-13T08:46:37.470Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/887/1*FuwZYdWzT4NdyLQtHerPFQ.png" /></figure><p>A word cloud is a popular and informative visualization for text processing. It shows most frequently repeating words in bigger and bold representation. Less important information in such cases becomes smaller words. But let’s start from the beginning.</p><p>The article is split into several parts. This page is part 3. Other parts are written by <a href="https://medium.com/@irmaspu">Irma Spudiene.</a> All parts are available here:<br><strong>Part1. </strong><a href="https://medium.com/@irmaspu/are-they-similar-a01f8b627540">Solution overview</a><br><strong>Part2.</strong> <a href="https://medium.com/@irmaspu/vision-ai-and-8-images-similar-to-yours-53dbbdeb2dd5"><em>Are they similar app</em> source code walk through</a><br><strong>Part3.</strong> <a href="https://medium.com/p/5018714da483">Cloudy in the clouds (this story)</a></p><p>How to implement wordcloud in the cloud (feels cloudy in the cloud)? In my story, I will describe the challenge I had with word cloud implementation in the <strong>Google Cloud Platform</strong>. The general requirement for this task was — the representation of near-real-time data in a nice visual way with an interactive word cloud. Visualization had to run for several hours.</p><p>I split this challenge into 2 sides:</p><ul><li>frontend</li><li>backend</li></ul><p>Let’s start from the beginning: as there were implementation and image recognition implemented in previous parts, there were some cool pictures that we decided to process, get labeled, and prepare wordcloud. For storing labeled data we chose to use a fast and efficient Google <strong>Firestore</strong>. After when data arrives it should be processed and available in UI for end-user. Processing functionality is implemented in cloud functions in order to have serverless products and the possibility to have event-based triggers on Google Cloud Storage bucket. When any image arrives in the bucket, the function gets triggered. <strong>Vision AI </strong>adds particular attributes and labels to provided <strong>Firestore</strong> collection.</p><pre>from google.cloud import vision<br>from google.cloud import firestore<br>import secrets<br>import proto</pre><pre>def on_image_upload(event, context):<br>    file_name = event[&#39;name&#39;]<br>    file_bucket = event[&#39;bucket&#39;]<br>    client = vision.ImageAnnotatorClient()<br>    image = vision.Image()<br>    image.source.gcs_image_uri = f&quot;gs://{file_bucket}/{file_name}&quot;<br>    response = client.label_detection(image=image)<br>    labels = response.label_annotations<br>    <br>    image_labels = {}<br>    for label in labels:<br>        image_labels[&quot;label_&quot; + secrets.token_hex(8)] = proto.Message.to_dict(label)</pre><pre>db = firestore.Client()<br>    db_ref = db.collection(u&#39;atsapp_labels&#39;)<br>    db_ref.add({ &#39;labels&#39; : image_labels})</pre><p>As not the biggest fan of frontend technologies, I have started with the backend. I had an initial data source which was a NoSQL database — <strong>Cloud Firestore</strong>. Started with a python client for <a href="https://googleapis.dev/python/firestore/latest/index.html">Firestore</a> and wordcloud library. All code was written in <strong>Vertex AI Workbench</strong> managed Jupyter-Lab instance. The idea was to take the data from Firestore, process it with a notebook, and upload generated word cloud visualization to the target in my case it was <strong>Google Cloud Storage </strong>(further GCS). How to apply it for near-real-time data? I used a simple cheat — <em>for</em> cycle with <em>sleep</em>. So data reading and generation of the word cloud image are scheduled just according to cycle elements and sleep time for one iteration.</p><pre>import time<br>import datetime<br>import os<br>import matplotlib.pyplot as plt<br>from wordcloud import WordCloud<br>from google.cloud import storage<br>import firebase_admin<br>from firebase_admin import credentials, firestore<br>from PIL import Image<br>import numpy as np<br>from pandas.io.json._normalize import nested_to_record</pre><pre># firebase<br>cred = credentials.Certificate(&#39;auth/yourKey.json&#39;)<br>firebase_admin.initialize_app(cred)<br>db = firestore.client()</pre><pre># gcs<br>client = storage.Client(project=&#39;yourProject&#39;)<br>bucket = client.get_bucket(&#39;yourBucket&#39;)<br>blobs = bucket.list_blobs(prefix=&#39;pics/&#39;)<br>    <br># save<br>strFile = &quot;my_plot.jpg&quot;</pre><pre># iterations<br>iterations = 7200</pre><pre>def firebase_reader():<br>    users_ref = db.collection(u&#39;atsapp_labels&#39;)<br>    docs = users_ref.stream()<br>    dict_temp = {}<br>    list_temp = []<br>    <br>    for doc in docs:<br>        attrlabels = doc.get(&#39;labels&#39;)<br>        flat = nested_to_record(attrlabels, sep=&#39;_&#39;)<br>        for key, value in flat.items():<br>            #print(key)<br>            if &#39;description&#39; in key:<br>                dict_temp[key] = value<br>                list_temp.append(dict_temp[key])<br>    return list_temp<br>        <br>def gcs_clearner():<br>    bucket = client.get_bucket(&#39;dj-reader&#39;)<br>    blobs = bucket.list_blobs(prefix=&#39;pics/&#39;)<br>    for blob in blobs:<br>        blob.delete()</pre><pre>def gcs_uploader():<br>    blob = bucket.blob(&#39;pics/my_plot.jpg&#39;)<br>    blob.upload_from_filename(&#39;my_plot.jpg&#39;, content_type=&#39;image/jpeg&#39;)</pre><pre>for count in range(iterations):<br>    time.sleep(2)<br>    list_of_dict_values = firebase_reader()<br>    str1 = &#39; &#39;.join(list_of_dict_values)<br>    mask = np.array(Image.open(&#39;comment.png&#39;))<br>    word_cloud = WordCloud(mask=mask,<br>                           collocations = False, <br>                           background_color = &#39;white&#39;,<br>                           width=mask.shape[1],<br>                           height=mask.shape[0]).generate(str1)<br>    word_cloud.to_file(strFile)<br>    gcs_uploader()</pre><p>Firestore structure:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d8mD4FXngN_bgzhlXoEo3g.png" /></figure><p>For the firebase client, you need to have auth keys for the service account. The service account should have roles/firebase.adminand storage read/write primitive roles. The generated key should be added in line:</p><pre>credentials.Certificate(&#39;auth/yourKey.json&#39;)</pre><p>Wordcloud has the possibility to be created in form as you wish. In this case, the “comment” image was used to create wordcloud:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/492/1*ps2UM5g24i1Rdqp9f4TOCA.png" /></figure><p>For the frontend, I used a simple flask application that was hosted in <strong>Google App Engine</strong>. The app reads the picture from GCS and represents it in index.html. The page has small CSS which reloads the page each <strong>60</strong> seconds. There is a small challenge, that when an application reads a picture from the GCS bucket, the picture might be in the renewal process, in such case temp picture like “reloading“ is represented.</p><pre>from flask import Flask<br>import matplotlib.pyplot as plt<br>import io<br>import base64<br>from google.cloud import storage<br>from PIL import Image<br>from io import BytesIO<br><br>app = Flask(__name__)<br><br>client = storage.Client.from_service_account_json(&#39;yourProject&#39;)<br>bucket = client.get_bucket(&#39;yourBucket&#39;)<br><br>@app.route(&#39;/&#39;)<br>def build_plot():<br>    blob = bucket.get_blob(&#39;pics/my_plot.jpg&#39;)<br>    try:<br>        pic = blob.download_as_bytes()<br>        img = BytesIO(pic)<br>        reloader = str(60)<br>    except:<br>        blob = bucket.get_blob(&#39;static/reload.png&#39;)<br>        pic = blob.download_as_bytes()<br>        img = BytesIO(pic)<br>        reloader = str(1)<br>        <br>    plot_url = base64.b64encode(img.getvalue()).decode()<br>    ret = &#39;&lt;meta http-equiv=&quot;refresh&quot; content=&quot;{}&quot; /&gt;&lt;p style=&quot;text-align:center;&quot;&gt;&lt;img src=&quot;data:image/png;base64,{}&quot;width=&quot;700&quot; height=&quot;700&quot;&gt;&lt;/p&gt;&#39;.format(reloader, plot_url)<br><br>    return ret<br><br>if __name__ == &#39;__main__&#39;:<br>    app.debug = True<br>    app.run()Graphical representation of the concept:</pre><p>If reload of the image makes overwrite — the app takes reload image, in case not to get an error. This happens rarely approx 30 iterations (if sleep 2 seconds).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/146/1*-0QjydtxTVroT9j9KXgyGg.png" /></figure><p>General overview (high-level diagram of solution architecture):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WdpmnUyG5wDQPf_jSBbZOg.png" /></figure><p>Challenge was completed with the final result:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/957/1*eEUOh4TUHgnvkTFc1ybg7g.gif" /></figure><p>There are multiple things that can be improved, like:</p><ul><li>Use triggers to process the data — that will allow avoiding “cycle/sleep” practice</li><li>Transfer backend code to Cloud Function or Dataflow</li><li>The frontend must be transferred to GKE, which should lower the costs</li><li>“Overwrite” while reading the generated image</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5018714da483" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[First steps with Apache Spark on K8s (part #3): GCP Spark Operator and auto scaling]]></title>
            <link>https://domisj.medium.com/first-steps-with-apache-spark-on-k8s-part-3-gcp-spark-operator-and-auto-scaling-cc0b95101f54?source=rss-b00dd99bdcc------2</link>
            <guid isPermaLink="false">https://medium.com/p/cc0b95101f54</guid>
            <category><![CDATA[spark-on-kubernetes]]></category>
            <category><![CDATA[spark]]></category>
            <dc:creator><![CDATA[Domas Jautakis]]></dc:creator>
            <pubDate>Mon, 01 Mar 2021 07:14:16 GMT</pubDate>
            <atom:updated>2021-03-01T07:14:16.161Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4JGUhwMVp7FCQ1-T_IHXuA.png" /></figure><p><em>All you will read here is personal opinion or lack of knowledge :) Please feel free to contact me for fixing incorrect parts.</em></p><p>In previous blogposts I have described <a href="https://domisj.medium.com/first-steps-with-apache-spark-on-k8s-preparation-of-environment-74f66868a0d0">how to setup environment for Kubernetes</a> and <a href="https://domisj.medium.com/first-steps-with-apache-spark-on-k8s-standalone-spark-auto-scaling-with-hpa-327f858f453b">how to auto scale Standalone Spark</a>. In this post let’s have a look on <a href="https://github.com/GoogleCloudPlatform/spark-on-k8s-operator">GCP Spark Operator</a>. Join my journey with K8s and Spark! Operator versions shown below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nY96R1ci1fLDfzBXeA2oKA.png" /></figure><p>Installation is pretty simple, just use helm. I’m not providing namespace as using default one:</p><pre>#1.</pre><pre>helm repo add spark-operator <a href="https://googlecloudplatform.github.io/spark-on-k8s-operator">https://googlecloudplatform.github.io/spark-on-k8s-operator</a></pre><pre>helm install gcp-spark-operator spark-operator/spark-operator</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/802/1*yc0KsIlO3dQy5nOWwyII0Q.png" /></figure><p>Double check if deployment was successful:</p><pre>#2.</pre><pre>kubectl get deployments<br>kubectl get po</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/702/1*TuQ4awC1zxmc-2QMnc5zmQ.png" /></figure><p>To understand how horizontal auto scaling works in GCP Spark Operator I will use several scripts:</p><pre>#3.</pre><pre>git clone <a href="https://github.com/domasjautakis/spark-operator-gcp.git">https://github.com/domasjautakis/spark-operator-gcp</a></pre><p>As the first step service account, cluster role binding and then operator should be created:</p><pre>#4.</pre><pre>kubectl apply -f spark-operator-gcp/1.rbac.yaml<br>kubectl apply -f spark-operator-gcp/2.spark-app-gcpoperator.yaml</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/801/1*02g38wDt4ohMe1MfaJt23A.png" /></figure><h3>Horizontal auto scaling</h3><p>Check example application:</p><pre>#5.</pre><pre>more spark-operator-gcp/3.spark-app-gcpoperator-scaled.yaml</pre><p>What is the difference between <a href="https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/master/examples/spark-pi.yaml">provided example</a> and this small dummy app?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/941/1*5ZHC551eVHTcXIGZl1Uafg.png" /></figure><p>The <a href="https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/master/docs/api-docs.md#dynamicallocation">dynamicAllocation </a>block is added in specification, which describes preferences of scaling. This block actually describes horizontal auto scaling parameters. In this case it’s set from 2 to 10 executors as required. Also arguments block is added which is used for input parameters. 10 — is relatively small random sample input parameter for this app, so it will scale, but not to the maximum replicas.</p><p>Most probably the best tool which grafically represents PODs scaling is Minikube dashboard, but in my case I will use simple console:</p><pre>#6.</pre><pre>kubectl apply -f spark-operator-gcp/3.spark-app-gcpoperator-scaled.yaml<br>kubectl get po</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*RqTZYIGiKXD3a_8vQZE8gw.png" /></figure><p>Next app is totally the same, just with input argument set to 100000 which gives a bit more load :)</p><pre>#7.</pre><pre>kubectl apply -f spark-operator-gcp/4.spark-app-gcpoperator-scaled.yaml<br>kubectl get po</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/741/1*r8SvQx8Mq9GyHIrkgbMM_A.png" /></figure><h3>Vertical auto scaling</h3><p>There is no such term as vertical auto scaling in K8s world, but my understanding and personal opinion is that POD can scale inside itself with provided resource and resource limits. Probably that might be possible only with operators:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/841/1*yQlR4vgKFNaPqCkkG1b17w.png" /></figure><p>I haven’t found option to set “vertical” scaling in executor POD — something like cores/coresLimit and memory/memoryLimit. Some similar <a href="https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/master/docs/api-docs.md#sparkpodspec">configuration items</a> exists, but actually they represents different functionality.</p><p><strong>Conclusion</strong>: horizontal auto scaling of Spark workers with GCP Spark operator is pretty obvious and simple. There were not HPA object type created during scaling — so that is a bit strange. Overall it’s great user experience, as it has decent documentation and wide community. Personally — love it!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cc0b95101f54" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[First steps with Apache Spark on K8s: standalone Spark auto scaling with HPA]]></title>
            <link>https://domisj.medium.com/first-steps-with-apache-spark-on-k8s-standalone-spark-auto-scaling-with-hpa-327f858f453b?source=rss-b00dd99bdcc------2</link>
            <guid isPermaLink="false">https://medium.com/p/327f858f453b</guid>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[spark-on-kubernetes]]></category>
            <dc:creator><![CDATA[Domas Jautakis]]></dc:creator>
            <pubDate>Sat, 27 Feb 2021 19:01:08 GMT</pubDate>
            <atom:updated>2021-02-28T06:54:31.803Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4JGUhwMVp7FCQ1-T_IHXuA.png" /></figure><h3>First steps with Apache Spark on K8s (part #2): standalone Spark auto scaling with HPA</h3><p><em>All you will read here is personal opinion or lack of knowledge :) Please feel free to contact me for fixing incorrect parts.</em></p><p>If you’re in this blogpost — most probably you know what Apache Spark and K8s is. Probably the most exciting feature which is possible under Kubernetes is auto scaling which allows to utilize resources more efficient. It’s possible to scale Spark in different manner, but most probably the most known is HPA. The more details can be found at: <a href="https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/">https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/710/1*G1rgr3k7WT4Nf5cazWPquw.png" /></figure><blockquote>picture taken from: <a href="https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/">https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/</a></blockquote><p>How to use HPA with Spark? The most efficient way is — Spark Operator, but in order to understand how to it works, let’s begin with standalone Spark. In this blogpost we will create HPA for worker node deployment.</p><p>If you have working environment with Kubernetes — it’s great, otherwise you can follow my previous post: <a href="https://domisj.medium.com/first-steps-with-apache-spark-on-k8s-preparation-of-environment-74f66868a0d0">First steps with apache spark on k8s preparation-of environment</a></p><p>I will begin with git clone the repo:</p><pre>#1.</pre><pre>git clone <a href="https://github.com/domasjautakis/standalone-spark-on-k8s.git">https://github.com/domasjautakis/standalone-spark-on-k8s.git</a></pre><p>This repo consist of:</p><ul><li>Main Dockerfile for master/worker container</li><li>Deployments for master and worker</li><li>Services — network service for Spark Web UI</li><li>Spark Proxy UI — which will allow us to get access to it outside of container. Module written by Alexis Seigneurin, more: <a href="https://github.com/aseigneurin/spark-ui-proxy">github</a></li></ul><p>Next step consist of Docker image build:</p><pre>#2.</pre><pre>cd standalone-spark-on-k8s/<br>docker build -f Dockerfile -t spark3:v1 .</pre><p>Verify that image was created successfully</p><pre>#3.</pre><pre>docker image ls</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1023/1*JCmfBX5QoJx8Vk1eTHudug.png" /></figure><p>Create spark-ui-proxy image and verify it:</p><pre>#4.</pre><pre>cd spark-ui-proxy/<br>docker build -f Dockerfile -t spark3proxy:v1 .<br>docker image ls</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/977/1*W7KIrFuxLB9eZwPnisRQtw.png" /></figure><p>Let’s proceed with deployments:</p><pre>#5.</pre><pre>kubectl apply -f ../services/spark-master-service.yaml<br>kubectl apply -f ../deployments/spark-master-deployment.yaml<br>sleep 5<br>kubectl apply -f ../deployments/spark-worker-deployment.yaml<br>sleep 5<br>kubectl apply -f ../deployments/spark-ui-proxy/service.yaml<br>kubectl apply -f ../deployments/spark-ui-proxy/deployment.yaml</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eLNEcNhkaeucRjJSbY0R9w.png" /></figure><p>Verify that PODs and services are running:</p><pre>#6.</pre><pre>kubectl get po<br>kubectl get svc<br>minikube ip</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/925/1*iycqYQiOrVWCLQ5PR-QDWw.png" /></figure><p>Now you can easily connect to Spark UI via minikube ip and spark-ui-proxy port. In my case it’s 31136. It might be different with other runs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NDvLg9gVxRe7gO2HF9v3fQ.png" /></figure><p>In the next step we will create HPA for spark worker deployment. Before that please verify that spark-worker deployment exists. After creation check if you can see hpa as well. In our case we’re setting maxpods to 6 and deployment will scale according cpu factor. When cpu in worker node will hit 10% — new POD will be added. I have set small factor in order to scale fast.</p><pre>#7. </pre><pre>kubectl get deployments<br>kubectl autoscale deployment spark-worker --max=6 --cpu-percent=10<br>kubectl get hpa</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/937/1*Wo0K86jYCV0dy8xO66XRSg.png" /></figure><p>In order to make some load, let’s connect to Spark worker POD and execute spark-pi app which is provided by default in example folder with input parameter 100000 of random samples.</p><pre>#8. </pre><pre>kubectl exec -it &quot;$(kubectl get pods | grep &quot;spark-worker&quot; | cut -d &#39; &#39; -f 1)&quot; -- /bin/bash<br>spark-submit --master spark://spark-master:7077 --conf spark.shuffle.service.enabled=true --conf spark.dynamicAllocation.enabled=true --class org.apache.spark.examples.SparkPi /opt/spark/examples/jars/spark-examples_2.12-3.0.2.jar 100000</pre><p>And magic starts. Be patient as HPA uses metrics-server it take cpu avg usage values so it might take some time. Duplicate ssh session and check hpa status and pods:</p><pre>#9.</pre><pre>kubectl get hpa</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/877/1*Ge-UJis_tWcYspGO8Ctt6A.png" /></figure><p>Verify in Spark UI:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/869/1*L0kNOOTCdodoe-YF8qxTJQ.png" /></figure><p>Wait while application will finish or stop it manually. It will take some time to scale it down.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/858/1*NGt3g52QSliT7c4f2M67gw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/846/1*P7synhf4aZsToB9-xqWi3w.png" /></figure><p><strong>Conclusion</strong>: it’s not efficient to run Spark in Standalone mode at Kubernetes, but it’s pretty awesome to understand how it scales. In next parts I will try to go through “vertical” and horizontal scaling in available spark operators.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=327f858f453b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[First steps with Apache Spark on K8s: preparation of environment]]></title>
            <link>https://domisj.medium.com/first-steps-with-apache-spark-on-k8s-preparation-of-environment-74f66868a0d0?source=rss-b00dd99bdcc------2</link>
            <guid isPermaLink="false">https://medium.com/p/74f66868a0d0</guid>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[spark-on-kubernetes]]></category>
            <dc:creator><![CDATA[Domas Jautakis]]></dc:creator>
            <pubDate>Sat, 27 Feb 2021 15:32:04 GMT</pubDate>
            <atom:updated>2021-03-01T06:56:10.447Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yof31xqNUL9zPT_EzaQhkA.png" /></figure><h3>First steps with Apache Spark on K8s (part #1): preparation of environment</h3><p><em>All you will read here is personal opinion or lack of knowledge :) Please feel free to contact me for fixing incorrect parts.</em></p><p>In order to start working with Spark on Kubernetes first step is to prepare sandbox. In my case I’m using Windows 10 based machine with Hyper-V. For this sandbox I can dedicate 16 Gigs and 8 threads.</p><pre>All scripts from this article can be found at: <a href="https://github.com/domasjautakis/docker-minikube">github.com/domasjautakis/docker-minikube</a></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/626/1*D17K7c4QFjKjDeesJ7Ml9g.png" /></figure><p>As OS I will use Centos 7 Minimal:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/481/1*pOxl7zAdAp7_BkSvleO6lA.png" /></figure><p>The first step of prep. — install all packages which might be required for installation or configuration.</p><pre>#1.</pre><pre>yum install dnf -y<br>yum install dnf-plugins-core -y<br>yum install wget -y<br>yum install java-11-openjdk-devel -y<br>yum install git -y</pre><p>Let’s create dedicated user and connect to it, as we’re going to make all installation from it.</p><pre>#2.</pre><pre>adduser minikubeuser<br>passwd minikubeuser<br>#**New password: <br>usermod -aG wheel minikubeuser<br>su - minikubeuser</pre><p>For the next step I will:</p><ul><li>will add dnf repo</li><li>install docker</li><li>start the service</li><li>add firewall masquerade</li><li>will change mode for docker.sock</li></ul><p>Mode change is needed in order to use docker socket.</p><pre>#3.</pre><pre>sudo dnf update -y<br>sudo setenforce 0<br>sudo sed -i --follow-symlinks &#39;s/SELINUX=enforcing/SELINUX=disabled/g&#39; /etc/sysconfig/selinux<br>sudo dnf config-manager --add-repo=<a href="https://download.docker.com/linux/centos/docker-ce.repo">https://download.docker.com/linux/centos/docker-ce.repo</a><br>sudo dnf install docker-ce -y<br>sudo systemctl start docker<br>sudo systemctl enable docker<br>sudo firewall-cmd --zone=public --add-masquerade --permanent<br>sudo firewall-cmd --reload<br>sudo chmod 666 /var/run/docker.sock<br>docker ps --no-trunc<br>docker image ls</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/820/1*75TFDZewRcI3hyDR4Kvtbg.png" /></figure><p><em>if you have some issues with </em><strong><em>docker ps</em></strong><em> — make sure that the latest step with change mode is completed. More can be found at: </em><a href="https://stackoverflow.com/questions/48957195/how-to-fix-docker-got-permission-denied-issue"><em>https://stackoverflow.com/questions/48957195/how-to-fix-docker-got-permission-denied-issue</em></a></p><p>In the next step I will install conntrack.</p><pre>#4.</pre><pre>sudo dnf install conntrack -y</pre><p>Let’s install minikube now:</p><pre>#5.</pre><pre>curl -LO <a href="https://storage.googleapis.com/kubernetes-release/release/%60curl">https://storage.googleapis.com/kubernetes-release/release/`curl</a> -s <a href="https://storage.googleapis.com/kubernetes-release/release/stable.txt%60/bin/linux/amd64/kubectl">https://storage.googleapis.com/kubernetes-release/release/stable.txt`/bin/linux/amd64/kubectl</a><br>chmod +x ./kubectl<br>sudo mv ./kubectl /usr/local/bin/kubectl<br>curl -Lo minikube <a href="https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64">https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64</a><br>chmod +x minikube<br>sudo mkdir -p /usr/local/bin/<br>sudo install minikube /usr/local/bin/</pre><p>It’s almost there. Next is start the minikube!</p><pre>#6.</pre><pre>minikube start --driver=none<br>minikube status</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*kaJBn9-Q8rhMapsJWBiYLw.png" /></figure><p>It’s worth to check which addons are enabled. For such functionality like HPA metrics-server might be required</p><pre>#7.</pre><pre>minikube addons list<br>minikube addons enable metrics-server<br>minikube addons list</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/663/1*vGdc_QJFoLjQ7NxYDCWiQA.png" /></figure><p>The final step in environment preparation — helm install:</p><pre>#8.</pre><pre>curl -fsSL -o get_helm.sh <a href="https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3">https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3</a><br>chmod 700 get_helm.sh<br>./get_helm.sh</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*NWoi4cBDdgDgu8XIuU1ZJA.png" /></figure><p>Done!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=74f66868a0d0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Comparison of Big Data storage layers: Delta vs Apache Hudi vs Apache Iceberg. Part#1]]></title>
            <link>https://domisj.medium.com/comparison-of-big-data-storage-layers-delta-vs-apache-hudi-vs-apache-iceberg-part-1-200599645a02?source=rss-b00dd99bdcc------2</link>
            <guid isPermaLink="false">https://medium.com/p/200599645a02</guid>
            <category><![CDATA[apache-iceberg]]></category>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[delta-lake]]></category>
            <category><![CDATA[apache-hudi]]></category>
            <dc:creator><![CDATA[Domas Jautakis]]></dc:creator>
            <pubDate>Mon, 27 Jan 2020 11:56:54 GMT</pubDate>
            <atom:updated>2021-02-27T14:22:57.206Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*305nEJHGZeTp6abnnwuUig.jpeg" /></figure><p><em>All you will read here is personal opinion or lack of knowledge :) Please feel free to contact me for fixing incorrect parts.</em></p><p>As data engineer who is passionated about <a href="https://spark.apache.org/"><strong>Apache Spark</strong></a> I decided to compare different and similar open-source projects like <a href="https://delta.io/"><strong>Delta</strong></a>, <a href="https://hudi.incubator.apache.org/"><strong>Hudi </strong></a>and <a href="https://iceberg.incubator.apache.org/"><strong>Iceberg</strong></a>. The idea is simple: prepare environment for all three technologies and compare them from Apache Spark and consumption perspectives including <a href="https://hive.apache.org/"><strong>Hive </strong></a>and Presto. e. Topic is splitted in three parts:</p><ul><li>Preparation of single node cluster including: Hadoop, Spark, Hive, Presto and all dependencies.</li><li>Test how Delta, Hudi, Iceberg will behave with updates, deletes, time-travels, schema merges. Also I will check how transaction logs looks like and what is the size difference with default configs and same volume of data. Please contact me if some other parts are interesting for you.</li><li>Consumption part from Apache Hive and Presto perspectives</li></ul><p>Let’s start from part 1!</p><h4>Preparation of single node cluster</h4><p>Versions:</p><pre>ubuntu-18.04.3-live-server-amd64<br>openjdk-8-jdk<br>scala-2.11.12<br>spark-2.4.4-bin-hadoop2.7<br>hadoop-2.7.7<br>apache-hive-2.3.6-bin<br>presto-server-329.tar<br>org.apache.iceberg:iceberg-spark-runtime:0.7.0-incubating<br>org.apache.hudi:hudi-spark-bundle:0.5.0-incubating<br>io.delta:delta-core_2.11:0.5.0<br>--<br>spark-3.0.0-preview2-bin-hadoop2.7 - let&#39;s check impact of spark version updagrade :)</pre><p>In Ubuntu i’m using user <strong>spuser</strong>. For this user I will generate authorization keys which will be needed for hadoop.</p><pre>ssh-keygen -t rsa -P &#39;&#39; -f ~/.ssh/id_rsa<br>cat ~/.ssh/id_rsa.pub &gt;&gt; ~/.ssh/authorized_keys<br>chmod 0600 ~/.ssh/authorized_keys</pre><p>The first step is install java 1.8 for Spark:</p><pre>#1.</pre><pre>sudo add-apt-repository ppa:openjdk-r/ppa<br>sudo apt-get update<br>sudo apt-get install openjdk-8-jdk<br>sudo update-alternatives --config java<br>sudo update-alternatives --config javac</pre><p>Make sure that your java version is 1.8</p><pre>#2.</pre><pre>spuser@acid:~$ java -version<br>openjdk version &quot;1.8.0_232&quot;<br>OpenJDK Runtime Environment (build 1.8.0_232-8u232-b09-0ubuntu1~16.04.1-b09)<br>OpenJDK 64-Bit Server VM (build 25.232-b09, mixed mode)</pre><p>Download all required packages</p><pre>#3.</pre><pre>mkdir downloads<br>cd downloads/<br>wget <a href="https://downloads.lightbend.com/scala/2.11.12/scala-2.11.12.deb">https://downloads.lightbend.com/scala/2.11.12/scala-2.11.12.deb</a><br>wget <a href="http://apache.mirror.vu.lt/apache/spark/spark-2.4.4/spark-2.4.4-bin-hadoop2.7.tgz">http://apache.mirror.vu.lt/apache/spark/spark-2.4.4/spark-2.4.4-bin-hadoop2.7.tgz</a><br>wget <a href="http://apache.mirror.vu.lt/apache/spark/spark-3.0.0-preview2/spark-3.0.0-preview2-bin-hadoop2.7.tgz">http://apache.mirror.vu.lt/apache/spark/spark-3.0.0-preview2/spark-3.0.0-preview2-bin-hadoop2.7.tgz</a><br>wget <a href="https://archive.apache.org/dist/hadoop/core/hadoop-2.7.7/hadoop-2.7.7.tar.gz">https://archive.apache.org/dist/hadoop/core/hadoop-2.7.7/hadoop-2.7.7.tar.gz</a><br>wget <a href="http://apache.mirror.vu.lt/apache/hive/hive-2.3.6/apache-hive-2.3.6-bin.tar.gz">http://apache.mirror.vu.lt/apache/hive/hive-2.3.6/apache-hive-2.3.6-bin.tar.gz</a><br>wget <a href="https://repo1.maven.org/maven2/io/prestosql/presto-cli/329/presto-cli-329-executable.jar">https://repo1.maven.org/maven2/io/prestosql/presto-cli/329/presto-cli-329-executable.jar</a><br>wget <a href="https://repo1.maven.org/maven2/io/prestosql/presto-server/329/presto-server-329.tar.gz">https://repo1.maven.org/maven2/io/prestosql/presto-server/329/presto-server-329.tar.gz</a></pre><p>Let’s check sizes what we downloaded:</p><pre>#4.</pre><pre>spuser@acid:~/downloads$ ll -h</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/683/1*9OvofwaXvC7OWubIvjDxfg.png" /></figure><p>Install scala</p><pre>#5.</pre><pre>sudo dpkg -i scala-2.11.12.deb</pre><p>I will unarchive all packages under directory /usr/local/. For particular versions symlinks will be created, for easier migration in the future:</p><pre>#6.</pre><pre>sudo tar -xzf apache-hive-2.3.6-bin.tar.gz -C /usr/local/<br>sudo tar -xzf hadoop-2.7.7.tar.gz -C /usr/local/<br>sudo tar -xzf spark-2.4.4-bin-hadoop2.7.tgz -C /usr/local/<br>sudo tar -xzf spark-3.0.0-preview2-bin-hadoop2.7.tgz -C /usr/local/<br>sudo tar -xzf presto-server-329.tar.gz -C /usr/local</pre><pre>sudo chown -R spuser /usr/local/apache-hive-2.3.6-bin/<br>sudo chown -R spuser /usr/local/hadoop-2.7.7/<br>sudo chown -R spuser /usr/local/spark-2.4.4-bin-hadoop2.7/<br>sudo chown -R spuser /usr/local/spark-3.0.0-preview2-bin-hadoop2.7/<br>sudo chown -R spuser /usr/local/presto-server-329/</pre><pre>cd /usr/local/<br>sudo ln -s /usr/local/apache-hive-2.3.6-bin/ /usr/local/hive<br>sudo chown -h spuser:spuser /usr/local/hive<br>sudo ln -s /usr/local/hadoop-2.7.7/ /usr/local/hadoop<br>sudo chown -h spuser:spuser /usr/local/hadoop<br>sudo ln -s /usr/local/spark-2.4.4-bin-hadoop2.7 /usr/local/spark<br>sudo chown -h spuser:spuser /usr/local/spark<br>sudo ln -s /usr/local/spark-3.0.0-preview2-bin-hadoop2.7 /usr/local/spark3<br>sudo chown -h spuser:spuser /usr/local/spark3<br>sudo ln -s /usr/local/presto-server-329 /usr/local/presto<br>sudo chown -h spuser:spuser /usr/local/presto<br> </pre><p>I will create several folders for logs and hdfs storage. To create some folders under the root is not the best practice, but anyway it’s sandbox:</p><pre>#7.</pre><pre>sudo mkdir /logs<br>sudo chown -R spuser /logs<br>mkdir /logs/hadoop</pre><pre>#Add dir for data<br>sudo mkdir /hadoop<br>sudo chown -R spuser /hadoop<br>mkdir -p /hadoop/hdfs/namenode<br>mkdir -p /hadoop/hdfs/datanode</pre><pre>#create tmp hadoop dir:<br>mkdir -p /tmp/hadoop</pre><p>Now let’s update env. variables — .bashrc:</p><pre>#8.</pre><pre>sudo nano ~/.bashrc<br>#Add entries in existing file:</pre><pre>export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64<br>export PATH=$PATH:$JAVA_HOME/bin</pre><pre>export HADOOP_HOME=/usr/local/hadoop<br>export HIVE_HOME=/usr/local/hive</pre><pre>export PATH=$PATH:$HADOOP_HOME/bin<br>export PATH=$PATH:$HADOOP_HOME/sbin<br>export PATH=$PATH:$HIVE_HOME/bin</pre><pre>export HADOOP_MAPRED_HOME=$HADOOP_HOME<br>export HADOOP_COMMON_HOME=$HADOOP_HOME<br>export HADOOP_HDFS_HOME=$HADOOP_HOME<br>export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop<br>export YARN_HOME=$HADOOP_HOME<br>export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native<br>export HADOOP_OPTS=&quot;-Djava.library.path=$HADOOP_HOME/lib/native&quot;<br>export HADOOP_LOG_DIR=/logs/hadoop</pre><pre>export SPARK_HOME=/usr/local/spark<br>export PATH=$PATH:$SPARK_HOME/bin</pre><pre>#Save it!</pre><pre>#Source it:<br>source ~/.bashrc</pre><h4>Hadoop related changes</h4><p>Changes in Hadoop configs, navigate to dir:</p><pre>#9.</pre><pre>cd /usr/local/hadoop/etc/hadoop</pre><p>hadoop-env.sh:</p><pre>#10.</pre><pre>#Comment existing JAVA_HOME and add new one: <br>export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64</pre><p>core-site.xml:</p><pre>#11.</pre><pre>&lt;configuration&gt;<br>&lt;property&gt;<br> &lt;name&gt;hadoop.tmp.dir&lt;/name&gt;<br>   &lt;value&gt;/tmp/hadoop&lt;/value&gt;<br>   &lt;description&gt;A base for other temporary directories.&lt;/description&gt;<br>&lt;/property&gt;<br> &lt;property&gt;<br>        &lt;name&gt;fs.defaultFS&lt;/name&gt;<br>        &lt;value&gt;hdfs://localhost:9000&lt;/value&gt;<br>    &lt;/property&gt;<br>&lt;/configuration&gt;</pre><p>mapred-site.xml:</p><pre>#12.</pre><pre>&lt;configuration&gt;<br>&lt;property&gt;<br>        &lt;name&gt;mapreduce.framework.name&lt;/name&gt;<br>        &lt;value&gt;yarn&lt;/value&gt;<br>    &lt;/property&gt;<br>&lt;/configuration&gt;</pre><p>hdfs-site.xml:</p><pre>#13.</pre><pre>&lt;configuration&gt;<br>&lt;property&gt;<br>        &lt;name&gt;dfs.replication&lt;/name&gt;<br>        &lt;value&gt;1&lt;/value&gt;<br>&lt;/property&gt;<br>&lt;property&gt;<br>   &lt;name&gt;dfs.namenode.name.dir&lt;/name&gt;<br>   &lt;value&gt;file:/hadoop/hdfs/namenode&lt;/value&gt;<br> &lt;/property&gt;<br> &lt;property&gt;<br>   &lt;name&gt;dfs.datanode.data.dir&lt;/name&gt;<br>   &lt;value&gt;file:/hadoop/hdfs/datanode&lt;/value&gt;<br> &lt;/property&gt;<br>&lt;/configuration&gt;</pre><p>yarn-site.xml:</p><pre>#14.</pre><pre>&lt;configuration&gt;<br>&lt;property&gt;<br>      &lt;name&gt;yarn.nodemanager.aux-services&lt;/name&gt;<br>      &lt;value&gt;mapreduce_shuffle&lt;/value&gt;<br>   &lt;/property&gt;<br>&lt;/configuration&gt;</pre><p>After this HDFS is ready to go, make format and start the services:</p><pre>#15.</pre><pre>hdfs namenode -format<br>start-all.sh</pre><p>Let’s check what is running:</p><pre>#16.</pre><pre>spuser@acid:/usr/local/hadoop/etc/hadoop$ jps<br>9890 DataNode<br>10275 ResourceManager<br>10115 SecondaryNameNode<br>10613 NodeManager<br>9705 NameNode<br>10732 Jps</pre><h4>Hive related changes</h4><p>Create hdfs directories for hive</p><pre>#17.</pre><pre>#Create HDFS dirs:<br>hdfs dfs -mkdir -p /user/hive/warehouse<br>hdfs dfs -mkdir /tmp<br>hdfs dfs -chmod g+w /user/hive/warehouse<br>hdfs dfs -chmod g+w /tmp</pre><p>Navigate to hive configuration directory:</p><pre>#18.</pre><pre>cd /usr/local/hive/conf</pre><p>Create hive-site.xml:</p><pre>#19.</pre><pre>&lt;configuration&gt;<br>&lt;property&gt;<br>&lt;name&gt;javax.jdo.option.ConnectionURL&lt;/name&gt;<br>&lt;value&gt;jdbc:derby:;databaseName=/usr/local/hive/metastore_db;create=true&lt;/value&gt;<br>&lt;description&gt;<br>JDBC connect string for a JDBC metastore.<br>To use SSL to encrypt/authenticate the connection, provide database-specific SSL flag in the connection URL.<br>For example, jdbc:postgresql://myhost/db?ssl=true for postgres database.<br>&lt;/description&gt;<br>&lt;/property&gt;<br>&lt;property&gt;<br>&lt;name&gt;hive.metastore.warehouse.dir&lt;/name&gt;<br>&lt;value&gt;/user/hive/warehouse&lt;/value&gt;<br>&lt;description&gt;location of default database for the warehouse&lt;/description&gt;<br>&lt;/property&gt;<br>&lt;property&gt;<br>&lt;name&gt;hive.metastore.uris&lt;/name&gt;<br>&lt;value/&gt;<br>&lt;description&gt;Thrift URI for the remote metastore. Used by metastore client to connect to remote metastore.&lt;/description&gt;<br>&lt;/property&gt;<br>&lt;property&gt;<br>&lt;name&gt;javax.jdo.option.ConnectionDriverName&lt;/name&gt;<br>&lt;value&gt;org.apache.derby.jdbc.EmbeddedDriver&lt;/value&gt;<br>&lt;description&gt;Driver class name for a JDBC metastore&lt;/description&gt;<br>&lt;/property&gt;<br>&lt;property&gt;<br>&lt;name&gt;javax.jdo.PersistenceManagerFactoryClass&lt;/name&gt;<br>&lt;value&gt;org.datanucleus.api.jdo.JDOPersistenceManagerFactory&lt;/value&gt;<br>&lt;description&gt;class implementing the jdo persistence&lt;/description&gt;<br>&lt;/property&gt;<br>&lt;property&gt;<br>&lt;name&gt;hive.metastore.schema.verification&lt;/name&gt;<br>&lt;value&gt;false&lt;/value&gt;<br>&lt;description/&gt;<br>&lt;/property&gt;<br>&lt;/configuration&gt;</pre><p>Create hive-env.sh</p><pre>#20.</pre><pre># The heap size of the jvm stared by hive shell script can be controlled via:<br>#<br>export HADOOP_HEAPSIZE=512<br>#<br># Larger heap size may be required when running queries over large number of files or partitions.<br># By default hive shell scripts use a heap size of 256 (MB).  Larger heap size would also be<br># appropriate for hive server (hwi etc).</pre><pre># Set HADOOP_HOME to point to a specific hadoop install directory<br>export HADOOP_HOME=/usr/local/hadoop</pre><pre># Hive Configuration Directory can be controlled by:<br>export HIVE_CONF_DIR=/usr/local/hive/conf</pre><pre># Folder containing extra ibraries required for hive compilation/execution can be controlled by:<br>export HIVE_AUX_JARS_PATH=/usr/local/hive/lib/*.jar</pre><p>Before create hive metastore, please update hive-schema-2.3.0.derby.sql otherwise iceberg will not be able to create table and you will get error like:</p><pre>#21.</pre><pre>ERROR metastore.RetryingHMSHandler: Retrying HMSHandler after 2000 ms (attempt 8 of 10) with error: javax.jdo.JDODataStoreException: Insert of object &quot;org.apache.hadoop.hive.metastore.model.MTable@604201a0&quot; using statement &quot;INSERT INTO TBLS (TBL_ID,OWNER,CREATE_TIME,SD_ID,TBL_NAME,VIEW_EXPANDED_TEXT,LAST_ACCESS_TIME,DB_ID,RETENTION,VIEW_ORIGINAL_TEXT,TBL_TYPE) VALUES (?,?,?,?,?,?,?,?,?,?,?)&quot; failed : Column &#39;IS_REWRITE_ENABLED&#39;  cannot accept a NULL value.</pre><p>Update hive-schema-2.3.0.derby.sql</p><pre>#22.</pre><pre>nano /usr/local/hive/scripts/metastore/upgrade/derby/hive-schema-2.3.0.derby.sql</pre><pre>#update statement: &quot;APP&quot;.&quot;TBLS&quot;<br>CREATE TABLE &quot;APP&quot;.&quot;TBLS&quot; (&quot;TBL_ID&quot; BIGINT NOT NULL, &quot;CREATE_TIME&quot; INTEGER NOT NULL, &quot;DB_ID&quot; BIGINT, &quot;LAST_ACCESS_TIME&quot; INTEGER NOT NULL, &quot;OWNER&quot; VARCHAR(767), &quot;RETENTION&quot; INTEGER NOT NULL, &quot;SD_ID&quot; BIGINT, &quot;TBL_NAME&quot; VARCHAR(256), &quot;TBL_TYPE&quot; VARCHAR(128), &quot;VIEW_EXPANDED_TEXT&quot; LONG VARCHAR, &quot;VIEW_ORIGINAL_TEXT&quot; LONG VARCHAR, &quot;IS_REWRITE_ENABLED&quot; CHAR(1) NOT NULL <strong>DEFAULT &#39;N&#39;</strong>);</pre><p>After this change let’s create hive metastore:</p><pre>#23.</pre><pre>schematool -initSchema -dbType derby --verbose</pre><p>Check if schema was created successful:</p><pre>#24.<br>...</pre><pre>beeline&gt; Initialization script completed<br>schemaTool completed</pre><p>Test Hive via CLI:</p><pre>#25.</pre><pre>hive -e &quot;show databases&quot;<br></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jnzH3pnPS3sq4zM0ELZvpg.png" /></figure><h4>Presto related changes</h4><p>Create config dir:</p><pre>#26. </pre><pre>mkdir -p /usr/local/presto/etc</pre><p>Create config file<em> /usr/local/presto/etc/config.propertie</em>s:</p><pre>#27.</pre><pre>coordinator=true<br>node-scheduler.include-coordinator=true<br>http-server.http.port=8080<br>query.max-memory=5GB<br>query.max-memory-per-node=1GB<br>query.max-total-memory-per-node=2GB<br>discovery-server.enabled=true<br>discovery.uri=<a href="http://localhost:8080/">http://localhost:8080</a></pre><p>Create jvm config file<em> /usr/local/presto/etc/jvm.propertie</em>s:</p><pre>#28.</pre><pre>-server<br>-Xmx16G<br>-XX:+UseG1GC<br>-XX:G1HeapRegionSize=32M<br>-XX:+UseGCOverheadLimit<br>-XX:+ExplicitGCInvokesConcurrent<br>-XX:+HeapDumpOnOutOfMemoryError<br>-XX:+ExitOnOutOfMemoryError</pre><p>Create node config file<em> /usr/local/presto/etc/node.properties</em>:</p><pre>#29.</pre><pre>node.environment=production<br>node.id=ffffffff-ffff-ffff-ffff-ffffffffffff<br>node.data-dir=/var/presto/data</pre><p>Create dir:</p><pre>#30.</pre><pre>sudo mkdir -p /var/presto/data<br>sudo chown spuser:spuser -h /var/presto<br>sudo chown spuser:spuser -h /var/presto/data</pre><p>Create catalog and hive config <em>/usr/local/presto/etc/catalog/hive.properties</em>:</p><pre>#31.</pre><pre>connector.name=hive-hadoop2<br>hive.metastore.uri=thrift://localhost:9083</pre><h4>Spark related changes</h4><p>Check scala version</p><pre>#32.</pre><pre>scala -version<br>#make sure that you can see something like: <br>Scala code runner version 2.11.12 -- Copyright 2002-2017, LAMP/EPFL<br>#otherwise get back to step #5.</pre><p>Navigate to spark config dir:</p><pre>#33.</pre><pre>cd /usr/local/spark/conf</pre><p>spark-env.sh:</p><pre>#34.</pre><pre>#add<br>export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop<br>export SPARK_CONF_DIR=/usr/local/spark/conf<br>export SPARK_LOCAL_IP=127.0.0.1</pre><p>Copy hive-site.xml, because we will test how delta, hudi, iceberg behavious with Hive and Presto.</p><pre>#35.</pre><pre>cp /usr/local/hive/conf/hive-site.xml /usr/local/spark/conf/</pre><p>Download all needed dependencies:</p><pre>#36.</pre><pre>spark-shell --packages org.apache.iceberg:iceberg-spark-runtime:0.7.0-incubating,org.apache.hudi:hudi-spark-bundle:0.5.0-incubating,io.delta:delta-core_2.11:0.5.0 --conf &#39;spark.serializer=org.apache.spark.serializer.KryoSerializer&#39;</pre><p>Let’s try to store dataframe with all 3 types to make it sure that it works.</p><p>Delta</p><pre>#37.</pre><pre>import org.apache.spark.sql.SaveMode._</pre><pre>spark.range(1000).toDF.write.format(&quot;delta&quot;).mode(Overwrite).save(&quot;/tmp/delta_tab01&quot;)</pre><p>Hudi</p><pre>#38.</pre><pre>import org.apache.spark.sql.SaveMode._<br>import org.apache.hudi.DataSourceWriteOptions._<br>import org.apache.hudi.config.HoodieWriteConfig._</pre><pre>spark.range(1000).write.format(&quot;org.apache.hudi&quot;).option(TABLE_NAME, &quot;hudi_tab01&quot;).option(PRECOMBINE_FIELD_OPT_KEY, &quot;id&quot;).option(RECORDKEY_FIELD_OPT_KEY, &quot;id&quot;).mode(Overwrite).save(&quot;/tmp/hudi_tab01&quot;)</pre><p>Iceberg</p><pre>#39.</pre><pre>import org.apache.iceberg.hive.HiveCatalog<br>import org.apache.iceberg.catalog._<br>import org.apache.iceberg.Schema<br>import org.apache.iceberg.types.Types._<br>import org.apache.iceberg.PartitionSpec<br>import org.apache.iceberg.spark.SparkSchemaUtil<br>import org.apache.iceberg.hadoop.HadoopTables</pre><pre>val name = TableIdentifier.of(&quot;default&quot;,&quot;iceberg_tab01&quot;);<br>val df1=spark.range(1000).toDF.withColumn(&quot;level&quot;,lit(&quot;1&quot;))<br>val df1_schema = SparkSchemaUtil.convert(df1.schema)<br>val partition_spec=PartitionSpec.builderFor(df1_schema).identity(&quot;level&quot;).build<br>val tables = new HadoopTables(spark.sessionState.newHadoopConf())<br>val table = tables.create(df1_schema, partition_spec, &quot;hdfs:/tmp/iceberg_tab01&quot;)</pre><pre>df1.write.format(&quot;iceberg&quot;).mode(&quot;append&quot;).save(&quot;hdfs:/tmp/iceberg_tab01&quot;)</pre><p>Let’s check results in HDFS:</p><pre>#40.</pre><pre>hdfs dfs -ls -h -R /tmp/delta* &amp;&amp; hdfs dfs -ls -h -R /tmp/hudi* &amp;&amp; hdfs dfs -ls -h -R /tmp/iceberg*</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mRStu3LFbyVtN72VxYzZtg.png" /></figure><p>It Works!</p><p>Other parts most probably will not be released! Following parts were written, but not published. Now it does not make sense to publish them as most of the products already has new versions and functionalities. I’m not eliminating possibility to re-write those parts in the future.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=200599645a02" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>