<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Tilak Patidar on Medium]]></title>
        <description><![CDATA[Stories by Tilak Patidar on Medium]]></description>
        <link>https://medium.com/@tilakpatidar?source=rss-f3db0362cb75------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*Az0AXbTkGgT-pVwbYUfQ-A.jpeg</url>
            <title>Stories by Tilak Patidar on Medium</title>
            <link>https://medium.com/@tilakpatidar?source=rss-f3db0362cb75------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 08:25:08 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@tilakpatidar/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Vectorized and Compiled Queries — Part 3]]></title>
            <link>https://medium.com/@tilakpatidar/vectorized-and-compiled-queries-part-3-807d71ec31a5?source=rss-f3db0362cb75------2</link>
            <guid isPermaLink="false">https://medium.com/p/807d71ec31a5</guid>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[computer-science]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[software]]></category>
            <category><![CDATA[database]]></category>
            <dc:creator><![CDATA[Tilak Patidar]]></dc:creator>
            <pubDate>Sun, 07 Oct 2018 11:49:27 GMT</pubDate>
            <atom:updated>2018-10-07T11:49:27.464Z</atom:updated>
            <content:encoded><![CDATA[<h3>Vectorized and Compiled Queries — Part 3</h3><blockquote>An unexamined query execution code is not worth executing. — Socrates the DBA</blockquote><p>In the previous <a href="https://medium.com/@tilakpatidar/vectorized-and-compiled-queries-part-1-37794c3860cc">Part 1</a> and <a href="https://medium.com/@tilakpatidar/vectorized-and-compiled-queries-part-2-cd0d91fa189f">Part 2</a> of this series, we learnt about the volcano iterator model which is a predecessor of vectorized and compiled queries. We also saw how it suffered from instruction cache miss issue.</p><p>Vectorization came up with the remedy that we should batch the data into vectors and apply our instructions on the vector instead of applying them one tuple at a time. This ameliorated the problem of instruction cache misses but it did not root out the problem completely.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1C3vaglP1gmirPZ7.gif" /><figcaption>Time to clean your query execution code so you only get what you need.</figcaption></figure><p>Compiled queries tried to solve it at the compiler level. The problem was seen in a new light as a compiler problem. It was realised that the generation of query execution code should have minimum branching and abstraction. In short, it should not have any unnecessary parts that will not execute.</p><p>Changes to the query compilation phase were proposed. Volcano iterator model which is very general and fits all kinds of queries was being shunned. It was proposed that the query generation and compilation should be very specific to the query hand. In short …</p><blockquote>They wanted to write a program which could generate query specific code. Removing all the clutter which are excessive branching instructions and abstractions.</blockquote><p><strong>Example: </strong>This is a simple query which will return the sum of ages across all the people.</p><p>SELECT SUM(age) FROM people WHERE age &gt; 10;</p><p>For volcano iterator model it will create at least 3 iterators. The code will be generic to suit any data type and table schema.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/121/1*zLCFnHXBDInEffDZfU65TQ.jpeg" /><figcaption>A typical volcano iterator model with scan, filter and aggregate.</figcaption></figure><p>On, the other hand a compiled query execution code will be much simpler.</p><pre><em>for </em>(<em>int </em>i = 0; i &lt; size; i += 1) {<br>  Person person = people[i];<br>  <em>int </em>age = person.age;<br>  <em>if </em>(age &gt; 10) {<br>    sum += age;<br>  }<br>}</pre><p>Generating query execution code tailored to the query also provides a way to slip in performance improvements like loop unrolling or loop pipelining discussed in <a href="https://medium.com/@tilakpatidar/vectorized-and-compiled-queries-part-2-cd0d91fa189f">part 2 </a>of this series.</p><pre><em>for </em>(<em>int </em>i = 0; i &lt; size; i += 4) {<br>  Person person1 = people[i];<br>  Person person2 = people[i + 2];<br>  Person person3 = people[i + 3];<br>  Person person4 = people[i + 4];<br>  <em>if </em>(person1.age &gt; 10) {<br>    sum += person1.age;<br>  }<br>  <em>if </em>(person2.age &gt; 10) {<br>    sum += person2.age;<br>  }<br>  <em>if </em>(person3.age &gt; 10) {<br>    sum += person3.age;<br>  }<br>  <em>if </em>(person4.age &gt; 10) {<br>    sum += person4.age;<br>  }<br>}</pre><p><strong>How compiled queries are achieved?</strong></p><ul><li>Identify various operators and their sequence of execution.</li><li>Combine various function calls into one.</li><li>Based on the storage format and hardware support apply optimizations such as loop unrolling, loop pipelining, SIMD instructions etc.</li></ul><p><strong>Advantages of compiled queries</strong></p><ul><li>No virtual function calls.</li><li>Data in CPU registers.</li><li>Compiler loop unrolling, loop pipelining and usage of SIMD.</li></ul><p><strong>Disadvantages of compiled queries</strong></p><ul><li>Compilation overhead. Generating custom code for queries takes a while. If data is small then this overhead sometimes might be more than the execution overhead itself. Thus, compiled queries are not widely used for OLTP queries they are more used for OLAP kind of queries.</li></ul><p>This is it, for now.</p><p>If you have anything to add or you spotted an error in my post, do not hesitate to contact me, I’ll be happy to make changes and learn a few things.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=807d71ec31a5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Vectorized and Compiled Queries — Part 2]]></title>
            <link>https://medium.com/@tilakpatidar/vectorized-and-compiled-queries-part-2-cd0d91fa189f?source=rss-f3db0362cb75------2</link>
            <guid isPermaLink="false">https://medium.com/p/cd0d91fa189f</guid>
            <category><![CDATA[software]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[computer-science]]></category>
            <category><![CDATA[bds]]></category>
            <dc:creator><![CDATA[Tilak Patidar]]></dc:creator>
            <pubDate>Sun, 07 Oct 2018 09:36:34 GMT</pubDate>
            <atom:updated>2018-10-07T11:50:11.298Z</atom:updated>
            <content:encoded><![CDATA[<h3>Vectorized and Compiled Queries — Part 2</h3><p>Let’s resume our investigation of vectorized and compiled queries. In <a href="https://medium.com/@tilakpatidar/vectorized-and-compiled-queries-part-1-37794c3860cc">part 1</a> of this series, we examined the volcano iterator model which was the precursor of both vectorized and compiled queries. We saw how it suffered from instruction cache misses and the two alternatives available to solve it are vectorized and compiled queries. In this part 2 of the series, we will discuss vectorized queries.</p><h3>Vectorized Queries</h3><p>So, how do vectorized queries tackle the problem of instruction cache misses discussed in <a href="https://medium.com/@tilakpatidar/vectorized-and-compiled-queries-part-1-37794c3860cc">part 1</a> of this series.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/982/1*krn3NPwCERy6hk0xKB5CZQ.png" /><figcaption>Vector instructions apply an instruction on a vector of data</figcaption></figure><p>Here is a non-fun wiki definition:</p><blockquote>Vectorization is the term for converting a scalar program to a vector program. Vectorized programs can run multiple operations from a single instruction, whereas scalar can only operate on pairs of operands at once.</blockquote><blockquote>Idea is to take advantage of Data Level Parallelism (DLP) within an algorithm via vector processing i.e., processing batches of rows together.</blockquote><p>In easier words:</p><blockquote>Operate on a vector (batch) of records using the single instruction.</blockquote><p>By, the single instruction we mean that you need not load the same instructions frequently for each row. Load them once and operate on a batch of records. Thus, instruction cache misses will be lessened as the execution of code is not happening for each row but for a batch of rows.</p><p>So, instead of reducing the branching and unnecessary abstractions due to general query execution code, vectorization tackles the problem of instruction cache miss by batching the rows. It does not weed out the problem but ameliorates it.</p><h3>Vectorization and Columnar Formats</h3><p>Vectorization techniques are popular for columnar formats. There are certain advantages that columnar formats provide which facilitate better performance of vectorized queries:</p><ul><li>Using columnar formats load only the columns required by the query. Thus, less I/O on reading from disk and more data could be loaded into the cache.</li><li>Applying an operator on a vector of a column is performant using the tight loops.</li><li>Data availability increases as the unused columns are not present in the cache. Having the data which is actually required for the query laid side by side like members of an array reduces cache misses.</li></ul><p>Applying vectorization techniques on a row by row file format would result in loading the unnecessary columns in the main memory from where the projection of the column has to be taken on which the vectorization has to be applied. Thus, vectorization techniques are popular with columnar formats.</p><h4>Vectorization techniques</h4><p>Vectorization exploits SIMD (Single Instruction Multiple Data) instructions which allows executing a single instruction on multiple data.</p><p>One SIMD instruction can process 16 integers at a time, thereby achieving Data Level Parallelism.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/634/0*mBG0wumVccuLuS8H.gif" /><figcaption>Addition of numbers using vectorization on SIMD vs scalar mode</figcaption></figure><p><strong>Loop Unrolling</strong></p><p>This technique exploits the fact that the data is a vector now and is being computed in a loop. In this loop to avoid the unnecessary increment instruction of counters and jumping in assembly code, the body of the loop could be unrolled as in expanded to reduce the number of increments required by the loop to finish computing the result of the vector.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DzUOgRismZmu-w2pzXld6A.png" /><figcaption>To reduce the number of loops the same loop is unrolled by 4</figcaption></figure><p>Loop unrolling is a compiler feature. Manually doing loop unrolling will result in less readable code. One downside of using loop unrolling is that the binary size will increase as the number of instructions will increase. Often loop unrolling is applied on nested loops where the inner loop is constant iteration size.</p><p><strong>Loop pipelining</strong></p><p>It allows executing the operations inside the loop in a concurrent manner. This results in the interleaving of instructions in the loop. The major benefit of this pipelining is that it reduces CPU stall.</p><p><strong>What is a CPU stall?</strong> Well, each instruction goes through a cycle of Fetch, Decode, Execute, Write. Often the fetch stage could result in a cache miss due which a stall is experienced in the execution cycle. Running operations in loop pipelining manner circumvent this stall as operations are running concurrently and unabated by other the fetch operations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/475/0*q3CqpSBg2UDwXL-U.png" /><figcaption>The ideal case of loop pipelining</figcaption></figure><h3>Summary</h3><blockquote>Vectorization is running instructions on a vector (batch) of data.</blockquote><blockquote>Vectorization does not solve the instruction cache issue by removing the unnecessary branching and abstractions from query execution code rather it just batches the data (vector) to reduce the number of instruction cache misses.</blockquote><blockquote>Using columnar formats with vectorization yields excellent performance and results in increased data availability in the cache, loading only required columns in the main memory and faster execution of an operator on a vector in a tight loop.</blockquote><blockquote>Vectorization exploits SIMD (Single Instruction Multiple Data) instructions which allows executing a single instruction on multiple data.</blockquote><blockquote>Further performance gains could be achieved using techniques such as loop unrolling and loop pipelining with vectorization.</blockquote><p>This is it, for now, we will discuss compiled queries in <a href="https://medium.com/@tilakpatidar/vectorized-and-compiled-queries-part-3-807d71ec31a5">part 3</a>.</p><p>If you have anything to add or you spotted an error in my post, do not hesitate to contact me, I’ll be happy to make changes and learn a few things.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cd0d91fa189f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Vectorized and Compiled Queries — Part 1]]></title>
            <link>https://medium.com/@tilakpatidar/vectorized-and-compiled-queries-part-1-37794c3860cc?source=rss-f3db0362cb75------2</link>
            <guid isPermaLink="false">https://medium.com/p/37794c3860cc</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[bds]]></category>
            <category><![CDATA[software]]></category>
            <category><![CDATA[computer-science]]></category>
            <category><![CDATA[database]]></category>
            <dc:creator><![CDATA[Tilak Patidar]]></dc:creator>
            <pubDate>Sun, 07 Oct 2018 04:25:26 GMT</pubDate>
            <atom:updated>2018-10-07T11:50:47.929Z</atom:updated>
            <content:encoded><![CDATA[<h3>Vectorized and Compiled Queries — Part 1</h3><p>While assessing databases or data computation engines you would have stumbled upon terms such as vectorization support, compile-time queries, or code generation suited according to the query. But what do they mean? Do they mean something? Let’s find out.</p><p>Before, going into what let’s ask the why first. Why vectorized or compiled queries are needed? To answer that let’s see who was the precursor of vectorized and compiled queries. It would be the volcano iterator model . . .</p><h3>Volcano iterator model</h3><p>Imagine a volcano it is like a cone or at least it is drawn like that by most kids including me :). The below one is definitely not drawn by a kid.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/0*_5TR68HmNq9gT1mb" /></figure><p>As the shape of the volcano suggests lava(data) is too much at the base and very less of it is fuming out of it.</p><p>Here is the non-fun wiki definition:</p><blockquote>The Volcano model (originally known as the Iterator Model) is the ‘classical’ evaluation strategy of an analytical DBMS query: Each relational-algebraic operator produces a <em>tuple stream</em>, and a consumer can <em>iterate </em>over its input streams. The tuple stream interface is essentially: ‘open’, ‘next’ and ‘close’; all operators offer the same interface, and the implementation is opaque. Each ‘next’ call produces a new tuple from the stream, if one is available. To obtain the query output, one “next-next-next”s on the final RA operator; that one will in turn use “next”s on its inputs to pull tuples allowing it to produce output tuples, etc. Some “next”s will take an extremely long time, since many “next”s on previous operators will be required before they emit any output. Example: SELECT max(v) FROM t; may need to go trough all of t in order to find that maximum.</blockquote><p>In easier words for the brain:</p><p><em>It is a chain of iterators and data flows through them when the topmost iterator calls next() on the iterator below it. Which results in propagation of .next() calls till the bottom-most iterator is called. Each iterator might apply some predicate or other operators. And as visualized in form of volcano data might get reduced as it flows up the iterator chain.</em></p><h4><strong>What is wrong with it?</strong></h4><p>Well, at the first glimpse nothing until people realized that it causes a lot of <strong>instruction cache misses</strong>.</p><h4><strong>What is instruction cache misses?</strong></h4><p>What is this madness you just volunteered yourself into?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*dkWMKiAqWGX6aIJS" /><figcaption>Not a food chain just how the data is guzzled down to the register for operations</figcaption></figure><p>So, consider your database uses a volcano iterator model. Your data resides on the disk from where your disk manager reads it and writes it into main memory on the request of buffer manager. This piece of information from the main memory will then be loaded into cache memory (L1 and L2). From these caches, the information will be moved into registers where the operations would be performed. However, both the data and the assembly instructions have to be loaded into the cache memory.</p><p>So, you have the instructions and the data residing in the cache. But, what if you have too many instructions? Your query was big and so was the generated execution code which translated to more assembly instructions. And a general query execution code resulted in more branching thus more useless instructions in the cache. But, hang on a second.</p><p><strong>What do you mean by a general query execution code?</strong></p><p>Imagine you are writing a code which supports all the kinds of queries and all data types. Writing such a code would require the right amount of abstractions and also a lot of if-then for handling various data types.</p><p><strong>Example: Writing a ‘greater than’ operator for your query execution engine.</strong></p><p>The greater than operator needs to know the data type. Date, strings, and ints will result in a different expectation of the behaviour and assembly instructions generated.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/459/0*C8BU_83-wvsH9awp.gif" /><figcaption>Remember the good ole’ switch statement? It causes loading more instructions than necessary.</figcaption></figure><p>Branching statements, in general, leads to loading instructions that are not needed because those instructions will never be executed if the condition does not satisfy.</p><blockquote>What if I don’t write a general code supporting everything? What if I write just enough code for my own query? What if my query execution engine generates just enough code for my own query?</blockquote><p>Those were the kind of questions that would have crossed the minds of the people who first pondered if <strong>compiled queries </strong>were possible.</p><p><em>And what if this kind of instruction cache misses occurs for each row?</em></p><p>As volcano model operates <strong>each row at a time</strong>.</p><p><strong>Well then we got two alternatives:</strong></p><ul><li><strong>Compiled Queries </strong>Generate instructions with less branching and specific to the query.</li><li><strong>Vectorized Queries </strong>Let the cache miss happen but not for each row, maybe for a batch of rows.</li></ul><p>So, now we have the idea of why compiled queries and vectorized queries were necessary. This is it, for now, we will discuss <a href="https://medium.com/@tilakpatidar/vectorized-and-compiled-queries-part-2-cd0d91fa189f">vectorized queries in part 2</a> followed by compiled queries in <a href="https://medium.com/@tilakpatidar/vectorized-and-compiled-queries-part-3-807d71ec31a5">part 3</a>.</p><p>If you have anything to add or you spotted an error in my post, do not hesitate to contact me, I’ll be happy to make changes and learn a few things.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=37794c3860cc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Streaming data from PostgreSQL to Kafka using Debezium]]></title>
            <link>https://medium.com/@tilakpatidar/streaming-data-from-postgresql-to-kafka-using-debezium-a14a2644906d?source=rss-f3db0362cb75------2</link>
            <guid isPermaLink="false">https://medium.com/p/a14a2644906d</guid>
            <category><![CDATA[debezium]]></category>
            <category><![CDATA[postgres]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[apache-kafka]]></category>
            <category><![CDATA[change-data-capture]]></category>
            <dc:creator><![CDATA[Tilak Patidar]]></dc:creator>
            <pubDate>Sat, 10 Mar 2018 11:06:08 GMT</pubDate>
            <atom:updated>2018-03-10T11:11:18.769Z</atom:updated>
            <content:encoded><![CDATA[<p>Using Change Data Capture (CDC) is a must in any application these days. No one wants to hear that the changes they made did not reflect in the analytics because the nightly or hourly sync job has not pulled or pushed the data. The common problem is that there are a raft amount of web applications which are OLTP and are often backed by a relational database such as Oracle, PostgreSQL, MySQL etc.</p><p>Performing real-time data analytics on these database systems requires usage of big joins and aggregations which results in locks as these database systems are ACID complaint and provide good isolation levels. These locks may be held for a long duration which could affect the performance of the application for the live users.</p><p>Thus, it makes sense to stream data into other teams of your organisation which could perform analytics on it using spark jobs, hive queries or whatever is your preferred framework for big data madness.</p><p>The following technologies will be used to accomplish capturing data change.</p><ol><li><strong>Apache Kafka</strong> — It will be used to create a messaging topic which will store the data changes happening in the database. <br><a href="https://kafka.apache.org/">https://kafka.apache.org/</a></li><li><strong>Kafka Connect</strong> — It is a tool used for scalable and reliable data streaming between Apache Kafka and other systems. It is used to define <em>connectors</em> which are capable of moving data from entire databases into and out of Kafka. The list of available connectors is available <a href="https://www.confluent.io/product/connectors/">here</a>.</li><li><strong>Debezium</strong> — It is a tool used to utilise the best underlying mechanism provided by the database system to convert the WALs into a data stream. The data from the database is then streamed into Kafka using Kafka Connect API.<br><a href="https://github.com/debezium/debezium">https://github.com/debezium/debezium</a></li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/571/1*5nXxBTW8WAUN4pwIET01ew.png" /><figcaption>Capturing data from PostgreSQL into Apache Kafka topics.</figcaption></figure><p>Debezium uses logical decoding feature available in PostgreSQL to extract all persistent changes to the database in an easy to understand format which can be interpreted without detailed knowledge of the database’s internal state. More on logical decoding could be found <a href="https://www.postgresql.org/docs/current/static/logicaldecoding-explanation.html">here</a>.</p><p>Once, the changed data is available to Debezium in an easy to understand format it uses Kafka Connect API to register itself as one of the connectors of a data source. Debezium performs checkpointing and only reads committed data from the transaction log.</p><h3>Let us run an example</h3><p>To run this example you will require docker.</p><p><strong>Start a PostgreSQL instance</strong></p><pre>docker run — name postgres -p 5000:5432 debezium/postgres</pre><p><strong>Start a Zookeeper instance</strong></p><pre>docker run -it — name zookeeper -p 2181:2181 -p 2888:2888 -p 3888:3888 debezium/zookeeper</pre><p><strong>Start a Kafka instance</strong></p><pre>docker run -it — name kafka -p 9092:9092 — link zookeeper:zookeeper debezium/kafka</pre><p><strong>Start a Debezium instance</strong></p><pre>docker run -it — name connect -p 8083:8083 -e GROUP_ID=1 -e CONFIG_STORAGE_TOPIC=my-connect-configs -e OFFSET_STORAGE_TOPIC=my-connect-offsets -e ADVERTISED_HOST_NAME=$(echo $DOCKER_HOST | cut -f3 -d’/’ | cut -f1 -d’:’) — link zookeeper:zookeeper — link postgres:postgres — link kafka:kafka debezium/connect</pre><p><strong>Connect to PostgreSQL and create a database to monitor</strong></p><pre>psql -h localhost -p 5000 -U postgres<br>CREATE DATABASE inventory;<br>CREATE TABLE dumb_table(id SERIAL PRIMARY KEY, name VARCHAR);</pre><h3><strong>What we just did?</strong></h3><p>We started PostgreSQL database and bound its port to 5000 for our system. We also started zookeeper and which is used by Apache Kafka to store consumer offsets. At last, we started a debezium instance in which we linked our existing containers i.e postgres, kafka and zookeeper. The linking will help in communicating across the containers.</p><p>Our setup is ready we just now need to register a connector to Kafka Connect.</p><p><strong>Create connector using Kafka Connect</strong></p><pre><br>curl -X POST -H “Accept:application/json” -H “Content-Type:application/json” localhost:8083/connectors/ -d ‘<br>{<br> “name”: “inventory-connector”,<br> “config”: {<br> “connector.class”: “io.debezium.connector.postgresql.PostgresConnector”,<br> “tasks.max”: “1”,<br> “database.hostname”: “postgres”,<br> “database.port”: “5432”,<br> “database.user”: “postgres”,<br> “database.password”: “postgres”,<br> “database.dbname” : “inventory”,<br> “database.server.name”: “dbserver1”,<br> “database.whitelist”: “inventory”,<br> “database.history.kafka.bootstrap.servers”: “kafka:9092”,<br> “database.history.kafka.topic”: “schema-changes.inventory”<br> }<br>}’</pre><p><strong>Verify the Connector is created</strong></p><pre>curl -X GET -H “Accept:application/json” localhost:8083/connectors/inventory-connector</pre><p><strong>Start a Kafka Console consumer to watch changes</strong></p><pre>docker run -it — name watcher — rm — link zookeeper:zookeeper debezium/kafka watch-topic -a -k dbserver1.public.dumb_table</pre><p>This console consumer is used for the purpose of an example. You have to write a Kafka Consumer for your production systems.</p><h3><strong>Result</strong></h3><p>Now issue some SQL inserts, updates and deletes from PSQL CLI. You will see some JSON like output in the console consumer.</p><p>This JSON like output is, in fact, an Apache Avro format which is a RPC and data serialisation framework developed within Apache’s Hadoop project. It uses JSON for defining data types and protocols. The best feature of Avro format is schema evolution. More about Avro <a href="https://avro.apache.org/">here</a>.</p><p>If you use a JSON pretty tool on the output you will find two main keys in the JSON which are <em>schema </em>and<em> payload</em>.</p><p>The <em>schema</em> key contains the schema for the<em> row, txn record, source </em>etc. The <em>payload</em> key contains the data change. The <em>payload</em> key also contains the before and after for the row in case the row was updated or deleted. It also contains the operation type which is useful for knowing the event type that is inserted, updated or deleted.</p><p>Also, note that if you change the structure of the table and then perform some inserts, updates or deletes the schema printed in the console consumer would also change and evolve to accommodate the data.</p><p>I hope this small example would have helped you to understand the tools available for performing change data capture from old relational database systems. Thank You.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a14a2644906d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Evolution of Linux Containers]]></title>
            <link>https://medium.com/@tilakpatidar/evolution-of-linux-containers-660fcfdefa3e?source=rss-f3db0362cb75------2</link>
            <guid isPermaLink="false">https://medium.com/p/660fcfdefa3e</guid>
            <category><![CDATA[linux]]></category>
            <category><![CDATA[virtualization]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[lxc]]></category>
            <dc:creator><![CDATA[Tilak Patidar]]></dc:creator>
            <pubDate>Wed, 14 Jun 2017 11:43:04 GMT</pubDate>
            <atom:updated>2017-06-14T11:53:16.072Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jUSlQ-npZqIJDHqRaOw8jQ.png" /></figure><h4>Understanding the containerization landscape</h4><p>Containerization is not a new concept in computer science. Due to Docker zeitgeist, they might seem like a new technology. But, the inception of Linux containers can be traced back to earlier 1980’s when for the first time <a href="https://en.wikipedia.org/wiki/Chroot"><strong>chroot</strong></a> system call was introduced in UNIX. It brought with it the ability to change the root filesystem for a process and it’s children. This was the first glimpse of <a href="https://en.wikipedia.org/wiki/Operating-system-level_virtualization">operating-system-level virtualization</a>. However, the movement didn’t pace up because the need for virtualization and scalability was not realised then.</p><p>The major difference between <a href="https://en.wikipedia.org/wiki/Operating-system-level_virtualization">operating-system-level virtualization</a> and traditional <a href="https://en.wikipedia.org/wiki/Hypervisor">hypervisor-based virtualization</a> is that running another OS on top of the host and emulating hardware is not required by the former while the latter has to do both. More on containers vs hypervisors <a href="https://www.linux.com/news/containers-vs-hypervisors-choosing-best-virtualization-technology">here</a>.</p><p>For someone like me who is trying to understand the entire landscape of <a href="https://www.docker.com/">Docker</a> or similar containerization technologies. It is essential to know why things are the way they are. By, tracing the entire timeline of Linux containers and Docker landscape we will be able to discover how the features that are available today in containers evolved over the time. However, the timeline stretches to many years and we will only discuss those who had the most impact on the container’s present and future.</p><p><strong>1979 — chroot</strong><br>UNIX introduced chroot in 1979, <a href="https://en.wikipedia.org/wiki/Chroot"><strong>chroot</strong></a> is an operating system call with the ability to change the filesystem root for a process and it’s children. This was the first building block for Linux containers whose existence without chroot is just not possible. This was built with the purpose of providing isolated disk space for each process.</p><p><strong>2000 — FreeBSD Jails</strong><br>FreeBSD Jails is one of the earliest complete containerization frameworks. It provided more than just disk space isolation. It is one of the earliest to provide user and networking isolation and other process sandboxing features.</p><p><strong>2006 — Process Containers</strong><br>Process Containers was implemented at Google for resource management on a group of processes. The main focus of these process containers was to provide isolation and resource usage control. In 2007, Google renamed it to Control Groups and merged it to the Linux kernel. <a href="https://en.wikipedia.org/wiki/Cgroups"><strong>Cgroups</strong></a> since then is available in all Linux kernels and is widely used for resource management on a collection of processes.</p><p><strong>2008 — LXC</strong><br><a href="https://en.wikipedia.org/wiki/LXC"><strong>LXC</strong></a> aka Linux Containers was a major milestone in containerization history. It was released with a complete implementation of OS-level virtualization. Back then, people were already using <strong>VMs</strong> for providing scalability and isolation but they were troubled about the overhead of running another OS. Built on top of <a href="https://en.wikipedia.org/wiki/Cgroups"><strong>cgroups</strong></a> and <a href="https://en.wikipedia.org/wiki/Linux_namespaces"><strong>Linux namespaces</strong></a> and language bindings available for many popular high-level programming languages it seemed like a new direction in containerization movement. A brief tutorial on LXC containers can be found <a href="http://blog.scottlowe.org/2013/11/25/a-brief-introduction-to-linux-containers-with-lxc/">here</a>.</p><h4><strong>LXD</strong></h4><p>It isn’t a complete rewrite of LXC. It uses LXC underneath. It is a container “hypervisor”. It is composed by a daemon, the command-line interface(LXC) and an OpenStack plugin. It was developed to provide more flexibility and features on top of LXC. The project is backed by Canonical and more info about it can be found <a href="https://linuxcontainers.org/lxd/introduction/">here</a>.</p><p><strong>2013 — LMCTFY</strong><br><a href="https://github.com/google/lmctfy"><strong>lmctfy</strong></a> stands for <em>“Let Me Contain That For You”</em>. It was started by Google to provide a container stack for applications running on Linux which has guaranteed performance, near zero overhead. Now, Google has decided to contribute core <a href="https://github.com/google/lmctfy"><strong>lmctfy</strong></a> concepts and abstractions to <a href="https://github.com/docker/libcontainer"><strong>libcontainer</strong></a>. As a result, now no active development is done in <a href="https://github.com/google/lmctfy"><strong>LMCTFY</strong></a>.<br>The <strong>libcontainer</strong> project was initially started by Docker and now it has been moved to <a href="https://www.opencontainers.org/"><strong>Open Container Foundation</strong></a>.</p><p><strong>2013 — Docker</strong><br><a href="https://www.docker.com/"><strong>Docker</strong></a> is currently the most popular and widely used container management system. Docker used <strong>LXC</strong> at the initial stages and later replaced <strong>LXC</strong> with its own library called <a href="https://github.com/docker/libcontainer"><strong>libcontainer</strong></a>. However, <a href="https://github.com/docker/libcontainer"><strong>libcontainer</strong></a> was also moved to <a href="https://github.com/opencontainers/runc"><strong>runC</strong></a> which was docker’s implementation of containers on top of <a href="https://www.opencontainers.org/"><strong>OCI</strong></a> specification. Docker was first to introduce a complete ecosystem for containers which includes an efficient, layered container image model, a global and local container registries, a CLI, etc. It also took an initiative to implement a container cluster management solution called <a href="https://docs.docker.com/engine/swarm/"><strong>Docker Swarm</strong></a>. The Docker ecosystem has grown into a collection of open source projects used for plumbing the infrastructure. The entire list of such projects is available <a href="https://www.docker.com/technologies/overview">here</a>.</p><p><strong>2014 — rkt(pronounced “rock-it”)</strong><br><a href="https://github.com/rkt/rkt"><strong>Rkt</strong></a> is started by <a href="https://coreos.com/"><strong>CoreOS</strong></a> for fixing some of the major security drawbacks they found in Docker. This project has always been a strong proponent of security for containers. Docker still has some security vulnerabilities but they are fixing them by each version. For example, earlier all Docker processes ran as root user but it was fixed in version 1.10. An excellent post comparing rkt and other container technologies can be found <a href="https://coreos.com/rkt/docs/latest/rkt-vs-other-projects.html">here</a>.</p><p><strong>2016 — Windows Containers</strong><br>Before, Microsoft’s initiative to add container support running Docker containers on windows was not possible. It was made possible by running a virtual machine on Windows which ran the containers. However, due to <a href="https://www.docker.com/microsoft"><strong>Windows containers</strong></a> now it is entirely possible to run Docker containers natively on Windows.</p><p>So, far we have seen how Linux containers have evolved over the years. However, the Linux container landscape is growing at a tremendous pace under the flagship of Docker.</p><p>The Docker project has always tried to break their monolithic codebases into small components. They have always abstracted components out of the Docker project into smaller open source projects. Major projects that came out of the Docker project is discussed below.</p><ol><li><a href="https://docs.docker.com/engine/"><strong>Docker Engine</strong></a> is used to create and run Docker containers.</li><li><a href="https://docs.docker.com/compose/"><strong>Docker Compose</strong></a> is used to define multi-container applications using a simple YML based configuration file.</li><li><a href="https://docs.docker.com/machine/"><strong>Docker Machine</strong></a> is used for easier docker provisioning.</li><li><a href="https://docs.docker.com/engine/swarm/"><strong>Docker Swarm</strong></a> is a native Docker clustering system.</li><li><a href="https://docs.docker.com/registry/"><strong>Docker Registry</strong></a> is an Open Source docker image distribution hub.</li><li>Docker initially was built on top of <strong>LXC</strong> containers. Docker started building <a href="https://github.com/docker/libcontainer"><strong>libcontainer</strong></a>. Developers from <a href="https://github.com/google/lmctfy"><strong>LMCTFY</strong></a> also started contributing to this project and <a href="https://github.com/google/lmctfy"><strong>LMCTFY</strong></a> was discontinued.</li><li><a href="https://www.opencontainers.org/"><strong>Open Container Initiative</strong></a> was started under the flagship of Docker. It provided <a href="https://github.com/opencontainers/runtime-spec"><strong>runtime-spec</strong></a> and <a href="https://github.com/opencontainers/image-spec"><strong>image-spec</strong></a> which provides specifications for the way the application containers are built. Docker donated the OCI specification to the Linux Foundation in 2015.</li><li><a href="https://github.com/docker/libcontainer"><strong>Libcontainer</strong></a> was moved to <a href="https://github.com/opencontainers/runc"><strong>runC</strong></a>. RunC is Docker’s effort to build containers complying with OCI specification. RunC is a CLI tool for spawning and running containers according to the OCI specification.</li><li><a href="https://containerd.io/"><strong>Containerd</strong></a> is used as an executor for containers that comply with OCI specifications. In Docker, it acts as a daemon to control runC containers. The relationship of containerd with other projects is summarized succinctly <a href="https://containerd.io/">here</a>.</li><li>Recently, added <a href="https://mobyproject.org/"><strong>Moby project</strong></a> is another example which reflects the amount of emphasis put on abstracting out components by Docker community. Moby project provides a framework to assemble specialized container systems without reinventing the wheel. It provides various standard components and a framework for assembling them into custom platforms.</li></ol><p>So, far we have discussed and traversed the containerization landscape. There are so many projects to cover but I tried to keep it simple and important. We saw how Linux containers evolved and how Docker is driving the containerization space by breaking its projects into different components. This provides an easier contribution to the containerization space and encourages people to build their own custom components.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=660fcfdefa3e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Hive docker image for running integration tests]]></title>
            <link>https://medium.com/@tilakpatidar/building-a-hive-docker-image-for-running-integration-tests-d26a8d058b4a?source=rss-f3db0362cb75------2</link>
            <guid isPermaLink="false">https://medium.com/p/d26a8d058b4a</guid>
            <category><![CDATA[docker-compose]]></category>
            <category><![CDATA[testing]]></category>
            <category><![CDATA[integration-testing]]></category>
            <category><![CDATA[hive]]></category>
            <category><![CDATA[docker]]></category>
            <dc:creator><![CDATA[Tilak Patidar]]></dc:creator>
            <pubDate>Fri, 26 May 2017 03:26:37 GMT</pubDate>
            <atom:updated>2017-05-26T03:26:37.992Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TyU4z6oGKrhnWivwrrFHGw.jpeg" /></figure><p>In our current project we needed a simple and efficient way to run our integration tests on our dev machines. Previously we were using a test environment to run our jars and check if everything goes right.</p><p>After, a little research we came up to a consensus to try out docker containers for installing Hive and other Hadoop related components. Cloudera provides a standalone docker image but it also had unwanted services for our integration tests. Also, we don’t need all the services at one time. For example, if we have to run tests against our Spark jobs we need HDFS, Spark, and if Hive related jobs are to be tested we need Hive, HDFS and Postgres. Docker compose gave us the capability to create separate docker containers as services which could be extended on the fly for the job to be tested.</p><p>However, the real use case of docker is not to run integration tests. Docker is used to ship your code with all the dependencies in a docker image. Our team is already using chef for provisioning and we needed a kind of VM for running our integration tests. And docker containers are better than using VMs.</p><p>In this post, I will explain how we used docker compose to extend services and run only the required portion of Cloudera services for running our integration tests. The source code is available at <a href="https://goo.gl/SyV7o6">https://goo.gl/SyV7o6</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/228/1*Rx1WIgFddlNZUoyt4P6qvQ.png" /><figcaption>Directory structure of our docker image</figcaption></figure><p>We have three docker images here, ubuntu acts as the base image followed by hadoop and hive_pg. Hive and Postgres are installed in same container due to configuration dependencies.</p><p>The docker-compose.yml file and ubuntu.yml are shared below.</p><pre><strong>version: </strong>&quot;2&quot;<br><br><strong>services:<br>    hadoop:<br>        extends:<br>            file: </strong>ubuntu/ubuntu.yml<br>            <strong>service: </strong>ubuntu<br>        <strong>build:<br>            context: </strong>./hadoop-2.6.0<br>            <strong>dockerfile: </strong>Dockerfile<br>        <strong>image: </strong>hadoop-2.6.0<br>        <strong>ports:<br>            </strong>- &quot;50010:50010&quot;<br>            - &quot;50020:50020&quot;<br>            - &quot;50070:50070&quot;<br>            - &quot;50075:50075&quot;<br>            - &quot;50090:50090&quot;<br>            - &quot;8020:8020&quot;<br>            - &quot;9000:9000&quot;<br>            - &quot;10020:10020&quot;<br>            - &quot;19888:19888&quot;<br>            - &quot;8030:8030&quot;<br>            - &quot;8031:8031&quot;<br>            - &quot;8032:8032&quot;<br>            - &quot;8033:8033&quot;<br>            - &quot;8040:8040&quot;<br>            - &quot;8042:8042&quot;<br>            - &quot;8088:8088&quot;<br>            - &quot;52023:22&quot;<br>    <strong>hive_pg:<br>        depends_on:<br>            </strong>- &quot;hadoop&quot;<br>        <strong>extends:<br>            file: </strong>ubuntu/ubuntu.yml<br>            <strong>service: </strong>ubuntu<br>        <strong>build:<br>            context: </strong>./hive_pg<br>            <strong>dockerfile: </strong>Dockerfile<br>        <strong>image: </strong>hive_pg<br>        <strong>ports:<br>            </strong>- &quot;10000:10000&quot;<br>            - &quot;10001:10001&quot;<br>            - &quot;10002:10002&quot;<br>            - &quot;10003:10003&quot;<br>            - &quot;9083:9083&quot;<br>            - &quot;50111:50111&quot;<br>            - &quot;52024:22&quot;<br>            - &quot;5432:5432&quot;</pre><p>This file declares two services hadoop and hive_pg both extending ubuntu service. Ubuntu service is the place where ssh config and required env variables are set. Also, it must be noticed that hive_pg depends on hadoop so to start hive_pg, hadoop container will be started first.</p><p>Docker compose file for base service ubuntu is shared below.</p><pre><strong>version: </strong>&#39;2&#39;<br><br><strong>services:<br>  ubuntu:<br>    env_file: </strong>./base.env<br>    <strong>image: </strong>ubuntu<br>    <strong>build:<br>        context: </strong>./<br>        <strong>dockerfile: </strong>Dockerfile<br>        <strong>args:<br>            </strong>- JAVA_HOME=/usr/lib/jvm/java-8-oracle<br>            - HIVE_HOME=/usr/local/hive<br>            - HIVE_CONF=/usr/local/hive/conf<br>            - HIVE_VERSION=1.1.0<br>            - HADOOP_VERSION=2.6.0<br>            - CDH_VERSION=5<br>            - CDH_EXACT_VERSION=5.11.0<br>            - POSTGRES_VERSION=9.5<br>            - POSTGRESQL_MAIN=/var/lib/postgresql/9.5/main/<br>            - POSTGRESQL_CONFIG_FILE=/var/lib/postgresql/9.5/main/postgresql.conf<br>            - POSTGRESQL_BIN=/usr/lib/postgresql/9.5/bin/postgres<br>            - PGPASSWORD=hive<br>            - HADOOP_HOME=/usr/local/hadoop<br>            - HADOOP_PREFIX=/usr/local/hadoop<br>            - HADOOP_COMMON_HOME=/usr/local/hadoop<br>            - HADOOP_HDFS_HOME=/usr/local/hadoop<br>            - HADOOP_MAPRED_HOME=/usr/local/hadoop<br>            - HADOOP_YARN_HOME=/usr/local/hadoop<br>            - HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop</pre><p>The key thing here to note is that we are passing env variables for the image build process from args key in build. The key env_file accepts a file path which contains env variables to be set once container boots up. The build time env variables passed should also be declared in the Dockerfile using ARG. A short example is given below.</p><pre><em>FROM postgres<br><br></em>#expected build args<br><em>ARG JAVA_HOME<br>ARG HIVE_HOME<br>ARG HIVE_CONF</em></pre><p>Using docker-compose a service can be started with all it’s dependencies.</p><pre>➜  <em>docker-compose up hive</em></pre><h3>Key Learnings</h3><blockquote>Containers are immutable, it can be stopped, destroyed, or replaced. For that reason, if you need to store data, do it in a volume.</blockquote><blockquote>Waiting for image to build after every small change can be frustrating. Always break your Dockerfile into services. It improves build time as only changed service needs to be rebuild if changes are made.</blockquote><blockquote>Download all big build tars beforehand instead of using wget or curl in the Dockerfile. And, then use ADD which will copy and extract the archive for you(not all archive formats are supported). This decreases time to build image when changes are made.</blockquote><blockquote>To expose ports to host you need to build an image with the EXPOSE directive. Also, you need to use -p arg while doing docker run or you can specify port mappings in docker-compose.yml as explained above.</blockquote><blockquote>If you have a SSH into a docker container your env variable will not be present. You have to append you env vars into .bashrc from your container entrypoint script.</blockquote><blockquote>Containers have their own IP address which could change on starting and stoping the container. If your application requires specific hostname and port use environment variables to set them.</blockquote><p>Checkout out the source code at <a href="https://goo.gl/SyV7o6">https://goo.gl/SyV7o6</a> for further digging. I hope this post enables you to leverage docker for running your Hive integration tests. Thank You for reading.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d26a8d058b4a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Handling exceptions in scala]]></title>
            <link>https://medium.com/@tilakpatidar/handling-exceptions-in-scala-dc8a8160eb64?source=rss-f3db0362cb75------2</link>
            <guid isPermaLink="false">https://medium.com/p/dc8a8160eb64</guid>
            <category><![CDATA[scala]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Tilak Patidar]]></dc:creator>
            <pubDate>Wed, 24 May 2017 03:17:26 GMT</pubDate>
            <atom:updated>2017-05-24T03:17:26.826Z</atom:updated>
            <content:encoded><![CDATA[<p>As scala runs on JVM it also inherits the bad language choices such as exceptions from java. Hate it or love it we know exceptions are here to stay. They have become an integral part of many languages and frameworks.</p><blockquote>When data is missing, pass null you must. And when something goes wrong, throw exceptions you must.</blockquote><p>However, this violates the referential integrity of a function which is a fundamental concept of functional programming paradigm. To deal with this scala opted for declaring all exceptions as unchecked. Yes, all exceptions are unchecked in scala i.e no throws like syntax. Making all exceptions unchecked delegates the responsibility to the programmer to wrap these exceptions into some object. Scala discourages the use of nulls and exceptions. To handle these situations some alternatives provided are: Option, Either, Try.</p><h3>Throwing and catching exceptions</h3><p>Below is the basic try/catch block similar to java. However, here the catch block is used as a matcher to declare cases for various types of exceptions.</p><p>The default case is _ whereas NonFatal is commonly used for all exceptions other than VirtualMachineError<em>, </em>OutOfMemoryError<em> </em>and other fatal errors.</p><pre><strong>import </strong>scala.util.control.<strong>NonFatal<br><br>object BasicExceptionDemo extends </strong>App {<br><br>  <strong>override def </strong>main(args: <strong>Array</strong>[String]): Unit = {<br>    <strong>lazy val </strong>zero = 0<br>    <strong>try </strong>{<br>      <strong>val </strong>result = zero / zero<br>    } <strong>catch </strong>{<br>      <strong>case </strong>e: <strong>ArithmeticException </strong>=&gt; <em>println</em>(&quot;caught arithmetic exception&quot;)<br>      <strong>case </strong><em>NonFatal</em>(t)            =&gt; <em>println</em>(&quot;non fatal caught&quot;)<br>      <strong>case </strong>_                      =&gt; <em>println</em>(&quot;something really weird happened&quot;)<br>    } <strong>finally </strong>{<br>      <em>println</em>(&quot;performing cleanup&quot;)<br>    }<br>  }<br>}</pre><h4>Options</h4><p>Options are super useful as a wrapper around null/None or any other default value.<em> </em>However, they can also be used to handle exceptions. The resulted value from an operation is provided if it ran successfully otherwise a None is returned.<em> </em>This looks decent enough functional approach to handle exceptions but, None does not convey what exceptional situation happened.</p><pre><strong>object ExceptionsUsingOption extends </strong>App {<br><br>  <strong>def </strong>divide(x: Int, y: Int): <strong>Option</strong>[Int] =<br>    <strong>try </strong>{<br>      <em>Option</em>[Int](x / y)<br>    } <strong>catch </strong>{<br>      <strong>case </strong>_ =&gt; <strong>None<br>    </strong>}<br><br>  <strong>override def </strong>main(args: <strong>Array</strong>[String]): Unit = {<br>    <strong>val </strong>result: <strong>Option</strong>[Int] = <em>divide</em>(42, 0)<br>    <em>println</em>(result.getOrElse(0))<br>  }<br>}</pre><h4><strong>Either</strong></h4><p>Either[L,R] has two implementations, Left[L] and Right[R]. Left[L] is used to store exceptional value, and Right[R] is used for valid results. Once again left is biased.</p><pre><strong>import </strong>scala.util.Try<br><br><strong>case class FailureReason</strong>(msg: String)<br><br><strong>object ExceptionsUsingEither extends </strong>App {<br><br>  <em>//By convention failure values are left in Either<br>  </em><strong>def </strong>divide(a: Int, b: Int): Either[<strong>FailureReason</strong>, Int] =<br>    <em>//Try(a / b).toEither<br>    </em><strong>try </strong>{<br>      <em>Right</em>(a / b)<br>    } <strong>catch </strong>{<br>      <strong>case </strong>_ =&gt; <em>Left</em>(FailureReason(&quot;Cannot divide by zero&quot;))<br>    }<br><br>  <strong>override def </strong>main(args: <strong>Array</strong>[String]): Unit =<br>    <em>divide</em>(5, 0).fold(<br>      error =&gt; <em>println</em>(&quot;failed&quot;),<br>      success =&gt; <em>println</em>(s&quot;<strong>$</strong>success&quot;)<br>    )<br>}</pre><p>However, Either solves the problem of wrapping the valid result and exception data into one data type but Either is an abstract concept not specialized for handling exceptions.</p><h4>Try</h4><p>Working with Try is similar to working with an Either. Try has two subclasses Success and Failure both describing the nature of the outcome of the operation. In Scala functions prone to exceptions return a Try value.</p><pre><strong>import </strong>java.io.{<strong>File</strong>, <strong>IOException</strong>}<br><br><strong>import </strong>scala.util.Try<br><br><strong>object BasicTryValueDemo extends </strong>App {<br><br>  <strong>def </strong>filePath(path: String): <strong>Try</strong>[<strong>File</strong>] =<br>    <em>// creating an File object with null throws NullPointerException which is an unchecked exception<br>    //Try[A] always returns either Success[A] or Failure[A]<br>    Try</em>(<strong>new </strong>File(path))<br><br>  <strong>override def </strong>main(args: <strong>Array</strong>[String]): Unit = {<br><br>    <em>//using a getOrElse to provide a safe value if function returns Failure[File]<br>    </em><strong>val </strong>file = <em>filePath</em>(<strong>null</strong>).getOrElse(<strong>new </strong>File(&quot;/&quot;))<br>    <em>println</em>(file.toString)<br><br>    <em>//assert failure<br>    assert</em>(<em>filePath</em>(<strong>null</strong>).isFailure)<br><br>  }<br><br>}</pre><p>In the above example, new File(path) which could throw an exception is called in Try. Any expression passed to Try is <strong><em>lazily evaluated</em></strong> and <strong>e<em>xecuted in the apply method</em></strong> of Try where all the NonFatal exceptions are caught. If an exception is raised a Failure is returned which wraps the handled exception. Else, a Success is returned wrapping the result of an operation.</p><p>Here, the key point to note is that Try catches only NonFatal exceptions. Thus, any JVM related exceptions are unhandled which broadly covers all the unchecked exceptions from java.</p><h4>Benefits of using Try</h4><p>Using Try provides you with the ability to use functions such as map, filter etc which are available in collections. Mapping a Try[A] which has a Success[A] to a Try[B] results in a Success[B]. If map is performed on Failure[A], the resulting Try[B] will be a Failure[B] but, containing the same exception as the Failure[A]</p><p>Also, using recover and recoverWith are useful to handle failure situations. Recover requires a partial function which matches an exception and returns the default value. Whereas, recoverWith is used to match an exception and substitute it with another Failure.</p><p>Using recover to provide a default value.</p><pre><strong>import </strong>java.io.<strong>File<br><br>import </strong>scala.util.Try<br><br><strong>object RecoverDemo extends </strong>App {<br><br>  <strong>def </strong>filePath(path: String): <strong>Try</strong>[<strong>File</strong>] =<br>    <em>//Creating new File with null results in NullPointerException which is an unchecked exception<br>    //Try[A] always returns either Success[A] or Failure[A]<br>    Try</em>(<strong>new </strong>File(path))<br><br>  <strong>override def </strong>main(args: <strong>Array</strong>[String]): Unit = {<br>    <strong>val </strong>filePaths = <em>List</em>(&quot;/&quot;, &quot;/home&quot;, &quot;/tmp&quot;, <strong>null</strong>)<br>    <strong>val </strong>fileObjs =<br>      filePaths<br>      <em>//Create List[Try[File]]<br>        </em>.map(<em>filePath</em>)<br>        <em>//Iterate over List[Try[File]]<br>        </em>.map(_.recover({<br>          <em>//recover for each Failure[File]<br>          </em><strong>case </strong>e: NullPointerException =&gt; <em>filePath</em>(&quot;/&quot;).get<br>        }))<br>    <em>println</em>(fileObjs)<br>  }<br>}</pre><p>Using recoverWith to catch and substitute the Failure value</p><pre><strong>import </strong>java.io.<strong>File<br><br>import </strong>scala.util.{Failure, Try}<br><br><strong>object RecoverWithDemo extends </strong>App {<br><br>  <strong>def </strong>filePath(path: String): <strong>Try</strong>[<strong>File</strong>] =<br>    <em>//Creating new File with null results in NullPointerException which is an unchecked exception<br>    //Try[A] always returns either Success[A] or Failure[A]<br>    Try</em>(<strong>new </strong>File(path))<br><br>  <strong>override def </strong>main(args: <strong>Array</strong>[String]): Unit = {<br>    <strong>val </strong>filePaths = <em>List</em>(&quot;/&quot;, &quot;/home&quot;, &quot;/tmp&quot;, <strong>null</strong>)<br>    <strong>val </strong>fileObjs =<br>      filePaths<br>      <em>//Create List[Try[File]]<br>        </em>.map(<em>filePath</em>)<br>        <em>//Iterate over List[Try[File]]<br>        </em>.map(_.recoverWith({<br>          <em>//recover with allows swapping your failure with another failure<br>          </em><strong>case </strong>e: NullPointerException =&gt; <em>Failure</em>(<strong>new </strong>IllegalArgumentException)<br>        }))<br>    <em>println</em>(fileObjs)<br>  }<br>}</pre><h4>Handling a List of Try</h4><p>Recently, while refactoring we encountered a case where a batch job was returning List[Try] and based on this we had to provide statuses such as Failure, PartialFailure and Success. To do this we used cats library, it provides powerful abstractions for functional programming in scala. Using cats library we converted our List[Try] into Try[List] .</p><pre><strong>import </strong>cats.<strong>implicits</strong>._<br><br><strong>import </strong>scala.util.{Failure, Success, Try}<br><br><strong>object CatsDemo extends </strong>App {<br><br>  <strong>override def </strong>main(args: <strong>Array</strong>[String]): Unit = {<br><br>    <em>// We have a List[Try[String]] but we want a Try[List[String]].<br>    // we want it to be a Failure if any of the Try was a Failure.<br>    </em><strong>val </strong>listOfTries: List[<strong>Try</strong>[String]] = <em>List</em>(<em>Failure</em>(<strong>new   </strong>Exception), <em>Success</em>(&quot;b&quot;), <em>Success</em>(&quot;c&quot;))<br>    <strong>val </strong>tryOfList: <strong>Try</strong>[List[String]] = listOfTries.sequence<em><br>    println</em>(tryOfList) <em>//Failure(new Exception)</em></pre><pre><strong>val </strong>listOfTriesOther: List[<strong>Try</strong>[String]] = <em>List</em>(<em>Success</em>(&quot;a&quot;), <em>Success</em>(&quot;b&quot;), <em>Success</em>(&quot;c&quot;))<br>    <strong>val </strong>tryOfListOther:<strong>Try</strong>[List[String]] = listOfTriesOther.sequence<em><br>    println</em>(listOfTriesOther) <em>//Success(List(a, b, c))</em></pre><pre>  }<br>}</pre><h3>Conclusion</h3><p>Exceptions are considered by most people as an anti-pattern. They result in ambiguous code flows, long method signatures, uncomposable (except by reverting to throws Exception). However, Scala handles exceptions in an exceptional way. By encapsulating computation result and exception if thrown, it maintains referential integrity. Availability of Option/Either/Try for exception handling provides a way to chain operations based on the computation’s Success or Failure.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dc8a8160eb64" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>