<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Abhilasha Sharma Gupta on Medium]]></title>
        <description><![CDATA[Stories by Abhilasha Sharma Gupta on Medium]]></description>
        <link>https://medium.com/@abhilashasharmagupta?source=rss-4a44aeb89b6c------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*fLrwY-XLjJuZL82F.</url>
            <title>Stories by Abhilasha Sharma Gupta on Medium</title>
            <link>https://medium.com/@abhilashasharmagupta?source=rss-4a44aeb89b6c------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 00:51:57 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@abhilashasharmagupta/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Apache Oozie: An introduction]]></title>
            <link>https://medium.com/@abhilashasharmagupta/apache-oozie-an-introduction-d0f899e2a99b?source=rss-4a44aeb89b6c------2</link>
            <guid isPermaLink="false">https://medium.com/p/d0f899e2a99b</guid>
            <category><![CDATA[apache]]></category>
            <category><![CDATA[oozie]]></category>
            <dc:creator><![CDATA[Abhilasha Sharma Gupta]]></dc:creator>
            <pubDate>Thu, 28 Jun 2018 00:33:37 GMT</pubDate>
            <atom:updated>2018-06-28T00:34:34.671Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>The Introduction<br></strong>When my manager told me I needed to implement a scheduled task to handle the “dreaded” GDPR shopper delete requests based on snapshots in hadoop, I was lost.</p><p>Where do I begin ? What is involved with this process ?</p><p>So, I started where most programmers begin and searched through the github repo for the company for anything that would give me an idea. In my search, I stumbled upon the magic word, OOZIE.</p><p>According to <a href="https://www.infoq.com/articles/introductionOozie">InfoQ</a>, Oozie is a Java Web-Application that runs in a Java servlet-container — Tomcat and uses a database to store:<br>Workflow definitions currently running workflow instances, including instance states and variables</p><p>Oozie workflow is a collection of actions (i.e. Hadoop Map/Reduce jobs, Pig jobs) arranged in a control dependency DAG (Direct Acyclic Graph), specifying a sequence of actions execution. This graph is specified in hPDL (a XML Process Definition Language).</p><p>Sounds promising !! I can define a Hadoop job via this mythical language. COOL.</p><h3>Creating Simple Oozie workflow</h3><p>To show Oozie usage, let us turn to a simple example. Lets say I have a hive query I want to execute to copy data from one table to another. To automate this process, lets create a simple Oozie Workflow</p><p><strong>Here is my sample HIVE query:</strong></p><blockquote>Insert into summaryTable Select action, count(action) from sourceTable<br>where to_date(actiondate) = date_sub(current_date(), 1)<br>group by action</blockquote><p>So, I saved the hive query with the name “summary_generator.hive” and copied it to hdfs</p><blockquote>hadoop fs -copyFromLocal –f summary_generator.hive /user/agupta/oozie/simple</blockquote><p>Now, I created a simple workflow which calls the HIVE query</p><blockquote>vi simple_workflow.xml</blockquote><p>create the workflow-app top level element in XML</p><blockquote>&lt;?xml version=”1.0&quot; encoding=”UTF-8&quot;?&gt;<br>&lt;workflow-app name=”simple_workflow” xmlns=”uri:oozie:workflow:0.4&quot;&gt;<br>&lt;/workflow-app&gt;</blockquote><p>Add the credentials element within the workflow-app element to allow Oozie access to the Hive metastore.</p><blockquote>&lt;credentials&gt;<br>&lt;credential name=”hcat” type=”hcat”&gt;<br>&lt;property&gt;<br>&lt;name&gt;hcat.metastore.uri&lt;/name&gt;<br>&lt;value&gt;${hcatMetastoreUri}&lt;/value&gt;<br>&lt;/property&gt;<br>&lt;property&gt;<br>&lt;name&gt;hcat.metastore.principal&lt;/name&gt;<br>&lt;value&gt;${hcatMetastorePrincipal}&lt;/value&gt;<br>&lt;/property&gt;<br>&lt;/credential&gt;<br>&lt;/credentials&gt;</blockquote><p>Add start and end elements within the workflow-app element to define the start and end points of your workflow. The start element will have an attribute “name” that will define the first workflow action which is “run_hive”. The end element has a name attribute that will be “success”.</p><blockquote>&lt;start to=”run_hive” /&gt;<br>&lt;end name=”success” /&gt;</blockquote><p>Define the “run_hive” hive action within the workflow-app element between the start and end elements. This action will run the hive script created in step 4. The hive action will include a script element that will contain the HDFS path to your Hive script that was created in step 4</p><blockquote>&lt;action name=”run_hive” cred=”hcat”&gt;<br>&lt;hive xmlns=”uri:oozie:hive-action:0.5&quot;&gt;<br>&lt;job-tracker&gt;${jobTracker}&lt;/job-tracker&gt;<br>&lt;name-node&gt;${nameNode}&lt;/name-node&gt;<br>&lt;job-xml&gt;${oozieShareHiveConf}&lt;/job-xml&gt;<br>&lt;script&gt;${nodeName}/user/agupta/oozie/simple/summary_generator.hive&lt;/script&gt;<br>&lt;/hive&gt;<br>&lt;ok to=”success” /&gt;<br>&lt;error to=”kill” /&gt;<br>&lt;/action&gt;</blockquote><p>Add a “kill” action within the workflow-app element and just before end element. The hive action will be directed to this step if an error occurs</p><blockquote>&lt;kill name=”kill”&gt;<br>&lt;message&gt;Action failed, error message:[${wf:errorMessage(wf:lastErrorNode())}]&lt;/message&gt;<br>&lt;/kill&gt;</blockquote><p>Save the file and exit out. Completed file should look like this</p><blockquote>&lt;?xml version=”1.0&quot; encoding=”UTF-8&quot;?&gt;<br>&lt;workflow-app name=”simple_workflow” xmlns=”uri:oozie:workflow:0.4&quot;&gt;<br>&lt;credentials&gt;<br>&lt;credential name=”hcat” type=”hcat”&gt;<br>&lt;property&gt;<br>&lt;name&gt;hcat.metastore.uri&lt;/name&gt;<br>&lt;value&gt;${hcatMetastoreUri}&lt;/value&gt;<br>&lt;/property&gt;<br>&lt;property&gt;<br>&lt;name&gt;hcat.metastore.principal&lt;/name&gt;<br>&lt;value&gt;${hcatMetastorePrincipal}&lt;/value&gt;<br>&lt;/property&gt;<br>&lt;/credential&gt;<br>&lt;/credentials&gt;<br>&lt;start to=”run_hive” /&gt;<br>&lt;action name=”run_hive” cred=”hcat”&gt;<br>&lt;hive xmlns=”uri:oozie:hive-action:0.5&quot;&gt;<br>&lt;job-tracker&gt;${jobTracker}&lt;/job-tracker&gt;<br>&lt;name-node&gt;${nameNode}&lt;/name-node&gt;<br>&lt;job-xml&gt;${oozieShareHiveConf}&lt;/job-xml&gt;<br>&lt;script&gt;${nameNode}/user/agupta/oozie/simple/summary_generator.hive&lt;/script&gt;<br>&lt;/hive&gt;<br>&lt;ok to=”success” /&gt;<br>&lt;error to=”kill” /&gt;<br>&lt;/action&gt;<br>&lt;kill name=”kill”&gt;<br>&lt;message&gt;Action failed, error message:[${wf:errorMessage(wf:lastErrorNode())}]&lt;/message&gt;<br>&lt;/kill&gt;<br>&lt;end name=”success” /&gt;<br>&lt;/workflow-app&gt;</blockquote><p>now copy the workflow to the hdfs</p><blockquote>hadoop fs -copyFromLocal –f simple_workflow.xml /user/agupta/oozie/simple</blockquote><p>Run your workflow by executing the Oozie run command. A workflow ID will be returned. (It’s a single command)</p><blockquote>oozie job -config /etc/oozie/conf/job-default.properties -Doozie.wf.application.path=/user/agupta/oozie/simple/simple_workflow.xml -run</blockquote><p>8. Check the status of your workflow and review the results</p><blockquote>oozie job -info &lt;Workflow ID&gt;<br>oozie job -info &lt;Workflow ID&gt; -verbose</blockquote><p>Use the URL returned in the verbose command above to view the results of your workflow on the Hadoop job tracker web page. Use the stdout logs to see the results of the hive query.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d0f899e2a99b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Apache Oozie]]></title>
            <link>https://medium.com/series/apache-oozie-c5adef2905c3?source=rss-4a44aeb89b6c------2</link>
            <guid isPermaLink="false">https://medium.com/p/c5adef2905c3</guid>
            <dc:creator><![CDATA[Abhilasha Sharma Gupta]]></dc:creator>
            <pubDate>Thu, 28 Jun 2018 00:26:59 GMT</pubDate>
            <atom:updated>2018-06-28T00:26:59.440Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>The Introduction</strong></p><p>When my manager told me I needed to implement a scheduled task to handle the “dreaded” GDPR shopper delete requests based on snapshots in hadoop, I was lost.</p><p>Where do I begin ? What is involved with this process ?</p><p>So, I started where most programmers begin and searched through the github repo for the company for anything that would give me an idea. In my search, I stumbled upon the magic word, OOZIE.</p><p>According to <a href="https://www.infoq.com/articles/introductionOozie">InfoQ</a>, Oozie is a Java Web-Application that runs in a Java servlet-container — Tomcat and uses a database to store:</p><ul><li>Workflow definitions</li><li>Currently running workflow instances, including instance states and variables</li></ul><p>Oozie workflow is a collection of actions (i.e. Hadoop Map/Reduce jobs, Pig jobs) arranged in a control dependency DAG (Direct Acyclic Graph), specifying a sequence of actions execution. This graph is specified in hPDL (a XML Process Definition Language).</p><p>Sounds promising !! I can define a Hadoop job via this mythical language. COOL.</p><p>Creating Simple Oozie workflow</p><p>To show Oozie usage, let us turn to a simple example. Lets say I have a hive query I want to execute to copy data from one table to another. To automate this process, lets create a simple Oozie Workflow</p><p>Here is my sample HIVE query:</p><blockquote>Insert into summaryTable Select action, count(action) from sourceTable</blockquote><blockquote>where to_date(actiondate) = date_sub(current_date(), 1)</blockquote><blockquote>group by action</blockquote><p>Save the hive query with the name “summary_generator.hive” and copy it to hdfs</p><p>hadoop fs -copyFromLocal –f summary_generator.hive /user/agupta/oozie/simple</p><p>Now, I created a simple workflow which calls the HIVE query</p><p>Create the workflow XML file.</p><p>vi simple_workflow.xml</p><p>create the workflow-app top level element in XML</p><p>&lt;?<strong>xml</strong> version=&quot;1.0&quot; encoding=&quot;UTF-8&quot;?&gt;</p><p>&lt;<strong>workflow-app</strong> name=&quot;simple_workflow&quot; xmlns=&quot;uri:oozie:workflow:0.4&quot;&gt;</p><p>&lt;/<strong>workflow-app</strong>&gt;</p><p>Add the credentials element <strong>within the workflow-app element</strong> to allow Oozie access to the Hive metastore.</p><p>&lt;<strong>credentials</strong>&gt;</p><p>&lt;<strong>credential</strong> name=&quot;hcat&quot; type=&quot;hcat&quot;&gt;</p><p>&lt;<strong>property</strong>&gt;</p><p>&lt;<strong>name</strong>&gt;hcat.metastore.uri&lt;/<strong>name</strong>&gt;</p><p>&lt;<strong>value</strong>&gt;${hcatMetastoreUri}&lt;/<strong>value</strong>&gt;</p><p>&lt;/<strong>property</strong>&gt;</p><p>&lt;<strong>property</strong>&gt;</p><p>&lt;<strong>name</strong>&gt;hcat.metastore.principal&lt;/<strong>name</strong>&gt;</p><p>&lt;<strong>value</strong>&gt;${hcatMetastorePrincipal}&lt;/<strong>value</strong>&gt;</p><p>&lt;/<strong>property</strong>&gt;</p><p>&lt;/<strong>credential</strong>&gt;</p><p>&lt;/<strong>credentials</strong>&gt;</p><p>Add start and end elements within the workflow-app element to define the start and end points of your workflow. The start element will have an attribute “name” that will define the first workflow action which is “run_hive”. The end element has a name attribute that will be “success”.</p><p>&lt;<strong>start</strong> to=&quot;run_hive&quot; /&gt;</p><p>&lt;<strong>end</strong> name=&quot;success&quot; /&gt;</p><p>Define the “run_hive” hive action <strong>within the workflow-app element between the start and end elements</strong>. This action will run the hive script created in step 4. The hive action will include a script element that will contain the HDFS path to your Hive script that was created in step 4</p><p>&lt;<strong>action</strong> name=&quot;run_hive&quot; cred=&quot;hcat&quot;&gt;</p><p>&lt;<strong>hive</strong> xmlns=&quot;uri:oozie:hive-action:0.5&quot;&gt;</p><p>&lt;<strong>job-tracker</strong>&gt;${jobTracker}&lt;/<strong>job-tracker</strong>&gt;</p><p>&lt;<strong>name-node</strong>&gt;${nameNode}&lt;/<strong>name-node</strong>&gt;</p><p>&lt;<strong>job-xml</strong>&gt;${oozieShareHiveConf}&lt;/<strong>job-xml</strong>&gt;</p><p>&lt;<strong>script</strong>&gt;hdfs://production/user/&lt;<strong>user</strong> name&gt;/oozie/simple/action_count.hive&lt;/<strong>script</strong>&gt;</p><p>&lt;/<strong>hive</strong>&gt;</p><p>&lt;<strong>ok</strong> to=&quot;success&quot; /&gt;</p><p>&lt;<strong>error</strong> to=&quot;kill&quot; /&gt;</p><p>&lt;/<strong>action</strong>&gt;</p><p>Add a “kill” action within the workflow-app element and just before end element. The hive action will be directed to this step if an error occurs</p><p>&lt;<strong>kill</strong> name=&quot;kill&quot;&gt;</p><p>&lt;<strong>message</strong>&gt;Action failed, error message:[${wf:errorMessage(wf:lastErrorNode())}]&lt;/<strong>message</strong>&gt;</p><p>&lt;/<strong>kill</strong>&gt;</p><p>Save the file and exit out.</p><p>Completed file should look like this</p><p>&lt;?<strong>xml</strong> version=&quot;1.0&quot; encoding=&quot;UTF-8&quot;?&gt;</p><p>&lt;<strong>workflow-app</strong> name=&quot;simple_workflow&quot; xmlns=&quot;uri:oozie:workflow:0.4&quot;&gt;</p><p>&lt;<strong>credentials</strong>&gt;</p><p>&lt;<strong>credential</strong> name=&quot;hcat&quot; type=&quot;hcat&quot;&gt;</p><p>&lt;<strong>property</strong>&gt;</p><p>&lt;<strong>name</strong>&gt;hcat.metastore.uri&lt;/<strong>name</strong>&gt;</p><p>&lt;<strong>value</strong>&gt;${hcatMetastoreUri}&lt;/<strong>value</strong>&gt;</p><p>&lt;/<strong>property</strong>&gt;</p><p>&lt;<strong>property</strong>&gt;</p><p>&lt;<strong>name</strong>&gt;hcat.metastore.principal&lt;/<strong>name</strong>&gt;</p><p>&lt;<strong>value</strong>&gt;${hcatMetastorePrincipal}&lt;/<strong>value</strong>&gt;</p><p>&lt;/<strong>property</strong>&gt;</p><p>&lt;/<strong>credential</strong>&gt;</p><p>&lt;/<strong>credentials</strong>&gt;</p><p>&lt;<strong>start</strong> to=&quot;run_hive&quot; /&gt;</p><p>&lt;<strong>action</strong> name=&quot;run_hive&quot; cred=&quot;hcat&quot;&gt;</p><p>&lt;<strong>hive</strong> xmlns=&quot;uri:oozie:hive-action:0.5&quot;&gt;</p><p>&lt;<strong>job-tracker</strong>&gt;${jobTracker}&lt;/<strong>job-tracker</strong>&gt;</p><p>&lt;<strong>name-node</strong>&gt;${nameNode}&lt;/<strong>name-node</strong>&gt;</p><p>&lt;<strong>job-xml</strong>&gt;${oozieShareHiveConf}&lt;/<strong>job-xml</strong>&gt;</p><p>&lt;<strong>script</strong>&gt;${nameNode}/user/agupta/oozie/simple/summary_generator.hive&lt;/<strong>script</strong>&gt;</p><p>&lt;/<strong>hive</strong>&gt;</p><p>&lt;<strong>ok</strong> to=&quot;success&quot; /&gt;</p><p>&lt;<strong>error</strong> to=&quot;kill&quot; /&gt;</p><p>&lt;/<strong>action</strong>&gt;</p><p>&lt;<strong>kill</strong> name=&quot;kill&quot;&gt;</p><p>&lt;<strong>message</strong>&gt;Action failed, error message:[${wf:errorMessage(wf:lastErrorNode())}]&lt;/<strong>message</strong>&gt;</p><p>&lt;/<strong>kill</strong>&gt;</p><p>&lt;<strong>end</strong> name=&quot;success&quot; /&gt;</p><p>&lt;/<strong>workflow-app</strong>&gt;</p><p>now copy the workflow to the hdfs</p><p>hadoop fs -copyFromLocal –f simple_workflow.xml /user/agupta/oozie/simple</p><p>7. : Run your workflow by executing the Oozie run command. A workflow ID will be returned. <em>(It’s a single command)</em></p><p>oozie job -config /etc/oozie/conf/job-default.properties -Doozie.wf.application.path=/user/&lt;user name&gt;/oozie/simple/simple_workflow.xml -run</p><p>8. Check the status of your workflow and review the results</p><p>oozie job -info &lt;Workflow ID&gt;</p><p>oozie job -info &lt;Workflow ID&gt; -verbose</p><p>Use the URL returned in the verbose command above to view the results of your workflow on the Hadoop job tracker web page. Use the stdout logs to see the results of the hive query.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c5adef2905c3" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>