<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Naveen Balusu on Medium]]></title>
        <description><![CDATA[Stories by Naveen Balusu on Medium]]></description>
        <link>https://medium.com/@naveenbalusu?source=rss-350416aa22d3------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*ZkV2MJGhlqIN9JlC.</url>
            <title>Stories by Naveen Balusu on Medium</title>
            <link>https://medium.com/@naveenbalusu?source=rss-350416aa22d3------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 09:37:09 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@naveenbalusu/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Monitoring sensitive data using Anomaly Detection]]></title>
            <link>https://medium.com/@naveenbalusu/monitoring-sensitive-data-using-anomaly-detection-e3cb7793ee78?source=rss-350416aa22d3------2</link>
            <guid isPermaLink="false">https://medium.com/p/e3cb7793ee78</guid>
            <category><![CDATA[cloudera]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[hadoop]]></category>
            <dc:creator><![CDATA[Naveen Balusu]]></dc:creator>
            <pubDate>Tue, 05 Mar 2024 21:45:09 GMT</pubDate>
            <atom:updated>2024-03-05T21:45:09.303Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DFdCt3Dux3YJsawQE_qd3g.png" /></figure><p>Over the years, the size of the data being stored by the organizations has been growing exponentially. This has increased concerns over the data privacy. With the threat of data breaches, organizations face the challenge of protecting users sensitive information including Phone Numbers, Emails etc. Also, the organizations need to stay in compliance with International regulations. With the ever-increasing data regulations and their complexity, it would be a daunting task for the organizations.</p><p>Kogni is a unique tool which does not only redact sensitive information but also discovers all the places they are stored. It uses Machine Learning to leverage Data Security and ensures that the organizations stay in compliance with industry guidelines and international regulations like NIST, GDPR, PCI etc.</p><p>Due to sheer volumes of data being stored, it is very difficult for the organizations to monitor all of the data. Kogni discovers all the places where the sensitive information is stored. A very useful feature of it can be to monitor the places where the sensitive information is stored and thereby adding an extra layer of security to all the sensitive data.</p><p>In this blog, we present a method to monitor sensitive data. We compare different methods of achieving this and advantages of one over the other.</p><p><strong>Approach:</strong> One plausible way of detecting the security breaches can be monitoring all the user actions which are accessing sensitive data. This can be done by analyzing various Audit logs. By detecting anomalies in these logs, we can detect security breaches on private data.</p><p>There are several audit logs that are generated by various processes within the Hadoop environment. Some processes generating audit logs:</p><ol><li>Cloudera Navigator</li><li>HDFS Audit logs</li><li>MapReduce Audit logs</li><li>Hive Audit logs</li><li>Impala Audit logs</li><li>YARN Audit logs</li><li>HBase Audit logs</li><li>Sentry Audit logs</li></ol><p><strong>Cloudera Navigator Audit logs:</strong> Cloudera Navigator provides a unified auditing service for all the major services(see all the services supported by Cloudera Navigator Auditing service <a href="https://www.cloudera.com/documentation/enterprise/5-5-x/topics/cn_iu_reqts.html">here</a>). A big catch is that if we solely rely on it, it imposes an extra infrastructural requirement to run the Anomaly Detection. Therefore, we will not solely rely on it but we will use it to improve our pipeline.</p><p>Advantages:</p><ol><li>Easy to query using API interface.</li><li>Easy to filter related logs for a query.</li><li>Can query logs for a specific timeframe.</li><li>Well structured.</li></ol><p>Disadvantages:</p><ol><li>Does not support Spark SQL and Spark file access.</li><li>When using Impala queries with Hue, Hue does not terminate its connection after the query is executed. As a result, the related audit logs are missing.</li><li>HiveCLI is not supported.</li><li>Imposes some infrastructural needs.</li></ol><p>Sample Cloudera Navigator Audit log:</p><pre>{<br>      &quot;timestamp&quot; : &quot;2018-02-05T23:40:59.000Z&quot;,<br>      &quot;service&quot; : &quot;CD-IMPALA-SpjIIzkY&quot;,<br>      &quot;username&quot; : &quot;root&quot;,<br>      &quot;ipAddress&quot; : &quot;10.0.0.58&quot;,<br>      &quot;command&quot; : &quot;CREATE_TABLE&quot;,<br>      &quot;resource&quot; : &quot;test:test3&quot;,<br>      &quot;operationText&quot; : &quot;create table test3(id int, phone String)&quot;,<br>      &quot;allowed&quot; : true,<br>      &quot;serviceValues&quot; : {<br>        &quot;operation_text&quot; : &quot;create table test3(id int, phone String)&quot;,<br>        &quot;database_name&quot; : &quot;test&quot;,<br>        &quot;query_id&quot; : &quot;204880ee6fccc9e8:9502f66e00000000&quot;,<br>        &quot;object_type&quot; : &quot;TABLE&quot;,<br>        &quot;session_id&quot; : &quot;374a675274aa2524:eba8aa57e61ca69a&quot;,<br>        &quot;privilege&quot; : &quot;CREATE&quot;,<br>        &quot;table_name&quot; : &quot;test3&quot;,<br>        &quot;status&quot; : &quot;&quot;<br>      }<br>}</pre><p><strong>HDFS Audit logs:</strong> HDFS audit logs records every file access on HDFS. Many of the relational databases are stored in files on HDFS. So, every related audit log related to the files will be logged.</p><p>Advantages:</p><ol><li>No infrastructural needs.</li><li>You can have every log related to every file access.</li></ol><p>Disadvantages:</p><ol><li>For services like Impala, if the data being accessed is already cached, the related HDFS Audit logs don’t show up.</li><li>Difficult to access them.</li><li>Need to handle log rollover.</li><li>Difficult to filter.</li><li>Too verbose that for even a small MapReduce job, a lot of related HDFS audit logs are generated.</li><li>Difficult to find queries/processes/services that are resulting in those audit logs.</li></ol><p>Sample HDFS Audit log:</p><pre>{<br>     “allowed”: true,<br>     “serviceName”: “hdfs”,<br>     “username”: “centos”,<br>     “src”: “/user/centos”,<br>     “eventTime”: 1512538718000,<br>     “ipAddress”: “10.0.15.3”,<br>     “operation”: “listStatus”,<br>     “dest”: null,<br>     “permissions”: null,<br>     “impersonator”: null,<br>     “delegationTokenId”: null<br>}</pre><p><strong>Impala Audit logs:</strong></p><p>Impala audit logs are generated on every node where the Impala daemon is running.</p><p>Advantages:</p><ol><li>Well structured.</li><li>Gives the query being executed.</li><li>No infrastructural needs.</li></ol><p>Disadvantages:</p><ol><li>Need to monitor all the nodes where Impala daemon is running.</li><li>Since this only records for Impala related Audit logs, we need to find other similar auditing services for other services as well.</li></ol><p>Sample Impala Audit log:</p><pre>{  <br>   &quot;1518626522508&quot;:{  <br>      &quot;query_id&quot;:&quot;51488b4f90313d18:2860e81700000000&quot;,<br>      &quot;session_id&quot;:&quot;b4fd37e314dd6a6:c822b00988ff0086&quot;,<br>      &quot;start_time&quot;:&quot;2018-02-14 16:42:02.475475000&quot;,<br>      &quot;authorization_failure&quot;:false,<br>      &quot;status&quot;:&quot;&quot;,<br>      &quot;user&quot;:&quot;root&quot;,<br>      &quot;impersonator&quot;:null,<br>      &quot;statement_type&quot;:&quot;CREATE_TABLE&quot;,<br>      &quot;network_address&quot;:&quot;10.0.0.58:36646&quot;,<br>      &quot;sql_statement&quot;:&quot;create table test5 (name String)&quot;,<br>      &quot;catalog_objects&quot;:[  <br>         {  <br>            &quot;name&quot;:&quot;test.test5&quot;,<br>            &quot;object_type&quot;:&quot;TABLE&quot;,<br>            &quot;privilege&quot;:&quot;CREATE&quot;<br>         }<br>      ]<br>   }<br>}</pre><p><strong>Hive Audit logs:</strong></p><p>Hive uses Hive metastore for service logging. Hive Audit logs are stored with the class name <em>org.apache.hadoop.hive.metastore.HiveMetaStore.audit. </em>These needs to be filtered out before processing further.</p><p>Advantages:</p><ol><li>All the Hive Audit logs are stored in the Master Node.</li><li>No infrastructural needs.</li></ol><p>Disadvantages:</p><ol><li>No information about Query being executed.</li><li>Logs need to be parsed.</li></ol><p>Sample Hive Audit log:</p><pre>2017-12-05 05:10:56,239 INFO org.apache.hadoop.hive.metastore.HiveMetaStore.audit: [pool-5-thread-1]: ugi=hue/ip-10-0-15-3.us-west-2.compute.internal@CLAIRVOYANT ip=/10.0.15.3 cmd=get_functions: db=cloudera_manager_metastore_canary_test_db_hive_hivemetastore_6b039170b5491248c0b846d658d38e58 pat=*</pre><p><strong>Some factors affecting Audit logs:</strong> We considered some common factors that may effect the Audit logs.</p><p><strong>Kerberos:</strong> Kerberos is a network authentication protocol which gives certain access to certain people based on Principal they belong to by generating tokens valid for a certain time. <br>Here we perform a detailed comparison of how the resulting Audit logs are affected when Kerberos is installed and when not installed.<br>HDFS Audit logs on Non-kerborized cluster: <br>{<br> “allowed”: true,<br> “serviceName”: “hdfs”,<br> “username”: “centos”,<br> “src”: “/user/centos”,<br> “eventTime”: 1512538718000,<br> “ipAddress”: “10.0.15.3”,<br> “operation”: “listStatus”,<br> “dest”: null,<br> “permissions”: null,<br> “impersonator”: null,<br> “delegationTokenId”: null<br>}<br>HDFS Audit logs on Kerborized Cluster: <br>{<br> “allowed”: true,<br> “serviceName”: “hdfs”,<br> “username”: “centos@CLAIRVOYANT”,<br> “src”: “/user/centos”,<br> “eventTime”: 1512429082151,<br> “ipAddress”: “10.0.15.3”,<br> “operation”: “listStatus”,<br> “dest”: null,<br> “permissions”: null,<br> “impersonator”: null,<br> “delegationTokenId”: null<br>}<br>Hive on Non-Kerborized Cluster:<br>2017–11–30 17:33:50,912 INFO org.apache.hadoop.hive.metastore.HiveMetaStore.audit: [pool-5-thread-21]: ugi=centos ip=10.0.15.3 cmd=source:10.0.15.3 get_table : db=test tbl=movies<br>Hive on Kerborized Cluster:<br>2017–12–05 04:52:32,209 INFO org.apache.hadoop.hive.metastore.HiveMetaStore.audit: [pool-5-thread-21]: ugi=centos@CLAIRVOYANT ip=10.0.15.3 cmd=source:10.0.15.3 get_table : db=test tbl=movies<br>Impala Audit logs on Non-kerborized Cluster:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/996/1*WH0hSXhWo2kD2oVh_OAZSQ.png" /><figcaption>Impala Audit log</figcaption></figure><p>Effect of Kerberos: The User field is being affected after installing Kerberos. On Kerberos cluster, the username field also contains the principal to which the user belongs.</p><p><strong>Connection Types:</strong> Different types of connections for a service also affecting the Auditing logs. For example, Hive can be accessed through Hue, HiveCLI, and Beeline. Similarly other services.</p><p>HDFS Audit log accessed from Hadoop FsShell:</p><pre>{<br>   &quot;allowed&quot;: true,<br>   &quot;serviceName&quot;: &quot;hdfs&quot;,<br>   &quot;username&quot;: &quot;centos&quot;,<br>   &quot;src&quot;: &quot;/user/centos&quot;,<br>   &quot;eventTime&quot;: 1512538718000,<br>   &quot;ipAddress&quot;: &quot;10.0.15.3&quot;,<br>   &quot;operation&quot;: &quot;listStatus&quot;,<br>   &quot;dest&quot;: null,<br>   &quot;permissions&quot;: null,<br>   &quot;impersonator&quot;: null,<br>   &quot;delegationTokenId&quot;: null<br>}</pre><p>HDFS Audit log accessed from Hue:</p><pre>{<br>   &quot;allowed&quot;: true,<br>   &quot;serviceName&quot;: &quot;hdfs&quot;,<br>   &quot;username&quot;: &quot;centos&quot;,<br>   &quot;src&quot;: &quot;/user/centos&quot;,<br>   &quot;eventTime&quot;: 1512538602501,<br>   &quot;ipAddress&quot;: &quot;10.0.15.3&quot;,<br>   &quot;operation&quot;: &quot;listStatus&quot;,<br>   &quot;dest&quot;: null,<br>   &quot;permissions&quot;: null,<br>   &quot;impersonator&quot;: &quot;hue&quot;,<br>   &quot;delegationTokenId&quot;: null<br>}</pre><p>Effect of Connections: For many services, the impersonator field seems to be effected. Similarly, we have performed various experiments for JDBC connections etc.</p><p><strong>Analyzing and Comparing various Audit logs:</strong></p><p>As we discussed above, there are various processes within the Big Data environment generating Audit logs. Here we analyze and compare them for simple queries.</p><p>Audit logs for Impala:</p><p>Query: “show databases”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/939/1*_iWDndVcst2HGMqI1wiRqA.png" /><figcaption>Query: “show databases”</figcaption></figure><p>There are no related HDFS Audit logs. This can be one of the scenarios where there is data access regardless of accessing files on HDFS.</p><p>Query: “select * from test2”. Reading the content from test2 table in test database.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*P0cJLEL9lPTZ4HLr96B-FA.png" /><figcaption>Query: “select * from test2”</figcaption></figure><p>Here, we analyze more into the Audit logs and compare some of them.</p><p>Query: “create table test5 (name String)”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MWvsHqp38c2R_VB39JR-0w.png" /><figcaption>Query: “create table test5 (name String)”</figcaption></figure><p>After executing and analyzing simple Impala queries (typically involving one or no tables), it is found that Navigator and Impala Queries are similarly structured giving almost the same information.</p><p>Navigator has a “service” field which can be used to filter out the logs while Impala Audit logs don’t have to be filtered.</p><p>The Query that was being executed was accessed through “operationText” field in Navigator logs while using “sql_statement” in Impala Audit logs.</p><p>Let’s do the same comparison for a little more complex queries like joins, where multiple tables are involved.</p><p>Query: “select t1.id, t1.name, t1.email, t2.phone from test4 t1 join test3 t2 on t1.id = t2.id”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UPacFYDhbnIV1-GQ9Pp6PA.png" /><figcaption>Query: Join statement on test4 and test3 tables</figcaption></figure><p>Though Impala Audit logs offer more information like port address, for our case currently we are not considering the port addresses in order to ease the development process.</p><p>Also, the multiple audit logs for each query is stored under one JSON object in Impala logs while Navigator produces multiple records. Similarly, multiple audit logs are produced for cases like invalidating metadata, other joins or cases where we are accessing multiple tables.</p><p><strong><em>Upcoming</em></strong>:</p><p>In this blog, we discussed the need and advantages of adding an extra layer of securit<strong>y for the sensitive data.</strong> We also discussed various Audit logs and effect of Kerberos on them.</p><p>In my next blog, we will be discussing Hive Audit logs, Pipeline of the Anomaly Detection, using Kogni API to identify sensitive information and method to identify the Anomalies.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e3cb7793ee78" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Apache Airflow CSD]]></title>
            <link>https://medium.com/clairvoyantblog/apache-airflow-csd-ac5b145d5e2d?source=rss-350416aa22d3------2</link>
            <guid isPermaLink="false">https://medium.com/p/ac5b145d5e2d</guid>
            <category><![CDATA[apache-airflow]]></category>
            <category><![CDATA[airflow]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[csd]]></category>
            <category><![CDATA[cloudera]]></category>
            <dc:creator><![CDATA[Naveen Balusu]]></dc:creator>
            <pubDate>Thu, 03 Jan 2019 20:50:08 GMT</pubDate>
            <atom:updated>2019-08-20T15:10:11.492Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uxS1wzlvSuzB8cy0dkuKCw.jpeg" /></figure><p><strong>What is Apache Airflow?</strong></p><p>Apache Airflow is a work-flow management system to programmatically author, schedule and monitor data pipelines. It is one of the most effective tools to manage workflows. In 2014, Airflow started as an internal project in Airbnb. Since then, the popularity of the Airflow has been growing and is being adopted by many major companies.</p><p>One of the advantages of the Airflow is its configurability. With a lot of independent processes coordinating together to build up the whole Airflow system, you can choose from various options like messaging queues etc.</p><p><strong>Motivation behind the Airflow CSD</strong></p><p>Installing Airflow has been a daunting task especially when many operating systems come preinstalled with Python 2.6. You would have to do a parallel installation of Python 2.7 without affecting the existing Python installation and use that to install Airflow.</p><p>One of the easiest ways to address this issue is to build a CSD for the Airflow and add it as a service within the Cloudera Manager like many other big data technologies (e.g., Apache Spark). It would not only simplify the installation process but also Airflow runs in its own container so that it wouldn’t affect the OS environment. Another added advantage is that specific versions of the Airflow can be activated and deactivated based on the requirement.</p><p>Here is the <a href="https://github.com/teamclairvoyant/apache-airflow-cloudera-csd">GitHub link</a> for the project.</p><p><strong>What is a Parcel?</strong></p><p>Parcels provide a way to distribute software to a managed cluster through Cloudera Manager. Each parcel is self-contained and will be installed in its own directory. This makes it easier to install and manage multiple versions of the services. Other advantages include keeping software consistent across all nodes, easy to upgrade and downgrade, easier to configure and manage the software across the cluster. You can learn more about parcels <a href="https://github.com/cloudera/cm_ext/wiki/Parcels:-What-and-Why%3Fhttps://github.com/cloudera/cm_ext/wiki/Parcels:-What-and-Why%3F">here</a>.</p><p><strong>What is a CSD?</strong></p><p>CSD offers an ability to manage software installed through parcels. CSD can be added as a service through Cloudera Manager. CSD is a self-contained jar file which has all the code to manage a particular service type in Cloudera Manager. CSD offers a simplified way to monitor service, its resource management, allocation, distribution, and its lifecycle management. You can learn more about CSD <a href="https://github.com/cloudera/cm_ext/wiki/CSD-Overview">here</a>.</p><p><strong>Versions of Airflow available</strong></p><p>A complete list of the versions currently available can be found on Github: <a href="https://github.com/teamclairvoyant/apache-airflow-cloudera-csd">https://github.com/teamclairvoyant/apache-airflow-cloudera-csd</a></p><p><strong>Installation</strong></p><p>To install Airflow as a CSD, you just have to follow these high level steps, which are detailed bellow:</p><ol><li>Install the Airflow Parcel</li><li>Setup the Database for Airflow</li><li>Install RabbitMQ.</li><li>Install the Airflow CSD.</li></ol><p><strong>Install the Airflow Parcel</strong></p><p>To install Airflow Parcel, go to Cloudera Manager and navigate to Hosts -&gt; Parcels -&gt; Configurations and add<a href="https://s3-us-west-2.amazonaws.com/archive.clairvoyantsoft.com/airflow/parcels/">http://archive.clairvoyantsoft.com/airflow/parcels/</a>&lt;airflow_version&gt;</p><p>Once, all the available versions of the parcels are listed, select the right version and click download. After successfully downloading, distribute and then install and activate the parcel.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uP61nzslPxVaUuQkZ54xNA.png" /><figcaption>Screenshot of Available parcels before installation</figcaption></figure><p><strong>Setup the Database for Airflow</strong></p><p>MySQL or PostgreSQL needs to be installed and setup in order to store Airflow metadata.</p><p>You can use the script from https://github.com/teamclairvoyant/hadoop-deployment-bash/blob/master/services/create_mysql_dbs-airflow.sh to download, install and configure the MySQL database using<br> sh create_mysql_dbs-airflow.sh --host &lt;hostname&gt; --user &lt;username&gt; --password &lt;password&gt;<br> <br> And, https://github.com/teamclairvoyant/hadoop-deployment-bash/blob/master/services/create_postgresql_dbs-airflow.sh to download, install and configure PostgreSQL using</p><p>sh create_postgresql_dbs-airflow.sh --host &lt;hostname&gt; --user &lt;username&gt; --password &lt;password&gt;</p><p>Alternatively, here is the high-level procedure to setup the database manually. <br> <br> 1. A database needs to be created.<br> 2. A database user needs to be created along with a password<br> 3. Grant all the privileges on the database to the newly created user. <br> <br> Example for PostgreSQL:<br> 1. Create a Database.</p><p>CREATE DATABASE &lt;database_name&gt;;<br> <br> 2. Create a Database User</p><p>CREATE USER &lt;username&gt; WITH ENCRYPTED PASSWORD &#39;&lt;password&gt;&#39;;<br> <br> 3. Grant privileges for the user on the database.</p><p>GRANT ALL PRIVILEGES ON DATABASE &lt;database_name&gt; to &lt;username&gt;;<br> <br><strong>Install the Airflow CSD</strong></p><p>To install the CSD, you can download the related jar file here. Download Airflow 1.9.0 <a href="https://s3-us-west-2.amazonaws.com/archive.clairvoyantsoft.com/airflow/csd/AIRFLOW-1.9.0.jar">here</a>. Copy the jar file to CSD installation location (by default its /opt/cloudera/csd) where Cloudera Manager server is installed. Restart the Cloudera Manager server using service cloudera-scm-server restart.</p><p>Within the Cloudera Manager navigate to Add Services -&gt; Airflow.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aX8m7T8OeVKEf8xu-MFvcA.png" /><figcaption>Screenshot of role selection screen</figcaption></figure><p>Deploy the roles on the nodes. Airflow Webserver, Worker, and Flower can be deployed on multiple nodes to increase the availability of the cluster. The information on number instances of each role is found in next section “Roles”. After successfully assigning the roles, you can click continue.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sNqz5ND0Wt7O-yWohavmlg.png" /><figcaption>Screenshot of configuration page during installation</figcaption></figure><p>You will be able to set some specify the database to store the Airflow metadata. Currently, only PostgreSQL and MySQL are only supported.</p><p>RabbitMQ hostname should be the same host that was specified on the previous page. You can also enter the credentials of existing RabbitMQ username and password or a new user will be created with the credentials provided if that specified does not exists.</p><p>You will be given some additional configurations like Expose config, Load examples, Airflow Home, Base Log Folder and whether the Dags are paused at initialization or not.</p><p>After providing the desired configurations, you can continue.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KEJGOMsXYdB9XKK2l6qw1w.png" /><figcaption>Installation screen</figcaption></figure><p>Installation consists of four phases.</p><p>First stage will ensure the required CDH versions and parcels are installed</p><p>Second phase will deploy the configurations that was specified before.</p><p>Third Phase will be initializing the Airflow database.</p><p>Fourth phase will start all the roles. More information about the roles is provided below.</p><p><strong>Roles</strong></p><p>Here are the roles defined in the Airflow CSD:</p><ol><li>Airflow Webserver</li><li>Airflow Scheduler</li><li>Airflow Worker</li><li>Airflow Flower</li><li>Kerberos</li><li>Gateway</li></ol><p><strong>Airflow Webserver:</strong> Used to start the Airflow Web UI. Webserver role can be deployed on more than instances. However, they will be the same and can be used for backup purposes.</p><p><strong>Airflow Scheduler:</strong> Used to schedule the Airflow jobs. This is limited to one instance to reduce the risk of duplicate jobs.</p><p><strong>Airflow Worker: </strong>Picks jobs from the message broker and execute them on the nodes. Multiple instances can be deployed.</p><p><strong>Airflow Flower: </strong>Used to monitor celery clusters. Multiple instances are supported</p><p><strong>Kerberos: </strong>Used to periodically renew Kerberos certificates forAirflow. It internally executes airflow kerberos. An external Kerberos Distribution Center must be setup. Multiple instances can be setup for load balancing purposes.</p><p><strong>Gateway: </strong>Update the configurations from the configurations tab into the airflow.cfg file. This is done through the update_cfg.sh file which is executed from the scriptRunner within the gateway role.</p><p><strong>Interactions between Roles</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/986/1*mGAPY0hNMo9OEhT28QlQFw.png" /></figure><p><strong>Configurations update</strong></p><p>In order to update the configurations, you can navigate to CSD and then configurations tab. Update the configurations as needed. Configurations can also be filtered based on the roles on the left-hand side. After updating, click deploy client configuration which will update the airflow.cfg file on all the nodes it was deployed.</p><p>To update the Airflow configurations, navigate to the CSD and click on the configuration tab. Here is a screenshot of the configurations page.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*I0t1RLnw3GIN1oQbnbHq8g.png" /><figcaption>Screenshot of the configurations page</figcaption></figure><p>You can filter the configurations based on the Role name under the scope section on the left hand side.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NOsRj3AsCjmCG21koPm6Zw.png" /><figcaption>Deploy Client configurations to update the Airflow.cfg</figcaption></figure><p>Deploying client configuration will update the <em>airflow.cfg</em> and restart the roles for the latest configurations to come into effect.</p><p><strong>Deploying DAGs</strong></p><p>In order to deploy a DAG copy the python file to DAG_FOLDER on all the nodes. The DAG will appear in the Airflow WebUI.</p><p><strong>Relavent Links</strong></p><ul><li>Apache Airflow CSD Project in Github: <a href="https://github.com/teamclairvoyant/apache-airflow-cloudera-csd">https://github.com/teamclairvoyant/apache-airflow-cloudera-csd</a></li><li>Apache Airflow Parcel Project in Github: <a href="https://github.com/teamclairvoyant/apache-airflow-cloudera-parcel">https://github.com/teamclairvoyant/apache-airflow-cloudera-parcel</a></li><li>Cloudera Extensibility Documentation Wiki: <a href="https://github.com/cloudera/cm_ext/wiki">https://github.com/cloudera/cm_ext/wiki</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ac5b145d5e2d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/clairvoyantblog/apache-airflow-csd-ac5b145d5e2d">Apache Airflow CSD</a> was originally published in <a href="https://medium.com/clairvoyantblog">Clairvoyant Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>