<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Kotti Sasikanth on Medium]]></title>
        <description><![CDATA[Stories by Kotti Sasikanth on Medium]]></description>
        <link>https://medium.com/@ksasi?source=rss-5be5ec29922c------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*eF3TglBtCf-4Ftcn.jpg</url>
            <title>Stories by Kotti Sasikanth on Medium</title>
            <link>https://medium.com/@ksasi?source=rss-5be5ec29922c------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 02:27:00 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ksasi/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Descriptive Statistics Final Project — Python]]></title>
            <link>https://medium.com/sasikanth-kotti/descriptive-statistics-final-project-python-fdb128422a08?source=rss-5be5ec29922c------2</link>
            <guid isPermaLink="false">https://medium.com/p/fdb128422a08</guid>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Kotti Sasikanth]]></dc:creator>
            <pubDate>Sun, 04 Dec 2016 05:07:13 GMT</pubDate>
            <atom:updated>2017-11-21T04:43:22.904Z</atom:updated>
            <content:encoded><![CDATA[<h4>Overview</h4><p>Welcome to the Descriptive Statistics Final Project! In this project, you will demonstrate what you have learned in this course by conducting an experiment dealing with drawing from a deck of playing cards and creating a writeup containing your findings.</p><p>Be sure to check through the <a href="https://www.google.com/url?q=https://docs.google.com/document/d/1gVCBsThTXBkUtMiccc4Rj5XT8I9h4ji0bpqTe2uUxRg/pub&amp;sa=D&amp;ust=1480730153149000&amp;usg=AFQjCNGUHJ1sEPRT8G0PPvcMOm8m2v8tgQ">project rubric</a> to self-assess and share with others who will give you feedback.</p><h4>Questions for Investigation</h4><p>This experiment will require the use of a standard deck of playing cards. This is a deck of fifty-two cards divided into four suits (spades (♠), hearts (♥), diamonds (♦), and clubs (♣)), each suit containing thirteen cards (Ace, numbers 2–10, and face cards Jack, Queen, and King). You can use either a physical deck of cards for this experiment or you may use a virtual deck of cards such as that found on random.org (<a href="https://www.google.com/url?q=http://www.random.org/playing-cards/&amp;sa=D&amp;ust=1480730153152000&amp;usg=AFQjCNGHJuQqlS9H34UbvveWII9361HUyw">http://www.random.org/playing-cards/</a>).</p><p>For the purposes of this task, assign each card a value: The Ace takes a value of 1, numbered cards take the value printed on the card, and the Jack, Queen, and King each take a value of 10.</p><p>1. First, create a histogram depicting the relative frequencies of the card values.</p><p>2. Now, we will get samples for a new distribution. To obtain a single sample, shuffle your deck of cards and draw three cards from it. (You will be sampling from the deck without replacement.) Record the cards that you have drawn and the sum of the three cards’ values. Replace the drawn cards back into the deck and repeat this sampling procedure a total of at least thirty times.</p><p>3. Let’s take a look at the distribution of the card sums. Report descriptive statistics for the samples you have drawn. Include at least two measures of central tendency and two measures of variability.</p><p>4. Create a histogram of the sampled card sums you have recorded. Compare its shape to that of the original distribution. How are they different, and can you explain why this is the case?</p><p>5. Make some estimates about values you will get on future draws. Within what range will you expect approximately 90% of your draw values to fall? What is the approximate probability that you will get a draw value of at least 20? Make sure you justify how you obtained your values.</p><h4>Investigation</h4><h4>1.Histogram depicting the relative frequencies of the card values.</h4><p>Deck_Cards.csv contains deck of fifty-two cards divided into four suits, along with assigning each card a value: The Ace takes a value of 1, numbered cards take the value printed on the card, and the Jack, Queen, and King each take a value of 10)</p><p>Below is the format:</p><p>cards,suits,value<br>A,S,1<br>2,S,2<br>3,S,3<br>4,S,4<br>5,S,5<br>6,S,6<br>7,S,7<br>8,S,8<br>9,S,9<br>10,S,10<br>J,S,10<br>Q,S,10<br>K,S,10<br>A,H,1<br>2,H,2<br>3,H,3<br>4,H,4<br>5,H,5<br>6,H,6<br>7,H,7<br>8,H,8<br>9,H,9<br>10,H,10<br>J,H,10<br>Q,H,10<br>K,H,10<br>A,D,1<br>2,D,2<br>3,D,3<br>4,D,4<br>5,D,5<br>6,D,6<br>7,D,7<br>8,D,8<br>9,D,9<br>10,D,10<br>J,D,10<br>Q,D,10<br>K,D,10<br>A,C,1<br>2,C,2<br>3,C,3<br>4,C,4<br>5,C,5<br>6,C,6<br>7,C,7<br>8,C,8<br>9,C,9<br>10,C,10<br>J,C,10<br>Q,C,10<br>K,C,10</p><h4>Load Data</h4><pre>import pandas as pd<br>import numpy as np</pre><pre>df = pd.read_csv(&quot;/Users/....../Documents/Deck_Cards.csv&quot;,header=&#39;infer&#39;)<br>print df.head(3)<br>print df.iloc[:,2].describe()</pre><pre>Output:<br>  cards suits value<br>0 A S 1<br>1 2 S 2<br>2 3 S 3<br>count   52.000000<br>mean     6.538462<br>std      3.183669<br>min      1.000000<br>25%      4.000000<br>50%      7.000000<br>75%     10.000000<br>max     10.000000<br>Name: value, dtype: float64</pre><h4>Plot Histogram</h4><pre>import matplotlib.pyplot as plt</pre><pre>fig = plt.figure()<br>ax = fig.add_subplot(211)<br>ax.hist(df[&#39;value&#39;],bins = 10,range=[0.5, 10.5],facecolor=&#39;g&#39;, align=&#39;mid&#39;)<br>ax.xaxis.set_ticks(np.arange(0, 12, 1))</pre><pre>plt.title(&#39;Relative Frequencies&#39;)<br>plt.xlabel(&#39;Values&#39;)<br>plt.ylabel(&#39;Count&#39;)<br>plt.grid(True)</pre><pre>ay = fig.add_subplot(212)<br>ay.boxplot(df[&#39;value&#39;])</pre><pre>plt.show()</pre><pre>Output:</pre><figure><img alt="histogram_1" src="https://cdn-images-1.medium.com/max/380/0*wd7tR8C3c3AG6TCd.png" /></figure><h4>Show Frequency Table</h4><pre>freq_table = df.groupby([&#39;value&#39;])<br>print freq_table.size()</pre><pre>Output:</pre><pre>value<br>1 4<br>2 4<br>3 4<br>4 4<br>5 4<br>6 4<br>7 4<br>8 4<br>9 4<br>10 16<br>dtype: int64</pre><h4>2.Sampling and Sampling distribution for sum and average of the three cards’ values</h4><pre>from random import sample</pre><pre>c1=[]<br>c2=[]<br>c3=[]<br>sum_s=[]<br>average_s=[]</pre><pre>for i in xrange(1000):<br> rindex = np.array(sample(xrange(len(df)),3))<br> dfr = df.ix[rindex]<br> c1.append(dfr.iloc[0,2])<br> c2.append(dfr.iloc[1,2])<br> c3.append(dfr.iloc[2,2])<br> sum_s.append(dfr.iloc[0,2]+dfr.iloc[1,2]+dfr.iloc[2,2])<br> average_s.append((dfr.iloc[0,2]+dfr.iloc[1,2]+dfr.iloc[2,2])/(3.0))</pre><pre>sampling_df = pd.DataFrame({&#39;card1&#39;:c1,&#39;card2&#39;:c2,&#39;card3&#39;:c3,&#39;sum_col&#39;:sum_s,&#39;average_col&#39;:average_s})</pre><pre>print sampling_df.head(3)</pre><pre>Output:</pre><pre>average_col card1     card2    card3     sum_col<br>0 7.000000     7         10       4         21<br>1 5.333333     4         10       2         16<br>2 7.000000     5          7       9         21</pre><pre>3.Distribution of the card sums. Descriptive statistics for the samples drawn.(At least two measures of central tendency and two measures of variability.)</pre><pre>print sampling_df[&#39;sum_col&#39;].describe()</pre><pre>Output:</pre><pre>count 1000.000000<br>mean 19.646000<br>std 5.328703<br>min 5.000000<br>25% 16.000000<br>50% 20.000000<br>75% 23.000000<br>max 30.000000<br>Name: sum_col, dtype: float64</pre><pre>4.Histogram of the sampled card sums and comparison of its shape to original distribution</pre><pre>fig = plt.figure()<br>ay = fig.add_subplot(211)<br>ay.hist(sampling_df[&#39;sum_col&#39;],bins = 25,range=[4, 34],facecolor=&#39;g&#39;, align=&#39;mid&#39;)<br>#ay.xaxis.set_ticks(np.arange(0, 40, 5))</pre><pre>plt.title(&#39;Sampling Distribution - SUM&#39;)<br>plt.xlabel(&#39;Values(SUM)&#39;)<br>plt.ylabel(&#39;Count&#39;)</pre><pre>ay = fig.add_subplot(212)<br>ay.boxplot(sampling_df[&#39;sum_col&#39;])</pre><pre>plt.show()</pre><pre>Output:</pre><figure><img alt="histogram_2" src="https://cdn-images-1.medium.com/max/393/0*ixJR1HT3HeWrA_LY.png" /></figure><pre>The shape of the sampled card sums is same as that of normal distribution and this is in accordance with Central Limit Theorem.</pre><pre>5.Descriptive statistics and histogram of Sampling distribution for Average of the three cards’ values</pre><pre>print sampling_df[&#39;average_col&#39;].describe()</pre><pre>Output:</pre><pre>count 1000.000000<br>mean 6.548667<br>std 1.776234<br>min 1.666667<br>25% 5.333333<br>50% 6.666667<br>75% 7.666667<br>max 10.000000<br>Name: average_col, dtype: float64</pre><pre>fig = plt.figure()<br>ay = fig.add_subplot(211)<br>ay.hist(sampling_df[&#39;average_col&#39;],bins = 40,range=[4, 34],facecolor=&#39;g&#39;, align=&#39;mid&#39;)<br>#ay.xaxis.set_ticks(np.arange(0, 40, 5))</pre><pre>plt.title(&#39;Sampling Distribution - AVG&#39;)<br>plt.xlabel(&#39;Values(AVG)&#39;)<br>plt.ylabel(&#39;Count&#39;)</pre><pre>ay = fig.add_subplot(212)<br>ay.boxplot(sampling_df[&#39;average_col&#39;])</pre><pre>plt.show()</pre><pre>Output:</pre><figure><img alt="histogram_3" src="https://cdn-images-1.medium.com/max/393/0*18y856ZWF9J5aVoW.png" /></figure><pre>The shape of theSampling distribution for Average of the three cards’ values is same as that of normal distribution and this is in accordance with Central Limit Theorem.</pre><pre>Also as perCentral Limit Theorem for Sampling distribution for Average of the three cards’ values,</pre><pre>Population SD/SD of Sampling distribution for Average of the three cards’ values is square root of sample size</pre><pre>3.183669/1.776234 = 1.7923702620262871 ~√ (3 )</pre><pre>6.Within what range will you expect approximately 90% of your draw values to fall? What is the approximate probability that you will get a draw value of at least 20? (For sampled card sums distribution)</pre><pre>90% of values will fall between the points corresponding to 5% and 95%.</pre><pre>import scipy.stats as sp</pre><pre>mean_sum_distribution = 19.481000<br>variance_sum_distribution = 5.435364</pre><pre>p1=sp.norm.ppf(0.05)*variance_sum_distribution+mean_sum_distribution</pre><pre>p2=sp.norm.ppf(0.95)*variance_sum_distribution+mean_sum_distribution</pre><pre>print &#39;It is expected approximately 90% of the draw values will fall between &#39;,p1,&#39; and &#39;,p2</pre><pre>Output:</pre><pre>It is expected approximately 90% of the draw values will fall between  10.5406218108  and  28.4213781892</pre><pre>Manual Calculation by referring <a href="https://s3.amazonaws.com/udacity-hosted-downloads/ZTable.jpg">Z Table</a> (For sampled card sums distribution):</pre><pre>95% is 1.64<br>5%  is -1.65<br>90% is between 95% and 5%</pre><pre>Mean = 19.481</pre><pre>Variance = 5.435364</pre><pre>Converting Z values to X</pre><pre>1.64*5.435364+19.481 = 28.39499696~28.4<br>-1.65*5.435364+19.481 = 10.512649400000003~10.51</pre><pre>The approximate probability that you will get a draw value of at most 20</pre><pre>p_20_atmost = sp.norm.cdf(z_20)</pre><pre>print z_20</pre><pre>print &#39;The approximate probability that we get a draw value of at least 20 is&#39;,(1-p_20_atmost)</pre><pre>Output:</pre><pre>0.0954857853126<br>The approximate probability that we get a draw value of at least 20 is 0.461964490173</pre><pre>Manual Calculation by referring <a href="https://s3.amazonaws.com/udacity-hosted-downloads/ZTable.jpg">Z Table</a> (For sampled card sums distribution):</pre><pre>Z-score = (X-Mu)/Sigma</pre><pre>Z-score for 20 = (20-19.481)/5.435364 = 0.09548578531263009, left of 20</pre><pre>To get more than 20 i.e. atleast 20</pre><pre>1 - Probability(Z-score for 20) i.e 1 - 0.5359 = 0.46409999999999996~0.464</pre><pre>References:</pre><pre>Descriptive Statistics Final Project -<a href="https://docs.google.com/document/d/1059JMJ9C5dn7vKUrmfWYle57Ai3Uk9PzxPQBGj5drjE/pub?embedded=true">https://docs.google.com/document/d/1059JMJ9C5dn7vKUrmfWYle57Ai3Uk9PzxPQBGj5drjE/pub?embedded=true</a><br>Python For Data Science Cheat Sheet - <a href="https://s3.amazonaws.com/assets.datacamp.com/blog_assets/PythonForDataScience.pdf">https://s3.amazonaws.com/assets.datacamp.com/blog_assets/PythonForDataScience.pdf</a><br>R-bloggers - <a href="https://www.r-bloggers.com/descriptive-statistics-final-project-with-python-r/">https://www.r-bloggers.com/descriptive-statistics-final-project-with-python-r/</a><br>Cheat Sheet for Exploratory Data Analysis in Python - <a href="https://www.analyticsvidhya.com/blog/2015/06/infographic-cheat-sheet-data-exploration-python/">https://www.analyticsvidhya.com/blog/2015/06/infographic-cheat-sheet-data-exploration-python/</a><br>Z Table - <a href="https://s3.amazonaws.com/udacity-hosted-downloads/ZTable.jpg">https://s3.amazonaws.com/udacity-hosted-downloads/ZTable.jpg</a><br>Cheat sheet: Data Visualisation in Python - <a href="https://www.analyticsvidhya.com/blog/2015/06/data-visualization-in-python-cheat-sheet/">https://www.analyticsvidhya.com/blog/2015/06/data-visualization-in-python-cheat-sheet/</a></pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fdb128422a08" width="1" height="1" alt=""><hr><p><a href="https://medium.com/sasikanth-kotti/descriptive-statistics-final-project-python-fdb128422a08">Descriptive Statistics Final Project — Python</a> was originally published in <a href="https://medium.com/sasikanth-kotti">Sasikanth Kotti</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Interesting Machine Learning Resources]]></title>
            <link>https://medium.com/sasikanth-kotti/interesting-machine-learning-resources-6d0628895072?source=rss-5be5ec29922c------2</link>
            <guid isPermaLink="false">https://medium.com/p/6d0628895072</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Kotti Sasikanth]]></dc:creator>
            <pubDate>Sun, 23 Oct 2016 04:47:10 GMT</pubDate>
            <atom:updated>2017-11-21T04:43:25.175Z</atom:updated>
            <content:encoded><![CDATA[<h4>Online Courses</h4><ul><li>Andrew Ng’s<a href="https://www.coursera.org/learn/machine-learning"> Stanford course on machine learning</a> is very popular and generally well-reviewed. It’s considered one of the best introductory courses in machine learning and will give you some rigorous preparation for delving into deep learning.</li><li>Udacity has a free,<a href="https://www.udacity.com/course/intro-to-machine-learning--ud120"> ten week introductory course in machine learning</a> that focuses on both theory and real-world applications. Again, it’s a decent preparatory course for those interested in eventually pursuing deep learning.</li><li>Andrej Karpathy’s “<a href="http://cs231n.stanford.edu/">CS231n: Convolutional Neural Networks for Visual Recognition</a>” at Stanford is challenging but well-done course in deep neural networks, and the syllabus and detailed course notes are available online.</li><li>Geoffrey Hinton’s course on “<a href="https://www.coursera.org/learn/neural-networks">Neural Networks for Machine Learning</a>” is good, and it’s taught by one of the godfathers of the field.</li><li>Udacity “<a href="https://www.udacity.com/course/machine-learning-engineer-nanodegree--nd009">Machine Learning Engineer Nanodegree</a>” is good and comprehensive course.</li></ul><h4>Blog Posts</h4><ul><li>What are the best ways to learn advanced Machine Learning outside academia? Click <a href="https://www.quora.com/What-are-the-best-ways-to-learn-advanced-Machine-Learning-outside-academia">here</a></li><li>Data Science and Robots <a href="https://brohrer.github.io/index.html">Brandon Rohrer</a></li><li>The Great A.I. Awakening. Click <a href="http://nyti.ms/2hE6XZ5">here</a></li><li>Nuts and Bolts of Applying Deep Learning by <a href="http://www.andrewng.org/">Andrew Ng</a> . Click <a href="https://kevinzakka.github.io/2016/09/26/applying-deep-learning/">here</a></li><li>ML Wave.Click <a href="https://mlwave.com/">here</a></li><li>Winning Tips on Machine Learning Competitions by Kazanova, Current Kaggle #3.Click <a href="https://mlwave.com/">here</a></li><li>FastML.Click <a href="https://mlwave.com/">here</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6d0628895072" width="1" height="1" alt=""><hr><p><a href="https://medium.com/sasikanth-kotti/interesting-machine-learning-resources-6d0628895072">Interesting Machine Learning Resources</a> was originally published in <a href="https://medium.com/sasikanth-kotti">Sasikanth Kotti</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[OBIEE and Pivotal HAWQ]]></title>
            <link>https://medium.com/sasikanth-kotti/obiee-and-pivotal-hawq-aa3c18778273?source=rss-5be5ec29922c------2</link>
            <guid isPermaLink="false">https://medium.com/p/aa3c18778273</guid>
            <category><![CDATA[hadoop]]></category>
            <category><![CDATA[business-intelligence]]></category>
            <category><![CDATA[obiee]]></category>
            <category><![CDATA[hdfs]]></category>
            <category><![CDATA[hortonworks-hdp]]></category>
            <dc:creator><![CDATA[Kotti Sasikanth]]></dc:creator>
            <pubDate>Fri, 14 Aug 2015 00:07:26 GMT</pubDate>
            <atom:updated>2017-11-21T04:43:26.396Z</atom:updated>
            <content:encoded><![CDATA[<p>Oracle Business Intelligence Enterprise Edition 11g (OBIEE) is a comprehensive business intelligence platform that delivers a full range of capabilities — including interactive dashboards, ad hoc queries, notifications and alerts, enterprise and financial reporting, scorecard and strategy management, business process invocation, search and collaboration, mobile, integrated systems management and more.</p><p>HAWQ is a parallel SQL query engine that combines the key technological advantages of the industry-leading Greenplum Database with the scalability and convenience of Hadoop. Using HAWQ functionality, you can interact with petabyte range data sets. HAWQ provides a complete, standards-compliant SQL interface.</p><p>Pivotal HAWQ enables data analysis for a variety of Hadoop-based data formats using the Pivotal Extension Framework (PXF), without duplicating or converting HBase files.</p><p>In this blog post we will look at setting up OBIEE to run SampleAppLite dashboards sourcing data from Hadoop using Pivotal HAWQ and PXF.</p><h4>Technology Stack :-</h4><ol><li>Oracle Business Intelligence 11g (11.1.1.9.0)</li><li>Pivotal HAWQ 1.3.0.3</li><li>AWS EC2 m3.large instance type</li><li>AWS EC2 t2.small instance type</li><li>Hortonworks HDP 2.2 Sandbox on Microsoft Azure(Basic A3 (4 Cores, 7 GB))</li></ol><h4>Initial Setup :-</h4><p>Setup OBIEE 11g(11.1.1.9.0) in AWS EC2 by following the steps in the blog post<em> </em><a href="http://www.rittmanmead.com/2014/09/obiee-sampleapp-in-the-cloud-importing-virtualbox-machines-to-aws-ec2/"><em>OBIEE SampleApp in The Cloud</em></a></p><p>Setup OBIEE 11g(11.1.1.9.0) Developer Client Tools in AWS EC2 t2.small instance</p><p>Launch Hortonworks HDP 2.2 Sandbox on Microsoft Azure by following the steps in the blog post <a href="http://hortonworks.com/blog/hortonworks-sandbox-azure/"><em>Getting Started with Hortonworks Sandbox on Azure</em></a></p><p>Setup Pivotal HAWQ and PXF by following steps in the blog post <a href="http://blog.pivotal.io/big-data-pivotal/p-o-v/pivotal-hawq-lands-in-the-hortonworks-sandbox"><em>Pivotal HAWQ Lands In The Hortonworks Sandbox</em></a></p><h4>Setting up SampleAppLite data store in HAWQ :-</h4><p>Step 1:- Placing the source XML files in HDFS</p><p>The SampleAppLite source data from the following XML files</p><p>SAMP_OFFICES_D.xml<br>SAMP_PRODUCTS_D.xml<br>SAMP_REVENUE_A.xml<br>SAMP_REVENUE_F.xml<br>SAMP_TARGETS_F.xml<br>SAMP_TIME_DAY_D.xml<br>SAMP_TIME_MTH_D.xml</p><p>Place the above xml files in HDFS</p><pre>Ex:- hdfs dfs -mkdir /obi_hdfs<br>     hdfs dfs -put SAMP_OFFICES_D.xml /obi_hdfs</pre><p>Step 2:- Transform XML files to tab delimited files with Pig</p><p>XML files are converted to tab delimited files using the Pig scripts shown below</p><pre>SAMP_Dimentions_Pig_Script :-</pre><pre>REGISTER piggybank.jar;<br>A = LOAD &#39;/obi_hdfs/SAMP_PRODUCTS_D.xml&#39; <br>USING org.apache.pig.piggybank.storage.XMLLoader(&#39;SAMP_PRODUCTS_D&#39;) <br>AS (doc:chararray);</pre><pre>clean = foreach A generate REPLACE(REPLACE(REPLACE(REPLACE(doc, &#39;\\u000D&#39;, &#39;|&#39;), &#39;\\u000A&#39;, &#39;|&#39;), &#39;&gt; &lt;&#39;, &#39;&gt; &lt;&#39;), &#39;\t&#39;, &#39; &#39;);<br>store clean into &#39;/obi_hdfs/SAMP_PRODUCTS_D&#39;;</pre><pre>B = LOAD &#39;/obi_hdfs/SAMP_OFFICES_D.xml&#39; <br>USING org.apache.pig.piggybank.storage.XMLLoader(&#39;SAMP_OFFICES_D&#39;) <br>AS (doc:chararray);</pre><pre>clean = foreach B generate REPLACE(REPLACE(REPLACE(REPLACE(doc, &#39;\\u000D&#39;, &#39;|&#39;), &#39;\\u000A&#39;, &#39;|&#39;), &#39;&gt; &lt;&#39;, &#39;&gt; &lt;&#39;), &#39;\t&#39;, &#39; &#39;);<br>store clean into &#39;/obi_hdfs/SAMP_OFFICES_D&#39;;</pre><pre>C = LOAD &#39;/obi_hdfs/SAMP_TIME_DAY_D.xml&#39; <br>USING org.apache.pig.piggybank.storage.XMLLoader(&#39;SAMP_TIME_DAY_D&#39;) <br>AS (doc:chararray);</pre><pre>clean = foreach C generate REPLACE(REPLACE(REPLACE(REPLACE(doc, &#39;\\u000D&#39;, &#39;|&#39;), &#39;\\u000A&#39;, &#39;|&#39;), &#39;&gt; &lt;&#39;, &#39;&gt; &lt;&#39;), &#39;\t&#39;, &#39; &#39;);<br>store clean into &#39;/obi_hdfs/SAMP_TIME_DAY_D&#39;;</pre><pre>D = LOAD &#39;/obi_hdfs/SAMP_TIME_MTH_D.xml&#39; <br>USING org.apache.pig.piggybank.storage.XMLLoader(&#39;SAMP_TIME_MTH_D&#39;) <br>AS (doc:chararray);</pre><pre>clean = foreach D generate REPLACE(REPLACE(REPLACE(REPLACE(doc, &#39;\\u000D&#39;, &#39;|&#39;), &#39;\\u000A&#39;, &#39;|&#39;), &#39;&gt; &lt;&#39;, &#39;&gt; &lt;&#39;), &#39;\t&#39;, &#39; &#39;);<br>store clean into &#39;/obi_hdfs/SAMP_TIME_MTH_D&#39;;</pre><pre>SAMP_Facts_Pig_Script :-</pre><pre>REGISTER piggybank.jar;<br>A = LOAD &#39;/obi_hdfs/SAMP_REVENUE_A.xml&#39; <br>USING org.apache.pig.piggybank.storage.XMLLoader(&#39;SAMP_REVENUE_A&#39;) <br>AS (doc:chararray);</pre><pre>clean = foreach A generate REPLACE(REPLACE(REPLACE(REPLACE(doc, &#39;\\u000D&#39;, &#39;|&#39;), &#39;\\u000A&#39;, &#39;|&#39;), &#39;&gt; &lt;&#39;, &#39;&gt; &lt;&#39;), &#39;\t&#39;, &#39; &#39;);<br>store clean into &#39;/obi_hdfs/SAMP_REVENUE_A&#39;;</pre><pre>B = LOAD &#39;/obi_hdfs/SAMP_REVENUE_F.xml&#39; <br>USING org.apache.pig.piggybank.storage.XMLLoader(&#39;SAMP_REVENUE_F&#39;) <br>AS (doc:chararray);</pre><pre>clean = foreach B generate REPLACE(REPLACE(REPLACE(REPLACE(doc, &#39;\\u000D&#39;, &#39;|&#39;), &#39;\\u000A&#39;, &#39;|&#39;), &#39;&gt; &lt;&#39;, &#39;&gt; &lt;&#39;), &#39;\t&#39;, &#39; &#39;);<br>store clean into &#39;/obi_hdfs/SAMP_REVENUE_F&#39;;</pre><pre>C = LOAD &#39;/obi_hdfs/SAMP_TARGETS_F.xml&#39; <br>USING org.apache.pig.piggybank.storage.XMLLoader(&#39;SAMP_TARGETS_F&#39;) <br>AS (doc:chararray);</pre><pre>clean = foreach C generate REPLACE(REPLACE(REPLACE(REPLACE(doc, &#39;\\u000D&#39;, &#39;|&#39;), &#39;\\u000A&#39;, &#39;|&#39;), &#39;&gt; &lt;&#39;, &#39;&gt; &lt;&#39;), &#39;\t&#39;, &#39; &#39;);<br>store clean into &#39;/obi_hdfs/SAMP_TARGETS_F&#39;;<br>Step 3 :- Create External tables in HAWQ</pre><pre>CREATE EXTERNAL TABLE SAMP_TIME_MTH_D_T<br>(<br>xml_data text<br>)<br>LOCATION (<br>&#39;pxf://sandbox.hortonworks.com:51200/obi_hdfs/SAMP_TIME_MTH_D/part*?profile=HdfsTextSimple&#39;<br>)<br>FORMAT &#39;TEXT&#39; (delimiter E&#39;\t&#39;);</pre><pre>CREATE EXTERNAL TABLE SAMP_TIME_DAY_D_T<br>(<br>xml_data text<br>)<br>LOCATION (<br>&#39;pxf://sandbox.hortonworks.com:51200/obi_hdfs/SAMP_TIME_DAY_D/part*?profile=HdfsTextSimple&#39;<br>)<br>FORMAT &#39;TEXT&#39; (delimiter E&#39;\t&#39;);</pre><pre>CREATE EXTERNAL TABLE SAMP_OFFICES_D_T<br>(<br>xml_data text<br>)<br>LOCATION (<br>&#39;pxf://sandbox.hortonworks.com:51200/obi_hdfs/SAMP_OFFICES_D/part*?profile=HdfsTextSimple&#39;<br>)<br>FORMAT &#39;TEXT&#39; (delimiter E&#39;\t&#39;);</pre><pre>CREATE EXTERNAL TABLE SAMP_PRODUCTS_D_T<br>(<br>xml_data text<br>)<br>LOCATION (<br>&#39;pxf://sandbox.hortonworks.com:51200/obi_hdfs/SAMP_PRODUCTS_D/part*?profile=HdfsTextSimple&#39;<br>)<br>FORMAT &#39;TEXT&#39; (delimiter E&#39;\t&#39;);</pre><pre>CREATE EXTERNAL TABLE SAMP_REVENUE_A_T<br>(<br>xml_data text<br>)<br>LOCATION (<br>&#39;pxf://sandbox.hortonworks.com:51200/obi_hdfs/SAMP_REVENUE_A/part*?profile=HdfsTextSimple&#39;<br>)<br>FORMAT &#39;TEXT&#39; (delimiter E&#39;\t&#39;);</pre><pre>CREATE EXTERNAL TABLE SAMP_REVENUE_F_T<br>(<br>xml_data text<br>)<br>LOCATION (<br>&#39;pxf://sandbox.hortonworks.com:51200/obi_hdfs/SAMP_REVENUE_F/part*?profile=HdfsTextSimple&#39;<br>)<br>FORMAT &#39;TEXT&#39; (delimiter E&#39;\t&#39;);</pre><pre>CREATE EXTERNAL TABLE SAMP_TARGETS_F_T<br>(<br>xml_data text<br>)<br>LOCATION (<br>&#39;pxf://sandbox.hortonworks.com:51200/obi_hdfs/SAMP_TARGETS_F/part*?profile=HdfsTextSimple&#39;<br>)<br>FORMAT &#39;TEXT&#39; (delimiter E&#39;\t&#39;);<br>Step 4:- Create simple SQL function to parse the data</pre><pre>CREATE OR REPLACE FUNCTION fn_extract_xml_value(p_tag text, p_xml text) RETURNS TEXT AS<br>$$<br>SELECT SPLIT_PART(SUBSTRING($2 FROM &#39;&lt;&#39; || $1 || &#39;&gt;(.*)&lt;/&#39; || $1 || &#39;&gt;&#39;), &#39;&lt;&#39;, 1)<br>$$<br>LANGUAGE SQL;</pre><pre>Step 5:- Create views in HAWQ</pre><pre>CREATE VIEW SAMP_TIME_MTH_D AS<br>SELECT <br>(fn_extract_xml_value(&#39;MTH_KEY&#39;, xml_data))::int AS MTH_KEY, <br>(fn_extract_xml_value(&#39;CAL_HALF&#39;, xml_data))::int AS CAL_HALF,<br>(fn_extract_xml_value(&#39;CAL_MONTH&#39;, xml_data))::int AS CAL_MONTH,<br>(fn_extract_xml_value(&#39;CAL_QTR&#39;, xml_data))::int AS CAL_QTR,<br>(fn_extract_xml_value(&#39;CAL_YEAR&#39;, xml_data))::int AS CAL_YEAR,<br>(fn_extract_xml_value(&#39;FSCL_HALF&#39;, xml_data))::int AS FSCL_HALF,<br>(fn_extract_xml_value(&#39;FSCL_MONTH&#39;, xml_data))::int AS FSCL_MONTH,<br>(fn_extract_xml_value(&#39;FSCL_QTR&#39;, xml_data))::int AS FSCL_QTR,<br>(fn_extract_xml_value(&#39;FSCL_YEAR&#39;, xml_data))::int AS FSCL_YEAR,<br>(fn_extract_xml_value(&#39;JULIAN_MONTH_NUM&#39;, xml_data))::int AS JULIAN_MONTH_NUM,<br>(fn_extract_xml_value(&#39;JULIAN_QTR_NUM&#39;, xml_data))::int AS JULIAN_QTR_NUM,<br>(fn_extract_xml_value(&#39;MONTH_AGO_WID&#39;, xml_data))::int AS MONTH_AGO_WID,<br>(fn_extract_xml_value(&#39;QUARTER_AGO_WID&#39;, xml_data))::int AS QUARTER_AGO_WID,<br>(fn_extract_xml_value(&#39;YEAR_AGO_WID&#39;, xml_data))::int AS YEAR_AGO_WID,<br>(fn_extract_xml_value(&#39;PER_NAME_HALF&#39;, xml_data))::text AS PER_NAME_HALF,<br>(fn_extract_xml_value(&#39;PER_NAME_MONTH&#39;, xml_data))::text AS PER_NAME_MONTH,<br>(fn_extract_xml_value(&#39;PER_NAME_QTR&#39;, xml_data))::text AS PER_NAME_QTR,<br>(fn_extract_xml_value(&#39;PER_NAME_YEAR&#39;, xml_data))::text AS PER_NAME_YEAR,<br>(fn_extract_xml_value(&#39;DAYS_IN_QTR&#39;, xml_data))::int AS DAYS_IN_QTR,<br>(fn_extract_xml_value(&#39;DAYS_IN_MTH&#39;, xml_data))::int AS DAYS_IN_MTH,<br>(fn_extract_xml_value(&#39;BEG_OF_QTR_WID&#39;, xml_data))::int AS BEG_OF_QTR_WID,<br>(fn_extract_xml_value(&#39;LOADED&#39;, xml_data))::int AS LOADED,<br>(fn_extract_xml_value(&#39;CAL_MONTH_END_DATE&#39;, xml_data))::date AS CAL_MONTH_END_DATE,<br>(fn_extract_xml_value(&#39;CAL_QTR_END_DATE&#39;, xml_data))::date AS CAL_QTR_END_DATE,<br>(fn_extract_xml_value(&#39;CAL_HALF_YEAR_END_DATE&#39;, xml_data))::date AS CAL_HALF_YEAR_END_DATE,<br>(fn_extract_xml_value(&#39;CAL_YEAR_END_DATE&#39;, xml_data))::date AS CAL_YEAR_END_DATE,<br>(fn_extract_xml_value(&#39;FSCL_MONTH_END_DATE&#39;, xml_data))::date AS FSCL_MONTH_END_DATE,<br>(fn_extract_xml_value(&#39;FSCL_QTR_END_DATE&#39;, xml_data))::date AS FSCL_QTR_END_DATE,<br>(fn_extract_xml_value(&#39;FSCL_HALF_YEAR_END_DATE&#39;, xml_data))::date AS FSCL_HALF_YEAR_END_DATE,<br>(fn_extract_xml_value(&#39;FSCL_YEAR_END_DATE&#39;, xml_data))::date AS FSCL_YEAR_END_DATE,<br>(fn_extract_xml_value(&#39;DAYS_IN_HALF_YEAR&#39;, xml_data))::int AS DAYS_IN_HALF_YEAR,<br>(fn_extract_xml_value(&#39;DAYS_IN_YEAR&#39;, xml_data))::int AS DAYS_IN_YEAR,<br>(fn_extract_xml_value(&#39;DAYS_IN_FSCL_MONTH&#39;, xml_data))::int AS DAYS_IN_FSCL_MONTH,<br>(fn_extract_xml_value(&#39;DAYS_IN_FSCL_QTR&#39;, xml_data))::int AS DAYS_IN_FSCL_QTR,<br>(fn_extract_xml_value(&#39;DAYS_IN_FSCL_HALF_YEAR&#39;, xml_data))::int AS DAYS_IN_FSCL_HALF_YEAR,<br>(fn_extract_xml_value(&#39;DAYS_IN_FSCL_YEAR&#39;, xml_data))::int AS DAYS_IN_FSCL_YEAR<br>FROM SAMP_TIME_MTH_D_T;</pre><pre>CREATE VIEW SAMP_TIME_DAY_D AS<br>SELECT <br>(fn_extract_xml_value(&#39;CALENDAR_DATE&#39;, xml_data))::date AS CALENDAR_DATE, <br>(fn_extract_xml_value(&#39;DAY_KEY&#39;, xml_data))::int AS DAY_KEY,<br>(fn_extract_xml_value(&#39;DAY_OF_MONTH&#39;, xml_data))::int AS DAY_OF_MONTH,<br>(fn_extract_xml_value(&#39;DAY_OF_WEEK&#39;, xml_data))::int AS DAY_OF_WEEK,<br>(fn_extract_xml_value(&#39;DAY_OF_YEAR&#39;, xml_data))::int AS DAY_OF_YEAR,<br>(fn_extract_xml_value(&#39;JULIAN_DAY_NUM&#39;, xml_data))::int AS JULIAN_DAY_NUM,<br>(fn_extract_xml_value(&#39;JULIAN_MONTH_NUM&#39;, xml_data))::int AS JULIAN_MONTH_NUM,<br>(fn_extract_xml_value(&#39;JULIAN_QTR_NUM&#39;, xml_data))::int AS JULIAN_QTR_NUM,<br>(fn_extract_xml_value(&#39;JULIAN_WEEK_NUM&#39;, xml_data))::int AS JULIAN_WEEK_NUM,<br>(fn_extract_xml_value(&#39;PER_NAME_HALF&#39;, xml_data))::text AS PER_NAME_HALF,<br>(fn_extract_xml_value(&#39;PER_NAME_MONTH&#39;, xml_data))::text AS PER_NAME_MONTH,<br>(fn_extract_xml_value(&#39;PER_NAME_QTR&#39;, xml_data))::text AS PER_NAME_QTR,<br>(fn_extract_xml_value(&#39;PER_NAME_WEEK&#39;, xml_data))::text AS PER_NAME_WEEK,<br>(fn_extract_xml_value(&#39;PER_NAME_YEAR&#39;, xml_data))::text AS PER_NAME_YEAR<br>FROM SAMP_TIME_DAY_D_T;</pre><pre>CREATE VIEW SAMP_OFFICES_D AS<br>SELECT <br>(fn_extract_xml_value(&#39;OFFICE_KEY&#39;, xml_data))::int AS OFFICE_KEY, <br>(fn_extract_xml_value(&#39;OFFICE_DSC&#39;, xml_data))::text AS OFFICE_DSC,<br>(fn_extract_xml_value(&#39;COMPANY&#39;, xml_data))::text AS COMPANY,<br>(fn_extract_xml_value(&#39;ORGANIZATION&#39;, xml_data))::text AS ORGANIZATION,<br>(fn_extract_xml_value(&#39;DEPARTMENT&#39;, xml_data))::text AS DEPARTMENT,<br>(fn_extract_xml_value(&#39;SEQUENCE&#39;, xml_data))::int AS SEQUENCE,<br>(fn_extract_xml_value(&#39;COMPANY_KEY&#39;, xml_data))::int AS COMPANY_KEY,<br>(fn_extract_xml_value(&#39;ORG_KEY&#39;, xml_data))::int AS ORG_KEY,<br>(fn_extract_xml_value(&#39;DEPT_KEY&#39;, xml_data))::int AS DEPT_KEY<br>FROM SAMP_OFFICES_D_T;</pre><pre>CREATE VIEW SAMP_PRODUCTS_D AS<br>SELECT <br>(fn_extract_xml_value(&#39;PROD_KEY&#39;, xml_data))::int AS PROD_KEY, <br>(fn_extract_xml_value(&#39;PROD_DSC&#39;, xml_data))::text AS PROD_DSC,<br>(fn_extract_xml_value(&#39;ATTRIBUTE_2&#39;, xml_data))::text AS ATTRIBUTE_2,<br>(fn_extract_xml_value(&#39;ATTRIBUTE_1&#39;, xml_data))::text AS ATTRIBUTE_1,<br>(fn_extract_xml_value(&#39;TYPE&#39;, xml_data))::text AS TYPE,<br>(fn_extract_xml_value(&#39;LOB&#39;, xml_data))::text AS LOB,<br>(fn_extract_xml_value(&#39;BRAND&#39;, xml_data))::text AS BRAND,<br>(fn_extract_xml_value(&#39;SEQUENCE&#39;, xml_data))::int AS SEQUENCE,<br>(fn_extract_xml_value(&#39;BRAND_KEY&#39;, xml_data))::int AS BRAND_KEY,<br>(fn_extract_xml_value(&#39;LOB_KEY&#39;, xml_data))::int AS LOB_KEY,<br>(fn_extract_xml_value(&#39;TYPE_KEY&#39;, xml_data))::int AS TYPE_KEY<br>FROM SAMP_PRODUCTS_D_T;</pre><pre>CREATE VIEW SAMP_REVENUE_F AS<br>SELECT <br>(fn_extract_xml_value(&#39;OFFICE_KEY&#39;, xml_data))::int AS OFFICE_KEY, <br>(fn_extract_xml_value(&#39;PROD_KEY&#39;, xml_data))::int AS PROD_KEY,<br>(fn_extract_xml_value(&#39;ORDER_NUMBER&#39;, xml_data))::int AS ORDER_NUMBER,<br>(fn_extract_xml_value(&#39;REVENUE&#39;, xml_data))::float8 AS REVENUE,<br>(fn_extract_xml_value(&#39;UNITS&#39;, xml_data))::int AS UNITS,<br>(fn_extract_xml_value(&#39;DISCNT_VALUE&#39;, xml_data))::float8 AS DISCNT_VALUE,<br>(fn_extract_xml_value(&#39;BILL_MTH_KEY&#39;, xml_data))::int AS BILL_MTH_KEY,<br>(fn_extract_xml_value(&#39;BILL_DAY_DT&#39;, xml_data))::date AS BILL_DAY_DT,<br>(fn_extract_xml_value(&#39;ORDER_DAY_DT&#39;, xml_data))::date AS ORDER_DAY_DT,<br>(fn_extract_xml_value(&#39;ORDER_STATUS&#39;, xml_data))::text AS ORDER_STATUS,<br>(fn_extract_xml_value(&#39;ORDER_TYPE&#39;, xml_data))::text AS ORDER_TYPE<br>FROM SAMP_REVENUE_F_T;</pre><pre>CREATE VIEW SAMP_REVENUE_A AS<br>SELECT <br>(fn_extract_xml_value(&#39;PROD_KEY&#39;, xml_data))::int AS PROD_KEY,<br>(fn_extract_xml_value(&#39;BILL_MTH_KEY&#39;, xml_data))::int AS BILL_MTH_KEY,<br>(fn_extract_xml_value(&#39;REVENUE&#39;, xml_data))::float8 AS REVENUE,<br>(fn_extract_xml_value(&#39;UNITS&#39;, xml_data))::int AS UNITS,<br>(fn_extract_xml_value(&#39;DISCNT_VALUE&#39;, xml_data))::float8 AS DISCNT_VALUE<br>FROM SAMP_REVENUE_A_T;</pre><pre>CREATE VIEW SAMP_TARGETS_F AS<br>SELECT <br>(fn_extract_xml_value(&#39;OFFICE_KEY&#39;, xml_data))::int AS OFFICE_KEY,<br>(fn_extract_xml_value(&#39;PROD_KEY&#39;, xml_data))::int AS PROD_KEY,<br>(fn_extract_xml_value(&#39;BILL_MTH_KEY&#39;, xml_data))::int AS BILL_MTH_KEY,<br>(fn_extract_xml_value(&#39;REVENUE&#39;, xml_data))::float8 AS REVENUE,<br>(fn_extract_xml_value(&#39;UNITS&#39;, xml_data))::int AS UNITS<br>FROM SAMP_TARGETS_F_T;<br>Step 6:- Configure HAWQ to allow connectivity from OBIEE instance</pre><pre>Enable connectivity from OBIEE instance by changing configuration file named pg_hba.conf as shown below</pre><pre>[gpadmin@sandbox gpseg-1]$ pwd<br>/data/hawq/master/gpseg-1<br>[gpadmin@sandbox gpseg-1]$ tail -30 pg_hba.conf<br># special characters must be quoted. Quoting one of the keywords<br># &quot;all&quot;, &quot;sameuser&quot;, &quot;samerole&quot; or &quot;replication&quot; makes the name lose<br># its special character, and just match a database or username with<br># that name.<br>#<br># This file is read on server startup and when the postmaster receives<br># a SIGHUP signal. If you edit the file on a running system, you have<br># to SIGHUP the postmaster for the changes to take effect. You can<br># use &quot;pg_ctl reload&quot; to do that.<br># Put your actual configuration here<br># ----------------------------------<br>#<br># If you want to allow non-local connections, you need to add more<br># &quot;host&quot; records. In that case you will also need to make PostgreSQL<br># listen on a non-local interface via the listen_addresses<br># configuration parameter, or via the -i or -h command line switches.<br># CAUTION: Configuring the system for local &quot;trust&quot; authentication allows<br># any local user to connect as any PostgreSQL user, including the database<br># superuser. If you do not trust all your local users, use another<br># authentication method.<br># TYPE DATABASE USER CIDR-ADDRESS METHOD<br># &quot;local&quot; is for Unix domain socket connections only<br># IPv4 local connections:<br># IPv6 local connections:<br>local all gpadmin ident<br>host all gpadmin 127.0.0.1/28 trust<br>host all gpadmin 10.0.0.4/32 trust<br>host all gpadmin ::1/128 trust<br>host all gpadmin fe80::20d:3aff:fe10:dee3/128 trust<br>host all gpadmin 0.0.0.0/0 password</pre><h4>Setting up SampleAppLite RPD for HAWQ</h4><pre>Step 1:- Setup OBIEE to HAWQ connectivity</pre><pre>Connectivity can be established by adding the below entry in odbc.ini file of OBIEE instance</pre><pre>[ODBC Data Sources]<br>HAWQ_DSN=DataDirect 7.1 Greenplum Wire Protocol</pre><pre>[HAWQ_DSN]<br>Driver=/app/oracle/biee/Oracle_BI1/common/ODBC/Merant/7.1.4/lib/ARgplm27.so<br>Description=DataDirect 7.1 Greenplum Wire Protocol<br>AlternateServers=<br>ApplicationUsingThreads=1<br>ConnectionReset=0<br>ConnectionRetryCount=0<br>ConnectionRetryDelay=3<br><strong>Database=gpadmin</strong><br>DefaultLongDataBuffLen=2048<br>EnableDescribeParam=0<br>EnableKeysetCursors=0<br>EncryptionMethod=0<br>ExtendedColumnMetadata=0<br>FailoverGranularity=0<br>FailoverMode=0<br>FailoverPreconnect=0<br>FetchTSWTZasTimestamp=0<br>FetchTWFSasTime=0<br><strong>HostName=&lt;hostname&gt;</strong><br>InitializationString=<br>KeyPassword=<br>KeysetCursorOptions=0<br>KeyStore=<br>KeyStorePassword=<br>LoadBalanceTimeout=0<br>LoadBalancing=0<br>LoginTimeout=15<br>LogonID=<br>MaxPoolSize=100<br>MinPoolSize=0<br>Password=<br>Pooling=0<br><strong>PortNumber=15432</strong><br>QueryTimeout=0<br>ReportCodepageConversionErrors=0<br>TransactionErrorBehavior=1<br>XMLDescribeType=-10<br>Please note that the &quot;DataDirect 7.1 Greenplum Wire Protocol&quot; drivers that are available along with the installation of OBIEE can be utilized for HAWQ connectivity</pre><pre>Step 2:- Change the SampleAppLite RPD connection pool to utilize HAWQ DSN</pre><pre>Shown below are RPD and HAWQ DSN setup on Windows server</pre><figure><img alt="RPD with HAWQ Connection Pool" src="https://cdn-images-1.medium.com/max/1024/0*ddQnRuEThAXj7HAA.png" /></figure><figure><img alt="ODBC Driver Manager" src="https://cdn-images-1.medium.com/max/819/0*pCVBLXMn3H0sbyw4.png" /></figure><figure><img alt="HAWQ_DSN_Windows" src="https://cdn-images-1.medium.com/max/819/0*kWWKAE7Mj0RMKiM0.png" /></figure><pre>Step 3:- Deploy the RPD and execute SampleAppLite dashboards</pre><figure><img alt="Dashboard Run" src="https://cdn-images-1.medium.com/max/1024/0*dPvAY88M9A7goh4E.png" /></figure><figure><img alt="Dashboard_Executed" src="https://cdn-images-1.medium.com/max/1024/0*60IagIvxWY8Hp_kZ.png" /></figure><figure><img alt="HAWQ Query OBIEE" src="https://cdn-images-1.medium.com/max/1024/0*zSM3JkQurwQGSh5r.png" /></figure><pre>As shown above, SQL is being sent from OBIEE to HAWQ. The is being processed by HAWQ and results from XML processed files stored in HDFS are displayed in OBIEE.</pre><pre>Conclusion</pre><pre>That’s all folks! I hope you enjoyed the guide. HAWQ opens up interesting ways to use HDFS as data store for variety of enterprise reporting requirements.Please feel free to post questions in the comments section.</pre><h4>Reference Blogs</h4><pre><a href="http://www.pivotalguru.com/?p=782">http://www.pivotalguru.com/?p=782</a></pre><pre><a href="http://blog.pivotal.io/big-data-pivotal/p-o-v/pivotal-hawq-lands-in-the-hortonworks-sandbox">http://blog.pivotal.io/big-data-pivotal/p-o-v/pivotal-hawq-lands-in-the-hortonworks-sandbox</a></pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=aa3c18778273" width="1" height="1" alt=""><hr><p><a href="https://medium.com/sasikanth-kotti/obiee-and-pivotal-hawq-aa3c18778273">OBIEE and Pivotal HAWQ</a> was originally published in <a href="https://medium.com/sasikanth-kotti">Sasikanth Kotti</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Running OBIEE 11g inside Docker]]></title>
            <link>https://medium.com/sasikanth-kotti/running-obiee-11g-inside-docker-49af68c07a3f?source=rss-5be5ec29922c------2</link>
            <guid isPermaLink="false">https://medium.com/p/49af68c07a3f</guid>
            <category><![CDATA[business-intelligence]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[containers]]></category>
            <category><![CDATA[obiee]]></category>
            <dc:creator><![CDATA[Kotti Sasikanth]]></dc:creator>
            <pubDate>Mon, 13 Apr 2015 07:35:43 GMT</pubDate>
            <atom:updated>2017-11-21T04:43:27.236Z</atom:updated>
            <content:encoded><![CDATA[<p>Oracle Business Intelligence Enterprise Edition 11g (OBIEE) is a comprehensive business intelligence platform that delivers a full range of capabilities — including interactive dashboards, ad hoc queries, notifications and alerts, enterprise and financial reporting, scorecard and strategy management, business process invocation, search and collaboration, mobile, integrated systems management and more.</p><p>Slowly enterprises are leaning towards deploying applications in the cloud.Few days agao, I have asked myself a question:Can I run OBIEE inside Docker as a container.The answer is yes we can setup and here’s how</p><h3>Creating the Docker images</h3><h4>Step 1: Preparing the docker host</h4><p>Before we attempt to build the image, configure the host so that docker uses a faster and bigger pool as below(https://jpetazzo.github.io/2014/01/29/docker-device-mapper-resize/):<br>1.Stop the Docker daemon<br>2.Wipe out /var/lib/docker<br>3.Create the storage directory: mkdir -p /var/lib/docker/devicemapper/devicemapper<br>4.Create a data symbolic link in that directory, pointing to the device:</p><pre>#ln -s /dev/xvdg /var/lib/docker/devicemapper/devicemapper</pre><pre>/var/lib/docker/devicemapper/devicemapper# ls -lrt<br> total 62856<br> lrwxrwxrwx 1 root root 9 Apr 5 19:44 data -&gt; /dev/xvdg<br> -rw------- 1 root root 2147483648 Apr 13 05:54 metadata<br> 5.Restart Docker<br> 6.Check with docker info that the Data Space Total value is correct.</pre><p>By default the docker creates images/containers who size is 10G.This will not help with building the image/containers running OBIEE as the setup files are huge.Googling for a solution turned up this page(http://blog.senthilthecoder.com/post/docker-devicemapper-imagesize/).It appears dm.basesize is the parameter that does the trick.<br>Below are the steps on how to increase dm.basesize parameter in Ubuntu.</p><p>1.Navigate to /etc/default<br>2.Add DOCKER_OPTS=” — storage-opt dm.basesize=60G” in docker.io file</p><pre># Docker Upstart and SysVinit configuration file<br> <br># Customize location of Docker binary (especially for development testing).<br>#DOCKER=&quot;/usr/local/bin/docker&quot;<br> <br># Use DOCKER_OPTS to modify the daemon startup options.<br>DOCKER_OPTS=&quot;--storage-opt dm.basesize=60G&quot;<br> <br># If you need Docker to use an HTTP proxy, it can also be specified here.<br>#export http_proxy=&quot;http://127.0.0.1:3128/&quot;<br> <br># This is also a handy place to tweak where Docker&#39;s temporary files go.<br>#export TMPDIR=&quot;/mnt/bigdrive/docker-tmp&quot;</pre><p>3.Restart Docker</p><h4>Step 2: Creating the docker image</h4><p>1.Setting up OBIEE 11g involves setting up RCU schemas in Oracle database<br>2.For this we can reuse existing docker image i.e. Oracle Express 11g R2 on Ubuntu 14.04.1 LTS in docker hub<br>3.Pull wnameless / oracle-xe-11g image and bring up docker container running oracle-xe-11g.(https://registry.hub.docker.com/u/wnameless/oracle-xe-11g/)<br>4.Obtain RCU from OTN(http://www.oracle.com/technetwork/middleware/bi-enterprise-edition/downloads/bi-downloads-1923016.html)<br>5.Run the RCU to create and populate DEV_BIPLATFORM and DEV_MDS schemas<br>6.Commit the container as a new image named obi_rep</p><pre>#sudo docker commit distracted_blackwell obi_rep</pre><p>7.Run a new container from obi_rep image</p><pre>#sudo docker run -d -p 49160:22 -p 49161:1521 --name obi_rep obi_rep</pre><p>8.Build the image for OBIEE using the Docker file.Please note the associated installation files for OBIEE and weblogic need to placed in the same directory<br>as that of Docker file.This directory also need to contain files for silent installation and configuration of weblogic and OBIEE i.e. <a href="https://github.com/ksasi/docker-obiee/blob/master/silent.xml">silent.xml</a>,<a href="https://github.com/ksasi/docker-obiee/blob/master/obi_install.rsp">obi_install.rsp</a>,<a href="https://github.com/ksasi/docker-obiee/blob/master/obi_config.rsp">obi_config.rsp</a> and <a href="https://github.com/ksasi/docker-obiee/blob/master/oraInst.loc">oraInst.loc</a></p><pre>#sudo docker build -t oracle_obi .</pre><p>9.During the above build oraclelinux official docker images are being used (https://registry.hub.docker.com/u/library/oraclelinux/). The above build is only limited to installation of OBIEE.Configuration happens when the container is created from the image.<br>10.Run the container from the above built image i.e. oracle_obi</p><pre>#sudo docker run --privileged -d -P --name obi_server -h OBIEE11117Linux -p 7001:7001 -p 9704:9704 --link obi_rep:obi_rep oracle_obi</pre><p>11.When the container starts the below configuration step gets executed.</p><pre>#CMD /opt/Middleware/weblogic/Oracle_BI1/bin/config.sh -silent -response /opt/Middleware/obi_config.rsp -ignoreSysPrereqs -waitforcompletion</pre><p>12.Inorder to bake the configuration permanently into the image, the commit the above container as a new image</p><pre>#sudo docker commit obi_server obi_server:version1</pre><p>13.The VIRTUAL SIZE of the image obi_server:version1 is nearly 39.6 GB<br>14.The size of the image can be reduced by following the blog post Shrinking Docker Images by Matt Williams (http://matthewkwilliams.com/index.php/2015/03/23/shrinking-docker-images/)<br>15.The name of the final image is obi_server:version2</p><h4>Step 3: Running the docker container</h4><p>1.Run the container from RCU database image</p><pre>#sudo docker run -d -p 49160:22 -p 49161:1521 --name obi_rep obi_rep</pre><p>2.Run the container from OBIEE 11g image</p><pre>#sudo docker run --privileged -d -P --name obi_server -h OBIEE11117Linux -p 7001:7001 -p 9704:9704 --link obi_rep:obi_rep oracle_obi:version2</pre><p>3.Login to the OBIEE11g container using nsenter (https://blog.docker.com/tag/nsenter/) to start weblogic and OBIEE11g services.</p><h4>Docker file</h4><pre>#Set the base image :<br>FROM oraclelinux:6.6</pre><pre>#File Author/Maintainer :<br>MAINTAINER Sasikanth Kotti &lt;kotti.sasikanth@gmail.com&gt;</pre><pre>#Update packages and install wget and man :<br>RUN yum update -y -q<br>RUN yum install -y -q wget man tar sudo zip unzip ping glibc make binutils gcc libaio bc perl binutils compat-libstd* elfutils-libelf elfutils-libelf-devel gcc gcc-c++<br>glibc glibc-common glibc-devel libaio libaio-devel libgcc libstdc++ libstd c++-devel make sysstat<br>RUN yum reinstall -y -q glibc-common<br>RUN localedef -c -i en_US -f UTF-8 en_US.UTF-8<br>RUN yum reinstall -y -q --exclude=filesystem-2.4.30-3.el6.x86_64 \*<br>RUN useradd obiee</pre><pre>#Set workdir to /opt/Middleware :<br>WORKDIR /opt/Middleware</pre><pre>#Obtain/download Java SE Development Kit 7u75 using wget :<br>RUN wget --no-check-certificate --no-cookies --header &quot;Cookie: oraclelicense=accept-securebackup-cookie&quot; <a href="http://download.oracle.com/otn-pub/java/jdk/7u71-b14/jdk-7u71-l">http://download.oracle.com/otn-pub/java/jdk/7u71-b14/jdk-7u71-l</a><br>inux-x64.tar.gz</pre><pre>#Set permissions to Middleware directory to enable obiee user perform operations :<br>RUN chmod 777 /opt/Middleware</pre><pre>#Set the username to obiee :<br>USER obiee</pre><pre>#Setup jdk1.7.0_71 and create softlink :<br>RUN gunzip jdk-7u71-linux-x64.tar.gz<br>RUN tar -xvf jdk-7u71-linux-x64.tar<br>RUN ln -s jdk1.7.0_71 current_java<br>RUN rm jdk-7u71-linux-x64.tar<br> <br>#Setup environment variables for java :<br>ENV JAVA_HOME /opt/Middleware/current_java<br>ENV PATH $PATH:/opt/Middleware/current_java:/opt/Middleware/current_java/bin<br> <br>#Add weblogic installation file and configuration file for silent installation :<br>ADD ./wls1036_generic.jar /opt/Middleware/<br>ADD ./silent.xml /opt/Middleware/<br> <br>#Silent Installation of weblogic :<br>RUN java -Xmx1024m -jar wls1036_generic.jar -mode=silent -silent_xml=/opt/Middleware/silent.xml<br> <br>ADD ./bi_linux_x86_111170_64_disk1_2of2.zip /opt/Middleware/<br>ADD ./bi_linux_x86_111170_64_disk1_1of2.zip /opt/Middleware/<br>ADD ./bi_linux_x86_111170_64_disk2_1of2.zip /opt/Middleware/<br>ADD ./bi_linux_x86_111170_64_disk2_2of2.zip /opt/Middleware/<br>ADD ./bi_linux_x86_111170_64_disk3.zip /opt/Middleware/<br>ADD ./obi_install.rsp /opt/Middleware/<br>ADD ./obi_config.rsp /opt/Middleware/<br>ADD ./oraInst.loc /opt/Middleware/<br> <br>#ADD ./oracle-xe-11.2.0-1.0.x86_64.rpm.zip /opt/Middleware/<br>#ADD ./xe.rsp /opt/Middleware/<br> <br>RUN unzip ./bi_linux_x86_111170_64_disk1_1of2.zip<br>RUN unzip ./bi_linux_x86_111170_64_disk1_2of2.zip<br>RUN unzip ./bi_linux_x86_111170_64_disk2_1of2.zip<br>RUN unzip ./bi_linux_x86_111170_64_disk2_2of2.zip<br>RUN unzip ./bi_linux_x86_111170_64_disk3.zip<br> <br>RUN rm bi_linux_x86_111170_64_disk1_1of2.zip bi_linux_x86_111170_64_disk1_2of2.zip bi_linux_x86_111170_64_disk2_1of2.zip bi_linux_x86_111170_64_disk2_2of2.zip bi_linux_x86_111170_64_disk3.zip<br> <br> <br>USER obiee<br>#Install OBIEE 11g<br>RUN /opt/Middleware/bishiphome/Disk1/runInstaller -silent -response /opt/Middleware/obi_install.rsp -invPtrLoc /opt/Middleware/oraInst.loc -ignoreSysPrereqs -waitforcompletion<br> <br>#Set the username to root and hostname to OBIEE11117Linux<br>USER root<br>ENV HOSTNAME OBIEE11117Linux<br>#RUN hostname OBIEE11117Linux<br>RUN hostname<br> <br>USER obiee<br>ENV LANG en_US.UTF-8<br> <br>CMD /opt/Middleware/weblogic/Oracle_BI1/bin/config.sh -silent -response /opt/Middleware/obi_config.rsp -ignoreSysPrereqs -waitforcompletion</pre><pre>Conclusion</pre><pre>That’s all folks! I hope you enjoyed the guide.Please feel free to post questions in the comments section.</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=49af68c07a3f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/sasikanth-kotti/running-obiee-11g-inside-docker-49af68c07a3f">Running OBIEE 11g inside Docker</a> was originally published in <a href="https://medium.com/sasikanth-kotti">Sasikanth Kotti</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Automatic memory allocation]]></title>
            <link>https://medium.com/sasikanth-kotti/automatic-memory-allocation-8dd9b5e79beb?source=rss-5be5ec29922c------2</link>
            <guid isPermaLink="false">https://medium.com/p/8dd9b5e79beb</guid>
            <category><![CDATA[data-center]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[rams]]></category>
            <category><![CDATA[memory-allocation]]></category>
            <dc:creator><![CDATA[Kotti Sasikanth]]></dc:creator>
            <pubDate>Sat, 09 Aug 2014 07:14:14 GMT</pubDate>
            <atom:updated>2017-11-21T04:44:50.491Z</atom:updated>
            <content:encoded><![CDATA[<h3>Automatic memory allocation — Thoughts</h3><p>In todays world ,due to the explosion of virtualization technology all the servers in most of the data centers are VMs i.e. Virtual Machines.There may be VMs running on a given physical machine along with a variety of applications running on each of the VMs. During the initial build process, an important parameter is the amount of RAM that can be allocated to each VM. The RAM allocated to each VM is static. This needs to be revisited periodically so as to increase/decrease the allocation per VM .The allocation depends on the usage of RAM,load on the VM,application type etc.However inorder to obtain better efficiencies, this RAM allocation per VM may be made automatic and dynamic.</p><p>To understand the above concept,lets assume that there are 2 VMs running on a given physical box.Total memory existing on the box is 5GB and each VM is allocated 2GB.In a given day, assuming the peak usage is at 1PM for one of the VMs and the other VM usage is almost idle.If the memory allocation is static, the RAM allocated to the idle VM is not being used effectively.However if the RAM allocation is automatic and dynamic, the busiest VM can get more RAM and the idle VM can get less RAM during peak usage period. And this can happen seamlessly in a virtualized environment.If this is implemented in a datacenter where there are thousands of servers, the efficiency achieved can be profound.Machine learning may be used for implementing automatic memory allocation discussed above.</p><p>The inspiration for the above thought/idea is obtained from the below white papers and blogs:</p><ol><li><a href="http://googleblog.blogspot.ca/2014/05/better-data-centers-through-machine.html">http://googleblog.blogspot.ca/2014/05/better-data-centers-through-machine.html</a></li><li><a href="https://code.facebook.com/posts/816473015039157/making-facebook-s-software-infrastructure-more-energy-efficient-with-autoscale/">https://code.facebook.com/posts/816473015039157/making-facebook-s-software-infrastructure-more-energy-efficient-with-autoscale/</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8dd9b5e79beb" width="1" height="1" alt=""><hr><p><a href="https://medium.com/sasikanth-kotti/automatic-memory-allocation-8dd9b5e79beb">Automatic memory allocation</a> was originally published in <a href="https://medium.com/sasikanth-kotti">Sasikanth Kotti</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>