<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Dereje Workneh on Medium]]></title>
        <description><![CDATA[Stories by Dereje Workneh on Medium]]></description>
        <link>https://medium.com/@derejeabera?source=rss-185df1f137b1------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*_F8HCVPvMz6Gvzpe.</url>
            <title>Stories by Dereje Workneh on Medium</title>
            <link>https://medium.com/@derejeabera?source=rss-185df1f137b1------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 22:25:45 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@derejeabera/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Why Machine Learning is a buzzword of the 21st Century]]></title>
            <link>https://medium.com/@derejeabera/why-machine-learning-is-a-buzzword-of-the-21st-century-d35015b88e97?source=rss-185df1f137b1------2</link>
            <guid isPermaLink="false">https://medium.com/p/d35015b88e97</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[innovation]]></category>
            <category><![CDATA[unsupervised-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[supervised-learning]]></category>
            <dc:creator><![CDATA[Dereje Workneh]]></dc:creator>
            <pubDate>Sat, 06 Jun 2020 14:04:01 GMT</pubDate>
            <atom:updated>2020-06-06T15:08:26.314Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/608/1*bU16NFe-6GkXUPW9J9-IDg.png" /></figure><p>In the old days, only companies had data. But nowadays, with the inventions of many things like the personal computer, high bandwidth networks, sensors, and social media, we generate huge data every micro-second and in everywhere. Whether we buy or sell a product, rent a movie, withdraw money from a bank, use a credit card, post on social media, write blogs etc. we create data. For a variety of purposes, we analyze those data in order to meet our goals. The technique used to analyze and manipulate the data is Machine Learning (ML).</p><p>Machine Learning is a class of techniques that can be used to analyze data or information in order to generalize and observe the patterns of that data or information. To predict the future value or behavior from those observations or patterns, it will then iteratively learn from data, unlike typical computer programs. The purpose of machine learning is to program computers to use sample data as a past experience or model, and use the patterns of this data to predict the future based on that data. Machine Learning does not only deal with database problems, it is also an application of artificial intelligence (AI) too. It helps us to find solutions to various problems in image recognition, voice or speech recognition, face recognition, biometrics authentication, medical diagnoses, agriculture, economics, computer networks, robotics, etc. Machine learning opens up a new era of computer science that often uses statistical techniques to give computers the ability to ”learn” with data, without being explicitly programmed.</p><p><strong>Types of ML</strong></p><p>The types and classification of Machine Learning Machine learning algorithms can be categorized according to the types of data, whether they give feedback or a learning signal. Four types of machine learning are:</p><p>1. Supervised machine learning — Supervised Learning uses labeled data to predict future label data or events with some given features. If the label data is continuous then it is called a regression problem and if label data is categorical it is a classification problem. Sometimes it is called binary classification.</p><p>2. Unsupervised machine learning — Unsupervised learning uses information that is neither classified nor labeled. This is called non-structure data. We can group similar types of things together from the data. We can call these either clustering or association problems.</p><p>3. Semi-supervised machine learning — When the data are a combination of supervised and unsupervised learning, mostly a large amount of unlabeled data and a smaller amount of label data, then the problem falls under the Semi-supervised learning category. Under this method, the system is able to considerably improve learning accuracy.</p><p>4. Reinforcement machine learning — Reinforcement learning interacts with environments and works through trial and error to find which actions provide the greatest rewards.</p><p><strong>Pros and Cons of ML</strong></p><p>Pros of ML:-</p><p>Machine learning applications are widely used in sectors such as financial, banking, healthcare, rental, education, economics, agriculture etc.</p><p>Giant tech companies are using machine learning to push relevant advertisements based on users past search behavior.</p><p>In dynamic or uncertain environments, machine learning can handle multidimensional and multi-variety of data.</p><p>Machine learning allows time cycle reduction and efficient utilization.</p><p>Machine learning includes processes that can lead to automation of tasks by increased uses of algorithms in different applications.</p><p>In machine learning, there are tools available to provide continuous quality improvements in large and complex process environments.</p><p>Cons of ML:</p><p>Acquisition is the major challenge of Machine Learning, for this data needs to be processed based on different algorithms before providing it as input to its respective algorithms. — Interpretation of results is also another major challenge to determine the effectiveness of machine learning.</p><p>Different machine learning techniques are tried based on which action or algorithm needs to be used and when it is to be used.</p><p>Machine learning needs lots of data.</p><p>A fixed pattern of data or historical data is another limitation of machine learning. As it works with statistical truths, it will be difficult to prove or predict anything without historical data.</p><p>Error diagnosis and correction is another limitation of machine learning.</p><p><strong>Summary</strong></p><p>In this fast living world, we need to manage all our work within a given time in this case if our system takes a few decisions to keep it updated with the resources is really necessary. Machine learning with the help of artificial intelligence solutions and other cognitive technologies makes it a new era in the field of development in computer science.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d35015b88e97" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Stock price prediction using ARIMA Model]]></title>
            <link>https://medium.com/@derejeabera/stock-price-prediction-using-arima-model-251ddb4ee52a?source=rss-185df1f137b1------2</link>
            <guid isPermaLink="false">https://medium.com/p/251ddb4ee52a</guid>
            <category><![CDATA[predictions]]></category>
            <category><![CDATA[time-series-model]]></category>
            <category><![CDATA[time-series-data]]></category>
            <category><![CDATA[stock-market]]></category>
            <dc:creator><![CDATA[Dereje Workneh]]></dc:creator>
            <pubDate>Sat, 06 Jun 2020 13:17:02 GMT</pubDate>
            <atom:updated>2020-06-06T13:17:02.996Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f5HauS_4NUgNXMmOXn2INA.jpeg" /></figure><p>Any kind of prediction is a difficult task in the real world, especially where the future is very dynamic. The stock market is highly volatile and unpredictable by nature. Therefore, investors are always taking risks in hopes of making a profit. People want to invest in the stock market and expect profit from their investments. There are many factors that influence stock prices, such as supply and demand, market trends, the global economy, corporate results, historical price, public sentiments, sensitive financial information, popularity (such as good or bad news related to a company name and product), all of which may result in an increase or decrease in the number of buyers etc. Even though one may analyze a lot of factors, it is still difficult to achieve a better performance in the stock market and to predict the future price in general. Predicting the price of a specific stock one day ahead is, by itself, a very complicated task. In this blogpost, next day stock prices are predicted for each of the individual days of a certain year. For each day, comparisons are made with the actual prices to validate the model. In this blogpost, we have been tasked with predicting the price of the Apple (AAPL) stock price and have been provided with historical data (time series data). This includes features like opening and closing stock prices, volume, date, and so on. A time series is a series of data that is collected over a period of time. Time series data are sequential data which follow some patterns. In order of time, data are points in an index or listed or graphed. Time series data are also called historical data or past data. Time series data are used for predicting a future value based on an historical value. This is called time series analysis. The daily closing price of stocks, heights of ocean tides, and counts of sunspots are some examples of time series data. Time series data are studied for several purposes, such as forecasting the future based on knowledge of the past, understanding of the phenomenon. Underlying measures, or simply succinctly describing the salient features of the series. Forecasting or predicting future prices of an observed time series plays an important role in nearly all fields of science, engineering, finance, business intelligence, economics, meteorology, telecommunications etc. To predict an outcome based on time series data, we can use a time series model which is called Auto Regressive Integrated Moving Average (ARIMA) is used as the machine learning technique to analyze and predict future stock prices based on historical prices.</p><p>Data Collection: — In this section, we will collect our data and load it into python. The data used for this blogpost was collected 5 years (2015–2020) of AAPL(Apple) Stock price data from Yahoo Finance, which you can <a href="https://finance.yahoo.com/quote/AAPL/history?period1=1432771200&amp;period2=1590624000&amp;interval=1d&amp;filter=history&amp;frequency=1d">download here</a>. We chose to use the Closing Value for our analysis. This is the workflow of the ARIMA model for this blogpost:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1016/1*6fObiWXMP7FDKzSIGXzb2g.png" /></figure><p>Let’s first import the required libraries :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/798/1*1-X9oKu-4vOXztBO9TFEgg.png" /></figure><p>Load the data</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/815/1*LFCPSTIEicHdimul-9M9-Q.png" /></figure><p><strong>Exploratory Data Analysis:</strong></p><p>Let’s check the data there is null or not and the shape of the data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-zMobOKP0kxnbE4J9HzElg.png" /></figure><p>Now, We have seen earlier that the data type for ‘Date’ is an object. So first of all we have to change the data type to datetime format otherwise we can not extract features from it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/814/1*mxTXvySHk6sQCvwwUCjxYA.png" /></figure><p>The closing price based on month</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*x3Z5lPI5CyyIaiUdt8Tksw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tDUwFhoXG-ilsAi9k8O4Mg.png" /><figcaption>Closing price based on closing price</figcaption></figure><p>We can see that there is an increase during the specified time frame.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7Ivd0NTFJOhGIgIhLAtDpA.png" /><figcaption>Bar graph</figcaption></figure><p>The correlation</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jir5DJa3W5wuaz5kaUF36A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HgHxmZFHclcPuepDi0qlSA.png" /></figure><p>Training and test datasets:- It is important we do not randomly pick training and testing datasets. In stock price prediction, we have to use the test data always the recent dataset give a better result for our prediction.</p><p>Training dataset is 80% of the total dataset while the test dataset the remaining 20%.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HWiFvWjDxRGpmrK-V1F-gQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jf2MM0bJ2JsuPxDi25Vb3Q.png" /><figcaption>Data distribution based on train and test datasets</figcaption></figure><h3>ARIMA model:</h3><p>ARIMA stands for Auto Regression Integrated Moving Average. It is specified by three ordered parameters (p,d,q). Where:</p><ul><li>p is the order of the autoregressive model(number of time lags)</li><li>d is the degree of differencing (number of times the data have had past values subtracted)</li><li>q is the order of moving average model. Before building an ARIMA model, we have to make sure our data is stationary.</li></ul><p>Before going to the ARIMA model, we have to make our data is stationarize. For a data to be stationarize:</p><ol><li>The mean of the series should not be a function of time.</li><li>The variance of the series should not be a function of time.</li><li>the covariance of the i th term and the (i + m) th term should not be a function of time.</li></ol><p>Because when running a linear regression the assumption is that all of the observations are all independent of each other. In a time series, however, we know that observations are time dependent. It turns out that a lot of nice results that hold for independent random variables (law of large numbers and central limit theorem to name a couple) hold for stationary random variables. So by making the data stationary, we can actually apply regression techniques to this time dependent variable.</p><p>There are two methods to check the stationarity of a time series. The first is by looking at the data. By visualizing the data it should be easy to identify a changing mean or variation in the data. For a more accurate assessment there is the Dickey-Fuller test. I won’t go into the specifics of this test, but if the ‘Test Statistic’ is greater than the ‘Critical Value’ than the time series is stationary. Below is code that will help you visualize the time series and test for stationarity.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8l864d1zIV0lr50mY4lrSQ.png" /></figure><p>We can easily see that the time series is not stationary, and our adf_test_stationarity function confirms what we see.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OyQm_fCTQ15celmo9wYmEg.png" /></figure><p>To transform our data more stationary . There are many transformations to stationarize our data such as deflation by CPI, logarithmic, first Difference, seasonal difference and seasonal adjustment and if you need to read more <a href="http://people.duke.edu/~rnau/whatuse.htm">here</a> is the link.</p><p>For this blogpost, we are using Plot the ACF and PACF charts and find the optimal parameters. The next step is to determine the tuning parameters of the model by looking at the autocorrelation and partial autocorrelation graphs. There are many rules and practice about how to select the appropriate AR, MA, SAR, and MAR terms for the model. The chart below provides a brief guide on how to read the autocorrelation and partial autocorrelation graphs to select the proper terms. The big issue as with all models is that you don’t want to overfit your model to the data by using too many terms.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/530/1*1bsHzEgC2Tsc911JffBuNg.png" /><figcaption>ACF plot</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/533/1*sTw4Ba31HTc1pIaRBfwH-w.png" /><figcaption>PACF plot</figcaption></figure><h3>Build Model:</h3><p>Below are the steps we follow for implementing auto ARIMA:</p><ol><li>Fit Auto ARIMA: Fit the model on the univariate series</li><li>Predict values on validation set: Make predictions on the validation set</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LJlsNec1Qh66lEUkGp1Wjw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bZk-V4H6h940DDSIa8hWcQ.png" /></figure><h3>Summary:</h3><p>The general steps to implement an ARIMA model are in time series data:–</p><ol><li><strong>Load the data:</strong> The first step for model building is of course to load the dataset</li><li><strong>Preprocessing:</strong> Depending on the dataset, the steps of preprocessing will be defined. This will include creating timestamps, converting the dtype of date/time column, making the series univariate, etc.</li><li><strong>Make series stationary:</strong> In order to satisfy the assumption, it is necessary to make the series stationary. This would include checking the stationarity of the series and performing required transformations.</li><li><strong>Determine d value:</strong> For making the series stationary, the number of times the difference operation was performed will be taken as the d value.</li><li><strong>Create ACF and PACF plots:</strong> This is the most important step in ARIMA implementation. ACF PACF plots are used to determine the input parameters for our ARIMA model.</li><li><strong>Determine the p and q values:</strong> Read the values of p and q from the plots in the previous step.</li><li><strong>Fit ARIMA model:</strong> Using the processed data and parameter values we calculated from the previous steps, fit the ARIMA model.</li><li><strong>Predict values on validation set:</strong> Predict the future values.</li></ol><h3>Final Words</h3><p>Thank you for the read and your time, if you like this story please hold the clap button. Also, I’ll be happy to share your feedback and see you the next blogpost!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=251ddb4ee52a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Predicting House Price in Ames, Iowa using Regression Model]]></title>
            <link>https://medium.com/@derejeabera/predicting-house-price-in-ames-iowa-using-regression-model-af0abe8f60e7?source=rss-185df1f137b1------2</link>
            <guid isPermaLink="false">https://medium.com/p/af0abe8f60e7</guid>
            <category><![CDATA[exploratory-data-analysis]]></category>
            <category><![CDATA[lasso-regression]]></category>
            <category><![CDATA[regression-modeling]]></category>
            <category><![CDATA[house-price-prediction]]></category>
            <category><![CDATA[regression]]></category>
            <dc:creator><![CDATA[Dereje Workneh]]></dc:creator>
            <pubDate>Mon, 18 May 2020 01:53:04 GMT</pubDate>
            <atom:updated>2020-05-18T01:53:04.567Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/473/1*q7M0eg2o5Q-gULrMPJMZWw.png" /></figure><p>As a data scientist working for a real estate investment firm interesting, our task is to develop a model to predict the selling price of a given home in Ames, Iowa from 2006 to 2010. Our employer hopes to use this information to help assess whether asking price of a house is higher or lower than the true value of the house. If the home is undervalued, it may be a good investment for the firm. One might wonder what drives the price of a house? The size of the house? Our motivation is to explore and analyze the house dataset to find the key features that influences the sale price and develop a model to predict in house price in Ames, Iowa.</p><p><strong>Dataset:</strong> As described on the Kaggle <a href="https://www.kaggle.com/c/dsi-us-11-project-2-regression-challenge/overview">competition</a> description page, the data for this project was compiled by Dean De Cock for educational purposes, and it includes 2930 observations and 82 predictor variables (house attributes) and one target variable (price). Data set contains information from the Ames Assessor’s Office used in computing assessed values for individual residential properties sold in Ames, IA from 2006 to 2010. The data has 82 columns which include 23 nominal, 23 ordinal, 14 discrete, and 20 continuous variables. result of the educational nature of the competition, the data was pre-split into a training set and a test set; the two datasets were given in the forms of csv files.</p><p>The objective of this project is to utilize stastical techniques models to predict the housing price for each home in the dataset. There are a clear that many predictors and different datasets, predicting response variables could fall under the following:</p><ul><li>location variables</li><li>age variables</li><li>lot/land variables</li><li>basement variables</li><li>roof variables</li><li>garage variables</li><li>kitchen variables</li><li>room/bathroom variables</li><li>utilities variables</li><li>external features (pools, porches, etc.) variables.</li></ul><p><strong>Work flow</strong> our steps towards creating a highly accurate model were as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*Iw6cj7HcbEvBRwQzv_IffA.png" /><figcaption>Work flow of predicting the house price of Ames, Iowa.</figcaption></figure><p><strong>Exploratory Data Analysis (EDA) and Data cleaning</strong></p><p>This is a step that we should always perform before trying to fit a model to the data, as this step we will often lead to important insight about our data.</p><p>The first step, we import libraries.</p><pre><strong>import</strong> <strong>warnings</strong><br><strong>import</strong> <strong>numpy</strong> <strong>as</strong> <strong>np</strong><br><strong>import</strong> <strong>pandas</strong> <strong>as</strong> <strong>pd</strong><br><strong>import</strong> <strong>matplotlib.pyplot</strong> <strong>as</strong> <strong>plt</strong><br><strong>import</strong> <strong>seaborn</strong> <strong>as</strong> <strong>sns</strong><br><strong>import</strong> <strong>mpl_toolkits</strong><br><br>warnings.filterwarnings(&#39;ignore&#39;)<br><strong>import</strong> <strong>scipy.stats</strong> <strong>as</strong> <strong>stats</strong><br><strong>from</strong> <strong>scipy.stats</strong> <strong>import</strong> skew,norm<br><strong>from</strong> <strong>scipy.stats.stats</strong> <strong>import</strong> pearsonr<br><strong>from</strong> <strong>sklearn.preprocessing</strong> <strong>import</strong> PolynomialFeatures, StandardScaler<br><strong>import</strong> <strong>copy</strong><br><br><br><br><em># Configure visual settings:</em><br>%matplotlib inline<br>sns.set_style(&#39;whitegrid&#39;)<br>sns.set_style(&#39;darkgrid&#39;)<br>plt.rcParams[&#39;figure.figsize&#39;] = (10.0, 8.0) <br>plt.style.use(&#39;ggplot&#39;)<br>np.random.seed(2018)<br>%load_ext autoreload<br>%autoreload 2<br>sns.set()</pre><p>Importing data from the datasets</p><pre>train = pd.read_csv(&#39;./datasets/train.csv&#39;)</pre><pre>test = pd.read_csv(&#39;./datasets/test.csv&#39;)</pre><p>Then we are going to check the number of rows and and columns of the train and test datasets.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RRFYLn32QZQGue1WuLTaCQ.png" /></figure><pre>train.dtypes</pre><pre>Id                int64<br>PID               int64<br>MS SubClass       int64<br>MS Zoning        object<br>Lot Frontage    float64<br>                 ...   <br>Misc Val          int64<br>Mo Sold           int64<br>Yr Sold           int64<br>Sale Type        object<br>SalePrice         int64<br>Length: 81, dtype: object</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*78mptsl94cCMETFuucCvZw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Q-WyUR29JeEtzWvTeSUONQ.png" /></figure><p>Let’s check the columns of train and columns datasets.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*twABii0CFNKCMbhShSxAhQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xa9acYOHIm-HWO52Ul8hnw.png" /></figure><p>Now, let us see the descriptive statistics of my data:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uDGdpFiCe2vyjk-pEM4Mhg.png" /></figure><p>The distribution of target variables with sales price</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1015/1*y9v8Rn8faQ2vf2e6NB44hw.png" /></figure><p>This graph showed that variables are right skewed(positive skewness) and show peakedness. As (linear) models fits better on normally distributed data , we require proper transformation. Therefore, we transform the skewed numeric features by taking log (feature +1 to make features more normal.</p><p>Some relational exploration for few selected numerical variables with outliers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Yz0zv8oiZG4SezzKp4Xwew.png" /><figcaption>Totalbsmtsf vs Salesprice</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NNiJrGavntBfJE6PVRYlMg.png" /><figcaption>GrLivArea vs SalesPrice</figcaption></figure><p>‘TotalBsmtSF’,’LotArea’ and ‘GrLivArea’ seem to be linearly related with ‘SalePrice’. Both relationships are positive, which means that as one variable increases, the other also increases. In the case of ‘TotalBsmtSF’ and ‘LotArea’ we can see that the slope of the linear relationship are particularly high.</p><p>Here is relational exploration for categorical features outliers. The OverallQual’ and ‘YearBuilt’ also seem to be related with ‘SalePrice’. The relationship seems to be stronger in the case of ‘OverallQual’, where the box plot shows how sales prices increase with the overall quality.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YkxD17YI4AMxEFxW6IcTHA.png" /><figcaption>Overllqual vs SalePrice</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fqSu9EMnfujC1t4u-alh8A.png" /><figcaption>Yearbuilt vs SalesPrice</figcaption></figure><p>Feature engineering: Undertaking the a natural log transformation of “SalesPrice”. The histogram of log sale prices definitely appeared more symmetrical with less extreme values. The resulting model is a log-linear model, that means a log dependent variable with linear explanatory variables.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ObOXFimJu-x6v6xc9uzDzQ.png" /><figcaption>Log transform SalesPrice distribution vs frequency</figcaption></figure><p>For regression models, we try to solve the following problem: given a processed list of features for a house, we would like to predict its potential sale price. Linear regression is a natural choice of baseline model for regression problems. So we first ran linear regression including all features. The model is then used to predict sale prices of houses given features in our test data and is compared to the actual sale prices of houses given in test data set. The performance was measured by Root Mean Square Error (rmse) of the predicted results and the actual results. Our baseline model generated a rmse of 0.87. Note that since the target variable SalePrice is log-transformed before model fitting, the resulting rmse is based on differences in the log-transformed sale prices, which accounts for the small values of rmse for regression models.</p><p>After using linear regression model as the baseline model, we included the regularization parameters in linear regression models to reduce overfitting. The two models are ridge and Lasso.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*C-anCszmsSMFFHayePBZLg.png" /><figcaption>Alpha vs rmse</figcaption></figure><p>we are implementing the ridge model, as the rmse decreases with increasing the number of features.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KQTeVFv-7u9Qki7XZDdV1w.png" /><figcaption>Coefficients of Selected in the Lasso Model</figcaption></figure><p>Coefficients of selected variables from the lasso model, and value of coefficients for each fitted covariate. The variable that has the greatest coefficient is GrLivArea (Continuous): Above grade (ground) living area square feet. This makes intuitive sense that the sale price of a real estate property is strongly correlated with its living area. The variable that has the greatest negative impact on the housing prices is functional_Sal: This offers a new perspective of the housing prices as well.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pvzACJrqCnj4GJv4cUfwNg.png" /><figcaption>Residuals from Lasso Model</figcaption></figure><p>A visual realization of residuals obtained from fitted lasso regression model, we can see that, for the predicted values, residuals are closely clustered around 0, which indicates that the model does not have significant misfit problem due to the randomly distributed residuals.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xrEpEruIjRkiDht9WEJsTA.png" /></figure><p>When compared with Ridge model, Lasso model’s train and test RMSE difference are smaller than Ridge model. This means Ridge model is more likely to be overfitted. Also, the RMSE of the Lasso model decreases much faster than the Ridge model. Because Lasso helped to the feature selection because it shrinks a relatively unimportant coefficient to zero.</p><p><strong>Conclusion and recommendations:</strong> The most valuable feature in this project is the GrLIvArea while functional_Sal and functional sale are least valuable features. In this project we are predicting the price of houses based the data using the linear regression models ridge and lasso. In ridge model, we are applying the alphas value which is between 0.05 to 100. The main tuning parameter for the ridge model is alpha-regularization parameter that measures how flexible our model is. The higher the regularization the less prone our model will be overfit. The result showed that lasso has a better performance than ridge because it shrinks a relatively unimportant coefficient to zero. Future work may compare Ames, Iowa to other housing markets or develop a model based on housing data from a larger region or at a national scope and data involving local policy changes and economic trends in the housing market. Adding more variables such as school zoning or transportation and commercial information would produce models with more predictive power. Additionally, time-series analysis to predict when is the best time (season) to buy a house.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=af0abe8f60e7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Pandas Merge DataFrames]]></title>
            <link>https://medium.com/@derejeabera/how-to-pandas-merge-dataframes-9bc69d248775?source=rss-185df1f137b1------2</link>
            <guid isPermaLink="false">https://medium.com/p/9bc69d248775</guid>
            <dc:creator><![CDATA[Dereje Workneh]]></dc:creator>
            <pubDate>Sun, 17 May 2020 03:26:42 GMT</pubDate>
            <atom:updated>2020-05-18T01:22:14.670Z</atom:updated>
            <content:encoded><![CDATA[<p>How to Merge DataFrames with Pandas</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/899/1*LNMs3u31tij3NZpmvd9ztg.png" /></figure><p>Due to the Python’s popularity in all directions of technologies in information technology is increasing from time to time. Therefore, mastering Python opens more opportunities in the industries. <a href="https://towardsdatascience.com/top-10-in-demand-programming-languages-to-learn-in-2020-4462eb7d8d3e">Python</a> is one of the most popular data science tools. One of the reasons for Python’s high popularity in data science is the pandas library.</p><p><strong>Pandas </strong>is a python library which provides high-performance, easy-to-use data structures and data analysis tool. It is the best data manipulation packages in Python, Pandas is the go-to package for intuitive and easy data analysis. The first time pandas was developed by Wes Mickinney (father of pandas) and released January 11, 2008. Like other python libraries pandas is used in many areas of academic and business such as finance, economics, statistics, etc.</p><p>During my stay in General Assembly Immersive course, I am so excited the beauty of pandas library in python. In this blog post, we will see how to use Pandas to merge different files format. The beauty of of pandas is remarkable number of files we can read in Python such as CSV, XLSX, ZIP, JSON, XML, HTML, Images, PDF are some of the common data formats.</p><p>Pandas give access for a simple way of combining together Series or DataFrames with different types of set logic for indexes.</p><p>In real world situations, the data that we want to use common in multiple files. We always need to combine these files into single DataFrame to analyze the data. The pandas package provides many methods for combining DataFrames including join, merge and concat.</p><p>To work through the examples below, we need to load the two files in two DataFrames.</p><pre><strong>import</strong> <strong>pandas</strong> <strong>as</strong> <strong>pd</strong></pre><p>We’ll be using the <a href="https://grouplens.org/datasets/movielens/">MovieLens</a> dataset for joining the two dataset. There are two CSVs (movies.csv and ratings.csv) that we&#39;ll eventually inner join.</p><pre>movies = pd.read_csv(&#39;../ml-latest-small/movies.csv&#39;)<br></pre><pre>ratings = pd.read_csv(&#39;../ml-latest-small/ratings.csv&#39;)<br></pre><p>Then we merge movies and ratings:</p><pre>df = pd.merge(ratings, movies, on=&#39;movieId&#39;)</pre><p>The pandas merge function supports two other join types:</p><ul><li>Right (outer) join: Invoked by passing how=&#39;right&#39; as an argument. Similar to a left join, except <em>all</em> rows from the right DataFrame are kept, while rows from the left DataFrame without matching join key(s) values are discarded.</li><li>Full (outer) join: Invoked by passing how=&#39;outer&#39; as an argument. This join type returns the all pairwise combinations of rows from both DataFrames; i.e., the result DataFrame will NaN where data is missing in one of the DataFrames. This join type is very rarely used.</li><li>Inner joins is the most common of join. An inner join combines two DataFrames based on a join key and returns a new DataFrame that contains <strong>only</strong> those rows that have matching values in <em>both</em> of the original DataFrames.</li><li>Left(outer)join: returns a DataFrame containing all rows of the left DataFrame. All the non-matching rows of the left DataFrame contain NaN for the columns in the right DataFrame. It is simply an inner join plus all the non-matching rows of the left DataFrame filled with NaN for columns of the right DataFrame.</li></ul><p>Concatenating DataFrames: We can use the concat function in pandas to append either columns or rows from one DataFrame to another. In other words concatenation operations along an axis while performing optional set logic (union or intersection) of indexes on the other axes. When we concatenate DataFrames, we need to specify the axis.</p><p><strong>Summery</strong></p><ul><li>Pandas’ merge and concat can be used to combine subsets of a DataFrame, or data from different files.</li><li>join function combines DataFrames based on index or columns</li><li>Joining two DataFrames can be done in multiple ways(left, right and inner) depending on what data must be in the final DataFrame.</li></ul><p>I hope this tip gives you a birds eye view of how to merge DataFrame in pandas. If you liked this blog post, please remember to check out all of the pandas <a href="https://pandas.pydata.org/pandas-docs/stable/">documentation</a> to see just how useful it can be to you!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9bc69d248775" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Long journey to become data scientist]]></title>
            <link>https://medium.com/@derejeabera/a-long-journey-to-become-data-scientist-112c380d8d32?source=rss-185df1f137b1------2</link>
            <guid isPermaLink="false">https://medium.com/p/112c380d8d32</guid>
            <category><![CDATA[data-scientist]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[physics]]></category>
            <dc:creator><![CDATA[Dereje Workneh]]></dc:creator>
            <pubDate>Sat, 16 May 2020 03:50:38 GMT</pubDate>
            <atom:updated>2020-05-16T04:33:25.387Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*bhmLzz-mMrQMgfKyWf4XTQ.png" /><figcaption>Physics: The Science of the Universe and Everything In It</figcaption></figure><p>The journey leading up to me move here is has been full of good and bad emotions for me, I crossed three continents and oceans to accomplish my dream.</p><p>By end of 2017, graduated from Addis Ababa University collaborated with University of Leeds, UK and University of Antwerp, Belgium Physics PhD , where I completed my thesis work in the three University laboratories entitled with “measuring concentration of ice nucleating particles in the atmosphere, particulate matters and gaseous pollutants in museums: insight from models and elemental analysis”. My thesis work focused on the study of aerosol nucleation dynamics and particulate matter formation in the atmosphere and their impact on environment. I have used several laser spectroscopy methods, instruments and simulation in the quantification and characterization of ice nucleating particles, particulate matter and gaseous pollutants. During PhD work I used different data sources like ground based instrument data, satellite data and aircraft data. I have also a strong background in computational science and well versed in various programming language for example Matlab, FORTRAN.</p><p>Before I joined, my role was as a teaching and research staff in NOAA center for atmospheric science and supervise graduate students at Howard University. I am also an adjunct professor of Physics at Montgomery College’s Science, engineering and Technology stream.</p><p>We are living the world of text message and video game. Informations are migrating all over the globe with out luggage and border checking, because we are living the fourth industrial age (information age). I always get a message from my former graduate and under graduate friends and asking me for advice on a career change to data science. I tell them to become a Physics Professor. when I went to any conference or training, everyone talk about data and want to be a data scientist, But while <strong>Harvard Business Review</strong> mentioned, data scientist job may be one of the sexiest job in the 21st century and one of the highest paid profession. Through time, they convinced me to shift to data science field. I am not new for data but I was using for academic purposes. In the meantime , I am reading and searching for a bootcamp where I learn to start a new skill. My search was successfully and getting a bootcamp around me which is General General Assembly’s name is one of the top ranked coding bootcamp by Course Report 2019–2020. And one of my friend also gave me a brief introduction about the curriculum and the job opportunity become General assembly’s Alumni (Data Science Immersive).</p><p>In this Data Science Immersive program, I hopefully to become a problem solver data scientist with strong background programming in Python, statistics and machine learning. My knowledge and skill will be an asset for my future employee company’s.</p><p>Here comes a bird eye-view of my journey how to join General Assembly, and become a data scientist.</p><h3>Final Words</h3><p>Thank you for the read, if you like this story please hold the clap button. Also, I’ll be happy to share your feedback.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=112c380d8d32" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>