<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Dhyey Joshi on Medium]]></title>
        <description><![CDATA[Stories by Dhyey Joshi on Medium]]></description>
        <link>https://medium.com/@dhyey-joshi12?source=rss-2ac64d1c310a------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*ZA-1rPiZVSM2qHHcIPwoag.jpeg</url>
            <title>Stories by Dhyey Joshi on Medium</title>
            <link>https://medium.com/@dhyey-joshi12?source=rss-2ac64d1c310a------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 16:23:07 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@dhyey-joshi12/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Boston Homes — (MLOps) Continuous Integration And Deployment (CI/CD) for ML Pipelines with GitHub…]]></title>
            <link>https://dhyey-joshi12.medium.com/boston-homes-mlops-continuous-integration-ci-cd-for-ml-pipelines-with-github-actions-and-791f5396162a?source=rss-2ac64d1c310a------2</link>
            <guid isPermaLink="false">https://medium.com/p/791f5396162a</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[heroku]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[github-actions]]></category>
            <dc:creator><![CDATA[Dhyey Joshi]]></dc:creator>
            <pubDate>Tue, 11 Apr 2023 12:42:19 GMT</pubDate>
            <atom:updated>2023-04-12T04:31:29.557Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Boston Homes — (MLOps) Continuous Integration And Deployment (CI/CD) for ML Pipelines with GitHub Actions and Heroku</strong></h3><h4>Build, train, and deploy machine learning models at scale</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*u501wVazprXACXvP.jpeg" /></figure><p><a href="https://dhyey-joshi12.medium.com/mlaas-ml-as-a-service-using-aws-sagemaker-to-solve-machine-learning-ai-problems-1210ecb7d568/">Previously</a>, I wrote a blog post that covered the topic of MLaas, which stands for machine learning as a service, specifically focusing on the services provided by AWS Sagemaker. The article discussed how AWS offers an easy-to-use platform for building, training, and deploying machine learning models through its resources and SageMaker services. An ensemble learning model was trained using the provided services to predict whether clients of a Portuguese banking institution would subscribe to a bank term deposit after multiple phone calls. In this blog post, we include a hands-on guide for building the model, creating a CI/CD pipeline using MLOps, Docker, and GitHub Actions, and deploying the application on Heroku. We will also apply the concepts learned to an original dataset called &quot;The Boston House Price.&quot;</p><p>Live Demo Website: <a href="https://bostonhomes.herokuapp.com/">https://bostonhomes.herokuapp.com/</a></p><p>(Please view the demonstration <a href="https://github.com/dhyeyjoshi/BostonHousePrice-Predictor/blob/main/Demo-Video.mp4">video </a>that will guide you on how to utilize this website and make a prediction on the price of a house in Boston.)</p><p>We will take the Housing dataset which contains information about different houses in Boston. This data was originally a part of the UCI Machine Learning Repository and has been removed now. We can also access this data from the sci-kit-learn library. There are 506 samples and 13 feature variables in this dataset. The objective is to predict the value of the prices of the house using the given features.</p><p>GitHub — Project Source Code: <a href="https://github.com/dhyeyjoshi/BostonHousePrice-Predictor/blob/main/Linear%20Regression%20ML%20Implementation.ipynb">Link</a></p><h3>PART-1: Finding A Better MODEL —</h3><p>So let’s get started.</p><p>First, we will import the required libraries.</p><pre>import pandas as pd<br>import numpy as np<br>import matplotlib.pyplot as plt<br>%matplotlib inline</pre><p>Next, we will load the housing data from the scikit-learn library and understand it.</p><p>We print the value of the boston_dataset to understand what it contains. print(boston_dataset.keys()) gives</p><pre>dict_keys([&#39;data&#39;, &#39;target&#39;, &#39;feature_names&#39;, &#39;DESCR&#39;])</pre><ul><li><em>data</em>: contains the information for various houses</li><li><em>target</em>: prices of the house</li><li><em>feature_names</em>: names of the features</li><li><em>DESCR</em>: describes the dataset</li></ul><p>To know more about the features used boston_dataset.DESCR A description of all the features is given below:</p><pre>CRIM: Per capita crime rate by town<br>ZN: Proportion of residential land zoned for lots over 25,000 sq. ft<br>INDUS: Proportion of non-retail business acres per town<br>CHAS: Charles River dummy variable (= 1 if tract bounds river; 0 otherwise)<br>NOX: Nitric oxide concentration (parts per 10 million)<br>RM: Average number of rooms per dwelling<br>AGE: Proportion of owner-occupied units built prior to 1940<br>DIS: Weighted distances to five Boston employment centers<br>RAD: Index of accessibility to radial highways<br>TAX: Full-value property tax rate per $10,000<br>PTRATIO: Pupil-teacher ratio by town<br>B: 1000(Bk — 0.63)², where Bk is the proportion of [people of African American descent] by town<br>LSTAT: Percentage of lower status of the population<br>MEDV: Median value of owner-occupied homes in $1000s</pre><p>The prices of the house indicated by the variable MEDV are our <strong><em>target variables</em></strong> and the remaining are the <strong><em>feature variables</em></strong> based on which we will predict the value of a house.</p><p>We will now load the data into a pandas data frame using pd.DataFrame. We then print the first 5 rows of the data using head()</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/830/1*X7iCv2JNPQFWRy2SBJKRsQ.png" /></figure><p>We can see that the target value MEDV is missing from the data. We create a new column of target values and add it to the data frame.</p><h4><strong>Data Preprocessing</strong></h4><p>After loading the data, it’s a good practice to see if there are any missing values in the data. We count the number of missing values for each feature using isnull()</p><pre>boston.isnull().sum()</pre><p>However, there are no missing values in this dataset as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/187/1*IZl_w6N0soAHc4h5vHmRiw.png" /></figure><h4><strong>Exploratory Data Analysis</strong></h4><p>Exploratory Data Analysis is a very important step before training the model. In this section, we will use some visualizations to understand the relationship of the target variable with other features.</p><p>Let’s first plot the distribution of the target variable MEDV. We will use the distplot function from the seaborn library.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/693/0*pKY0kJ1c_WoLZif0.png" /></figure><p>We see that the values MEDV are distributed normally with few outliers.</p><pre>import seaborn as sns<br>sns.pairplot(dataset)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dvVI3JnEv3FGLJFrvaM19Q.png" /></figure><p>In order to gain a clearer understanding of this graphical illustration, we will generate several plots to compare it with the price characteristic.</p><pre>plt.scatter(dataset[&#39;CRIM&#39;],dataset[&#39;Price&#39;])<br>plt.xlabel(&quot;Crime Rate&quot;)<br>plt.ylabel(&quot;Price&quot;)<br></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/382/1*TQ0NgoP6u0n6oWk03SY3VQ.png" /></figure><pre>import seaborn as sns<br>sns.regplot(x=&quot;RM&quot;,y=&quot;Price&quot;,data=dataset)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/383/1*3E22Lc2mTUKU4Na0p_wndw.png" /></figure><pre>sns.regplot(x=&quot;LSTAT&quot;,y=&quot;Price&quot;,data=dataset)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/382/1*V7rNYnVqIAUXckZ1rpf33g.png" /></figure><h4>Observations:</h4><ul><li>The prices increase as the value of RM increases linearly. There are few outliers and the data seems to be capped at 50.</li><li>The prices tend to decrease with an increase in LSTAT. Though it doesn’t look to be following exactly a linear line.</li></ul><pre>sns.regplot(x=&quot;PTRATIO&quot;,y=&quot;Price&quot;,data=dataset)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/382/1*p4C3EImiXJNQv7RSSd_cDw.png" /></figure><p>Next, we create a correlation matrix that measures the linear relationships between the variables. The correlation matrix can be formed by using the corr function from the panda&#39;s data frame library. We will use the heatmap function from the Seaborn library to plot the correlation matrix.</p><pre>correlation_matrix = boston.corr().round(2)<br># annot = True to print the values inside the square<br>sns.heatmap(data=correlation_matrix, annot=True)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/669/0*umo4h0V0ixpMpDmE.png" /></figure><p>The correlation coefficient ranges from -1 to 1. If the value is close to 1, it means that there is a strong positive correlation between the two variables. When it is close to -1, the variables have a strong negative correlation.</p><h4>Observations:</h4><ul><li>To fit a linear regression model, we select those features which have a high correlation with our target variable MEDV. By looking at the correlation matrix we can see that RM has a strong positive correlation with MEDV (0.7) whereas LSTAT has a high negative correlation with MEDV(-0.74).</li><li>An important point in selecting features for a linear regression model is to check for multi-co-linearity. The features RAD, TAX have a correlation of 0.91. These feature pairs are strongly correlated to each other. We should not select both these features together for training the model. Check this for an explanation. The same goes for the features DIS and AGE which have a correlation of -0.75.</li></ul><h4>Preparing the data for training the model</h4><p>While it is generally recommended to use only highly correlated columns with the target column, we will be using all available columns this time since we will be training our model using the XgBoost Algorithm.</p><pre>## Independent and Dependent features<br><br>X=dataset.iloc[:,:-1]<br>y=dataset.iloc[:,-1]</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/985/1*CpdOUMYN1DXXiqn8PBrKZg.png" /></figure><h4>Splitting the data into training and testing sets</h4><p>Next, we split the data into training and testing sets. We train the model with 70% of the samples and test with the remaining 30%. <em>We do this to assess the model’s performance on unseen data</em>. To split the data we use train_test_split the function provided by the sci-kit-learn library. We finally print the sizes of our training and test set to verify if the splitting has occurred properly.</p><pre>##Train Test Split<br>from sklearn.model_selection import train_test_split<br>X_train,X_test,y_train,y_test=train_test_split(X,y,test_size=0.3,random_state=42)</pre><p>Standardization of the dataset —</p><pre>## Standardize the dataset<br>from sklearn.preprocessing import StandardScaler<br>scaler=StandardScaler()<br><br>X_train=scaler.fit_transform(X_train)<br>X_test=scaler.transform(X_test)<br><br>print(X_train)<br>print(X_test)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/877/1*l5mkqd_rvk60Lt0TLPW3YQ.png" /></figure><h4>Training and testing the model</h4><p>I developed several regression models including <strong>Linear Regression</strong>, <strong>Gradient Boosting Algorithm</strong>, and <strong>Extreme Gradient Boosting Algorithm</strong> to train our model on both the training and test data sets. After comparing the performance of all these models, we found that the <strong>XGBoost Algorithm</strong> provided the <strong>highest accuracy</strong> and the<strong> lowest loss curve </strong>in predicting house prices.</p><pre>## Hyper Parameter Optimization<br><br>params={<br> &quot;learning_rate&quot;    : [0.05, 0.10, 0.15, 0.20, 0.25, 0.30 ] ,<br> &quot;max_depth&quot;        : [ 3, 4, 5, 6, 8, 10, 12, 15],<br> &quot;min_child_weight&quot; : [ 1, 3, 5, 7 ],<br> &quot;gamma&quot;            : [ 0.0, 0.1, 0.2 , 0.3, 0.4 ],<br> &quot;colsample_bytree&quot; : [ 0.3, 0.4, 0.5 , 0.7 ]<br>    <br>}</pre><pre>## Hyperparameter optimization using RandomizedSearchCV<br>from sklearn.model_selection import RandomizedSearchCV, GridSearchCV<br>import xgboost</pre><pre>def timer(start_time=None):<br>    if not start_time:<br>        start_time = datetime.now()<br>        return start_time<br>    elif start_time:<br>        thour, temp_sec = divmod((datetime.now() - start_time).total_seconds(), 3600)<br>        tmin, tsec = divmod(temp_sec, 60)<br>        print(&#39;\n Time taken: %i hours %i minutes and %s seconds.&#39; % (thour, tmin, round(tsec, 2)))</pre><pre>xgbc = xgboost.XGBRegressor()<br><br>random_search = RandomizedSearchCV( xgbc , param_distributions = params , n_iter=5 , scoring=&#39;neg_mean_squared_error&#39; , n_jobs=-1 , cv=5 , verbose = 3)<br><br>from datetime import datetime<br># Here we go<br>start_time = timer(None) # timing starts from this point for &quot;start_time&quot; variable<br>random_search.fit(X,y)<br>timer(start_time) # timing ends here for &quot;start_time&quot; variable</pre><pre>random_search.best_estimator_</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/772/1*zLvKbzIFjC9U67aQKs4LDA.png" /></figure><pre>random_search.best_params_</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/320/1*Gn57GUIiT4_rA0uVaYlF9g.png" /></figure><pre>from  xgboost import XGBRegressor<br>xgbc = XGBRegressor(base_score=0.5, booster=&#39;gbtree&#39;, colsample_bylevel=1,<br>             colsample_bynode=1, colsample_bytree=0.7, enable_categorical=False,<br>             gamma=0.4, gpu_id=-1, importance_type=None,<br>             interaction_constraints=&#39;&#39;, learning_rate=0.15, max_delta_step=0,<br>             max_depth=3, min_child_weight=5,<br>             monotone_constraints=&#39;()&#39;, n_estimators=100, n_jobs=8,<br>             num_parallel_tree=1, predictor=&#39;auto&#39;, random_state=0, reg_alpha=0,<br>             reg_lambda=1, scale_pos_weight=1, subsample=1, tree_method=&#39;exact&#39;,<br>             validate_parameters=1, verbosity=None)<br>xgbc.fit(X_train,y_train)</pre><pre>y_xgbcPred = xgbc.predict(X_test)</pre><pre>Trainscore = xgbc.score(X_train,y_train)*100<br>Testscore  = xgbc.score(X_test,y_test)*100<br>print(&quot;Training Score  :&quot; , Trainscore)<br>print(&quot;Testing Score  :&quot; , Testscore)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/282/1*WefEXojcWSrKYoaguL4u1w.png" /></figure><pre>from sklearn.model_selection import cross_val_score<br>score = cross_val_score(xgbc,X_train,y_train,cv=10)<br>print(&quot;Cross Validation Average Score : &quot; , score.mean())</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/397/1*amxpoATW1niPk-np3QwELg.png" /></figure><h4>Residual Values (Residuals) in Regression Analysis</h4><pre>## Residuals<br>residuals_gbr=y_test-y_gbrPred<br><br>residuals_gbr</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/424/1*5w2iJDGfJSxyupMWIMAgDQ.png" /></figure><pre>## Plot this residuals <br><br>sns.displot(residuals_gbr,kind=&quot;kde&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/352/1*4_tGOX5VzudK1-m2kGBEfg.png" /></figure><pre>## Scatter plot with respect to prediction and residuals<br>## uniform distribution<br>plt.scatter(y_gbrPred,residuals)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/377/1*98gl-MZkxMLiPEmWGHlsAg.png" /></figure><h4>Pickling file for Model Deployment</h4><p>Pickling a file refers to the process of serializing a Python object hierarchy into a byte stream, which can be later deserialized to recreate the original object. In the context of model deployment, pickling a trained machine learning model allows us to save it to a file and later load it into memory to make predictions on new data without having to retrain the model. This can save time and computational resources, making it a common practice for deploying machine learning models in production environments.</p><pre>import pickle<br>pickle.dump(scaler,open(&#39;scaling.pkl&#39;,&#39;wb&#39;))<br><br>pickle.dump(GBR,open(&#39;regmodel.pkl&#39;,&#39;wb&#39;))<br></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/709/1*KEZTKSCNwyUWOM9CXwj7Gw.png" /></figure><h3>PART-2: MLOps Pipeline with GitHub Actions and Heroku —</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HFz7YCEEWdb8RMc3vRCz9Q.jpeg" /></figure><p>You’ll deploy your Flask Application to Heroku with GitHub Actions.</p><p>You’ll implement the MLOps pipeline, Continuous Integration, and Continuous Deployment with easy-to-follow, step-by-step explanations.</p><p>You’ll love it.</p><h4>CONTENTS</h4><p><a href="#f430"><strong>Introduction</strong></a></p><p><a href="#4e2c"><strong>CI (Continuous Integration) Part</strong></a></p><p><a href="#5f12"><strong>CD (Continuous Deployment) Part</strong></a></p><p><a href="#4aea"><strong>Conclusion</strong></a></p><p><a href="#4613"><strong>Code</strong></a></p><h4>Introduction</h4><p>In this article, we will discuss how to implement an end-to-end MLOps pipeline using GitHub Actions and Heroku.</p><p><strong>GitHub Actions</strong>:</p><p>Based on the <a href="https://github.com/features/actions">GitHub Actions web page</a> :</p><blockquote><em>GitHub Actions makes it easy to automate all your software workflows, now with </em><strong><em>world-class CI/CD</em></strong><em>. Build, test, and deploy your code right from GitHub. Make code reviews, branch management, and issue triaging work the way you want.</em></blockquote><p>Let’s use it to get a world-class CI/CD pipeline for our ML model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/706/0*wFkQP1iCtoU5VqE0.png" /></figure><h4>CI (Continuous Integration) Part</h4><p>For this CI part, I used the heart-failure dataset.</p><p>You can find a detailed discussion on <a href="#9abf"><strong>EDA and ML models here</strong>.</a></p><p>Based on the final XGboost model, a basic Flask App has been developed.</p><pre>import json<br>import pickle<br>import math <br><br>from flask import Flask,request,app,jsonify,url_for,render_template<br>import numpy as np<br>import pandas as pd<br><br>app=Flask(__name__)<br>## Load the model<br>regmodel=pickle.load(open(&#39;regmodel.pkl&#39;,&#39;rb&#39;))<br>scalar=pickle.load(open(&#39;scaling.pkl&#39;,&#39;rb&#39;))<br>@app.route(&#39;/&#39;)<br>def home():<br>    return render_template(&#39;home.html&#39;)<br>@app.route(&#39;/templates/property-grid.html&#39;)<br>def property():<br>    return render_template(&#39;property-grid.html&#39;)<br>@app.route(&#39;/templates/contact.html&#39;)<br>def contact():<br>    return render_template(&#39;contact.html&#39;)<br><br><br><br>@app.route(&#39;/predict_api&#39;,methods=[&#39;POST&#39;])<br>def predict_api():<br>    data=request.json[&#39;data&#39;]<br>    print(data)<br>    print(np.array(list(data.values())).reshape(1,-1))<br>    new_data=scalar.transform(np.array(list(data.values())).reshape(1,-1))<br>    output=regmodel.predict(new_data)<br>    print(output[0])<br>    return jsonify(output[0])<br><br>@app.route(&#39;/predict&#39;,methods=[&#39;POST&#39;])<br>def predict():<br>    data=[float(x) for x in request.form.values()]<br>    final_input=scalar.transform(np.array(data).reshape(1,-1))<br>    output=regmodel.predict(final_input)[0]<br>    output = math.ceil(output) <br>    return render_template(&quot;home.html&quot;,prediction_text=&quot;The House price prediction is {}$.&quot;.format(output))<br><br>if __name__==&quot;__main__&quot;:<br>    app.run(debug=True)</pre><p>And we need to add requirements.txt to be sure that the same libraries should be installed when deployment starts on the GitHub side.</p><pre>Flask<br>scikit-learn==1.0.1<br>pandas<br>numpy<br>matplotlib<br>gunicorn</pre><p>I have developed a demo website named “BostonHomes” for demonstration purposes, utilizing HTML, CSS, Javascript, and Bootstrap. The website has been saved in a<a href="https://github.com/dhyeyjoshi/BostonHousePrice-Predictor/tree/main/templates"> template folder</a>.</p><pre>&lt;!DOCTYPE html&gt;<br>&lt;html lang=&quot;en&quot;&gt;<br><br>&lt;head&gt;<br>  &lt;meta charset=&quot;utf-8&quot;&gt;<br>  &lt;meta content=&quot;width=device-width, initial-scale=1.0&quot; name=&quot;viewport&quot;&gt;<br><br>  &lt;title&gt;BostonHomes&lt;/title&gt;<br>  &lt;meta content=&quot;&quot; name=&quot;description&quot;&gt;<br>  &lt;meta content=&quot;&quot; name=&quot;keywords&quot;&gt;<br><br>  &lt;!-- Favicons --&gt;<br>  &lt;link href=&quot;/static/assets/img/favicon.png&quot; rel=&quot;icon&quot;&gt;<br>  &lt;link href=&quot;/static/assets/img/apple-touch-icon.png&quot; rel=&quot;apple-touch-icon&quot;&gt;<br><br>  &lt;!-- Google Fonts --&gt;<br>  &lt;link href=&quot;https://fonts.googleapis.com/css?family=Poppins:300,400,500,600,700&quot; rel=&quot;stylesheet&quot;&gt;<br><br>  &lt;!-- Vendor CSS Files --&gt;<br>  &lt;link href=&quot;/static/assets/vendor/animate.css/animate.min.css&quot; rel=&quot;stylesheet&quot;&gt;<br>  &lt;link href=&quot;/static/assets/vendor/bootstrap/css/bootstrap.min.css&quot; rel=&quot;stylesheet&quot;&gt;<br>  &lt;link href=&quot;/static/assets/vendor/bootstrap-icons/bootstrap-icons.css&quot; rel=&quot;stylesheet&quot;&gt;<br>  &lt;link href=&quot;/static/assets/vendor/swiper/swiper-bundle.min.css&quot; rel=&quot;stylesheet&quot;&gt;<br><br>  &lt;!-- Template Main CSS File --&gt;<br>  &lt;link href=&quot;/static/assets/css/style.css&quot; rel=&quot;stylesheet&quot;&gt;<br><br>&lt;/head&gt;<br><br>&lt;body&gt;<br><br>  &lt;!-- ======= Property Search Section ======= --&gt;<br><br>  &lt;!--/ Form Search Star /--&gt;<br>  &lt;div class=&quot;box-collapse&quot;&gt;<br>    &lt;span class=&quot;close-box-collapse right-boxed bi bi-x&quot;&gt;&lt;/span&gt;<br>    <br>    &lt;div class=&quot;title-box-d&quot;&gt;<br>      &lt;h3 class=&quot;title-d&quot;&gt;Find the Estimated Cost <br>of Your Dream Home &lt;/h3&gt;<br>    &lt;/div&gt;<br>    &lt;div class=&quot;box-collapse-wrap form&quot;&gt;<br>      &lt;form class=&quot;form-a&quot; action=&quot;{{ url_for(&#39;predict&#39;)}}&quot;method=&quot;post&quot;&gt;<br>        &lt;div class=&quot;row&quot;&gt;<br>          &lt;div class=&quot;col-md-12 mb-2&quot;&gt;<br>            &lt;div class=&quot;form-group&quot;&gt;<br>              &lt;label style=&quot;padding-bottom: 2%;&quot; for=&quot;Type&quot;&gt;CRIM - per capita crime rate by town&lt;/label&gt;<br>              &lt;input type=&quot;text&quot; name=&quot;CRIM&quot; placeholder=&quot;CRIM&quot; required=&quot;required&quot; class=&quot;form-control form-control-lg form-control-a&quot;&gt;<br>            &lt;/div&gt;<br>          &lt;/div&gt;<br>          &lt;div class=&quot;col-md-12 mb-2&quot;&gt;<br>            &lt;div class=&quot;form-group&quot;&gt;<br>              &lt;label style=&quot;padding-bottom: 2%;&quot; for=&quot;Type&quot;&gt;ZN - proportion of residential land zoned&lt;/label&gt;<br>              &lt;input type=&quot;text&quot; name=&quot;ZN&quot; placeholder=&quot;ZN&quot; required=&quot;required&quot; class=&quot;form-control form-control-lg form-control-a&quot;&gt;<br>            &lt;/div&gt;<br>          &lt;/div&gt;<br>          &lt;div class=&quot;col-md-12 mb-2&quot;&gt;<br>            &lt;div class=&quot;form-group&quot;&gt;<br>              &lt;label style=&quot;padding-bottom: 2%;&quot; for=&quot;Type&quot;&gt;INDUS - proportion of non-retail business acres per town&lt;/label&gt;<br>              &lt;input type=&quot;text&quot; name=&quot;INDUS&quot; placeholder=&quot;INDUS&quot; required=&quot;required&quot; class=&quot;form-control form-control-lg form-control-a&quot;&gt;<br>            &lt;/div&gt;<br>          &lt;/div&gt;<br>          &lt;div class=&quot;col-md-12 mb-2&quot;&gt;<br>            &lt;div class=&quot;form-group&quot;&gt;<br>              &lt;label style=&quot;padding-bottom: 2%;&quot; for=&quot;Type&quot;&gt;CHAS - Charles River dummy variable (1 if tract bounds river; 0 otherwise)&lt;/label&gt;<br>              &lt;input type=&quot;text&quot; name=&quot;CHAS&quot; placeholder=&quot;CHAS&quot; required=&quot;required&quot; class=&quot;form-control form-control-lg form-control-a&quot;&gt;<br>            &lt;/div&gt;<br>          &lt;/div&gt;<br>          &lt;div class=&quot;col-md-12 mb-2&quot;&gt;<br>            &lt;div class=&quot;form-group&quot;&gt;<br>              &lt;label style=&quot;padding-bottom: 2%;&quot; for=&quot;Type&quot;&gt;NOX - nitric oxides concentration (parts per 10 million)&lt;/label&gt;<br>              &lt;input type=&quot;text&quot; name=&quot;NOX&quot; placeholder=&quot;NOX&quot; required=&quot;required&quot; class=&quot;form-control form-control-lg form-control-a&quot;&gt;<br>            &lt;/div&gt;<br>          &lt;/div&gt;<br>          &lt;div class=&quot;col-md-12 mb-2&quot;&gt;<br>            &lt;div class=&quot;form-group&quot;&gt;<br>              &lt;label style=&quot;padding-bottom: 2%;&quot; for=&quot;Type&quot;&gt;RM - average number of rooms per dwelling&lt;/label&gt;<br>              &lt;input type=&quot;text&quot; name=&quot;RM&quot; placeholder=&quot;RM&quot; required=&quot;required&quot; class=&quot;form-control form-control-lg form-control-a&quot;&gt;<br>            &lt;/div&gt;<br>          &lt;/div&gt;<br>          &lt;div class=&quot;col-md-12 mb-2&quot;&gt;<br>            &lt;div class=&quot;form-group&quot;&gt;<br>              &lt;label style=&quot;padding-bottom: 2%;&quot; for=&quot;Type&quot;&gt;AGE - proportion of owner-occupied units built prior to 1940&lt;/label&gt;<br>              &lt;input type=&quot;text&quot; name=&quot;Age&quot; placeholder=&quot;Age&quot; required=&quot;required&quot; class=&quot;form-control form-control-lg form-control-a&quot;&gt;<br>            &lt;/div&gt;<br>          &lt;/div&gt;<br>          &lt;div class=&quot;col-md-12 mb-2&quot;&gt;<br>            &lt;div class=&quot;form-group&quot;&gt;<br>              &lt;label style=&quot;padding-bottom: 2%;&quot; for=&quot;Type&quot;&gt;DIS - weighted distances to five Boston employment centres&lt;/label&gt;<br>              &lt;input type=&quot;text&quot; name=&quot;DIS&quot; placeholder=&quot;DIS&quot; required=&quot;required&quot; class=&quot;form-control form-control-lg form-control-a&quot;&gt;<br>            &lt;/div&gt;<br>          &lt;/div&gt;<br>          &lt;div class=&quot;col-md-12 mb-2&quot;&gt;<br>            &lt;div class=&quot;form-group&quot;&gt;<br>              &lt;label style=&quot;padding-bottom: 2%;&quot; for=&quot;Type&quot;&gt;RAD - index of accessibility to radial highways&lt;/label&gt;<br>              &lt;input type=&quot;text&quot; name=&quot;RAD&quot; placeholder=&quot;RAD&quot; required=&quot;required&quot; class=&quot;form-control form-control-lg form-control-a&quot;&gt;<br>            &lt;/div&gt;<br>          &lt;/div&gt;<br>          &lt;div class=&quot;col-md-12 mb-2&quot;&gt;<br>            &lt;div class=&quot;form-group&quot;&gt;<br>              &lt;label style=&quot;padding-bottom: 2%;&quot; for=&quot;Type&quot;&gt;TAX - full-value property-tax rate per $10,000&lt;/label&gt;<br>              &lt;input type=&quot;text&quot; name=&quot;TAX&quot; placeholder=&quot;TAX&quot; required=&quot;required&quot; class=&quot;form-control form-control-lg form-control-a&quot;&gt;<br>            &lt;/div&gt;<br>          &lt;/div&gt;<br>          &lt;div class=&quot;col-md-12 mb-2&quot;&gt;<br>            &lt;div class=&quot;form-group&quot;&gt;<br>              &lt;label style=&quot;padding-bottom: 2%;&quot; for=&quot;Type&quot;&gt;PTRATIO - pupil-teacher ratio by town&lt;/label&gt;<br>              &lt;input type=&quot;text&quot; name=&quot;PTRATIO&quot; placeholder=&quot;PTRATIO&quot; required=&quot;required&quot; class=&quot;form-control form-control-lg form-control-a&quot;&gt;<br>            &lt;/div&gt;<br>          &lt;/div&gt;<br>          &lt;div class=&quot;col-md-12 mb-2&quot;&gt;<br>            &lt;div class=&quot;form-group&quot;&gt;<br>              &lt;label style=&quot;padding-bottom: 2%;&quot; for=&quot;Type&quot;&gt;B - 1000(Bk - 0.63)^2 where Bk is the proportion of blacks by town&lt;/label&gt;<br>              &lt;input type=&quot;text&quot; name=&quot;B&quot; placeholder=&quot;B&quot; required=&quot;required&quot; class=&quot;form-control form-control-lg form-control-a&quot;&gt;<br>            &lt;/div&gt;<br>          &lt;/div&gt;<br>          &lt;div class=&quot;col-md-12 mb-2&quot;&gt;<br>            &lt;div class=&quot;form-group&quot;&gt;<br>              &lt;label style=&quot;padding-bottom: 2%;&quot; for=&quot;Type&quot;&gt;LSTAT - % lower status of the population&lt;/label&gt;<br>              &lt;input type=&quot;text&quot; name=&quot;LSTAT&quot; placeholder=&quot;LSTAT&quot; required=&quot;required&quot; class=&quot;form-control form-control-lg form-control-a&quot;&gt;<br>            &lt;/div&gt;<br>          &lt;/div&gt;<br>          &amp;nbsp;        <br>          &lt;div class=&quot;col-md-12&quot;&gt;<br>            &lt;button  type=&quot;submit&quot; class=&quot;btn btn-b&quot;&gt;Estimate Price&lt;/button&gt;<br>          &lt;/div&gt;<br><br>          &lt;div class=&quot;title-box-d&quot;&gt;<br>            <br>          {{prediction_text}}<br>            <br>          &lt;/div&gt;<br>        &lt;/div&gt;<br>      &lt;/form&gt;<br>      <br>    &lt;/div&gt;<br>    <br>  &lt;/div&gt;&lt;!-- End Property Search Section --&gt;&gt;<br><br>  &lt;!-- ======= Header/Navbar ======= --&gt;<br>  &lt;nav class=&quot;navbar navbar-default navbar-trans navbar-expand-lg fixed-top&quot;&gt;<br>    &lt;div class=&quot;container&quot;&gt;<br>      &lt;button class=&quot;navbar-toggler collapsed&quot; type=&quot;button&quot; data-bs-toggle=&quot;collapse&quot; data-bs-target=&quot;#navbarDefault&quot; aria-controls=&quot;navbarDefault&quot; aria-expanded=&quot;false&quot; aria-label=&quot;Toggle navigation&quot;&gt;<br>        &lt;span&gt;&lt;/span&gt;<br>        &lt;span&gt;&lt;/span&gt;<br>        &lt;span&gt;&lt;/span&gt;<br>      &lt;/button&gt;<br>      &lt;a class=&quot;navbar-brand text-brand&quot; href=&quot;/&quot;&gt;Boston&lt;span class=&quot;color-b&quot;&gt;Homes&lt;/span&gt;&lt;/a&gt;<br><br>      &lt;div class=&quot;navbar-collapse collapse justify-content-center&quot; id=&quot;navbarDefault&quot;&gt;<br>        &lt;ul class=&quot;navbar-nav&quot;&gt;<br><br>          &lt;li class=&quot;nav-item&quot;&gt;<br>            &lt;a class=&quot;nav-link active&quot; href=&quot;/&quot; &gt;Home&lt;/a&gt;<br>          &lt;/li&gt;<br><br>          &lt;li class=&quot;nav-item&quot;&gt;<br>            &lt;a class=&quot;nav-link &quot; href=&quot;/templates/property-grid.html&quot;&gt;Property&lt;/a&gt;<br>          &lt;/li&gt;<br>          <br>          &lt;li class=&quot;nav-item&quot;&gt;<br>            &lt;a class=&quot;nav-link &quot; href=&quot;/templates/contact.html&quot;&gt;Contact&lt;/a&gt;<br>          &lt;/li&gt;<br>        &lt;/ul&gt;<br>      &lt;/div&gt;<br><br>      &lt;button type=&quot;button&quot; class=&quot;btn btn-b-n navbar-toggle-box navbar-toggle-box-collapse&quot; data-bs-toggle=&quot;collapse&quot; data-bs-target=&quot;#navbarTogglerDemo01&quot;&gt;<br>        &lt;i class=&quot;bi bi-search&quot;&gt;&lt;/i&gt;<br>      &lt;/button&gt;<br><br>    &lt;/div&gt;<br>  &lt;/nav&gt;&lt;!-- End Header/Navbar --&gt;<br><br>  &lt;!-- ======= Intro Section ======= --&gt;<br>  &lt;div class=&quot;intro intro-carousel swiper position-relative&quot;&gt;<br><br>    &lt;div class=&quot;swiper-wrapper&quot;&gt;<br><br>      &lt;div class=&quot;swiper-slide carousel-item-a intro-item bg-image&quot; style=&quot;background-image: url(https://medium.com/static/assets/img/slide-1.jpg)&quot;&gt;<br>        &lt;div class=&quot;overlay overlay-a&quot;&gt;&lt;/div&gt;<br>        &lt;div class=&quot;intro-content display-table&quot;&gt;<br>          &lt;div class=&quot;table-cell&quot;&gt;<br>            &lt;div class=&quot;container&quot;&gt;<br>              &lt;div class=&quot;row&quot;&gt;<br>                &lt;div class=&quot;col-lg-8&quot;&gt;<br>                  &lt;div class=&quot;intro-body&quot;&gt;<br>                    &lt;p class=&quot;intro-title-top&quot;&gt;Massachusetts,Boston<br>                      &lt;br&gt; 67542<br>                    &lt;/p&gt;<br>                    &lt;h1 class=&quot;intro-title mb-4 &quot;&gt;<br>                      &lt;span class=&quot;color-b&quot;&gt;205 &lt;/span&gt; Avalon Brighton<br>                      &lt;br&gt; Wharf Street<br>                    &lt;/h1&gt;<br>                    &lt;p class=&quot;intro-subtitle intro-price&quot;&gt;<br>                      &lt;a href=&quot;#&quot;&gt;&lt;span class=&quot;price-a&quot;&gt;rent | $ 12.000&lt;/span&gt;&lt;/a&gt;<br>                    &lt;/p&gt;<br>                  &lt;/div&gt;<br>                &lt;/div&gt;<br>              &lt;/div&gt;<br>            &lt;/div&gt;<br>          &lt;/div&gt;<br>        &lt;/div&gt;<br>      &lt;/div&gt;<br>      &lt;div class=&quot;swiper-slide carousel-item-a intro-item bg-image&quot; style=&quot;background-image: url(https://medium.com/static/assets/img/slide-2.jpg)&quot;&gt;<br>        &lt;div class=&quot;overlay overlay-a&quot;&gt;&lt;/div&gt;<br>        &lt;div class=&quot;intro-content display-table&quot;&gt;<br>          &lt;div class=&quot;table-cell&quot;&gt;<br>            &lt;div class=&quot;container&quot;&gt;<br>              &lt;div class=&quot;row&quot;&gt;<br>                &lt;div class=&quot;col-lg-8&quot;&gt;<br>                  &lt;div class=&quot;intro-body&quot;&gt;<br>                    &lt;p class=&quot;intro-title-top&quot;&gt;Massachusetts,Boston<br>                      &lt;br&gt; 78345<br>                    &lt;/p&gt;<br>                    &lt;h1 class=&quot;intro-title mb-4&quot;&gt;<br>                      &lt;span class=&quot;color-b&quot;&gt;108 &lt;/span&gt; Rino<br>                      &lt;br&gt; Venda Road Five<br>                    &lt;/h1&gt;<br>                    &lt;p class=&quot;intro-subtitle intro-price&quot;&gt;<br>                      &lt;a href=&quot;#&quot;&gt;&lt;span class=&quot;price-a&quot;&gt;rent | $ 22.000&lt;/span&gt;&lt;/a&gt;<br>                    &lt;/p&gt;<br>                  &lt;/div&gt;<br>                &lt;/div&gt;<br>              &lt;/div&gt;<br>            &lt;/div&gt;<br>          &lt;/div&gt;<br>        &lt;/div&gt;<br>      &lt;/div&gt;<br>      &lt;div class=&quot;swiper-slide carousel-item-a intro-item bg-image&quot; style=&quot;background-image: url(https://medium.com/static/assets/img/slide-3.jpg)&quot;&gt;<br>        &lt;div class=&quot;overlay overlay-a&quot;&gt;&lt;/div&gt;<br>        &lt;div class=&quot;intro-content display-table&quot;&gt;<br>          &lt;div class=&quot;table-cell&quot;&gt;<br>            &lt;div class=&quot;container&quot;&gt;<br>              &lt;div class=&quot;row&quot;&gt;<br>                &lt;div class=&quot;col-lg-8&quot;&gt;<br>                  &lt;div class=&quot;intro-body&quot;&gt;<br>                    &lt;p class=&quot;intro-title-top&quot;&gt;Massachusetts,Boston<br>                      &lt;br&gt; 34682<br>                    &lt;/p&gt;<br>                    &lt;h1 class=&quot;intro-title mb-4&quot;&gt;<br>                      &lt;span class=&quot;color-b&quot;&gt;102 &lt;/span&gt; Alira<br>                      &lt;br&gt; Roan Road One<br>                    &lt;/h1&gt;<br>                    &lt;p class=&quot;intro-subtitle intro-price&quot;&gt;<br>                      &lt;a href=&quot;#&quot;&gt;&lt;span class=&quot;price-a&quot;&gt;rent | $ 31.000&lt;/span&gt;&lt;/a&gt;<br>                    &lt;/p&gt;<br>                  &lt;/div&gt;<br>                &lt;/div&gt;<br>              &lt;/div&gt;<br>            &lt;/div&gt;<br>          &lt;/div&gt;<br>        &lt;/div&gt;<br>      &lt;/div&gt;<br>    &lt;/div&gt;<br>    &lt;div class=&quot;swiper-pagination&quot;&gt;&lt;/div&gt;<br>  &lt;/div&gt;&lt;!-- End Intro Section --&gt;<br><br>  &lt;main id=&quot;main&quot;&gt;<br><br>    &lt;!-- ======= Services Section ======= --&gt;<br>    &lt;section class=&quot;section-services section-t8&quot;&gt;<br>      &lt;div class=&quot;container&quot;&gt;<br>        &lt;div class=&quot;row&quot;&gt;<br>          &lt;div class=&quot;col-md-12&quot;&gt;<br>            &lt;div class=&quot;title-wrap d-flex justify-content-between&quot;&gt;<br>              &lt;div class=&quot;title-box&quot;&gt;<br>                &lt;h2 class=&quot;title-a&quot;&gt;Our Services&lt;/h2&gt;<br>              &lt;/div&gt;<br>            &lt;/div&gt;<br>          &lt;/div&gt;<br>        &lt;/div&gt;<br>        &lt;div class=&quot;row&quot;&gt;<br>          &lt;div class=&quot;col-md-4&quot;&gt;<br>            &lt;div class=&quot;card-box-c foo&quot;&gt;<br>              &lt;div class=&quot;card-header-c d-flex&quot;&gt;<br>                &lt;div class=&quot;card-box-ico&quot;&gt;<br>                  &lt;span class=&quot;bi bi-cart&quot;&gt;&lt;/span&gt;<br>                &lt;/div&gt;<br>                &lt;div class=&quot;card-title-c align-self-center&quot;&gt;<br>                  &lt;h2 class=&quot;title-c&quot;&gt;Lifestyle&lt;/h2&gt;<br>                &lt;/div&gt;<br>              &lt;/div&gt;<br>              &lt;div class=&quot;card-body-c&quot;&gt;<br>                &lt;p class=&quot;content-c&quot;&gt;<br>                  Sed porttitor lectus nibh. Cras ultricies ligula sed magna dictum porta. Praesent sapien massa,<br>                  convallis a pellentesque<br>                  nec, egestas non nisi.<br>                &lt;/p&gt;<br>              &lt;/div&gt;<br>              &lt;div class=&quot;card-footer-c&quot;&gt;<br>                &lt;a href=&quot;#&quot; class=&quot;link-c link-icon&quot;&gt;Read more<br>                  &lt;span class=&quot;bi bi-chevron-right&quot;&gt;&lt;/span&gt;<br>                &lt;/a&gt;<br>              &lt;/div&gt;<br>            &lt;/div&gt;<br>          &lt;/div&gt;<br>          &lt;div class=&quot;col-md-4&quot;&gt;<br>            &lt;div class=&quot;card-box-c foo&quot;&gt;<br>              &lt;div class=&quot;card-header-c d-flex&quot;&gt;<br>                &lt;div class=&quot;card-box-ico&quot;&gt;<br>                  &lt;span class=&quot;bi bi-calendar4-week&quot;&gt;&lt;/span&gt;<br>                &lt;/div&gt;<br>                &lt;div class=&quot;card-title-c align-self-center&quot;&gt;<br>                  &lt;h2 class=&quot;title-c&quot;&gt;Loans&lt;/h2&gt;<br>                &lt;/div&gt;<br>              &lt;/div&gt;<br>              &lt;div class=&quot;card-body-c&quot;&gt;<br>                &lt;p class=&quot;content-c&quot;&gt;<br>                  Nulla porttitor accumsan tincidunt. Curabitur aliquet quam id dui posuere blandit. Mauris blandit<br>                  aliquet elit, eget tincidunt<br>                  nibh pulvinar a.<br>                &lt;/p&gt;<br>              &lt;/div&gt;<br>              &lt;div class=&quot;card-footer-c&quot;&gt;<br>                &lt;a href=&quot;#&quot; class=&quot;link-c link-icon&quot;&gt;Read more<br>                  &lt;span class=&quot;bi bi-calendar4-week&quot;&gt;&lt;/span&gt;<br>                &lt;/a&gt;<br>              &lt;/div&gt;<br>            &lt;/div&gt;<br>          &lt;/div&gt;<br>          &lt;div class=&quot;col-md-4&quot;&gt;<br>            &lt;div class=&quot;card-box-c foo&quot;&gt;<br>              &lt;div class=&quot;card-header-c d-flex&quot;&gt;<br>                &lt;div class=&quot;card-box-ico&quot;&gt;<br>                  &lt;span class=&quot;bi bi-card-checklist&quot;&gt;&lt;/span&gt;<br>                &lt;/div&gt;<br>                &lt;div class=&quot;card-title-c align-self-center&quot;&gt;<br>                  &lt;h2 class=&quot;title-c&quot;&gt;Sell&lt;/h2&gt;<br>                &lt;/div&gt;<br>              &lt;/div&gt;<br>              &lt;div class=&quot;card-body-c&quot;&gt;<br>                &lt;p class=&quot;content-c&quot;&gt;<br>                  Sed porttitor lectus nibh. Cras ultricies ligula sed magna dictum porta. Praesent sapien massa,<br>                  convallis a pellentesque<br>                  nec, egestas non nisi.<br>                &lt;/p&gt;<br>              &lt;/div&gt;<br>              &lt;div class=&quot;card-footer-c&quot;&gt;<br>                &lt;a href=&quot;#&quot; class=&quot;link-c link-icon&quot;&gt;Read more<br>                  &lt;span class=&quot;bi bi-chevron-right&quot;&gt;&lt;/span&gt;<br>                &lt;/a&gt;<br>              &lt;/div&gt;<br>            &lt;/div&gt;<br>          &lt;/div&gt;<br>        &lt;/div&gt;<br>      &lt;/div&gt;<br>    &lt;/section&gt;&lt;!-- End Services Section --&gt;<br><br>    &lt;!-- ======= Latest Properties Section ======= --&gt;<br>    &lt;section class=&quot;section-property section-t8&quot;&gt;<br>      &lt;div class=&quot;container&quot;&gt;<br>        &lt;div class=&quot;row&quot;&gt;<br>          &lt;div class=&quot;col-md-12&quot;&gt;<br>            &lt;div class=&quot;title-wrap d-flex justify-content-between&quot;&gt;<br>              &lt;div class=&quot;title-box&quot;&gt;<br>                &lt;h2 class=&quot;title-a&quot;&gt;Latest Properties&lt;/h2&gt;<br>              &lt;/div&gt;<br>              &lt;div class=&quot;title-link&quot;&gt;<br>                &lt;a href=&quot;property-grid.html&quot;&gt;All Property<br>                  &lt;span class=&quot;bi bi-chevron-right&quot;&gt;&lt;/span&gt;<br>                &lt;/a&gt;<br>              &lt;/div&gt;<br>            &lt;/div&gt;<br>          &lt;/div&gt;<br>        &lt;/div&gt;<br><br>        &lt;div id=&quot;property-carousel&quot; class=&quot;swiper&quot;&gt;<br>          &lt;div class=&quot;swiper-wrapper&quot;&gt;<br><br>            &lt;div class=&quot;carousel-item-b swiper-slide&quot;&gt;<br>              &lt;div class=&quot;card-box-a card-shadow&quot;&gt;<br>                &lt;div class=&quot;img-box-a&quot;&gt;<br>                  &lt;img src=&quot;/static/assets/img/property-6.jpg&quot; alt=&quot;&quot; class=&quot;img-a img-fluid&quot;&gt;<br>                &lt;/div&gt;<br>                &lt;div class=&quot;card-overlay&quot;&gt;<br>                  &lt;div class=&quot;card-overlay-a-content&quot;&gt;<br>                    &lt;div class=&quot;card-header-a&quot;&gt;<br>                      &lt;h2 class=&quot;card-title-a&quot;&gt;<br>                        &lt;a href=&quot;property-single.html&quot;&gt;206 Mount<br>                          &lt;br /&gt; Olive Road Two&lt;/a&gt;<br>                      &lt;/h2&gt;<br>                    &lt;/div&gt;<br>                    &lt;div class=&quot;card-body-a&quot;&gt;<br>                      &lt;div class=&quot;price-box d-flex&quot;&gt;<br>                        &lt;span class=&quot;price-a&quot;&gt;rent | $ 12.000&lt;/span&gt;<br>                      &lt;/div&gt;<br>                      &lt;a href=&quot;#&quot; class=&quot;link-a&quot;&gt;Click here to view<br>                        &lt;span class=&quot;bi bi-chevron-right&quot;&gt;&lt;/span&gt;<br>                      &lt;/a&gt;<br>                    &lt;/div&gt;<br>                    &lt;div class=&quot;card-footer-a&quot;&gt;<br>                      &lt;ul class=&quot;card-info d-flex justify-content-around&quot;&gt;<br>                        &lt;li&gt;<br>                          &lt;h4 class=&quot;card-info-title&quot;&gt;Area&lt;/h4&gt;<br>                          &lt;span&gt;340m<br>                            &lt;sup&gt;2&lt;/sup&gt;<br>                          &lt;/span&gt;<br>                        &lt;/li&gt;<br>                        &lt;li&gt;<br>                          &lt;h4 class=&quot;card-info-title&quot;&gt;Beds&lt;/h4&gt;<br>                          &lt;span&gt;2&lt;/span&gt;<br>                        &lt;/li&gt;<br>                        &lt;li&gt;<br>                          &lt;h4 class=&quot;card-info-title&quot;&gt;Baths&lt;/h4&gt;<br>                          &lt;span&gt;4&lt;/span&gt;<br>                        &lt;/li&gt;<br>                        &lt;li&gt;<br>                          &lt;h4 class=&quot;card-info-title&quot;&gt;Garages&lt;/h4&gt;<br>                          &lt;span&gt;1&lt;/span&gt;<br>                        &lt;/li&gt;<br>                      &lt;/ul&gt;<br>                    &lt;/div&gt;<br>                  &lt;/div&gt;<br>                &lt;/div&gt;<br>              &lt;/div&gt;<br>            &lt;/div&gt;&lt;!-- End carousel item --&gt;<br><br>            &lt;div class=&quot;carousel-item-b swiper-slide&quot;&gt;<br>              &lt;div class=&quot;card-box-a card-shadow&quot;&gt;<br>                &lt;div class=&quot;img-box-a&quot;&gt;<br>                  &lt;img src=&quot;/static/assets/img/property-3.jpg&quot; alt=&quot;&quot; class=&quot;img-a img-fluid&quot;&gt;<br>                &lt;/div&gt;<br>                &lt;div class=&quot;card-overlay&quot;&gt;<br>                  &lt;div class=&quot;card-overlay-a-content&quot;&gt;<br>                    &lt;div class=&quot;card-header-a&quot;&gt;<br>                      &lt;h2 class=&quot;card-title-a&quot;&gt;<br>                        &lt;a href=&quot;property-single.html&quot;&gt;157 West<br>                          &lt;br /&gt; Central Park&lt;/a&gt;<br>                      &lt;/h2&gt;<br>                    &lt;/div&gt;<br>                    &lt;div class=&quot;card-body-a&quot;&gt;<br>                      &lt;div class=&quot;price-box d-flex&quot;&gt;<br>                        &lt;span class=&quot;price-a&quot;&gt;rent | $ 12.000&lt;/span&gt;<br>                      &lt;/div&gt;<br>                      &lt;a href=&quot;property-single.html&quot; class=&quot;link-a&quot;&gt;Click here to view<br>                        &lt;span class=&quot;bi bi-chevron-right&quot;&gt;&lt;/span&gt;<br>                      &lt;/a&gt;<br>                    &lt;/div&gt;<br>                    &lt;div class=&quot;card-footer-a&quot;&gt;<br>                      &lt;ul class=&quot;card-info d-flex justify-content-around&quot;&gt;<br>                        &lt;li&gt;<br>                          &lt;h4 class=&quot;card-info-title&quot;&gt;Area&lt;/h4&gt;<br>                          &lt;span&gt;340m<br>                            &lt;sup&gt;2&lt;/sup&gt;<br>                          &lt;/span&gt;<br>                        &lt;/li&gt;<br>                        &lt;li&gt;<br>                          &lt;h4 class=&quot;card-info-title&quot;&gt;Beds&lt;/h4&gt;<br>                          &lt;span&gt;2&lt;/span&gt;<br>                        &lt;/li&gt;<br>                        &lt;li&gt;<br>                          &lt;h4 class=&quot;card-info-title&quot;&gt;Baths&lt;/h4&gt;<br>                          &lt;span&gt;4&lt;/span&gt;<br>                        &lt;/li&gt;<br>                        &lt;li&gt;<br>                          &lt;h4 class=&quot;card-info-title&quot;&gt;Garages&lt;/h4&gt;<br>                          &lt;span&gt;1&lt;/span&gt;<br>                        &lt;/li&gt;<br>                      &lt;/ul&gt;<br>                    &lt;/div&gt;<br>                  &lt;/div&gt;<br>                &lt;/div&gt;<br>              &lt;/div&gt;<br>            &lt;/div&gt;&lt;!-- End carousel item --&gt;<br><br>            &lt;div class=&quot;carousel-item-b swiper-slide&quot;&gt;<br>              &lt;div class=&quot;card-box-a card-shadow&quot;&gt;<br>                &lt;div class=&quot;img-box-a&quot;&gt;<br>                  &lt;img src=&quot;/static/assets/img/property-7.jpg&quot; alt=&quot;&quot; class=&quot;img-a img-fluid&quot;&gt;<br>                &lt;/div&gt;<br>                &lt;div class=&quot;card-overlay&quot;&gt;<br>                  &lt;div class=&quot;card-overlay-a-content&quot;&gt;<br>                    &lt;div class=&quot;card-header-a&quot;&gt;<br>                      &lt;h2 class=&quot;card-title-a&quot;&gt;<br>                        &lt;a href=&quot;property-single.html&quot;&gt;245 Azabu<br>                          &lt;br /&gt; Nishi Park let&lt;/a&gt;<br>                      &lt;/h2&gt;<br>                    &lt;/div&gt;<br>                    &lt;div class=&quot;card-body-a&quot;&gt;<br>                      &lt;div class=&quot;price-box d-flex&quot;&gt;<br>                        &lt;span class=&quot;price-a&quot;&gt;rent | $ 12.000&lt;/span&gt;<br>                      &lt;/div&gt;<br>                      &lt;a href=&quot;property-single.html&quot; class=&quot;link-a&quot;&gt;Click here to view<br>                        &lt;span class=&quot;bi bi-chevron-right&quot;&gt;&lt;/span&gt;<br>                      &lt;/a&gt;<br>                    &lt;/div&gt;<br>                    &lt;div class=&quot;card-footer-a&quot;&gt;<br>                      &lt;ul class=&quot;card-info d-flex justify-content-around&quot;&gt;<br>                        &lt;li&gt;<br>                          &lt;h4 class=&quot;card-info-title&quot;&gt;Area&lt;/h4&gt;<br>                          &lt;span&gt;340m<br>                            &lt;sup&gt;2&lt;/sup&gt;<br>                          &lt;/span&gt;<br>                        &lt;/li&gt;<br>                        &lt;li&gt;<br>                          &lt;h4 class=&quot;card-info-title&quot;&gt;Beds&lt;/h4&gt;<br>                          &lt;span&gt;2&lt;/span&gt;<br>                        &lt;/li&gt;<br>                        &lt;li&gt;<br>                          &lt;h4 class=&quot;card-info-title&quot;&gt;Baths&lt;/h4&gt;<br>                          &lt;span&gt;4&lt;/span&gt;<br>                        &lt;/li&gt;<br>                        &lt;li&gt;<br>                          &lt;h4 class=&quot;card-info-title&quot;&gt;Garages&lt;/h4&gt;<br>                          &lt;span&gt;1&lt;/span&gt;<br>                        &lt;/li&gt;<br>                      &lt;/ul&gt;<br>                    &lt;/div&gt;<br>                  &lt;/div&gt;<br>                &lt;/div&gt;<br>              &lt;/div&gt;<br>            &lt;/div&gt;&lt;!-- End carousel item --&gt;<br><br>            &lt;div class=&quot;carousel-item-b swiper-slide&quot;&gt;<br>              &lt;div class=&quot;card-box-a card-shadow&quot;&gt;<br>                &lt;div class=&quot;img-box-a&quot;&gt;<br>                  &lt;img src=&quot;/static/assets/img/property-10.jpg&quot; alt=&quot;&quot; class=&quot;img-a img-fluid&quot;&gt;<br>                &lt;/div&gt;<br>                &lt;div class=&quot;card-overlay&quot;&gt;<br>                  &lt;div class=&quot;card-overlay-a-content&quot;&gt;<br>                    &lt;div class=&quot;card-header-a&quot;&gt;<br>                      &lt;h2 class=&quot;card-title-a&quot;&gt;<br>                        &lt;a href=&quot;property-single.html&quot;&gt;204 Montal<br>                          &lt;br /&gt; South Bela Two&lt;/a&gt;<br>                      &lt;/h2&gt;<br>                    &lt;/div&gt;<br>                    &lt;div class=&quot;card-body-a&quot;&gt;<br>                      &lt;div class=&quot;price-box d-flex&quot;&gt;<br>                        &lt;span class=&quot;price-a&quot;&gt;rent | $ 12.000&lt;/span&gt;<br>                      &lt;/div&gt;<br>                      &lt;a href=&quot;property-single.html&quot; class=&quot;link-a&quot;&gt;Click here to view<br>                        &lt;span class=&quot;bi bi-chevron-right&quot;&gt;&lt;/span&gt;<br>                      &lt;/a&gt;<br>                    &lt;/div&gt;<br>                    &lt;div class=&quot;card-footer-a&quot;&gt;<br>                      &lt;ul class=&quot;card-info d-flex justify-content-around&quot;&gt;<br>                        &lt;li&gt;<br>                          &lt;h4 class=&quot;card-info-title&quot;&gt;Area&lt;/h4&gt;<br>                          &lt;span&gt;340m<br>                            &lt;sup&gt;2&lt;/sup&gt;<br>                          &lt;/span&gt;<br>                        &lt;/li&gt;<br>                        &lt;li&gt;<br>                          &lt;h4 class=&quot;card-info-title&quot;&gt;Beds&lt;/h4&gt;<br>                          &lt;span&gt;2&lt;/span&gt;<br>                        &lt;/li&gt;<br>                        &lt;li&gt;<br>                          &lt;h4 class=&quot;card-info-title&quot;&gt;Baths&lt;/h4&gt;<br>                          &lt;span&gt;4&lt;/span&gt;<br>                        &lt;/li&gt;<br>                        &lt;li&gt;<br>                          &lt;h4 class=&quot;card-info-title&quot;&gt;Garages&lt;/h4&gt;<br>                          &lt;span&gt;1&lt;/span&gt;<br>                        &lt;/li&gt;<br>                      &lt;/ul&gt;<br>                    &lt;/div&gt;<br>                  &lt;/div&gt;<br>                &lt;/div&gt;<br>              &lt;/div&gt;<br>            &lt;/div&gt;&lt;!-- End carousel item --&gt;<br>          &lt;/div&gt;<br>        &lt;/div&gt;<br>        &lt;div class=&quot;propery-carousel-pagination carousel-pagination&quot;&gt;&lt;/div&gt;<br><br>      &lt;/div&gt;<br>    &lt;/section&gt;&lt;!-- End Latest Properties Section --&gt;<br><br>    &lt;!-- ======= Agents Section ======= --&gt;<br>    &lt;section class=&quot;section-agents section-t8&quot;&gt;<br>      &lt;div class=&quot;container&quot;&gt;<br>        &lt;div class=&quot;row&quot;&gt;<br>          &lt;div class=&quot;col-md-12&quot;&gt;<br>            &lt;div class=&quot;title-wrap d-flex justify-content-between&quot;&gt;<br>              &lt;div class=&quot;title-box&quot;&gt;<br>                &lt;h2 class=&quot;title-a&quot;&gt;Best Agents&lt;/h2&gt;<br>              &lt;/div&gt;<br>              &lt;div class=&quot;title-link&quot;&gt;<br>                &lt;a href=&quot;agents-grid.html&quot;&gt;All Agents<br>                  &lt;span class=&quot;bi bi-chevron-right&quot;&gt;&lt;/span&gt;<br>                &lt;/a&gt;<br>              &lt;/div&gt;<br>            &lt;/div&gt;<br>          &lt;/div&gt;<br>        &lt;/div&gt;<br>        &lt;div class=&quot;row&quot;&gt;<br>          &lt;div class=&quot;col-md-4&quot;&gt;<br>            &lt;div class=&quot;card-box-d&quot;&gt;<br>              &lt;div class=&quot;card-img-d&quot;&gt;<br>                &lt;img src=&quot;/static/assets/img/agent-4.jpg&quot; alt=&quot;&quot; class=&quot;img-d img-fluid&quot;&gt;<br>              &lt;/div&gt;<br>              &lt;div class=&quot;card-overlay card-overlay-hover&quot;&gt;<br>                &lt;div class=&quot;card-header-d&quot;&gt;<br>                  &lt;div class=&quot;card-title-d align-self-center&quot;&gt;<br>                    &lt;h3 class=&quot;title-d&quot;&gt;<br>                      &lt;a href=&quot;agent-single.html&quot; class=&quot;link-two&quot;&gt;Margaret Sotillo<br>                        &lt;br&gt; Escala&lt;/a&gt;<br>                    &lt;/h3&gt;<br>                  &lt;/div&gt;<br>                &lt;/div&gt;<br>                &lt;div class=&quot;card-body-d&quot;&gt;<br>                  &lt;p class=&quot;content-d color-text-a&quot;&gt;<br>                    Sed porttitor lectus nibh, Cras ultricies ligula sed magna dictum porta two.<br>                  &lt;/p&gt;<br>                  &lt;div class=&quot;info-agents color-a&quot;&gt;<br>                    &lt;p&gt;<br>                      &lt;strong&gt;Phone: &lt;/strong&gt; +54 356 945234<br>                    &lt;/p&gt;<br>                    &lt;p&gt;<br>                      &lt;strong&gt;Email: &lt;/strong&gt; agents@example.com<br>                    &lt;/p&gt;<br>                  &lt;/div&gt;<br>                &lt;/div&gt;<br>                &lt;div class=&quot;card-footer-d&quot;&gt;<br>                  &lt;div class=&quot;socials-footer d-flex justify-content-center&quot;&gt;<br>                    &lt;ul class=&quot;list-inline&quot;&gt;<br>                      &lt;li class=&quot;list-inline-item&quot;&gt;<br>                        &lt;a href=&quot;#&quot; class=&quot;link-one&quot;&gt;<br>                          &lt;i class=&quot;bi bi-facebook&quot; aria-hidden=&quot;true&quot;&gt;&lt;/i&gt;<br>                        &lt;/a&gt;<br>                      &lt;/li&gt;<br>                      &lt;li class=&quot;list-inline-item&quot;&gt;<br>                        &lt;a href=&quot;#&quot; class=&quot;link-one&quot;&gt;<br>                          &lt;i class=&quot;bi bi-twitter&quot; aria-hidden=&quot;true&quot;&gt;&lt;/i&gt;<br>                        &lt;/a&gt;<br>                      &lt;/li&gt;<br>                      &lt;li class=&quot;list-inline-item&quot;&gt;<br>                        &lt;a href=&quot;#&quot; class=&quot;link-one&quot;&gt;<br>                          &lt;i class=&quot;bi bi-instagram&quot; aria-hidden=&quot;true&quot;&gt;&lt;/i&gt;<br>                        &lt;/a&gt;<br>                      &lt;/li&gt;<br>                      &lt;li class=&quot;list-inline-item&quot;&gt;<br>                        &lt;a href=&quot;#&quot; class=&quot;link-one&quot;&gt;<br>                          &lt;i class=&quot;bi bi-linkedin&quot; aria-hidden=&quot;true&quot;&gt;&lt;/i&gt;<br>                        &lt;/a&gt;<br>                      &lt;/li&gt;<br>                    &lt;/ul&gt;<br>                  &lt;/div&gt;<br>                &lt;/div&gt;<br>              &lt;/div&gt;<br>            &lt;/div&gt;<br>          &lt;/div&gt;<br>          &lt;div class=&quot;col-md-4&quot;&gt;<br>            &lt;div class=&quot;card-box-d&quot;&gt;<br>              &lt;div class=&quot;card-img-d&quot;&gt;<br>                &lt;img src=&quot;/static/assets/img/agent-1.jpg&quot; alt=&quot;&quot; class=&quot;img-d img-fluid&quot;&gt;<br>              &lt;/div&gt;<br>              &lt;div class=&quot;card-overlay card-overlay-hover&quot;&gt;<br>                &lt;div class=&quot;card-header-d&quot;&gt;<br>                  &lt;div class=&quot;card-title-d align-self-center&quot;&gt;<br>                    &lt;h3 class=&quot;title-d&quot;&gt;<br>                      &lt;a href=&quot;agent-single.html&quot; class=&quot;link-two&quot;&gt;Stiven Spilver<br>                        &lt;br&gt; Darw&lt;/a&gt;<br>                    &lt;/h3&gt;<br>                  &lt;/div&gt;<br>                &lt;/div&gt;<br>                &lt;div class=&quot;card-body-d&quot;&gt;<br>                  &lt;p class=&quot;content-d color-text-a&quot;&gt;<br>                    Sed porttitor lectus nibh, Cras ultricies ligula sed magna dictum porta two.<br>                  &lt;/p&gt;<br>                  &lt;div class=&quot;info-agents color-a&quot;&gt;<br>                    &lt;p&gt;<br>                      &lt;strong&gt;Phone: &lt;/strong&gt; +54 356 945234<br>                    &lt;/p&gt;<br>                    &lt;p&gt;<br>                      &lt;strong&gt;Email: &lt;/strong&gt; agents@example.com<br>                    &lt;/p&gt;<br>                  &lt;/div&gt;<br>                &lt;/div&gt;<br>                &lt;div class=&quot;card-footer-d&quot;&gt;<br>                  &lt;div class=&quot;socials-footer d-flex justify-content-center&quot;&gt;<br>                    &lt;ul class=&quot;list-inline&quot;&gt;<br>                      &lt;li class=&quot;list-inline-item&quot;&gt;<br>                        &lt;a href=&quot;#&quot; class=&quot;link-one&quot;&gt;<br>                          &lt;i class=&quot;bi bi-facebook&quot; aria-hidden=&quot;true&quot;&gt;&lt;/i&gt;<br>                        &lt;/a&gt;<br>                      &lt;/li&gt;<br>                      &lt;li class=&quot;list-inline-item&quot;&gt;<br>                        &lt;a href=&quot;#&quot; class=&quot;link-one&quot;&gt;<br>                          &lt;i class=&quot;bi bi-twitter&quot; aria-hidden=&quot;true&quot;&gt;&lt;/i&gt;<br>                        &lt;/a&gt;<br>                      &lt;/li&gt;<br>                      &lt;li class=&quot;list-inline-item&quot;&gt;<br>                        &lt;a href=&quot;#&quot; class=&quot;link-one&quot;&gt;<br>                          &lt;i class=&quot;bi bi-instagram&quot; aria-hidden=&quot;true&quot;&gt;&lt;/i&gt;<br>                        &lt;/a&gt;<br>                      &lt;/li&gt;<br>                      &lt;li class=&quot;list-inline-item&quot;&gt;<br>                        &lt;a href=&quot;#&quot; class=&quot;link-one&quot;&gt;<br>                          &lt;i class=&quot;bi bi-linkedin&quot; aria-hidden=&quot;true&quot;&gt;&lt;/i&gt;<br>                        &lt;/a&gt;<br>                      &lt;/li&gt;<br>                    &lt;/ul&gt;<br>                  &lt;/div&gt;<br>                &lt;/div&gt;<br>              &lt;/div&gt;<br>            &lt;/div&gt;<br>          &lt;/div&gt;<br>          &lt;div class=&quot;col-md-4&quot;&gt;<br>            &lt;div class=&quot;card-box-d&quot;&gt;<br>              &lt;div class=&quot;card-img-d&quot;&gt;<br>                &lt;img src=&quot;/static/assets/img/agent-5.jpg&quot; alt=&quot;&quot; class=&quot;img-d img-fluid&quot;&gt;<br>              &lt;/div&gt;<br>              &lt;div class=&quot;card-overlay card-overlay-hover&quot;&gt;<br>                &lt;div class=&quot;card-header-d&quot;&gt;<br>                  &lt;div class=&quot;card-title-d align-self-center&quot;&gt;<br>                    &lt;h3 class=&quot;title-d&quot;&gt;<br>                      &lt;a href=&quot;agent-single.html&quot; class=&quot;link-two&quot;&gt;Emma Toledo<br>                        &lt;br&gt; Cascada&lt;/a&gt;<br>                    &lt;/h3&gt;<br>                  &lt;/div&gt;<br>                &lt;/div&gt;<br>                &lt;div class=&quot;card-body-d&quot;&gt;<br>                  &lt;p class=&quot;content-d color-text-a&quot;&gt;<br>                    Sed porttitor lectus nibh, Cras ultricies ligula sed magna dictum porta two.<br>                  &lt;/p&gt;<br>                  &lt;div class=&quot;info-agents color-a&quot;&gt;<br>                    &lt;p&gt;<br>                      &lt;strong&gt;Phone: &lt;/strong&gt; +54 356 945234<br>                    &lt;/p&gt;<br>                    &lt;p&gt;<br>                      &lt;strong&gt;Email: &lt;/strong&gt; agents@example.com<br>                    &lt;/p&gt;<br>                  &lt;/div&gt;<br>                &lt;/div&gt;<br>                &lt;div class=&quot;card-footer-d&quot;&gt;<br>                  &lt;div class=&quot;socials-footer d-flex justify-content-center&quot;&gt;<br>                    &lt;ul class=&quot;list-inline&quot;&gt;<br>                      &lt;li class=&quot;list-inline-item&quot;&gt;<br>                        &lt;a href=&quot;#&quot; class=&quot;link-one&quot;&gt;<br>                          &lt;i class=&quot;bi bi-facebook&quot; aria-hidden=&quot;true&quot;&gt;&lt;/i&gt;<br>                        &lt;/a&gt;<br>                      &lt;/li&gt;<br>                      &lt;li class=&quot;list-inline-item&quot;&gt;<br>                        &lt;a href=&quot;#&quot; class=&quot;link-one&quot;&gt;<br>                          &lt;i class=&quot;bi bi-twitter&quot; aria-hidden=&quot;true&quot;&gt;&lt;/i&gt;<br>                        &lt;/a&gt;<br>                      &lt;/li&gt;<br>                      &lt;li class=&quot;list-inline-item&quot;&gt;<br>                        &lt;a href=&quot;#&quot; class=&quot;link-one&quot;&gt;<br>                          &lt;i class=&quot;bi bi-instagram&quot; aria-hidden=&quot;true&quot;&gt;&lt;/i&gt;<br>                        &lt;/a&gt;<br>                      &lt;/li&gt;<br>                      &lt;li class=&quot;list-inline-item&quot;&gt;<br>                        &lt;a href=&quot;#&quot; class=&quot;link-one&quot;&gt;<br>                          &lt;i class=&quot;bi bi-linkedin&quot; aria-hidden=&quot;true&quot;&gt;&lt;/i&gt;<br>                        &lt;/a&gt;<br>                      &lt;/li&gt;<br>                    &lt;/ul&gt;<br>                  &lt;/div&gt;<br>                &lt;/div&gt;<br>              &lt;/div&gt;<br>            &lt;/div&gt;<br>          &lt;/div&gt;<br>        &lt;/div&gt;<br>      &lt;/div&gt;<br>    &lt;/section&gt;&lt;!-- End Agents Section --&gt;<br><br>    &lt;!-- ======= Latest News Section ======= --&gt;<br>    &lt;section class=&quot;section-news section-t8&quot;&gt;<br>      &lt;div class=&quot;container&quot;&gt;<br>        &lt;div class=&quot;row&quot;&gt;<br>          &lt;div class=&quot;col-md-12&quot;&gt;<br>            &lt;div class=&quot;title-wrap d-flex justify-content-between&quot;&gt;<br>              &lt;div class=&quot;title-box&quot;&gt;<br>                &lt;h2 class=&quot;title-a&quot;&gt;Latest News&lt;/h2&gt;<br>              &lt;/div&gt;<br>              &lt;div class=&quot;title-link&quot;&gt;<br>                &lt;a href=&quot;blog-grid.html&quot;&gt;All News<br>                  &lt;span class=&quot;bi bi-chevron-right&quot;&gt;&lt;/span&gt;<br>                &lt;/a&gt;<br>              &lt;/div&gt;<br>            &lt;/div&gt;<br>          &lt;/div&gt;<br>        &lt;/div&gt;<br><br>        &lt;div id=&quot;news-carousel&quot; class=&quot;swiper&quot;&gt;<br>          &lt;div class=&quot;swiper-wrapper&quot;&gt;<br><br>            &lt;div class=&quot;carousel-item-c swiper-slide&quot;&gt;<br>              &lt;div class=&quot;card-box-b card-shadow news-box&quot;&gt;<br>                &lt;div class=&quot;img-box-b&quot;&gt;<br>                  &lt;img src=&quot;/static/assets/img/post-2.jpg&quot; alt=&quot;&quot; class=&quot;img-b img-fluid&quot;&gt;<br>                &lt;/div&gt;<br>                &lt;div class=&quot;card-overlay&quot;&gt;<br>                  &lt;div class=&quot;card-header-b&quot;&gt;<br>                    &lt;div class=&quot;card-category-b&quot;&gt;<br>                      &lt;a href=&quot;#&quot; class=&quot;category-b&quot;&gt;House&lt;/a&gt;<br>                    &lt;/div&gt;<br>                    &lt;div class=&quot;card-title-b&quot;&gt;<br>                      &lt;h2 class=&quot;title-2&quot;&gt;<br>                        &lt;a href=&quot;blog-single.html&quot;&gt;House is comming<br>                          &lt;br&gt; new&lt;/a&gt;<br>                      &lt;/h2&gt;<br>                    &lt;/div&gt;<br>                    &lt;div class=&quot;card-date&quot;&gt;<br>                      &lt;span class=&quot;date-b&quot;&gt;18 Sep. 2017&lt;/span&gt;<br>                    &lt;/div&gt;<br>                  &lt;/div&gt;<br>                &lt;/div&gt;<br>              &lt;/div&gt;<br>            &lt;/div&gt;&lt;!-- End carousel item --&gt;<br><br>            &lt;div class=&quot;carousel-item-c swiper-slide&quot;&gt;<br>              &lt;div class=&quot;card-box-b card-shadow news-box&quot;&gt;<br>                &lt;div class=&quot;img-box-b&quot;&gt;<br>                  &lt;img src=&quot;/static/assets/img/post-5.jpg&quot; alt=&quot;&quot; class=&quot;img-b img-fluid&quot;&gt;<br>                &lt;/div&gt;<br>                &lt;div class=&quot;card-overlay&quot;&gt;<br>                  &lt;div class=&quot;card-header-b&quot;&gt;<br>                    &lt;div class=&quot;card-category-b&quot;&gt;<br>                      &lt;a href=&quot;#&quot; class=&quot;category-b&quot;&gt;Travel&lt;/a&gt;<br>                    &lt;/div&gt;<br>                    &lt;div class=&quot;card-title-b&quot;&gt;<br>                      &lt;h2 class=&quot;title-2&quot;&gt;<br>                        &lt;a href=&quot;blog-single.html&quot;&gt;Travel is comming<br>                          &lt;br&gt; new&lt;/a&gt;<br>                      &lt;/h2&gt;<br>                    &lt;/div&gt;<br>                    &lt;div class=&quot;card-date&quot;&gt;<br>                      &lt;span class=&quot;date-b&quot;&gt;18 Sep. 2017&lt;/span&gt;<br>                    &lt;/div&gt;<br>                  &lt;/div&gt;<br>                &lt;/div&gt;<br>              &lt;/div&gt;<br>            &lt;/div&gt;&lt;!-- End carousel item --&gt;<br><br>            &lt;div class=&quot;carousel-item-c swiper-slide&quot;&gt;<br>              &lt;div class=&quot;card-box-b card-shadow news-box&quot;&gt;<br>                &lt;div class=&quot;img-box-b&quot;&gt;<br>                  &lt;img src=&quot;/static/assets/img/post-7.jpg&quot; alt=&quot;&quot; class=&quot;img-b img-fluid&quot;&gt;<br>                &lt;/div&gt;<br>                &lt;div class=&quot;card-overlay&quot;&gt;<br>                  &lt;div class=&quot;card-header-b&quot;&gt;<br>                    &lt;div class=&quot;card-category-b&quot;&gt;<br>                      &lt;a href=&quot;#&quot; class=&quot;category-b&quot;&gt;Park&lt;/a&gt;<br>                    &lt;/div&gt;<br>                    &lt;div class=&quot;card-title-b&quot;&gt;<br>                      &lt;h2 class=&quot;title-2&quot;&gt;<br>                        &lt;a href=&quot;blog-single.html&quot;&gt;Park is comming<br>                          &lt;br&gt; new&lt;/a&gt;<br>                      &lt;/h2&gt;<br>                    &lt;/div&gt;<br>                    &lt;div class=&quot;card-date&quot;&gt;<br>                      &lt;span class=&quot;date-b&quot;&gt;18 Sep. 2017&lt;/span&gt;<br>                    &lt;/div&gt;<br>                  &lt;/div&gt;<br>                &lt;/div&gt;<br>              &lt;/div&gt;<br>            &lt;/div&gt;&lt;!-- End carousel item --&gt;<br><br>            &lt;div class=&quot;carousel-item-c swiper-slide&quot;&gt;<br>              &lt;div class=&quot;card-box-b card-shadow news-box&quot;&gt;<br>                &lt;div class=&quot;img-box-b&quot;&gt;<br>                  &lt;img src=&quot;/static/assets/img/post-3.jpg&quot; alt=&quot;&quot; class=&quot;img-b img-fluid&quot;&gt;<br>                &lt;/div&gt;<br>                &lt;div class=&quot;card-overlay&quot;&gt;<br>                  &lt;div class=&quot;card-header-b&quot;&gt;<br>                    &lt;div class=&quot;card-category-b&quot;&gt;<br>                      &lt;a href=&quot;#&quot; class=&quot;category-b&quot;&gt;Travel&lt;/a&gt;<br>                    &lt;/div&gt;<br>                    &lt;div class=&quot;card-title-b&quot;&gt;<br>                      &lt;h2 class=&quot;title-2&quot;&gt;<br>                        &lt;a href=&quot;#&quot;&gt;Travel is comming<br>                          &lt;br&gt; new&lt;/a&gt;<br>                      &lt;/h2&gt;<br>                    &lt;/div&gt;<br>                    &lt;div class=&quot;card-date&quot;&gt;<br>                      &lt;span class=&quot;date-b&quot;&gt;18 Sep. 2017&lt;/span&gt;<br>                    &lt;/div&gt;<br>                  &lt;/div&gt;<br>                &lt;/div&gt;<br>              &lt;/div&gt;<br>            &lt;/div&gt;&lt;!-- End carousel item --&gt;<br><br>          &lt;/div&gt;<br>        &lt;/div&gt;<br><br>        &lt;div class=&quot;news-carousel-pagination carousel-pagination&quot;&gt;&lt;/div&gt;<br>      &lt;/div&gt;<br>    &lt;/section&gt;&lt;!-- End Latest News Section --&gt;<br><br>    &lt;!-- ======= Testimonials Section ======= --&gt;<br>    &lt;section class=&quot;section-testimonials section-t8 nav-arrow-a&quot;&gt;<br>      &lt;div class=&quot;container&quot;&gt;<br>        &lt;div class=&quot;row&quot;&gt;<br>          &lt;div class=&quot;col-md-12&quot;&gt;<br>            &lt;div class=&quot;title-wrap d-flex justify-content-between&quot;&gt;<br>              &lt;div class=&quot;title-box&quot;&gt;<br>                &lt;h2 class=&quot;title-a&quot;&gt;Testimonials&lt;/h2&gt;<br>              &lt;/div&gt;<br>            &lt;/div&gt;<br>          &lt;/div&gt;<br>        &lt;/div&gt;<br><br>        &lt;div id=&quot;testimonial-carousel&quot; class=&quot;swiper&quot;&gt;<br>          &lt;div class=&quot;swiper-wrapper&quot;&gt;<br><br>            &lt;div class=&quot;carousel-item-a swiper-slide&quot;&gt;<br>              &lt;div class=&quot;testimonials-box&quot;&gt;<br>                &lt;div class=&quot;row&quot;&gt;<br>                  &lt;div class=&quot;col-sm-12 col-md-6&quot;&gt;<br>                    &lt;div class=&quot;testimonial-img&quot;&gt;<br>                      &lt;img src=&quot;/static/assets/img/testimonial-1.jpg&quot; alt=&quot;&quot; class=&quot;img-fluid&quot;&gt;<br>                    &lt;/div&gt;<br>                  &lt;/div&gt;<br>                  &lt;div class=&quot;col-sm-12 col-md-6&quot;&gt;<br>                    &lt;div class=&quot;testimonial-ico&quot;&gt;<br>                      &lt;i class=&quot;bi bi-chat-quote-fill&quot;&gt;&lt;/i&gt;<br>                    &lt;/div&gt;<br>                    &lt;div class=&quot;testimonials-content&quot;&gt;<br>                      &lt;p class=&quot;testimonial-text&quot;&gt;<br>                        Lorem ipsum dolor sit amet, consectetur adipisicing elit. Omnis, cupiditate ea nam praesentium<br>                        debitis hic ber quibusdam<br>                        voluptatibus officia expedita corpori.<br>                      &lt;/p&gt;<br>                    &lt;/div&gt;<br>                    &lt;div class=&quot;testimonial-author-box&quot;&gt;<br>                      &lt;img src=&quot;/static/assets/img/mini-testimonial-1.jpg&quot; alt=&quot;&quot; class=&quot;testimonial-avatar&quot;&gt;<br>                      &lt;h5 class=&quot;testimonial-author&quot;&gt;Albert &amp; Erika&lt;/h5&gt;<br>                    &lt;/div&gt;<br>                  &lt;/div&gt;<br>                &lt;/div&gt;<br>              &lt;/div&gt;<br>            &lt;/div&gt;&lt;!-- End carousel item --&gt;<br><br>            &lt;div class=&quot;carousel-item-a swiper-slide&quot;&gt;<br>              &lt;div class=&quot;testimonials-box&quot;&gt;<br>                &lt;div class=&quot;row&quot;&gt;<br>                  &lt;div class=&quot;col-sm-12 col-md-6&quot;&gt;<br>                    &lt;div class=&quot;testimonial-img&quot;&gt;<br>                      &lt;img src=&quot;/static/assets/img/testimonial-2.jpg&quot; alt=&quot;&quot; class=&quot;img-fluid&quot;&gt;<br>                    &lt;/div&gt;<br>                  &lt;/div&gt;<br>                  &lt;div class=&quot;col-sm-12 col-md-6&quot;&gt;<br>                    &lt;div class=&quot;testimonial-ico&quot;&gt;<br>                      &lt;i class=&quot;bi bi-chat-quote-fill&quot;&gt;&lt;/i&gt;<br>                    &lt;/div&gt;<br>                    &lt;div class=&quot;testimonials-content&quot;&gt;<br>                      &lt;p class=&quot;testimonial-text&quot;&gt;<br>                        Lorem ipsum dolor sit amet, consectetur adipisicing elit. Omnis, cupiditate ea nam praesentium<br>                        debitis hic ber quibusdam<br>                        voluptatibus officia expedita corpori.<br>                      &lt;/p&gt;<br>                    &lt;/div&gt;<br>                    &lt;div class=&quot;testimonial-author-box&quot;&gt;<br>                      &lt;img src=&quot;/static/assets/img/mini-testimonial-2.jpg&quot; alt=&quot;&quot; class=&quot;testimonial-avatar&quot;&gt;<br>                      &lt;h5 class=&quot;testimonial-author&quot;&gt;Pablo &amp; Emma&lt;/h5&gt;<br>                    &lt;/div&gt;<br>                  &lt;/div&gt;<br>                &lt;/div&gt;<br>              &lt;/div&gt;<br>            &lt;/div&gt;&lt;!-- End carousel item --&gt;<br><br>          &lt;/div&gt;<br>        &lt;/div&gt;<br>        &lt;div class=&quot;testimonial-carousel-pagination carousel-pagination&quot;&gt;&lt;/div&gt;<br><br>      &lt;/div&gt;<br>    &lt;/section&gt;&lt;!-- End Testimonials Section --&gt;<br><br>  &lt;/main&gt;&lt;!-- End #main --&gt;<br><br>  &lt;!-- ======= Footer ======= --&gt;<br>  &lt;section class=&quot;section-footer&quot;&gt;<br>    &lt;div class=&quot;container&quot;&gt;<br>      &lt;div class=&quot;row&quot;&gt;<br>        &lt;div class=&quot;col-sm-12 col-md-4&quot;&gt;<br>          &lt;div class=&quot;widget-a&quot;&gt;<br>            &lt;div class=&quot;w-header-a&quot;&gt;<br>              &lt;h3 class=&quot;w-title-a text-brand&quot;&gt;BostonHomes&lt;/h3&gt;<br>            &lt;/div&gt;<br>            &lt;div class=&quot;w-body-a&quot;&gt;<br>              &lt;p class=&quot;w-text-a color-text-a&quot;&gt;<br>                Committed to an unparalleled level of service, Boston Homes Real Estate brings quality customers and residences together. We creatively utilize every tool available, from the latest technologies to direct personal interaction. We listen, work efficiently and maintain a comprehensive understanding of the changing market place. With a long history of integrity and respect, we are dedicated to developing lasting relationships with customers and the community. We offer a  steadfast commitment to leadership and vision for the future of the industry.<br>              &lt;/p&gt;<br>            &lt;/div&gt;<br>            &lt;div class=&quot;w-footer-a&quot;&gt;<br>              &lt;ul class=&quot;list-unstyled&quot;&gt;<br>                &lt;li class=&quot;color-a&quot;&gt;<br>                  &lt;span class=&quot;color-text-a&quot;&gt;Phone .&lt;/span&gt; dhyey1974@gmail.com<br>                &lt;/li&gt;<br>                &lt;li class=&quot;color-a&quot;&gt;<br>                  &lt;span class=&quot;color-text-a&quot;&gt;Email .&lt;/span&gt; +91 997 4959775<br>                &lt;/li&gt;<br>              &lt;/ul&gt;<br>            &lt;/div&gt;<br>          &lt;/div&gt;<br>        &lt;/div&gt;<br>        &lt;div class=&quot;col-sm-12 col-md-4 section-md-t3&quot;&gt;<br>          &lt;div class=&quot;widget-a&quot;&gt;<br>            &lt;div class=&quot;w-header-a&quot;&gt;<br>              &lt;h3 class=&quot;w-title-a text-brand&quot;&gt;The Company&lt;/h3&gt;<br>            &lt;/div&gt;<br>            &lt;div class=&quot;w-body-a&quot;&gt;<br>              &lt;div class=&quot;w-body-a&quot;&gt;<br>                &lt;ul class=&quot;list-unstyled&quot;&gt;<br>                  &lt;li class=&quot;item-list-a&quot;&gt;<br>                    &lt;i class=&quot;bi bi-chevron-right&quot;&gt;&lt;/i&gt; &lt;a href=&quot;#&quot;&gt;Site Map&lt;/a&gt;<br>                  &lt;/li&gt;<br>                  &lt;li class=&quot;item-list-a&quot;&gt;<br>                    &lt;i class=&quot;bi bi-chevron-right&quot;&gt;&lt;/i&gt; &lt;a href=&quot;#&quot;&gt;Legal&lt;/a&gt;<br>                  &lt;/li&gt;<br>                  &lt;li class=&quot;item-list-a&quot;&gt;<br>                    &lt;i class=&quot;bi bi-chevron-right&quot;&gt;&lt;/i&gt; &lt;a href=&quot;#&quot;&gt;Agent Admin&lt;/a&gt;<br>                  &lt;/li&gt;<br>                  &lt;li class=&quot;item-list-a&quot;&gt;<br>                    &lt;i class=&quot;bi bi-chevron-right&quot;&gt;&lt;/i&gt; &lt;a href=&quot;#&quot;&gt;Careers&lt;/a&gt;<br>                  &lt;/li&gt;<br>                  &lt;li class=&quot;item-list-a&quot;&gt;<br>                    &lt;i class=&quot;bi bi-chevron-right&quot;&gt;&lt;/i&gt; &lt;a href=&quot;#&quot;&gt;Affiliate&lt;/a&gt;<br>                  &lt;/li&gt;<br>                  &lt;li class=&quot;item-list-a&quot;&gt;<br>                    &lt;i class=&quot;bi bi-chevron-right&quot;&gt;&lt;/i&gt; &lt;a href=&quot;#&quot;&gt;Privacy Policy&lt;/a&gt;<br>                  &lt;/li&gt;<br>                &lt;/ul&gt;<br>              &lt;/div&gt;<br>            &lt;/div&gt;<br>          &lt;/div&gt;<br>        &lt;/div&gt;<br>        &lt;div class=&quot;col-sm-12 col-md-4 section-md-t3&quot;&gt;<br>          &lt;div class=&quot;widget-a&quot;&gt;<br>            &lt;div class=&quot;w-header-a&quot;&gt;<br>              &lt;h3 class=&quot;w-title-a text-brand&quot;&gt;International sites&lt;/h3&gt;<br>            &lt;/div&gt;<br>            &lt;div class=&quot;w-body-a&quot;&gt;<br>              &lt;ul class=&quot;list-unstyled&quot;&gt;<br>                &lt;li class=&quot;item-list-a&quot;&gt;<br>                  &lt;i class=&quot;bi bi-chevron-right&quot;&gt;&lt;/i&gt; &lt;a href=&quot;#&quot;&gt;Venezuela&lt;/a&gt;<br>                &lt;/li&gt;<br>                &lt;li class=&quot;item-list-a&quot;&gt;<br>                  &lt;i class=&quot;bi bi-chevron-right&quot;&gt;&lt;/i&gt; &lt;a href=&quot;#&quot;&gt;China&lt;/a&gt;<br>                &lt;/li&gt;<br>                &lt;li class=&quot;item-list-a&quot;&gt;<br>                  &lt;i class=&quot;bi bi-chevron-right&quot;&gt;&lt;/i&gt; &lt;a href=&quot;#&quot;&gt;Hong Kong&lt;/a&gt;<br>                &lt;/li&gt;<br>                &lt;li class=&quot;item-list-a&quot;&gt;<br>                  &lt;i class=&quot;bi bi-chevron-right&quot;&gt;&lt;/i&gt; &lt;a href=&quot;#&quot;&gt;Argentina&lt;/a&gt;<br>                &lt;/li&gt;<br>                &lt;li class=&quot;item-list-a&quot;&gt;<br>                  &lt;i class=&quot;bi bi-chevron-right&quot;&gt;&lt;/i&gt; &lt;a href=&quot;#&quot;&gt;Singapore&lt;/a&gt;<br>                &lt;/li&gt;<br>                &lt;li class=&quot;item-list-a&quot;&gt;<br>                  &lt;i class=&quot;bi bi-chevron-right&quot;&gt;&lt;/i&gt; &lt;a href=&quot;#&quot;&gt;Philippines&lt;/a&gt;<br>                &lt;/li&gt;<br>              &lt;/ul&gt;<br>            &lt;/div&gt;<br>          &lt;/div&gt;<br>        &lt;/div&gt;<br>      &lt;/div&gt;<br>    &lt;/div&gt;<br>  &lt;/section&gt;<br>  &lt;footer&gt;<br>    &lt;div class=&quot;container&quot;&gt;<br>      &lt;div class=&quot;row&quot;&gt;<br>        &lt;div class=&quot;col-md-12&quot;&gt;<br>          &lt;nav class=&quot;nav-footer&quot;&gt;<br>            &lt;ul class=&quot;list-inline&quot;&gt;<br>              &lt;li class=&quot;list-inline-item&quot;&gt;<br>                &lt;a href=&quot;#&quot;&gt;Home&lt;/a&gt;<br>              &lt;/li&gt;<br>              &lt;li class=&quot;list-inline-item&quot;&gt;<br>                &lt;a href=&quot;#&quot;&gt;Property&lt;/a&gt;<br>              &lt;/li&gt;<br>              &lt;li class=&quot;list-inline-item&quot;&gt;<br>                &lt;a href=&quot;#&quot;&gt;Contact&lt;/a&gt;<br>              &lt;/li&gt;<br>            &lt;/ul&gt;<br>          &lt;/nav&gt;<br>          &lt;div class=&quot;socials-a&quot;&gt;<br>            &lt;ul class=&quot;list-inline&quot;&gt;<br>              &lt;li class=&quot;list-inline-item&quot;&gt;<br>                &lt;a href=&quot;#&quot;&gt;<br>                  &lt;i class=&quot;bi bi-facebook&quot; aria-hidden=&quot;true&quot;&gt;&lt;/i&gt;<br>                &lt;/a&gt;<br>              &lt;/li&gt;<br>              &lt;li class=&quot;list-inline-item&quot;&gt;<br>                &lt;a href=&quot;#&quot;&gt;<br>                  &lt;i class=&quot;bi bi-twitter&quot; aria-hidden=&quot;true&quot;&gt;&lt;/i&gt;<br>                &lt;/a&gt;<br>              &lt;/li&gt;<br>              &lt;li class=&quot;list-inline-item&quot;&gt;<br>                &lt;a href=&quot;#&quot;&gt;<br>                  &lt;i class=&quot;bi bi-instagram&quot; aria-hidden=&quot;true&quot;&gt;&lt;/i&gt;<br>                &lt;/a&gt;<br>              &lt;/li&gt;<br>              &lt;li class=&quot;list-inline-item&quot;&gt;<br>                &lt;a href=&quot;#&quot;&gt;<br>                  &lt;i class=&quot;bi bi-linkedin&quot; aria-hidden=&quot;true&quot;&gt;&lt;/i&gt;<br>                &lt;/a&gt;<br>              &lt;/li&gt;<br>            &lt;/ul&gt;<br>          &lt;/div&gt;<br>          &lt;div class=&quot;copyright-footer&quot;&gt;<br>            &lt;p class=&quot;copyright color-text-a&quot;&gt;<br>              &amp;copy; Copyright<br>              &lt;span class=&quot;color-a&quot;&gt;BostonHomes&lt;/span&gt; All Rights Reserved.<br>            &lt;/p&gt;<br>          &lt;/div&gt;<br>          &lt;div class=&quot;credits&quot;&gt;<br>            Designed by Dhyey Joshi&lt;/a&gt;<br>          &lt;/div&gt;<br>        &lt;/div&gt;<br>      &lt;/div&gt;<br>    &lt;/div&gt;<br>  &lt;/footer&gt;&lt;!-- End  Footer --&gt;<br><br>  &lt;div id=&quot;preloader&quot;&gt;&lt;/div&gt;<br>  &lt;a href=&quot;#&quot; class=&quot;back-to-top d-flex align-items-center justify-content-center&quot;&gt;&lt;i class=&quot;bi bi-arrow-up-short&quot;&gt;&lt;/i&gt;&lt;/a&gt;<br><br>  &lt;!-- Vendor JS Files --&gt;<br>  &lt;script src=&quot;/static/assets/vendor/bootstrap/js/bootstrap.bundle.min.js&quot;&gt;&lt;/script&gt;<br>  &lt;script src=&quot;/static/assets/vendor/swiper/swiper-bundle.min.js&quot;&gt;&lt;/script&gt;<br>  &lt;script src=&quot;/static/assets/vendor/php-email-form/validate.js&quot;&gt;&lt;/script&gt;<br><br>  &lt;!-- Template Main JS File --&gt;<br>  &lt;script src=&quot;/static/assets/js/main.js&quot;&gt;&lt;/script&gt;<br><br>&lt;/body&gt;<br><br>&lt;/html&gt;</pre><p>So we can move on to the GitHub part.</p><p>I created a new directory named “.github” and within that directory, I created a subdirectory called “workflow”. In the “workflow” directory, I created a YAML file. These directories were set up in a manner that enables GitHub to seamlessly interact with my local model directory.</p><p>Only one final touch needs to be done before starting the Continuous Integration part, which is the <a href="https://github.com/dhyeyjoshi/BostonHousePrice-Predictor/blob/main/.github/workflows/main.yaml"><strong>YAML file</strong></a>.</p><p>YAML file lets GitHub know exactly what needs to be done at each step of the way.</p><pre># Your workflow name.<br>name: Deploy to heroku.<br><br># Run workflow on every push to main branch.<br>on:<br>  push:<br>    branches: [main]<br><br># Your workflows jobs.<br>jobs:<br>  build:<br>    runs-on: ubuntu-latest<br>    steps:<br>      # Check-out your repository.<br>      - name: Checkout<br>        uses: actions/checkout@v2<br><br><br>### ⬇ IMPORTANT PART ⬇ ###<br><br>      - name: Build, Push and Release a Docker container to Heroku. # Your custom step name<br>        uses: gonuit/heroku-docker-deploy@v1.3.3 # GitHub action name (leave it as it is).<br>        with:<br>          # Below you must provide variables for your Heroku app.<br><br>          # The email address associated with your Heroku account.<br>          # If you don&#39;t want to use repository secrets (which is recommended) you can do:<br>          # email: my.email@example.com<br>          email: ${{ secrets.HEROKU_EMAIL }}<br>          <br>          # Heroku API key associated with provided user&#39;s email.<br>          # Api Key is available under your Heroku account settings.<br>          heroku_api_key: ${{ secrets.HEROKU_API_KEY }}<br>          <br>          # Name of the heroku application to which the build is to be sent.<br>          heroku_app_name: ${{ secrets.HEROKU_APP_NAME }}<br><br>          # (Optional, default: &quot;./&quot;)<br>          # Dockerfile directory.<br>          # For example, if you have a Dockerfile in the root of your project, leave it as follows:<br>          dockerfile_directory: ./<br><br>          # (Optional, default: &quot;Dockerfile&quot;)<br>          # Dockerfile name.<br>          dockerfile_name: Dockerfile<br><br>          # (Optional, default: &quot;&quot;)<br>          # Additional options of docker build command.<br>          docker_options: &quot;--no-cache&quot;<br><br>          # (Optional, default: &quot;web&quot;)<br>          # Select the process type for which you want the docker container to be uploaded.<br>          # By default, this argument is set to &quot;web&quot;.<br>          # For more information look at https://devcenter.heroku.com/articles/process-model<br>          process_type: web<br>          <br>   <br>          <br>### ⬆ IMPORTANT PART ⬆ ###</pre><p>Let’s push it to see the Continuous Integration part in the action.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YHoIURbGcwZe3-08wun_Rg.png" /></figure><p>So far everything is OK with our CI part of the MLOps pipeline and we got our badge.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/252/0*OQB0hdH-_tIVyBgu.png" /></figure><p>So we are ready to move on to the second part of our MLOps pipeline.</p><h3>CD (Continuous Deployment) Part</h3><p>For this part, we will use Heroku as a cloud application platform.</p><p>Heroku is easy to use and free platform as a service (PaaS) that enables developers to build, run, and operate applications entirely in the cloud.</p><p>Using our Flask application in the Heroku environment we need to create Procfile and put this one-line command in it.</p><pre>web gunicorn app:app</pre><p>Let’s go to the Heroku website and create a new application.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PxrA2M9UeCTbxOZTNWUdpA.png" /></figure><p>We need to make the connection between our GitHub repo and Heroku for the CI-CD pipeline.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*v15fXm_RCPF_KAWJkonO4w.png" /></figure><p>And then we enable automatic deployment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/706/0*PsrWdGdjCME2dwEf.png" /></figure><p>We need to get an authentication token from Heroku to connect directly from GitHub to start Continuous Deployment.</p><p>We need this token when creating keys in GitHub.<br>So move on to GitHub and create keys.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-9LmZrTVIuERbo4eASL27Q.png" /></figure><p>We select a new repository secret and add the token which we got from Heroku.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/174/0*JanYA3hvbGyV4AHq.png" /></figure><p>We need one more key to make a connection with Heroku, let’s add that one also.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/253/1*SkGDf28kRxoymusdfI8ATg.png" /></figure><p>We need one more key to make a connection with Heroku using Email, let’s add that one also.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/261/1*8YLtp-T6zSM8cJrnP9BXjw.png" /></figure><p>We are almost ready to start our end-to-end MLOps pipeline.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ODD012_BtdwZzQaFUCf4Cw.png" /></figure><p>We just need to add GitHub keys to the YAML file as a variable and finalize the <a href="https://github.com/dhyeyjoshi/BostonHousePrice-Predictor/blob/main/.github/workflows/main.yaml">YAML file</a>.</p><p>Let’s follow the GitHub actions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mIA3RzRREBg37d06VZZ83Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GEKoVQjusHmlxHuQ6KvHFA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KOVnDQj3x9Dr_aEGGQlbCA.png" /></figure><p>So far everything is OK with our CD part of the MLOps pipeline and we got our New badge.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/557/1*iR5zpbWzWOohUAAJFSD2cw.png" /></figure><p>Everything seems OK. Let’s see the Heroku part and open the application.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*onmb42TzsIvmYoYRD7bQAw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*W9k_1y9EWxDqXVGrNMqVAw.png" /></figure><p>Check out the demo of the application — <a href="https://bostonhomes.herokuapp.com/">https://bostonhomes.herokuapp.com/</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2M2LL7MRnddpYvaZPnvq4w.png" /></figure><p>How to use this application? — <a href="https://github.com/dhyeyjoshi/BostonHousePrice-Predictor/blob/main/Demo-Video.mp4">Demo Video</a> (Click here)</p><p>Verifying whether the application’s API is functioning correctly?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*biZrKfzhro34K9zEZi67-A.gif" /></figure><p><strong>Conclusion</strong></p><p>By using GitHub actions and Heroku we have created end to end MLOps pipeline.</p><p>I hope that it helps.</p><p>By the way, when you like the topic, you can show it by supporting <strong>👏</strong></p><p>Feel free to leave a comment. Thanks for your time.</p><p><strong>Code</strong></p><p>The code can be downloaded from <a href="https://github.com/dhyeyjoshi/BostonHousePrice-Predictor"><strong>here</strong></a>.</p><p>If you enjoy reading my content,<a href="https://dhyey-joshi12.medium.com/"> <em>please consider following me</em></a>.</p><p>I hope that you find this article and explanation useful. Stay tuned for more data science-related posts in the future.</p><blockquote>All the best 🤘</blockquote><p>Feel free to connect and give ur suggestions:</p><p><a href="https://www.linkedin.com/in/dhyey-joshi12/">https://www.linkedin.com/in/dhyey-joshi12/</a></p><p><a href="https://github.com/dhyeyjoshi">https://github.com/dhyeyjoshi</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=791f5396162a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[“MLaaS” — ML-as-a-Service — Using AWS SageMaker to Solve Machine Learning & AI Problems]]></title>
            <link>https://dhyey-joshi12.medium.com/mlaas-ml-as-a-service-using-aws-sagemaker-to-solve-machine-learning-ai-problems-1210ecb7d568?source=rss-2ac64d1c310a------2</link>
            <guid isPermaLink="false">https://medium.com/p/1210ecb7d568</guid>
            <category><![CDATA[aws-s3]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[amazon-sagemaker]]></category>
            <dc:creator><![CDATA[Dhyey Joshi]]></dc:creator>
            <pubDate>Sat, 11 Feb 2023 08:20:11 GMT</pubDate>
            <atom:updated>2023-04-07T10:40:54.306Z</atom:updated>
            <content:encoded><![CDATA[<h3>“MLaaS” — ML-as-a-Service — Using AWS SageMaker to Solve Machine Learning &amp; AI Problems</h3><h4>Build, train, and deploy machine learning models at scale</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b18tPQfsJjkcuJVlMHBOIQ.png" /><figcaption>Photo by <a href="https://www.youtube.com/@sightlineinnovation6224">Sightline Innovation</a> on the website</figcaption></figure><p>This article is an introduction to Amazon AWS SageMaker. Cloud platforms like Amazon Web Services (AWS) and Google Cloud Platform (GCP) are leading the industry in terms of building Machine Learning (ML) solutions for clients and customers. These “ML-as-a-Service” cloud services are expensive compared to building them on your own, but they save a lot of time and make life easier — delivering in months rather than quarters is worth the premium to keep your clients/customers happy.</p><p>The Amazon AWS SageMaker console can be found here:<br><a href="https://console.aws.amazon.com/sagemaker">https://console.aws.amazon.com/sagemaker</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d-2ZtAtkZ4-1qZHMAdXCFQ.png" /><figcaption>Image by Author</figcaption></figure><p>I studied Machine Learning during undergrad so I was familiar with <a href="https://jupyter.org/">Jupyter notebooks</a>. I later discovered <a href="https://colab.research.google.com/">Google Colab</a> and <a href="https://colab.research.google.com/signup">Google Colab Pro</a>. <a href="https://colab.research.google.com/">Google Colab</a> is a free, quick and easy way to run notebooks in the cloud without having the hassle of installing anything. There are however limits on public use which you may hit. <a href="https://colab.research.google.com/signup">Google Colab Pro</a> currently costs £9.99 a month which has increased limits and hardware. In terms of Machine Learning and AI in the cloud (“<strong>ML-as-a-service</strong>”), the “<strong>boss level</strong>” services are <a href="https://aws.amazon.com/sagemaker">Amazon AWS SageMaker</a>, <a href="https://aws.amazon.com/sagemaker/studio">Amazon AWS SageMaker Studio</a> and <a href="https://aws.amazon.com/rekognition">Amazon AWS Rekognition</a>. As you would expect from Amazon AWS the scaling, performance and workflows are insanely good but come at a steep cost.</p><p>In this blog, my objective is to enlighten you by answering the ‘WHY’, ‘HOW’ and ‘WHAT’ sequences of questions for selecting the best platform for a significant majority of the machine learning solutions in enterprises that we solve in the present day. I would then like to connect ‘<strong>AWS Sagemaker</strong>’ as a valid candidate for the problem and emphasize the reasons outlined with ‘Why do we need AWS Sagemaker?’.</p><p>Let’s start with the ‘WHY’.</p><p><strong>TAKE 1 — WHY do we need a platform for developing Machine Learning solutions?</strong></p><p>The major pain points associated with a machine learning project dramatically change as the scale of the project increases. A typical workflow for a machine learning project is demonstrated below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/794/0*yrE-3LPcwvI6udft.png" /><figcaption>An illustration of a typical Machine Learning project workflow</figcaption></figure><p>We can typically categorize the list of associated tasks into 3 broad areas -</p><ol><li><strong>Build</strong>: Here we define the problem, gather data, analyze it and then clean, transform and engineer the data into the desired form.</li><li><strong>Train</strong>: We leverage one or more machine learning algorithms and train (fit) the model to learn the patterns from the engineered data. Based on the model’s performance, we tune hyperparameters and retrain and repeat the process until we have fairly acceptable results.</li><li><strong>Deploy</strong>: We deploy the model into a production system where it caters its services to the larger ecosystem.</li></ol><p>This entire process is highly iterative and changes can be expected to loop back the progress to any state in the entire process (as shown in the diagram). In most of the Proof of Concepts (PoC) we experiment with and the available study resources over the internet (blogs/tutorials/books/courses), this entire process is highly simplified. The biggest abstraction rendered in this process is in the dimension of ‘<strong>scale</strong>’. As the scale of ‘data’, ‘training iterations’ and ‘inference’ post-deployment increases, this entire workflow gets overly complex and eventually evolves into 3 independent workflows.</p><p>Consider a fairly complex ML problem we would deal with in an enterprise. Say, you have a very large dataset with millions of records (~10GB — 1TB) to process and would need at least a few hundred iterations (say, for a deep neural network) and around 100,000 API calls per minute once the service is deployed. Then, the vanilla PoCs we developed during experimentation is no longer a comparison for the use case. Each group (Build, Train, Deploy) would need a different kind of compute infrastructure, a different approach to execute the problem and a distinct set of disciplines to accomplish. Figure (below), helps in understanding how each of these groups within the workflow, fork out as an independent branch with diverse requirements of skills and infrastructure to execute. <strong>The build</strong> phase requires a combination of Data Engineering and Data Analytics. <strong>The training</strong> phase requires Machine Learning skills and finally, the <strong>Deploy</strong> phase requires primarily Software Engineering skills and a few ML skills. Each of these phases would need a person with a completely different skill set.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/794/0*ICp-Nqh47gApTngG.png" /><figcaption>Image by <a href="https://blog.jojomoolayil.com/">Jojo John Moolayil</a></figcaption></figure><p>The most effective way to solve large ML problems is by aiding a data scientist with the necessary software skills in a neat abstract yet effective way to deliver an ML solution as a highly scalable web service (API). The software development team can integrate the API into the required software systems and abstract the ML service as just another service wrapped around an API. (Software engineers love APIs). Therefore, we need a platform that can enable a data scientist with the necessary tools to independently execute a machine learning project in a truly end-to-end way.</p><p><strong>TASK 2 — HOW can a platform solve this problem?</strong></p><p>we need a platform where the data scientist will be able to leverage his existing skills to engineer and study data, train and tune ML models and finally deploy the model as a web service by dynamically provisioning the required hardware, orchestrating the entire flow and transition for execution with simple abstraction and provide a robust solution that can scale and meet demands elastically.</p><p><strong>TASK 3 — What platform works best for me?</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/794/0*NM_7Qhs96EEU5GVw.jpeg" /><figcaption>source: AWS</figcaption></figure><p>Here, I can say,<strong> AWS Sagemaker</strong> fits best for us. It provides Jupyter NoteBooks running R/Python kernels with a compute instance that we can choose as per our data engineering requirements on demand. We can visualize, process, clean and transform the data into our required forms using the traditional methods we use (say Pandas + Matplotlib or R +ggplot2 or other popular combinations). Post-data engineering, we can train the models using a different compute instance based on the model’s compute demand, say memory-optimized or GPU-enabled. Leverage smart default high-performance hyperparameter tuning settings for a variety of models. Leverage performance-optimized algorithms from the rich AWS library or bring our algorithms through industry-standard containers. Also, deploy the trained model as an API, again using a different compute instance appropriate to meet business requirements and scale elastically. And the entire process of provisioning hardware instances, running high-capacity data jobs, orchestrating the entire flow with simple commands while abstracting the mammoth complexities and finally enabling serverless elastic deployment works with a few lines of code and yet is cost-effective. Sagemaker is a game-changing solution for the enterprise.</p><p>I will walk you through Amazon AWS’s tutorial and how to get started including my implementation and observations. I’m based in India so I usually would select the “<strong>Asia Pacific (Mumbai) ap-south-1</strong>” region. This region doesn’t seem to have all the features supported properly. To avoid problems using the regions in their official documentation: “<strong>N. California us-west-1</strong>”, “<strong>Oregon</strong> <strong>us-west-2</strong>”, “<strong>N.</strong> <strong>Virginia us-east-1</strong>”, “<strong>Ohio</strong> <strong>us-east-2</strong>”, or “<strong>Europe (Ireland) eu-west-1</strong>”.</p><h3>Getting Started</h3><p>You will need to sign up for an AWS account if you don’t already have one.</p><p>From the Amazon AWS SageMaker console open the “<strong>Notebook</strong>” menu, then select “<strong>Notebook instances</strong>”, then click on “<strong>Create notebook instance</strong>”.</p><p>For my project, I am presently working on the “<strong>N. Virginia us-east-1</strong>” area. Because this area includes around 71 data centres, when I selected “us-east-1,” AWS Service Provider offered me access to 71 data centers. All of the data centres’ capabilities, services, and resources will be available to me. AWS will provide me with a server with IP addresses as soon as I set up an instance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HrFmiNgBE6TRsmnbmvcP8g.png" /><figcaption>Overview of Amazon SageMaker &gt; Notebook Instances</figcaption></figure><p>The creation form looks like this.</p><p>Step-1 Give the Notebook Instance a distinct name. ( Maximum of 63 alphanumeric characters. Can include hyphens (-), but not spaces. Must be unique within your account in an AWS Region. )</p><p>Step-2 Choose any notebook instance type that meets the requirements of the application. This parameter will improve the application’s response time and page load time. (Page load time (measured in seconds) is the time it takes to download and show the whole content of a web page in the browser window.)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_jmP_LogL4iG8ljoEm-o1A.png" /><figcaption>Image by Author</figcaption></figure><p>Step-3 (Optional) Additional <a href="https://docs.aws.amazon.com/sagemaker/latest/dg/ei.html">Elastic Inference</a> parameter that allows extensive resources like CPU, RAM, GPU, etc. to scale your application as the workload on the application rises.</p><p>In terms of this tutorial, all you need to do is give your instance name. I called mine “<strong>BankApplication</strong>”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tfZ30AdveDZ6EKaxmEipSA.png" /><figcaption>Image by Author</figcaption></figure><blockquote>The most important topic to comprehend is “IAM roles”</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*2YNyeji7gMabPT2k77sKiA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*NvqsqhiVOMAbqzEgkkgG5w.png" /><figcaption>Image by AWS</figcaption></figure><p>We need to understand IAM Roles since the next step will be to create S3 buckets to store data of our bank application. Loading data from an Amazon Simple Storage Service (Amazon S3) bucket requires an <strong>AWS Identity and Access Management</strong> (IAM) role that has access to the bucket. Amazon Neptune assumes this role to load the data. You can load encrypted data from Amazon S3 if it was encrypted using the Amazon S3 SSE-S3 mode.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a63B0y3QjqnF3cELc_IHOg.png" /><figcaption>Image by Author</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tVbD5yNXFB0pGiGHDxhAJg.png" /><figcaption>Image by Author</figcaption></figure><p>Just click “<strong>Create role</strong>”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D98NHXmA6ZHzMV0gowzH_Q.png" /><figcaption>Image by Author</figcaption></figure><p>Then on the “<strong>Create notebook instance</strong>” form click the “<strong>Create notebook instance</strong>” button at the bottom.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1__QBLjGa20ElVlCKX4oNw.png" /><figcaption>Image by Author</figcaption></figure><p>Once it is ready click on your “<strong>BankApplication</strong>” notebook, then “<strong>Open Jupyter</strong>” under “<strong>Actions</strong>”.</p><p>Now it will start looking familiar to you :)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R0IG4pvormzyVb7ELbMo6A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-VF_cLr5pbj4jl6nZdO_-Q.png" /><figcaption>Image by Author</figcaption></figure><p>You will want to create a notebook. Click on the “<strong>New</strong>” drop-down and select “<strong>conda_python3</strong>”. A file called “<strong>Untitled. ipynb</strong>” will be created but you can save it to another filename if you wish.</p><p>Amazon has provided the code required for your first code cell. This code imports the required libraries and defines the environment variables you need to prepare the data, train the ML model, and deploy the ML model.</p><pre>!pip install --upgrade boto3<br><br># import libraries<br>import boto3, re, sys, math, json, os, sagemaker, urllib.request<br>from sagemaker import get_execution_role<br>import numpy as np                                <br>import pandas as pd                               <br>import matplotlib.pyplot as plt                   <br>from IPython.display import Image                 <br>from IPython.display import display               <br>from time import gmtime, strftime                 <br>from sagemaker.predictor import csv_serializer<br># Define IAM role<br>role = get_execution_role()<br>prefix = &#39;sagemaker/DEMO-xgboost-dm&#39;<br>containers = {&#39;us-west-2&#39;: &#39;433757028032.dkr.ecr.us-west-2.amazonaws.com/xgboost:latest&#39;,<br>              &#39;us-east-1&#39;: &#39;811284229777.dkr.ecr.us-east-1.amazonaws.com/xgboost:latest&#39;,<br>              &#39;us-east-2&#39;: &#39;825641698319.dkr.ecr.us-east-2.amazonaws.com/xgboost:latest&#39;,<br>              &#39;eu-west-1&#39;: &#39;685385470294.dkr.ecr.eu-west-1.amazonaws.com/xgboost:latest&#39;} # each region has its XGBoost container<br>my_region = boto3.session.Session().region_name # set the region of the instance<br>print(&quot;Success - the MySageMakerInstance is in the &quot; + my_region + &quot; region. You will use the &quot; + containers[my_region] + &quot; container for your SageMaker endpoint.&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rYwl7mbdax5nA9M1WaUV_g.png" /><figcaption>Image by Author</figcaption></figure><p>You will need an Amazon AWS S3 bucket to store data. <strong>Please note this code creates a new S3 bucket, you don’t need to create one in advance!</strong></p><pre>import sagemaker<br>import boto3<br>from sagemaker.amazon.amazon_estimator import get_image_uri <br>from sagemaker.session import s3_input, Session<br><br>bucket_name = &#39;bankingdatabase&#39; # &lt;--- CHANGE THIS VARIABLE TO A UNIQUE NAME FOR YOUR BUCKET<br>my_region = boto3.session.Session().region_name # set the region of the instance<br>print(my_region)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SwE-9_1mHrTZhSAcJeypuA.png" /><figcaption>Image by Author</figcaption></figure><p>Creating S3 Bucket titled “banking database<strong>”</strong>.</p><pre>s3 = boto3.resource(&#39;s3&#39;)<br>try:<br>    if  my_region == &#39;us-east-1&#39;:<br>        s3.create_bucket(Bucket=bucket_name)<br>    print(&#39;S3 bucket created successfully&#39;)<br>except Exception as e:<br>    print(&#39;S3 error: &#39;,e)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YuAP2oLu2EtzEUo00P36Ag.png" /><figcaption>Image by Author</figcaption></figure><p>Checking whether the <a href="https://s3.console.aws.amazon.com/s3/buckets?region=us-east-1"><strong>S3 bucket</strong></a> is created or not.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Qu1wjfsYYIRRpJAd3Rnv3Q.png" /><figcaption>Image by Author</figcaption></figure><pre># set an output path where the trained model will be saved<br>prefix = &#39;xgboost-as-a-built-in-algo&#39;<br>output_path =&#39;s3://{}/{}/output&#39;.format(bucket_name, prefix)<br>print(output_path)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ery9tVs0d9s7GCmlTnftmA.png" /><figcaption>Image by Author</figcaption></figure><h3>Preparing our data</h3><p>This is some really handy code that will download the banking dataset from the URL and load it into the local environment.</p><pre>import pandas as pd<br>import urllib<br>try:<br>    urllib.request.urlretrieve (&quot;https://d1.awsstatic.com/tmt/build-train-deploy-machine-learning-model-sagemaker/bank_clean.27f01fbbdf43271788427f3682996ae29ceca05d.csv&quot;, &quot;bank_clean.csv&quot;)<br>    print(&#39;Success: downloaded bank_clean.csv.&#39;)<br>except Exception as e:<br>    print(&#39;Data load error: &#39;,e)<br><br>try:<br>    model_data = pd.read_csv(&#39;./bank_clean.csv&#39;,index_col=0)<br>    print(&#39;Success: Data loaded into dataframe.&#39;)<br>except Exception as e:<br>    print(&#39;Data load error: &#39;,e)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7BVVy337Kd7_cDabljbSIw.png" /><figcaption>Image by Author</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uLRw1o7oPGze6HnbQ57j2g.png" /><figcaption>Image by Author — Overview of the Bank_clean Dataset</figcaption></figure><p>This pulls down a CSV from a remote site and reads it into a Pandas data frame. The code above will download a file called “<strong>bank_clean.csv</strong>” and you will find it in your Jupyter notebooks instance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8vnQXEwjH73aA2UTDvBPiw.png" /><figcaption>Image by Author</figcaption></figure><h3>Data related to direct marketing campaigns of a Portuguese Banking Institution.</h3><p>The marketing campaigns were based on phone calls. Often, more than one contact with the same client was required, to access if the product (bank term deposit) would be (‘yes’) or not (‘no’) subscribed.</p><p>I initially wanted to use a Kaggle dataset for this tutorial but they don’t support retrieving the CSVs directly so the code above won’t work it’s still really useful and wanted to show you in case you hadn’t seen it or realized it was possible.</p><p>As you are aware there are many ways to split our data into a train and test split. In most cases, it’s usually done on a 70/30% split. The common ways to do the split are to use filtering within the data frame itself or to use Sklearn’s model selection, the”<strong>train_test_split</strong>” function. Amazon’s example uses NumPy. I had not seen this method before and it’s worth bringing it to your attention to add another tool to your belt.</p><pre>### Train Test split<br><br>import numpy as np<br>train_data, test_data = np.split(model_data.sample(frac=1, random_state=1729), [int(0.7 * len(model_data))])<br>print(train_data.shape, test_data.shape)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jRCfmsUWFKBKsTvg-Lr11g.png" /><figcaption>Image by Author</figcaption></figure><h3>Modelling our data</h3><p>This section is mostly extracted from the official Amazon tutorial:<br><a href="https://aws.amazon.com/getting-started/hands-on/build-train-deploy-machine-learning-model-sagemaker">https://aws.amazon.com/getting-started/hands-on/build-train-deploy-machine-learning-model-sagemaker</a></p><p>This code reformats the header and first column of the training data and then loads the data from the S3 bucket. This step is required to use the Amazon SageMaker pre-built XGBoost algorithm.</p><p>*<strong><em>Note: If you’re using the Amazon tutorial code, you’ll get an error here. The attribute “s3_input” will be removed from the next version of the Sagemaker module. SageMaker Python SDK v2 renames the “s3_input” class to “TrainingInput.”</em></strong></p><pre>### Saving Train And Test Into Buckets<br>## We start with Train Data<br>import os<br>pd.concat([train_data[&#39;y_yes&#39;], train_data.drop([&#39;y_no&#39;, &#39;y_yes&#39;], <br>                                                axis=1)], <br>                                                axis=1).to_csv(&#39;train.csv&#39;, index=False, header=False)<br>boto3.Session().resource(&#39;s3&#39;).Bucket(bucket_name).Object(os.path.join(prefix, &#39;train/train.csv&#39;)).upload_file(&#39;train.csv&#39;)<br>s3_input_train = sagemaker.TrainingInput(s3_data=&#39;s3://{}/{}/train&#39;.format(bucket_name, prefix), content_type=&#39;csv&#39;)</pre><pre># Test Data Into Buckets<br>pd.concat([test_data[&#39;y_yes&#39;], test_data.drop([&#39;y_no&#39;, &#39;y_yes&#39;], axis=1)], axis=1).to_csv(&#39;test.csv&#39;, index=False, header=False)<br>boto3.Session().resource(&#39;s3&#39;).Bucket(bucket_name).Object(os.path.join(prefix, &#39;test/test.csv&#39;)).upload_file(&#39;test.csv&#39;)<br>s3_input_test = sagemaker.TrainingInput(s3_data=&#39;s3://{}/{}/test&#39;.format(bucket_name, prefix), content_type=&#39;csv&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xZLKU64h7LEsXAHV58cvTQ.png" /><figcaption>Image by Author</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tFpgnyLb-gR_UhQmXqsqow.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_S-X4jMnxX-H5EgfdAWcbA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7TBm6mZtQ8fWE1Mr6cNhPw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xQna2sDM7kaXOne7gaQWXA.png" /><figcaption>Image by Author</figcaption></figure><p>Set up the Amazon SageMaker session, create an instance of the XGBoost model (an estimator), and define the model’s hyperparameters.</p><pre># this line automatically looks for the XGBoost image URI and builds an XGBoost container.<br># specify the repo_version depending on your preference.<br>container = get_image_uri(boto3.Session().region_name,<br>                          &#39;xgboost&#39;, <br>                          repo_version=&#39;1.0-1&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GtjB6xRD3H8qcK5v1-jSzA.png" /><figcaption>Image by Author</figcaption></figure><blockquote><strong><em>Warning: ‘get_image_uri’ method will be deprecated in favour of the ‘ImageURIProvider’ class in SageMaker Python SDK v2.</em></strong></blockquote><p>The hyperparameters shown below are selected after trial and error. This model can have many hyperparameters and finding the best combination of parameters can be treated as a search problem. The two best strategies for Hyperparameter tuning are:</p><ul><li><a href="https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html">GridSearchCV</a></li><li><a href="https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.RandomizedSearchCV.html">RandomizedSearchCV</a></li></ul><p><strong><em>* Find the Best Hyperparameters for your model on your local PC rather than here. Warning: If you attempt to do so, Amazon may charge you for using their resources.</em></strong></p><pre># initialize hyperparameters<br>hyperparameters = {<br>        &quot;max_depth&quot;:&quot;5&quot;,<br>        &quot;eta&quot;:&quot;0.2&quot;,<br>        &quot;gamma&quot;:&quot;4&quot;,<br>        &quot;min_child_weight&quot;:&quot;6&quot;,<br>        &quot;subsample&quot;:&quot;0.7&quot;,<br>        &quot;objective&quot;:&quot;binary:logistic&quot;,<br>        &quot;num_round&quot;:50<br>        }</pre><pre># construct a SageMaker estimator that calls the xgboost-container<br>estimator = sagemaker.estimator.Estimator(image_name=container, <br>                                          hyperparameters=hyperparameters,<br>                                          role=sagemaker.get_execution_role(),<br>                                          train_instance_count=4, <br>                                          train_instance_type=&#39;ml.m5.2xlarge&#39;, <br>                                          train_volume_size=5, # 5 GB <br>                                          output_path=output_path,<br>                                          train_use_spot_instances=True,<br>                                          train_max_run=300,<br>                                          train_max_wait=600)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*C4qdu7w6p5tyiQUbO5uuag.png" /><figcaption>Image by Author</figcaption></figure><p>This code trains the model using gradient optimization on a, “<strong>ml.m2.2xlarge</strong>” instance. After a few minutes, you should see the training logs generated in your Jupyter notebook.</p><h4>Charges Applicable Following This Step</h4><p>Be aware the next part will have a cost associated. Amazon says the 1 minute of processing is only about $1 but just warning not to run this if you don’t want to incur that cost.</p><p>You can play with these three parameters which will <strong>reduce the billing cost of training the model</strong>.</p><blockquote>train_use_spot_instances=True,<br>train_max_run=300,<br>train_max_wait=600</blockquote><pre>estimator.fit({&#39;train&#39;: s3_input_train,&#39;validation&#39;: s3_input_test})</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_LZRvqvOMn-93r1GcEMPOQ.png" /><figcaption>Image by Author</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*YbXLggP9grgLhXOszeHS7Q.png" /><figcaption>Image by Author</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/611/1*P1qAcC2rzpYjvoHrRrktFQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/706/1*wimS597VGK3Njw7YOwxDJw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/441/1*XVxJsd7zBXW0cG3aP99fTg.png" /><figcaption>Output folder created &gt; Our Model &gt; model.tar.gz</figcaption></figure><h3>Deploy the model</h3><p>This code deploys the model on a server and creates a SageMaker endpoint that you can access. This step may take a few minutes to complete. When I ran this step myself it wasn’t obvious at first that something was happening. You will slowly see dashes going across the screen which is the progress bar for the function. It completes with an exclamation mark.</p><pre>xgb_predictor = estimator.deploy(initial_instance_count=1,instance_type=&#39;ml.m4.xlarge&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Jd59pJtk9MVg78uy7FegCQ.png" /><figcaption>Image by Author</figcaption></figure><p>To predict whether customers in the test data enrolled for the bank product or not.</p><pre>from sagemaker.predictor import csv_serializer<br>test_data_array = test_data.drop([&#39;y_no&#39;, &#39;y_yes&#39;], axis=1).values #load the data into an array<br>xgb_predictor.content_type = &#39;text/csv&#39; # set the data type for an inference<br>xgb_predictor.serializer = csv_serializer # set the serializer type<br>predictions = xgb_predictor.predict(test_data_array).decode(&#39;utf-8&#39;) # predict!<br>predictions_array = np.fromstring(predictions[1:], sep=&#39;,&#39;) # and turn the prediction into an array<br>print(predictions_array.shape)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qYPMZBHfh1JjXrS06ibIjw.png" /><figcaption>Image by Author</figcaption></figure><h3>Evaluate model performance</h3><p>This code compares the actual vs. predicted values in a table called a confusion matrix.</p><p>Based on the prediction, we can conclude that you predicted a customer will enroll for a certificate of deposit accurately for 91% of customers in the test data, with a precision of 66% for enrolled and 91% for didn’t enroll.</p><pre>cm = pd.crosstab(index=test_data[&#39;y_yes&#39;], columns=np.round(predictions_array), rownames=[&#39;Observed&#39;], colnames=[&#39;Predicted&#39;])<br>tn = cm.iloc[0,0]; fn = cm.iloc[1,0]; tp = cm.iloc[1,1]; fp = cm.iloc[0,1]; p = (tp+tn)/(tp+tn+fp+fn)*100<br>print(&quot;\n{0:&lt;20}{1:&lt;4.1f}%\n&quot;.format(&quot;Overall Classification Rate: &quot;, p))<br>print(&quot;{0:&lt;15}{1:&lt;15}{2:&gt;8}&quot;.format(&quot;Predicted&quot;, &quot;No Purchase&quot;, &quot;Purchase&quot;))<br>print(&quot;Observed&quot;)<br>print(&quot;{0:&lt;15}{1:&lt;2.0f}% ({2:&lt;}){3:&gt;6.0f}% ({4:&lt;})&quot;.format(&quot;No Purchase&quot;, tn/(tn+fn)*100,tn, fp/(tp+fp)*100, fp))<br>print(&quot;{0:&lt;16}{1:&lt;1.0f}% ({2:&lt;}){3:&gt;7.0f}% ({4:&lt;}) \n&quot;.format(&quot;Purchase&quot;, fn/(tn+fn)*100,fn, tp/(tp+fp)*100, tp))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/455/1*tVl0H1qD3d2hlGt5bVGFpg.png" /><figcaption>Image by Author</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*s6WVbL5dk_eamPC3k-N5SA.png" /><figcaption>Image by Author</figcaption></figure><h3>Clean up</h3><p>This next step is important as you want to make sure you clean up afterwards and not incur unnecessary costs.</p><h4>Deleting The Endpoints</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fhSoDSK8ngZU_5PJo_flUQ.png" /><figcaption>Image by Author</figcaption></figure><ol><li>Delete your end-point and model.</li></ol><pre>xgb_predictor.delete_endpoint()<br>xgb_predictor.delete_model()</pre><p>2. Delete the training artifacts in your S3 bucket (unless you would prefer to keep them).</p><pre>bucket_to_delete = boto3.resource(&#39;s3&#39;).Bucket(bucket_name) bucket_to_delete.objects.all().delete()</pre><p>3. Back in the <a href="https://console.aws.amazon.com/sagemaker">Amazon AWS SageMaker console</a>, open “<strong>Notebook</strong>” from the menu, then select “<strong>Notebook instances</strong>”. Select your notebook instance, and in the “<strong>Actions</strong>” drop-down select “<strong>Stop</strong>”.</p><p>4. Once the notebook instance has stopped you can select “<strong>Actions</strong>” again and then select “<strong>Delete</strong>”.</p><h4>Concluding thoughts</h4><p><strong>Pros</strong></p><p>AWS Sagemaker has been a great deal for most data scientists who would want to accomplish a truly end-to-end ML solution. It takes care of abstracting a ton of software development skills necessary to accomplish the task while still being highly effective and flexible and cost-effective. Most importantly, it helps you focus on the core ML experiments and supplements the remainder necessary skills with easy abstracted tools similar to our existing workflow.</p><p><strong>Cons</strong></p><p><a href="https://console.aws.amazon.com/sagemaker">Amazon AWS SageMaker</a> is more aimed at really intensive workflows or projects with huge budgets. I mean if it cost $1 for less than a minute of processing for a tutorial one could only imagine how much it would cost for hours, days, or weeks of processing!</p><p>According to me, ML-as-a-service seems to be the buzz at the moment and I suspect companies may be interested in candidates with SageMaker experience. It doesn’t seem all that difficult to me. Amazon AWS seems to have taken a lot of the complexity away. It’s probably worth going through the tutorial and trying it out.</p><p>I hope that you find this article and explanation useful. Stay tuned for more data science-related posts in the future.</p><p>Feel free to connect and give ur suggestions:</p><p><a href="https://www.linkedin.com/in/dhyey-joshi12/">https://www.linkedin.com/in/dhyey-joshi12/</a></p><p><a href="https://github.com/dhyeyjoshi">https://github.com/dhyeyjoshi</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1210ecb7d568" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Image Segmentation — Multimodal Brain Tumor Semantic Segmentation Using 3D U-Net Architecture]]></title>
            <link>https://dhyey-joshi12.medium.com/image-segmentation-multimodal-brain-tumor-semantic-segmentation-using-3d-u-net-architecture-f4e73b7091be?source=rss-2ac64d1c310a------2</link>
            <guid isPermaLink="false">https://medium.com/p/f4e73b7091be</guid>
            <category><![CDATA[image-segmentation]]></category>
            <category><![CDATA[brain-tumor]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[mri-scan]]></category>
            <category><![CDATA[healthcare]]></category>
            <dc:creator><![CDATA[Dhyey Joshi]]></dc:creator>
            <pubDate>Sat, 07 Jan 2023 19:57:30 GMT</pubDate>
            <atom:updated>2023-02-20T12:57:11.189Z</atom:updated>
            <content:encoded><![CDATA[<h3>Image Segmentation — Multimodal Brain Tumor Semantic Segmentation Using 3D U-Net Architecture</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JcjDkXjzAR0F8MJYiJdYwA.png" /><figcaption>Fig(1) Difference between Object Detection, Semantic and Instance Segmentation</figcaption></figure><p>In digital image processing and computer vision, image segmentation is the process of partitioning a digital image into multiple segments. The goal of segmentation is to simplify and/or change the representation of an image into something more meaningful and easier to analyze.</p><h3>What is Image Segmentation?</h3><p>Image segmentation is a sub-domain of computer vision and digital image processing which aims at grouping similar regions or segments of an image under their respective class labels. Since the entire process is digital, a representation of the analog image in the form of pixels is available, making the task of forming segments equivalent to that of grouping pixels. Image segmentation is an extension of image classification where, in addition to classification, we perform localization. Image segmentation thus is a superset of image classification with the model pinpointing where a corresponding object is present by outlining the object’s boundary.</p><blockquote><strong><em>“Attaining deeper understanding of images for machine learning models</em>“</strong></blockquote><p>Some computer vision models can detect objects, determine their shape and predict the direction in which they will travel. For example, such models are at work in self-driving cars. Three important tasks undertaken by computer vision are classification, object detection and image segmentation.</p><h4>Classification</h4><p>Classification is a machine-learning task for determining which objects are in an image or video. It refers to training machine learning models with the intent of finding out which classes (objects) are present.</p><p>Classification is useful at the yes-no level of deciding whether an image contains an object/anomaly or not. A separate task from the classification is localization, or determining the position of the classified objects in the image or video.</p><h4>Object Detection</h4><p>Object detection combines classification and localization to determine what objects are in the image or video and specify where they are in the image. It applies classification to distinct objects and uses bounding boxes, as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-VMbSQqFWVx3Nvwirs4oXA.png" /><figcaption>Fig(2) Original Image VS Object Detected with ‘Red’ Bounding Box</figcaption></figure><p>Object detection is useful in identifying objects in an image or video. Below, the image on the left illustrates classification, in which the classes Donut and Coffee are identified. The image on the right illustrates object detection by surrounding the members of each class — donuts and coffee — with a bounding box.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2Ge-yrz66lwLUAv8o5j6HQ.png" /><figcaption>Fig(2) Original Image with two objects VS Two Object Detected with ‘Red’ Bounding Box</figcaption></figure><p><strong><em>Where we can use an object detection system?</em></strong></p><p>Use cases for object detection include facial detection with any post-detection analysis; for example, expression detection, age estimation or drowsiness detection. Many real-time object detection applications exist for traffic management, such as vehicle detection systems based on traffic scenes.</p><p>As described above, the most popular approaches to computer vision are classification and object detection to identify objects present in an image and specify their position. But many use cases call for analyzing images at a lower level than that. That is where I<strong><em>mage Segmentation</em></strong> comes in.</p><h4>The process of Image Segmentation</h4><p>Any image consists of both practical and useless information, depending on the user’s interest. Image segmentation separates an image into regions, each with its particular shape and border, delineating potentially meaningful areas for further processing, like classification and object detection. The parts may not take up the entire image, but the goal of image segmentation is to highlight foreground elements and make it easier to evaluate them. Image segmentation provides pixel-by-pixel details of an object, making it different from classification and object detection.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/681/1*9KkUSaHQc5iJbMCHbt30fQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/683/1*prjVGLjndg_cPXcKlotQsQ.png" /><figcaption>Fig(3) Classification, Object Detection and Segmentation</figcaption></figure><p>In computer vision, most image segmentation models consist of an encoder-decoder network as compared to a single encoder network in classifiers.</p><p>The encoder encodes a latent space representation of the input which the decoder decodes to form segment maps, or in other words maps outlining each object’s location in the image.</p><p>A typical segment map looks something like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/475/1*qMeigBefRxORg6Q48r73kg.png" /><figcaption>Fig(4) Beauty of (Semantic) Image Segmentation</figcaption></figure><h3>Image Segmentation Algorithms</h3><p>Image segmentation techniques use different algorithms.</p><p>a. Edge Detection Segmentation: Makes use of discontinuous local features of an image to detect edges and hence define a boundary of the object.</p><p>b. Mask R-CNN: Gives three outputs for each object in the image: its class, bounding box coordinates, and object mask.</p><p>c. Segmentation based on Clustering: Divides the pixels of the image into homogeneous clusters.</p><p>d. Region-Based Segmentation: Separates the objects into different regions based on a threshold value(s).</p><p>To solve segmentation problems in a given domain, it is usually necessary to combine algorithms and techniques with specific knowledge of the domain.</p><h3>Types of Image Segmentation Tasks</h3><p>Image segmentation tasks can be classified into three groups based on the amount and type of information they convey.</p><p>While semantic segmentation segments out a broad boundary of objects belonging to a particular class, instance segmentation provides a segment map for each object it views in the image, without any idea of the class the object belongs to.</p><p>Panoptic segmentation is by far the most informative, being the conjugation of instance and semantic segmentation tasks. Panoptic segmentation gives us the segment maps of all the objects of any particular class present in the image.</p><p>Let’s explore these tasks in greater detail.</p><h3>Semantic segmentation</h3><p>Semantic segmentation refers to the classification of pixels in an image into semantic classes. Pixels belonging to a particular class are simply classified to that class with no other information or context taken into consideration.</p><p>As might be expected, it is a poorly defined problem statement when there are closely grouped multiple instances of the same class in the image. An image of a crowd in a street would have a semantic segmentation model to predict the entire crowd region as belonging to the “pedestrian” class, thus providing very little in-depth detail or information on the image.</p><h3>Instance segmentation</h3><p>Instance segmentation models classify pixels into categories on the basis of “instances” rather than classes.</p><p>An instance segmentation algorithm has no idea of the class a classified region belongs to but can segregate overlapping or very similar object regions on the basis of their boundaries.</p><p>If the same image of a crowd we talked about before is fed to an instance segmentation model, the model would be able to segregate each person from the crowd as well as the surrounding objects (ideally), but would not be able to predict what each region/object is an instance of.</p><h3>Panoptic segmentation</h3><p>Panoptic segmentation, the most recently developed segmentation task, can be expressed as the combination of semantic segmentation and instance segmentation where each instance of an object in the image is segregated and the object’s identity is predicted.</p><p>Panoptic segmentation algorithms find large-scale applicability in popular tasks like self-driving cars where a huge amount of information about the immediate surroundings must be captured with the help of a stream of images.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JrhA_oGQGTrXXmTaTV94xw.png" /></figure><h3>Traditional Image Segmentation Techniques</h3><p>Image segmentation originally started from Digital Image Processing coupled with optimization algorithms. These primitive algorithms made use of methods like region growing and snakes algorithm where they set up initial regions and the algorithm compared pixel values to gain an idea of the segment map.</p><p>These methods took a local view of the features in an image and focused on local differences and gradients in pixels.</p><p>Algorithms that took a global view of the input image came much later on with methods like adaptive thresholding, Otsu’s algorithm, and clustering algorithms being proposed amongst classical image processing methods.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gp5S81WRUQq-6XszvPBxhg.png" /></figure><h3>Thresholding</h3><p>Thresholding is one of the easiest methods of image segmentation where a threshold is set for dividing pixels into two classes. Pixels that have values greater than the threshold value are set to 1 while pixels with values lesser than the threshold value are set to 0.</p><p>The image is thus converted into a binary map, resulting in the process often termed binarization. Image thresholding is very useful in case the difference in pixel values between the two target classes is very high, and it is easy to choose an average value as the threshold.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/653/1*BcCMTwnoiMSk4HEM2vG1JA.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/653/1*exHSo_XG-tLntGYoeYkqKw.jpeg" /></figure><p>Thresholding is often used for image binarization so that further algorithms like contour detection and identification that work only on binary images can be used.</p><h3>Region-Based Segmentation</h3><p>Region-based segmentation algorithms work by looking for similarities between adjacent pixels and grouping them under a common class.</p><p>Typically, the segmentation procedure starts with some pixels set as seed pixels, and the algorithm works by detecting the immediate boundaries of the seed pixels and classifying them as similar or dissimilar.</p><p>The immediate neighbours are then treated as seeds and the steps are repeated till the entire image is segmented. An example of a similar algorithm is the popular watershed algorithm for segmentation that works by starting from the local maxima of the euclidean distance map and grows under the constraint that no two seeds can be classified as belonging to the same region or segment map.</p><h3>Edge Segmentation</h3><p>Edge segmentation, also called edge detection, is the task of detecting edges in images.</p><p>From a segmentation-based viewpoint, we can say that edge detection corresponds to classifying which pixels in an image are edge pixels and singling out those edge pixels under a separate class correspondingly.</p><p>Edge detection is generally performed by using special filters that give us edges of the image upon convolution. These filters are calculated by dedicated algorithms that work on estimating image gradients in the x and y coordinates of the spatial plane.</p><p>An example of edge detection using the Canny edge detection algorithm, one of the most popular edge detection algorithms is shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*akGysXs3B2zXOTK-sZrrzw.jpeg" /></figure><h3>Clustering-based Segmentation</h3><p>Modern segmentation procedures that depend on image processing techniques generally make use of clustering algorithms for segmentation.</p><p>Clustering algorithms perform better than their counterparts and can provide reasonably good segments in a small amount of time. Popular algorithms like the K-means clustering algorithms are unsupervised algorithms that work by clustering pixels with common attributes together as belonging to a particular segment.</p><p>K-means clustering, in particular, takes all the pixels into consideration and clusters them into “k” classes. Differing from region-growing methods, clustering-based methods do not need a seed point to start segmenting from.</p><h3>Deep Learning-based methods</h3><p>Semantic segmentation models provide segment maps as outputs corresponding to the inputs they are fed.</p><p>These segment maps are often n-channelled with n being the number of classes the model is supposed to segment. Each of these n-channels is binary in nature with object locations being “filled” with ones and empty regions consisting of zeros. The ground truth map is a single channel integer array the same size as the input and has a range of “n”, with each segment “filled” with the index value of the corresponding classes (classes are indexed from 0 to n-1).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IYr_Yd-6_ip-G8v8X0rDCQ.png" /></figure><p>The model output in an “n-channel” binary format is also known as a two-dimensional one-hot encoded representation of the predictions.</p><p>Neural networks that perform segmentation typically use an encoder-decoder structure where the encoder is followed by a bottleneck and a decoder or upsampling layers directly from the bottleneck (like in the FCN).</p><h3>Convolutional Encoder-Decoder Architecture</h3><p>Encoder decoder architectures for semantic segmentation became popular with the onset of works like SegNet (by Badrinarayanan <em>et. a.</em>) in 2015.</p><p>SegNet proposes the use of a combination of convolutional and downsampling blocks to squeeze information into a bottleneck and form a representation of the input. The decoder then reconstructs input information to form a segment map highlighting regions on the input and grouping them under their classes.</p><p>Finally, the decoder has a sigmoid activation at the end that squeezes the output in the range (0,1).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pc5i4GLbHKsy_wIlmSCEVA.png" /></figure><p>SegNet was accompanied by the release of another independent segmentation work at the same time, U-Net ( by Ronnerberger <em>et. al.</em>), which first introduced skip connections in Deep Learning as a solution for the loss of information observed in downsampling layers of typical encoder-decoder networks.</p><p>Skip connections are connections that go from the encoder directly to the decoder without passing through the bottleneck.</p><p>In other words, feature maps at various levels of encoded representations are captured and concatenated to feature maps in the decoder. This helps to reduce data loss by aggressive pooling and downsampling as done in the encoder blocks of an encoder-decoder architecture.</p><p>Skip Connections were a big hit, specifically in the domain of medical imaging, with U-Net providing state-of-the-art results in cell segmentation for the diagnosis of diseases.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UA523aHaKV1QAq1xC1cNyQ.png" /></figure><p>Following UNet, DeepLab by Facebook served as a milestone, providing state-of-the-art results on semantic segmentation.</p><p>DeepLab made use of atrous convolutions replacing simple pooling operations and preventing significant information loss while downsampling. They further introduced multi-scale feature extraction with the help of Atrous Spatial Pyramid Pooling to help the network segment objects regardless of their sizes.</p><p>To recover boundary information, one of the most important parts of semantic as well as instance segmentation, they made use of fully connected Conditional Random Fields (CRFs).</p><p>Coupling the fine-grained localization accuracy of CRFs, the recognition capacity of CNNs led DeepLab to provide highly accurate segment maps, beating methods like FCNs and SegNet by a wide margin.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/361/1*1-I5mdOvlMy0Uq9fOr0pmQ.png" /></figure><p>Papers like SegNet, U-Net, and DeepLab laid the groundwork for future work like Mask-RCNN, the DeepLab series by Facebook, and works like PspNet and GSCNN.</p><h3>Applications of Image Segmentation</h3><p>Image segmentation is an important step in artificial vision. Machines need to divide visual data into segments for segment-specific processing to take place.</p><p>Image segmentation thus finds its way into prominent fields like Robotics, Medical Imaging, Autonomous Vehicles, and Intelligent Video Analytics.</p><p>Apart from these applications, Image segmentation is also used by satellites on aerial imagery for segmenting out roads, buildings, and trees.</p><p>Here are a few of the most popular real-world use cases of image segmentation.</p><h3>Robotics (Machine Vision)</h3><p>Image segmentation aids machine perception and locomotion by pointing out objects in their path of motion, enabling them to change paths effectively and understand the context of their environment.</p><p>Apart from locomotion, segmentation of images helps machines segregate the objects they are working with and enables them to interact with real-world objects using only vision as a reference. This allows the machine to be useful almost anywhere without much constraint.</p><ul><li>Instance segmentation for robotic grasping</li><li>Recycling object picking</li><li>Autonomous navigation and SLAM</li></ul><h3>Medical imaging</h3><p>Medical Imaging is an important domain of computer vision that focuses on the diagnosis of diseases from visual data, both in the form of simple visual data and biomedical scans.</p><p>Segmentation forms an important role in medical imaging as it helps doctors identify possible malignant features in images in a fast and accurate manner.</p><p>Using image segmentation, the diagnosis of diseases can not only be speeded up but can also be made cheaper, thereby benefiting thousands across the globe.</p><ul><li>X-Ray segmentation</li><li>CT scan organ segmentation</li><li>Dental instance segmentation</li><li>Digital pathology cell segmentation</li><li>Surgical video annotation</li></ul><h3>Smart Cities</h3><p>Smart Cities often have CCTV cameras for real-time monitoring of pedestrians, traffic, and crime. This monitoring can be easily automated with the help of image segmentation.</p><p>With AI-based monitoring, crimes can be reported faster, road accidents can be followed up with immediate ambulances, and speeding cars can be easily caught and penalized.</p><p>The use of image segmentation and AI-based monitoring can thus improve the lifestyle of people.</p><ul><li>Pedestrian detection</li><li>Traffic analytics</li><li>License plate detection</li><li>Video Surveillance</li></ul><h3>Self Driving Cars</h3><p>Self Driving cars are one of the biggest applications of image segmentation with the planning of routes and movement depending heavily on it.</p><p>Semantic and instance segmentation helps these vehicles to identify road patterns and other vehicles, thereby enabling a hassle-free and smooth ride.</p><ul><li>Drivable surface semantic segmentation</li><li>Car and pedestrian instance segmentation</li><li>In-vehicle object detection (stuff left behind by passengers)</li><li>Pothole detection and segmentation</li></ul><h3>Metrics to Evaluate your Semantic Segmentation Model</h3><p>When evaluating a standard machine learning model, we usually classify our predictions into four categories: true positives, false positives, true negatives, and false negatives. However, for the dense prediction task of image segmentation, it’s not immediately clear what counts as a “true positive” and, more generally, how we can evaluate our predictions. In this post, I’ll discuss common methods for evaluating both semantic and instance segmentation techniques.</p><h4>1. Intersection-Over-Union (Jaccard Index)</h4><p>The Intersection over Union (IoU) metric, also referred to as the Jaccard index, is essentially a method to quantify the percent overlap between the target mask and our prediction output. This metric is closely related to the Dice coefficient which is often used as a loss function during training.</p><p>Quite simply, the IoU metric measures the number of pixels common between the target and prediction masks divided by the total number of pixels present across <em>both</em> masks.</p><p>IoU = target ∩ prediction / target ∪ prediction</p><p>As a visual example, let’s suppose we’re tasked with calculating the IoU score of the following prediction, given the ground truth labeled mask.</p><p>We can calculate this easily using Numpy.</p><pre>intersection = np.logical_and(target, prediction)<br>union = np.logical_or(target, prediction)<br>iou_score = np.sum(intersection) / np.sum(union)</pre><p>The IoU score is calculated for each class separately and then <strong>averaged over all classes</strong> to provide a global, mean IoU score of our semantic segmentation prediction.</p><h4>2. Pixel Accuracy</h4><p>An alternative metric to evaluate a semantic segmentation is to simply report the percent of pixels in the image which were correctly classified. The pixel accuracy is commonly reported for each class separately as well as globally across all classes.</p><p>When considering the per-class pixel accuracy we’re essentially evaluating a binary mask; a true positive represents a pixel that is correctly predicted to belong to the given class (according to the target mask) whereas a true negative represents a pixel that is correctly identified as not belonging to the given class.</p><p>accuracy = TP+TN / TP+TN+FP+FN</p><p>This metric can sometimes provide misleading results when the class representation is small within the image, as the measure will be biased in mainly reporting how well you identify negative case (ie. where the class is not present).</p><h4>3. Dice Coefficient (F1 Score)</h4><p>Dice coefficient is very similar to Jaccard’s Index. Dice coefficient double counts the intersection(TP).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/786/1*qbKt9zyHOV1kf4fqFyL-aw.jpeg" /></figure><p>Dice coefficient is a measure of overlap between two masks.1 indicates a perfect overlap while 0 indicates no overlap.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/786/1*05nhB-OoBL5jIso17fWB7A.jpeg" /></figure><p>Dice Loss = 1 — Dice Coefficient. Easy!</p><p>We calculate the gradient of Dice Loss in backpropagation.</p><blockquote>Why is Dice Loss used instead of Jaccard’s? Because Dice is easily differentiable and Jaccard’s is not.</blockquote><h4>Prerequisites Before Implementing Image Segmentation</h4><ol><li>Basic to Advance Python Programming Language</li><li>Digital Image Processing Fundamentals.</li><li>Segmentation of Grey level &amp; RGB images.</li><li>Detection of edges and lines in 2D/3D images. (Sobel operator, Prewitt operator, Robert operator, Canny edge detector, Laplacian of Gaussian etc.)</li><li>Images Enhancement Techniques.</li><li>Familiar with image processing and annotation tools like ImageJ, zen, and Apeer.</li><li>Basics of computer vision and feature detection.</li><li>Basic understanding of Deep Learning Architecture ( Lenet, Alexnet, VGG, InceptionV3 etc..)</li></ol><p>Let&#39;s begin Project 01:</p><h3>Multimodal Brain Tumor Semantic Segmentation Using 3D U-Net</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MRIfbaoX1jVYIYwlnTBJZw.gif" /></figure><p>Manual identification of brain tumours is an error-prone and tedious process for radiologists; therefore, it is crucial to adopt an automated system. The binary classification process, such as malignant or benign is relatively trivial; whereas, the multimodal brain tumours classification (T1, T2, T1CE, and Flair) is a challenging task for radiologists. Here, we present an automated multimodal classification method using 3D- Unet deep learning architecture for brain tumour-type classification. The proposed method begins with linear contrast stretching on a 3D Layered Image using edge-based histogram equalization. In the second step, feature extraction is performed. By utilizing the 2D -Unet model and converting it into 3D- Unet to handle the 3D Volume of data for feature extraction. Then combined evaluation matrix (Pixel Accuracy, Intersection-Over-Union (Jaccard Index) and Dice Coefficient (F1 Score)) were fed to 3D-Unet for final image segmentation. The proposed method was validated on the BraTS 2020 datasets and achieved an accuracy of 92.5% and a mean IOU of around 0.834.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JQf_wTdD9ewstZY-Z5cwBQ.png" /><figcaption>Fig(5) Methodology (Work-flow) Semantic segmentation on MRI Brain Tumor Scans.</figcaption></figure><blockquote><strong><em>Dataset Information:</em></strong></blockquote><p>I used the BraTS dataset 2020 samples for training and BraTS dataset 2021 samples for validation.</p><p>URL 1: <a href="https://www.kaggle.com/datasets/awsaf49/brats2020-training-data">https://www.kaggle.com/datasets/awsaf49/brats2020-training-data</a></p><p>URL 2: <a href="https://www.kaggle.com/datasets/dschettler8845/brats-2021-task1">https://www.kaggle.com/datasets/dschettler8845/brats-2021-task1</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sphzhDfGA2Dfux3wmD3Rag.png" /><figcaption>Fig(6) Dataset source from Kaggle. .</figcaption></figure><ul><li>Multimodal scans are available as NIFTI files (.nii.gz).</li><li>Four ‘channels’ of information — 4 different volumes of the same region.</li></ul><p>i. Native (T1).</p><p>ii. Post-contrast T1-weighted (T1CE).</p><p>iii. T2-weighted (T2).</p><p>iv. T2 Fluid Attenuated Inversion Recovery (FLAIR) volumes.</p><ul><li>All the imaging datasets have been segmented manually and were approved by experienced neuro-radiologists.</li><li>Annotations (labels):</li></ul><p>Label 0: Unlabeled volume</p><p>Label 1: Necrotic and non-enhancing tumour core (NCR/NET)</p><p>Label 2: Peritumoral edema (ED)</p><p><strong>Label 3: Missing <em>(No pixels in all the volumes contain label 3)</em></strong></p><p>Label 4: GD-enhancing tumour (ET)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/710/1*UqycuHSM6yKpW_lj4m42Sw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*MpqfGE1B3S60Wj6lZGnxhw.gif" /><figcaption>Fig(7) The four volumes of the same region are represented by the images above, and the last image is for ‘Mask.’</figcaption></figure><blockquote><strong>Proposed Methodology:</strong></blockquote><p><strong>Step 1: Get the data ready.</strong></p><ul><li>Download the dataset and unzip it.</li><li>Install nibabel library to handle nii files (https://pypi.org/project/nibabel/)</li></ul><blockquote>NII format is created by Neuroimaging Informatics Technology Initiative. It is commonly used to store magnetic resonance imaging (MRI) data.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/615/1*YqeONShZSBZUvggIkzJ9bQ.png" /><figcaption>Fig(8) 371 such folders</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/784/1*VmkX6dSrXpr4SzsTU0Ynhw.png" /><figcaption>Fig(9) Each folder contains a total of five NII Files. Out of which 4 files are the different volumes of the same region of the brain tumour and one file is titled ‘_seg.nii ’ is the mask.</figcaption></figure><p>After obtaining the dataset, proceed with the procedures below to prepare it for training.</p><ul><li>Scale all volumes (using MinMaxScaler).</li><li>Combine the three non-native volumes (T2, T1CE and Flair) into a single multi-channel volume.</li><li>Reassign pixels of value 4 to value 3 (as 3 is missing from the original labels).</li><li>Crop volumes to remove useless blank regions around the actual volume of interest (Crop to 128x128x128).</li><li>Drop all volumes where the amount of annotated data is less than a certain percentage (1%). (To maximize training on real labelled volumes).</li><li>Save all useful volumes to the local drive as NumPy arrays (npy).</li></ul><pre>import numpy as np<br>import nibabel as nib<br>import glob<br>from tensorflow.keras.utils import to_categorical<br>import matplotlib.pyplot as plt<br>from tifffile import imsave<br><br>from sklearn.preprocessing import MinMaxScaler<br>scaler = MinMaxScaler()<br><br>### Understanding the single image data and applying the operations<br><br># Loading Nibabel Images<br><br>TRAIN_DATASET_PATH = &#39;BraTS2020_TrainingData/MICCAI_BraTS2020_TrainingData/&#39;<br>#VALIDATION_DATASET_PATH = &#39;BraTS2020_ValidationData/MICCAI_BraTS2020_ValidationData&#39;<br><br>test_image_flair=nib.load(TRAIN_DATASET_PATH + &#39;BraTS20_Training_355/BraTS20_Training_355_flair.nii&#39;).get_fdata()<br>print(test_image_flair.max())<br><br>#Scalers are applied to 1D so let us reshape and then reshape back to original shape. <br>test_image_flair=scaler.fit_transform(test_image_flair.reshape(-1, test_image_flair.shape[-1])).reshape(test_image_flair.shape)<br><br>print(test_image_flair.max())<br><br>test_image_t1=nib.load(TRAIN_DATASET_PATH + &#39;BraTS20_Training_355/BraTS20_Training_355_t1.nii&#39;).get_fdata()<br>test_image_t1=scaler.fit_transform(test_image_t1.reshape(-1, test_image_t1.shape[-1])).reshape(test_image_t1.shape)<br><br>test_image_t1ce=nib.load(TRAIN_DATASET_PATH + &#39;BraTS20_Training_355/BraTS20_Training_355_t1ce.nii&#39;).get_fdata()<br>test_image_t1ce=scaler.fit_transform(test_image_t1ce.reshape(-1, test_image_t1ce.shape[-1])).reshape(test_image_t1ce.shape)<br><br>test_image_t2=nib.load(TRAIN_DATASET_PATH + &#39;BraTS20_Training_355/BraTS20_Training_355_t2.nii&#39;).get_fdata()<br>test_image_t2=scaler.fit_transform(test_image_t2.reshape(-1, test_image_t2.shape[-1])).reshape(test_image_t2.shape)<br><br># Loading Mask<br><br>test_mask=nib.load(TRAIN_DATASET_PATH + &#39;BraTS20_Training_355/BraTS20_Training_355_seg.nii&#39;).get_fdata()<br><br>test_mask=test_mask.astype(np.uint8)<br><br>print(np.unique(test_mask))  #0, 1, 2, 4 (Need to reencode to 0, 1, 2, 3)<br><br>test_mask[test_mask==4] = 3  #Reassign mask values 4 to 3<br>print(np.unique(test_mask)) <br><br># Visualizing random Brain images and their segmented Mask<br><br>import random<br>n_slice=random.randint(0, test_mask.shape[2])<br><br>plt.figure(figsize=(12, 8))<br><br>plt.subplot(231)<br>plt.imshow(test_image_flair[:,:,n_slice], cmap=&#39;gray&#39;)<br>plt.title(&#39;Image flair&#39;)<br>plt.subplot(232)<br>plt.imshow(test_image_t1[:,:,n_slice], cmap=&#39;gray&#39;)<br>plt.title(&#39;Image t1&#39;)<br>plt.subplot(233)<br>plt.imshow(test_image_t1ce[:,:,n_slice], cmap=&#39;gray&#39;)<br>plt.title(&#39;Image t1ce&#39;)<br>plt.subplot(234)<br>plt.imshow(test_image_t2[:,:,n_slice], cmap=&#39;gray&#39;)<br>plt.title(&#39;Image t2&#39;)<br>plt.subplot(235)<br>plt.imshow(test_mask[:,:,n_slice])<br>plt.title(&#39;Mask&#39;)<br>plt.show()<br><br># Now combining images to channels and divide them to patches<br><br><br>combined_x = np.stack([test_image_flair, test_image_t1ce, test_image_t2], axis=3)<br><br>combined_x.shape<br><br>combined_x=combined_x[56:184, 56:184, 13:141] #Crop to 128x128x128x4<br><br>#Do the same for mask<br>test_mask = test_mask[56:184, 56:184, 13:141]<br><br># Plot the images, but now you will see images little bit zoomed as unwanted edges are croped<br><br>n_slice=random.randint(0, test_mask.shape[2])<br>plt.figure(figsize=(12, 8))<br><br>plt.subplot(221)<br>plt.imshow(combined_x[:,:,n_slice, 0], cmap=&#39;gray&#39;)<br>plt.title(&#39;Image flair&#39;)<br><br>plt.subplot(222)<br>plt.imshow(combined_x[:,:,n_slice, 1], cmap=&#39;gray&#39;)<br>plt.title(&#39;Image t1ce&#39;)<br><br>plt.subplot(223)<br>plt.imshow(combined_x[:,:,n_slice, 2], cmap=&#39;gray&#39;)<br>plt.title(&#39;Image t2&#39;)<br><br>plt.subplot(224)<br>plt.imshow(test_mask[:,:,n_slice])<br>plt.title(&#39;Mask&#39;)<br><br>plt.show()<br><br>Saving the combined image mask into numpy array or as a tif file.<br><br>imsave(&#39;BraTS2020_TrainingData/combined255.tif&#39;, combined_x)<br>np.save(&#39;BraTS2020_TrainingData/combined255.npy&#39;, combined_x)<br><br>my_img=np.load(&#39;BraTS2020_TrainingData/combined255.npy&#39;)<br><br>my_img == combined_x # Both have same data</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/609/1*O8ClW1HHYssGtyWsHoRv7A.png" /><figcaption>Fig(10) Images little bit zoomed as unwanted edges are cropped (128x128x128)</figcaption></figure><pre>### Applying the same operations on all the images inside BraTS Training Data Folder<br><br><br># # # images lists harley<br><br>#t1_list = sorted(glob.glob(&#39;BraTS2020_TrainingData/MICCAI_BraTS2020_TrainingData/*/*t1.nii&#39;))<br><br>t2_list = sorted(glob.glob(&#39;BraTS2020_TrainingData/MICCAI_BraTS2020_TrainingData/*/*t2.nii&#39;))<br><br>t1ce_list = sorted(glob.glob(&#39;BraTS2020_TrainingData/MICCAI_BraTS2020_TrainingData/*/*t1ce.nii&#39;))<br><br>flair_list = sorted(glob.glob(&#39;BraTS2020_TrainingData/MICCAI_BraTS2020_TrainingData/*/*flair.nii&#39;))<br><br>mask_list = sorted(glob.glob(&#39;BraTS2020_TrainingData/MICCAI_BraTS2020_TrainingData/*/*seg.nii&#39;))<br><br>print(t2_list[1]) <br>print()<br>print(t1ce_list[1])<br>print()<br>print(flair_list[1])<br>print()<br>print(mask_list[1])<br><br>for img in range(len(t2_list)):   #Using t2_list as all lists are of same size<br>    print(&quot;Now preparing image and masks number: &quot;, img)<br>    <br>#     Normalizing the pixel values of image between 0 to 1 using Minmax Scaler<br><br>    temp_image_t2=nib.load(t2_list[img]).get_fdata()<br>    temp_image_t2=scaler.fit_transform(temp_image_t2.reshape(-1, temp_image_t2.shape[-1])).reshape(temp_image_t2.shape)<br>   <br>    temp_image_t1ce=nib.load(t1ce_list[img]).get_fdata()<br>    temp_image_t1ce=scaler.fit_transform(temp_image_t1ce.reshape(-1, temp_image_t1ce.shape[-1])).reshape(temp_image_t1ce.shape)<br>   <br>    temp_image_flair=nib.load(flair_list[img]).get_fdata()<br>    temp_image_flair=scaler.fit_transform(temp_image_flair.reshape(-1, temp_image_flair.shape[-1])).reshape(temp_image_flair.shape)<br>    <br>    temp_mask=nib.load(mask_list[img]).get_fdata()<br>    temp_mask=temp_mask.astype(np.uint8)<br>    temp_mask[temp_mask==4] = 3  #Reassign mask values 4 to 3<br>    <br>#     print(np.unique(temp_mask))<br><br>#     Combining t2_image, t1ce_image and flair_image into single combined_image with three channels t2,t1ce &amp; flair <br>    <br>    temp_combined_images = np.stack([temp_image_flair, temp_image_t1ce, temp_image_t2], axis=3)<br>        <br>#     croping the image into 128*128*128*3 - zooming the image and removing extra edges of image<br><br>    temp_combined_images=temp_combined_images[56:184, 56:184, 13:141] #random trial and error<br>    <br>    temp_mask = temp_mask[56:184, 56:184, 13:141] #random trial and error<br>    <br>#     Removing unwanted images having mask segmentation is less than 1% ~ 0.01 <br>    <br>    val, counts = np.unique(temp_mask, return_counts=True)<br>    <br>    if (1 - (counts[0]/counts.sum())) &gt; 0.01:  #At least 0.5% useful volume with labels that are not 0<br>        print(&quot;Save Me&quot;)<br>        <br>#         saving masks of each image which contain labels 0,1,2,3 in it in Numpy array format<br><br>        temp_mask= to_categorical(temp_mask, num_classes=4) # 0,1,2,3 classes<br>    <br>        np.save(&#39;BraTS2020_TrainingData/input_data_3channels/images/image_&#39;+str(img)+&#39;.npy&#39;, temp_combined_images)<br>        np.save(&#39;BraTS2020_TrainingData/input_data_3channels/masks/mask_&#39;+str(img)+&#39;.npy&#39;, temp_mask)<br>        <br>    else:<br>        print(&quot;I am useless&quot;)<br><br>print(temp_combined_images.shape)<br>print(temp_mask.shape)<br><br>## Splitting the data into train, test and val<br><br># !pip install split-folders<br><br>import splitfolders  # or import split_folders<br><br>input_folder = &#39;BraTS2020_TrainingData/input_data_3channels/&#39;<br>output_folder = &#39;BraTS2020_TrainingData/input_data_128/&#39;<br><br># Split with a ratio.<br># To only split into training and validation set, set a tuple to `ratio`, i.e, `(.8, .2)`.<br>splitfolders.ratio(input_folder, output=output_folder, seed=42, ratio=(.75, .25), group_prefix=None) # default values</pre><ul><li>Split image and mask volumes into train and validation datasets.</li></ul><p><strong>Step 2: Define the custom data generator.</strong></p><ul><li>Keras image data generator only works with jpg, png, and tif images. It will not recognize npy files. We need to define a custom generator to load our data from the disk.</li></ul><pre># Custom Data Generator <br><br>import os<br>import numpy as np<br>from matplotlib import pyplot as plt<br>import random<br><br># Define fuction to load the image file of format &quot;npy&quot;<br># Taking input as image directory and image list<br><br>def load_img(img_dir, img_list):<br>    images=[]<br>    for i, image_name in enumerate(img_list):    <br>        if (image_name.split(&#39;.&#39;)[1] == &#39;npy&#39;):<br>            <br>            image = np.load(img_dir+image_name)<br>                      <br>            images.append(image)<br>    images = np.array(images)<br>    <br>    return(images)<br><br># Custom Data Generator <br># Input as image directory, image list, mask directory and mask list<br># Setting batch size<br><br>def imageLoader(img_dir, img_list, mask_dir, mask_list, batch_size):<br><br>    L = len(img_list)<br><br>    #keras needs the generator infinite, so we will use while true  <br>    while True:<br><br>        batch_start = 0<br>        batch_end = batch_size<br><br>        while batch_start &lt; L:<br>            limit = min(batch_end, L)<br>                       <br>            X = load_img(img_dir, img_list[batch_start:limit])<br>            Y = load_img(mask_dir, mask_list[batch_start:limit])<br><br>            yield (X,Y) #a tuple with two numpy arrays with batch_size samples     <br><br>            batch_start += batch_size   <br>            batch_end += batch_size<br><br># Test the data generator<br><br>train_img_dir = &quot;BraTS2020_TrainingData/input_data_128/train/images/&quot;<br>train_mask_dir = &quot;BraTS2020_TrainingData/input_data_128/train/masks/&quot;<br>train_img_list=os.listdir(train_img_dir)<br>train_mask_list = os.listdir(train_mask_dir)<br><br>batch_size = 2<br><br>train_img_datagen = imageLoader(train_img_dir, train_img_list, <br>                                train_mask_dir, train_mask_list, batch_size)<br><br><br>#Verify generator.... In python 3 next() is renamed as __next__()<br>img, msk = train_img_datagen.__next__()<br><br>img.shape[0]<br><br>img_num = random.randint(0,img.shape[0]-1)<br><br>test_img=img[img_num] <br><br>test_mask=msk[img_num]<br><br>test_mask=np.argmax(test_mask, axis=3)<br><br>test_mask<br><br>n_slice=random.randint(0, test_mask.shape[2])<br><br>plt.figure(figsize=(12, 8))<br><br>plt.subplot(221)<br>plt.imshow(test_img[:,:,n_slice, 0], cmap=&#39;gray&#39;)<br>plt.title(&#39;Image flair&#39;)<br>plt.subplot(222)<br>plt.imshow(test_img[:,:,n_slice, 1], cmap=&#39;gray&#39;)<br>plt.title(&#39;Image t1ce&#39;)<br>plt.subplot(223)<br>plt.imshow(test_img[:,:,n_slice, 2], cmap=&#39;gray&#39;)<br>plt.title(&#39;Image t2&#39;)<br>plt.subplot(224)<br>plt.imshow(test_mask[:,:,n_slice])<br>plt.title(&#39;Mask&#39;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/609/1*ZjyI0ngWnE7KeGjqx16zTw.png" /><figcaption>Fig(11) Viewing random training images (npy) and their mask from the train folder.</figcaption></figure><blockquote>We previously integrated three channels (volumes) of image information onto a single (combined) image to utilise for training. (The ‘t1’ channel was removed since it was unnecessary and had less characteristics.)</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*54blqTcQX-OwDSKgvZXQIQ.png" /><figcaption>Fig(12) Combined Image Layers into a single image and used that image for training the model.</figcaption></figure><h4><strong>One Image — Contain 4 Different Volumes. Thus the size of an image increases. (Top View)</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/434/1*NUYkq2LGlw1_FULez3x-_g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/443/1*UngyjYeLpUwhYzT16pY7pw.png" /><figcaption><strong>Before VS After</strong></figcaption></figure><p><strong>Step 3: Define the 3D U-net model.</strong></p><ul><li>Extend the standard 2D U-Net into 3D OR</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/980/1*kmYyhuK9Z5kOb7ef2TJ-0g.png" /><figcaption>Fig(13) Defining 3D U-net architecture with input size of “ 128x128x3 “</figcaption></figure><pre># u-net model <br><br>from keras.models import Model<br>from keras.layers import Input, Conv3D, MaxPooling3D, UpSampling3D, concatenate, Conv3DTranspose, BatchNormalization, Dropout, Lambda<br>from keras import backend as K<br>from keras.optimizers import Adam<br>from keras.metrics import MeanIoU<br>import keras as k<br><br>kernel_initializer = &#39;he_uniform&#39;<br><br><br>def jacard_coef(y_true, y_pred):<br>    y_true_f = K.flatten(y_true)<br>    y_pred_f = K.flatten(y_pred)<br>    intersection = K.sum(y_true_f * y_pred_f)<br>    return (intersection + 1.0) / (K.sum(y_true_f) + K.sum(y_pred_f) - intersection + 1.0)<br><br><br>def jacard_coef_loss(y_true, y_pred):<br>    return -jacard_coef(y_true, y_pred)<br><br>################################################################<br>def simple_unet_model(IMG_HEIGHT, IMG_WIDTH, IMG_DEPTH, IMG_CHANNELS, num_classes):<br>#Build the model<br>    inputs = Input((IMG_HEIGHT, IMG_WIDTH, IMG_DEPTH, IMG_CHANNELS))<br>    #s = Lambda(lambda x: x / 255)(inputs)   #No need for this if we normalize our inputs beforehand<br>    s = inputs<br><br>    #Contraction path<br>    c1 = Conv3D(16, (3, 3, 3), activation=&#39;relu&#39;, kernel_initializer=kernel_initializer, padding=&#39;same&#39;)(s)<br>    c1 = Dropout(0.1)(c1)<br>    c1 = Conv3D(16, (3, 3, 3), activation=&#39;relu&#39;, kernel_initializer=kernel_initializer, padding=&#39;same&#39;)(c1)<br>    p1 = MaxPooling3D((2, 2, 2))(c1)<br>    <br>    c2 = Conv3D(32, (3, 3, 3), activation=&#39;relu&#39;, kernel_initializer=kernel_initializer, padding=&#39;same&#39;)(p1)<br>    c2 = Dropout(0.1)(c2)<br>    c2 = Conv3D(32, (3, 3, 3), activation=&#39;relu&#39;, kernel_initializer=kernel_initializer, padding=&#39;same&#39;)(c2)<br>    p2 = MaxPooling3D((2, 2, 2))(c2)<br>     <br>    c3 = Conv3D(64, (3, 3, 3), activation=&#39;relu&#39;, kernel_initializer=kernel_initializer, padding=&#39;same&#39;)(p2)<br>    c3 = Dropout(0.2)(c3)<br>    c3 = Conv3D(64, (3, 3, 3), activation=&#39;relu&#39;, kernel_initializer=kernel_initializer, padding=&#39;same&#39;)(c3)<br>    p3 = MaxPooling3D((2, 2, 2))(c3)<br>     <br>    c4 = Conv3D(128, (3, 3, 3), activation=&#39;relu&#39;, kernel_initializer=kernel_initializer, padding=&#39;same&#39;)(p3)<br>    c4 = Dropout(0.2)(c4)<br>    c4 = Conv3D(128, (3, 3, 3), activation=&#39;relu&#39;, kernel_initializer=kernel_initializer, padding=&#39;same&#39;)(c4)<br>    p4 = MaxPooling3D(pool_size=(2, 2, 2))(c4)<br>     <br>    c5 = Conv3D(256, (3, 3, 3), activation=&#39;relu&#39;, kernel_initializer=kernel_initializer, padding=&#39;same&#39;)(p4)<br>    c5 = Dropout(0.3)(c5)<br>    c5 = Conv3D(256, (3, 3, 3), activation=&#39;relu&#39;, kernel_initializer=kernel_initializer, padding=&#39;same&#39;)(c5)<br>    <br>    #Expansive path <br>    u6 = Conv3DTranspose(128, (2, 2, 2), strides=(2, 2, 2), padding=&#39;same&#39;)(c5)<br>    u6 = concatenate([u6, c4])<br>    c6 = Conv3D(128, (3, 3, 3), activation=&#39;relu&#39;, kernel_initializer=kernel_initializer, padding=&#39;same&#39;)(u6)<br>    c6 = Dropout(0.2)(c6)<br>    c6 = Conv3D(128, (3, 3, 3), activation=&#39;relu&#39;, kernel_initializer=kernel_initializer, padding=&#39;same&#39;)(c6)<br>     <br>    u7 = Conv3DTranspose(64, (2, 2, 2), strides=(2, 2, 2), padding=&#39;same&#39;)(c6)<br>    u7 = concatenate([u7, c3])<br>    c7 = Conv3D(64, (3, 3, 3), activation=&#39;relu&#39;, kernel_initializer=kernel_initializer, padding=&#39;same&#39;)(u7)<br>    c7 = Dropout(0.2)(c7)<br>    c7 = Conv3D(64, (3, 3, 3), activation=&#39;relu&#39;, kernel_initializer=kernel_initializer, padding=&#39;same&#39;)(c7)<br>     <br>    u8 = Conv3DTranspose(32, (2, 2, 2), strides=(2, 2, 2), padding=&#39;same&#39;)(c7)<br>    u8 = concatenate([u8, c2])<br>    c8 = Conv3D(32, (3, 3, 3), activation=&#39;relu&#39;, kernel_initializer=kernel_initializer, padding=&#39;same&#39;)(u8)<br>    c8 = Dropout(0.1)(c8)<br>    c8 = Conv3D(32, (3, 3, 3), activation=&#39;relu&#39;, kernel_initializer=kernel_initializer, padding=&#39;same&#39;)(c8)<br>     <br>    u9 = Conv3DTranspose(16, (2, 2, 2), strides=(2, 2, 2), padding=&#39;same&#39;)(c8)<br>    u9 = concatenate([u9, c1])<br>    c9 = Conv3D(16, (3, 3, 3), activation=&#39;relu&#39;, kernel_initializer=kernel_initializer, padding=&#39;same&#39;)(u9)<br>    c9 = Dropout(0.1)(c9)<br>    c9 = Conv3D(16, (3, 3, 3), activation=&#39;relu&#39;, kernel_initializer=kernel_initializer, padding=&#39;same&#39;)(c9)<br>     <br>    outputs = Conv3D(num_classes, (1, 1, 1), activation=&#39;softmax&#39;)(c9)<br>     <br>    model = Model(inputs=[inputs], outputs=[outputs])<br>    #compile model outside of this function to make it flexible. <br>    model.summary()<br>    <br>    return model<br><br>model = simple_unet_model(128, 128, 128, 3, 4)<br><br>print(model.input_shape)<br>print(model.output_shape)</pre><ul><li>Use the 3D segmentation models library <em>(pip install segmentation-models-3D). </em>You can also read the documentation on how to use this library for image segmentation. The best part of using this library is they provide an inbuilt function to calculate Loss, IOU, dice Coiffient etc. Thus it becomes easy for a developer to validate the model using different performance evaluation matrices.</li></ul><p><strong>Step 4: Training the 3D U-net model and predicting tumours from Brain MRI Scans.</strong></p><ul><li>Train by loading the images in batches using our custom generator</li><li>Predict and plot the data for visualization.</li></ul><pre># Training the Unet - model<br><br>import os<br>import numpy as np<br>#import tensorflow as tf<br>import keras<br>from matplotlib import pyplot as plt<br>import glob<br>import random<br><br>####################################################<br>train_img_dir = &quot;BraTS2020_TrainingData/input_data_128/train/images/&quot;<br>train_mask_dir = &quot;BraTS2020_TrainingData/input_data_128/train/masks/&quot;<br><br>img_list = os.listdir(train_img_dir)<br>msk_list = os.listdir(train_mask_dir)<br><br>num_images = len(os.listdir(train_img_dir))<br><br>img_num = random.randint(0,num_images-1)<br>test_img = np.load(train_img_dir+img_list[img_num])<br>test_mask = np.load(train_mask_dir+msk_list[img_num])<br>test_mask = np.argmax(test_mask, axis=3)<br><br>n_slice=random.randint(0, test_mask.shape[2])<br>plt.figure(figsize=(12, 8))<br><br>plt.subplot(221)<br>plt.imshow(test_img[:,:,n_slice, 0], cmap=&#39;gray&#39;)<br>plt.title(&#39;Image flair&#39;)<br>plt.subplot(222)<br>plt.imshow(test_img[:,:,n_slice, 1], cmap=&#39;gray&#39;)<br>plt.title(&#39;Image t1ce&#39;)<br>plt.subplot(223)<br>plt.imshow(test_img[:,:,n_slice, 2], cmap=&#39;gray&#39;)<br>plt.title(&#39;Image t2&#39;)<br>plt.subplot(224)<br>plt.imshow(test_mask[:,:,n_slice])<br>plt.title(&#39;Mask&#39;)<br>plt.show()<br><br>##############################################################<br>#Define the image generators for training and validation<br><br>train_img_dir = &quot;BraTS2020_TrainingData/input_data_128/train/images/&quot;<br>train_mask_dir = &quot;BraTS2020_TrainingData/input_data_128/train/masks/&quot;<br><br>val_img_dir = &quot;BraTS2020_TrainingData/input_data_128/val/images/&quot;<br>val_mask_dir = &quot;BraTS2020_TrainingData/input_data_128/val/masks/&quot;<br><br>train_img_list=os.listdir(train_img_dir)<br>train_mask_list = os.listdir(train_mask_dir)<br><br>val_img_list=os.listdir(val_img_dir)<br>val_mask_list = os.listdir(val_mask_dir)<br><br>########################################################################<br>batch_size = 2<br><br>train_img_datagen = imageLoader(train_img_dir, train_img_list, <br>                                train_mask_dir, train_mask_list, batch_size)<br><br>val_img_datagen = imageLoader(val_img_dir, val_img_list, <br>                                val_mask_dir, val_mask_list, batch_size)<br><br>#Verify generator.... In python 3 next() is renamed as __next__()<br>img, msk = train_img_datagen.__next__()<br><br>img_num = random.randint(0,img.shape[0]-1)<br>test_img=img[img_num]<br>test_mask=msk[img_num]<br>test_mask=np.argmax(test_mask, axis=3)<br><br>n_slice=random.randint(0, test_mask.shape[2])<br>plt.figure(figsize=(12, 8))<br><br>plt.subplot(221)<br>plt.imshow(test_img[:,:,n_slice, 0], cmap=&#39;gray&#39;)<br>plt.title(&#39;Image flair&#39;)<br>plt.subplot(222)<br>plt.imshow(test_img[:,:,n_slice, 1], cmap=&#39;gray&#39;)<br>plt.title(&#39;Image t1ce&#39;)<br>plt.subplot(223)<br>plt.imshow(test_img[:,:,n_slice, 2], cmap=&#39;gray&#39;)<br>plt.title(&#39;Image t2&#39;)<br>plt.subplot(224)<br>plt.imshow(test_mask[:,:,n_slice])<br>plt.title(&#39;Mask&#39;)<br>plt.show()<br><br>#######################################################################<br>#Fit the model <br><br>steps_per_epoch = len(train_img_list)//batch_size<br>val_steps_per_epoch = len(val_img_list)//batch_size</pre><pre>model = simple_unet_model(IMG_HEIGHT=128, <br>                          IMG_WIDTH=128, <br>                          IMG_DEPTH=128, <br>                          IMG_CHANNELS=3, <br>                          num_classes=4)<br><br>LR = 0.0001<br>optim = keras.optimizers.Adam(LR)<br><br>metrics = [&#39;accuracy&#39;, k.metrics.MeanIoU(num_classes=4), jacard_coef]<br><br>model.compile(optimizer = optim, loss = [jacard_coef_loss], metrics = metrics)<br><br>model.summary()<br><br>print(model.input_shape)<br>print(model.output_shape)<br><br>val_steps_per_epoch<br><br>history=model.fit(train_img_datagen,<br>          steps_per_epoch=25,<br>          epochs=100,<br>          verbose=1,<br>          validation_data=val_img_datagen,<br>          validation_steps=15,<br>          )<br><br>#plot the training and validation IoU and loss at each epoch<br>loss = history.history[&#39;loss&#39;]<br>val_loss = history.history[&#39;val_loss&#39;]<br>epochs = range(1, len(loss) + 1)<br>plt.plot(epochs, loss, &#39;y&#39;, label=&#39;Training loss&#39;)<br>plt.plot(epochs, val_loss, &#39;r&#39;, label=&#39;Validation loss&#39;)<br>plt.title(&#39;Training and validation loss&#39;)<br>plt.xlabel(&#39;Epochs&#39;)<br>plt.ylabel(&#39;Loss&#39;)<br>plt.legend()<br>plt.show()<br><br>acc = history.history[&#39;accuracy&#39;]<br>val_acc = history.history[&#39;val_accuracy&#39;]<br><br>plt.plot(epochs, acc, &#39;y&#39;, label=&#39;Training accuracy&#39;)<br>plt.plot(epochs, val_acc, &#39;r&#39;, label=&#39;Validation accuracy&#39;)<br>plt.title(&#39;Training and validation accuracy&#39;)<br>plt.xlabel(&#39;Epochs&#39;)<br>plt.ylabel(&#39;Accuracy&#39;)<br>plt.legend()<br>plt.show()<br><br><br># from keras.models import load_model<br><br># #Load model for prediction or continue training<br><br># #For continuing training....<br># #The following gives an error: Unknown loss function: dice_loss_plus_1focal_loss<br># #This is because the model does not save loss function and metrics. So to compile and <br># #continue training we need to provide these as custom_objects.<br># my_model = load_model(&#39;saved_models/brats_3d_100epochs_simple_unet_weighted_dice.hdf5&#39;)<br><br># #So let us add the loss as custom object... but the following throws another error...<br># #Unknown metric function: iou_score<br># my_model = load_model(&#39;saved_models/brats_3d_100epochs_simple_unet_weighted_dice.hdf5&#39;, <br>#                       custom_objects={&#39;dice_loss_plus_1focal_loss&#39;: total_loss})<br><br># #Now, let us add the iou_score function we used during our initial training<br># my_model = load_model(&#39;saved_models/brats_3d_100epochs_simple_unet_weighted_dice.hdf5&#39;, <br>#                       custom_objects={&#39;dice_loss_plus_1focal_loss&#39;: total_loss,<br>#                                       &#39;iou_score&#39;:sm.metrics.IOUScore(threshold=0.5)})<br><br># #Now all set to continue the training process. <br># history2=my_model.fit(train_img_datagen,<br>#           steps_per_epoch=steps_per_epoch,<br>#           epochs=1,<br>#           verbose=1,<br>#           validation_data=val_img_datagen,<br>#           validation_steps=val_steps_per_epoch,<br>#           )<br><br>from keras.metrics import MeanIoU<br><br>batch_size=8 #Check IoU for a batch of images<br>test_img_datagen = imageLoader(val_img_dir, val_img_list, <br>                                val_mask_dir, val_mask_list, batch_size)<br><br>#Verify generator.... In python 3 next() is renamed as __next__()<br>test_image_batch, test_mask_batch = test_img_datagen.__next__()<br><br>test_mask_batch_argmax = np.argmax(test_mask_batch, axis=4)<br>test_pred_batch = model.predict(test_image_batch)<br>test_pred_batch_argmax = np.argmax(test_pred_batch, axis=4)<br><br>n_classes = 4<br>IOU_keras = MeanIoU(num_classes=n_classes)  <br>IOU_keras.update_state(test_pred_batch_argmax, test_mask_batch_argmax)<br>print(&quot;Mean IoU =&quot;, IOU_keras.result().numpy())<br><br>#Try images: <br>img_num = 82<br><br>test_img = np.load(&quot;BraTS2020_TrainingData/input_data_128/val/images/image_&quot;+str(img_num)+&quot;.npy&quot;)<br><br>test_mask = np.load(&quot;BraTS2020_TrainingData/input_data_128/val/masks/mask_&quot;+str(img_num)+&quot;.npy&quot;)<br>test_mask_argmax=np.argmax(test_mask, axis=3)<br><br>test_img_input = np.expand_dims(test_img, axis=0)<br>test_prediction = model.predict(test_img_input)<br>test_prediction_argmax=np.argmax(test_prediction, axis=4)[0,:,:,:]<br><br>#Plot individual slices from test predictions for verification<br>from matplotlib import pyplot as plt<br>import random<br><br>#n_slice=random.randint(0, test_prediction_argmax.shape[2])<br>n_slice = 55<br>plt.figure(figsize=(12, 8))<br>plt.subplot(231)<br>plt.title(&#39;Testing Image&#39;)<br>plt.imshow(test_img[:,:,n_slice,1], cmap=&#39;gray&#39;)<br>plt.subplot(232)<br>plt.title(&#39;Testing Label&#39;)<br>plt.imshow(test_mask_argmax[:,:,n_slice])<br>plt.subplot(233)<br>plt.title(&#39;Prediction on test image&#39;)<br>plt.imshow(test_prediction_argmax[:,:, n_slice])<br>plt.show()</pre><h4><strong>Results:</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*T1b2bd5hE1cpo872Z-tOOg.jpeg" /><figcaption>Fig(14) Tumour predicted in Brain (MRI) and I got Mean IOU: 0.834 with an accuracy of 93%</figcaption></figure><p>I hope that you find this article and explanation useful. Stay tuned for more data science-related posts in the future.</p><p>Feel free to connect and give ur suggestions:</p><p><a href="https://www.linkedin.com/in/dhyey-joshi12/">https://www.linkedin.com/in/dhyey-joshi12/</a></p><p><a href="https://github.com/dhyeyjoshi">https://github.com/dhyeyjoshi</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f4e73b7091be" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Machine Learning- Level 0- Exploratory Data Analysis]]></title>
            <link>https://medium.com/geekculture/what-is-exploratory-data-analysis-8d972a83ae78?source=rss-2ac64d1c310a------2</link>
            <guid isPermaLink="false">https://medium.com/p/8d972a83ae78</guid>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[algorithms]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[future]]></category>
            <dc:creator><![CDATA[Dhyey Joshi]]></dc:creator>
            <pubDate>Wed, 16 Jun 2021 14:45:38 GMT</pubDate>
            <atom:updated>2023-01-05T09:27:37.863Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*Dpj2jVEtGIEc1q_sgr0n1Q.gif" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*dMY-1Ze_MduzMz_t.png" /></figure><h3>What is Exploratory Data Analysis?</h3><blockquote><em>Exploratory Data Analysis refers to the critical process of performing initial investigations on data so as to discover patterns,to spot anomalies,to test hypothesis and to check assumptions with the help of summary statistics and graphical representations.</em></blockquote><p>It is a good practice to understand the data first and try to gather as many insights from it. EDA is all about making sense of data in hand, before getting them dirty with it.</p><p><strong>EDA explained using sample Data set:</strong></p><p>To share my understanding of the concept and techniques I know, I’ll take an example of a white variant of the Wine Quality data set which is available on the UCI Machine Learning Repository, and try to catch hold of as many insights from the data set using EDA.</p><p>To starts with, I imported the necessary libraries (for this example Pandas, NumPy, Matplotlib, and Seaborn) and loaded the data set.</p><p>Note: Whatever inferences I could extract, I’ve mentioned with bullet points.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/575/0*cWObBJgTvZ8XaI2k.png" /></figure><ul><li>Original data are separated by the delimiter “; “ in the given data set.</li><li>To take a closer look at the data took the help of the “ .head()” function of the panda’s library which returns the first five observations of the data set. Similarly “.tail()” returns the last five observations of the data set.</li></ul><p>I found out the total number of rows and columns in the data set using “.shape”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/582/0*P4HpP0HTcnZgSV6k.png" /></figure><ul><li>The dataset comprises 4898 observations and 12 characteristics.</li><li>Out of which one is the dependent variable and the rest 11 are independent variables — Physico-chemical characteristics.</li></ul><p>It is also a good practice to know the columns and their corresponding data types and find whether they contain null values.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/590/0*IzHRwODUsi2oWFT7.png" /></figure><ul><li>Data has an only float and integer values.</li><li>No variable column has null/missing values.</li></ul><p>The describe() function in pandas is very handy in getting various summary statistics. This function returns the count, mean, standard deviation, minimum and maximum values, and the quantiles of the data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/590/0*0H9i5T6AnwrgFIpu.png" /></figure><ul><li>Here as you can notice mean value is less than the median value of each column which is represented by 50%(50th percentile) in the index column.</li><li>There is notably a large difference between 75th %tile and max values of predictors “residual sugar”,” free sulfur dioxide”,” total sulfur dioxide”.</li><li>Thus observations 1 and 2 suggest that there are extreme values-Outliers in our data set.</li></ul><p>Few key insights just by looking at dependent variable are as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/567/0*7Ge0fYOY9vhY-JJ3.PNG" /></figure><ul><li>Target variable/Dependent variable is discrete and categorical in nature.</li><li>“quality” score scale ranges from 1 to 10; where 1 being poor and 10 being the best.</li><li>1,2 &amp; 10 Quality ratings are not given by any observation. Only scores obtained are between 3 to 9.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/579/0*lPzsv1FiMAWNy2gZ.PNG" /></figure><ul><li>This tells us the count of each quality score in descending order.</li><li>“quality” has the most values concentrated in categories 5, 6, and 7.</li><li>Only a few observations were made for categories 3 &amp; 9.</li></ul><p><strong>Data Visualization</strong></p><p>I got a good glimpse of the data. But that’s the thing with Data Science the more you get involved the harder it is for you to stop exploring. Let’s now explore data with beautiful graphs. Python has a visualization library, Seaborn which build on top of Matplotlib. It provides very attractive statistical graphs to perform both Univariate, Bivariate, and Multivariate analysis.</p><p>Let’s just take a simple overlook over the three types of data analysis.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*YzeeY2GVFOh3lccQ.PNG" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*CkHJ05yGPZmTvGso.PNG" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*zxMbizQ5BaIt0M1V.PNG" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*V_s3faW6gc4BIncl.PNG" /></figure><h3>Visualizing Data using Matplotlib &amp; Seaborn</h3><p>Import the matplotlib library.</p><p>1. Area Plot</p><p>Area plots are stacked by default. And to produce a stacked area plot, each column must be either all positive or all negative values (any, i.e. not a number, values will default to 0). To produce an unstacked plot, set parameter stacked to a value False.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*TijLBDuXaOMdownm.png" /></figure><p>2. Histograms</p><p>A histogram is a way of representing the <em>frequency</em> distribution of a numeric dataset. The way it works is it partitions the x-axis into <em>bins</em>, assigns each data point in our dataset to a bin, and then counts the number of data points that have been assigned to each bin. So the y-axis is the frequency or the number of data points in each bin. Note that we can change the bin size and usually one needs to tweak it so that the distribution is displayed nicely.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/657/0*Gr_Rc_SaCx5HuOVD.png" /></figure><p>3.Bar Charts (Dataframe)</p><p>A bar plot is a way of representing data where the <em>length</em> of the bars represents the magnitude/size of the feature/variable. Bar graphs usually represent numerical and categorical variables grouped in intervals.</p><p>To create a bar plot, we can pass one of two arguments via kind parameter in plot():</p><ul><li>kind=bar creates a <em>vertical</em> bar plot</li><li>kind=barh creates a <em>horizontal</em> bar plot</li></ul><p><strong>Vertical bar plot</strong></p><p>In vertical bar graphs, the x-axis is used for labeling, and the length of bars on the y-axis corresponds to the magnitude of the variable being measured. Vertical bar graphs are particularly useful in analyzing time series data. One disadvantage is that they lack space for text labeling at the foot of each bar.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*azdQ_cz3MFtq8xv5.PNG" /></figure><p>To use linear regression for modeling, it’s necessary to remove correlated variables to improve your model. One can find correlations using pandas “.corr()” function and can visualize the correlation matrix using a heatmap in seaborn.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*BarobDIPMeKH6nb8.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*D2zpu8oGbwwft7pa.png" /></figure><p>4.Box Plot</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*gfIowgZsX5F2GAwp.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*tlCTe_BFef08dq-8.jpeg" /></figure><p>5.Scatter Plot</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/386/0*wEQ17zcN6C5vmR4h.png" /></figure><p>6.Waffle Chart</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/488/0*1TSGNlA7msaev1-A.png" /></figure><p>7.Folium Library</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/781/0*YYGOY1C931e1_ZBO.png" /></figure><h3>Exploratory Data Analysis &amp; Data Visualization in just one second</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6DM3jqXeh9kyPJPQ.png" /></figure><blockquote><strong><em>DataPrep Library- Perform Faster EDA Within No Time</em></strong></blockquote><p>DataPrep.EDA is the fastest and the easiest EDA (Exploratory Data Analysis) tool in Python. — <a href="https://pypi.org/project/dataprep/">https://pypi.org/project/dataprep/</a></p><p>Want to know more about this library — <a href="https://dataprep.ai/">https://dataprep.ai/</a></p><p>Step1: !pip install dataprep</p><p>Step2: from dataprep.datasets import load_dataset</p><p>from dataprep.eda import create_report</p><p>df = load_dataset(“_Dataset-Name_”)</p><p>create_report(df)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8Sd0mv3LIkw90nLJ.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8d972a83ae78" width="1" height="1" alt=""><hr><p><a href="https://medium.com/geekculture/what-is-exploratory-data-analysis-8d972a83ae78">Machine Learning- Level 0- Exploratory Data Analysis</a> was originally published in <a href="https://medium.com/geekculture">Geek Culture</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Mastering Linux Shell Scripting]]></title>
            <link>https://dhyey-joshi12.medium.com/mastering-linux-shell-scripting-966c06ea0720?source=rss-2ac64d1c310a------2</link>
            <guid isPermaLink="false">https://medium.com/p/966c06ea0720</guid>
            <category><![CDATA[life]]></category>
            <category><![CDATA[linux]]></category>
            <category><![CDATA[life-lessons]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[script]]></category>
            <dc:creator><![CDATA[Dhyey Joshi]]></dc:creator>
            <pubDate>Mon, 03 May 2021 20:54:04 GMT</pubDate>
            <atom:updated>2021-05-04T12:13:22.029Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iWEsJYTcNnsMBlXa-BBf9g.gif" /></figure><h3>The What and Why of Scripting with Bash</h3><p><strong>Welcome to the what and why of bash scripting. In this chapter, you will discover the types of shells in Linux and why we chose bash. You will learn what bash is, how to write your first bash script, and how to run it. Also, you will see how to configure Linux editors, such as vim and nano, in order to type your code.</strong></p><p><strong>Like in any other scripting language, variables are the basic blocks of coding. You will learn how to declare variables such as integers, strings, and arrays. Furthermore, you will learn how to export these variables and extend their scope outside the running process.</strong></p><p><strong>Finally, you will see how to visually debug your code using Visual Studio Code.</strong></p><p><strong>We will cover the following topics in this article:</strong></p><ul><li><strong>Types of Linux shells</strong></li><li><strong>What is bash scripting?</strong></li><li><strong>The bash command hierarchy</strong></li><li><strong>Creating and executing scripts</strong></li><li><strong>Set Environmental Variable</strong></li><li><strong>Running the script with arguments</strong></li><li><strong>Using the test shell built-in</strong></li><li><strong>Creating conditional statements using If statement</strong></li><li><strong>Test command with the If Statement</strong></li><li><strong>UNIX / Linux: How to lock or disable a user account</strong></li><li><strong>Solving First Real Industry Use — Creating your first automation bash shell script.</strong></li><li><strong>Solving Second Real Industry Use Case Of Scripting Used By Data Analyst To Create Data Model For Better Performance Of Website/Webapp</strong></li><li><strong>Getting started with AWK Command to advance level.</strong></li><li><strong>Creating and Deleting multiple files using Bash Shell Scripting Commands</strong></li><li><strong>What is the difference between a function and a command?</strong></li><li><strong>How to create a function in shell i.e. “Shell Function”?</strong></li><li><strong>Linux Shell — The case…esac Statement.</strong></li><li><strong>Extra Information: Linux Set Command.</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MCXHPDV0daSWuXUo3pAcIg.png" /></figure><h3>Types of Linux shells</h3><p><strong>As you know, Linux consists of some major parts, such as the kernel, the shell, and the GUI interface (Gnome, KDE, and so on).</strong></p><p><strong>The shell translates your commands and sends them to the system. Most Linux distributions are shipped with many shells.</strong></p><p><strong>Every shell has its own features, and some of them are very popular among developers today. These are some of the popular ones:</strong></p><ul><li><strong>Sh shell: This is called the Bourne shell, this was developed at AT&amp;T Labs in the 70s by a guy named Stephen Bourne. This shell offers many features.</strong></li><li><strong>Bash shell: Also called the Bourne again shell, this is very popular and compatible with sh shell scripts, so you can run your sh scripts without changing them. We are going to use this shell in this article.</strong></li><li><strong>Ksh shell: Also called the Korn shell, this is compatible with sh and bash. Ksh offers some enhancements over the Bourne shell.</strong></li><li><strong>Csh and tcsh: Linux was built using the C language and that drove developers at Berkeley University to develop a C-style shell in which the syntax is similar to the C language. Tcsh adds some minor enhancements to csh.</strong></li></ul><p><strong>Now we know the types of shells and we know that we are going to use bash, so what is bash scripting?</strong></p><h3>What is bash scripting?</h3><p><strong>The basic idea of bash scripting is to execute multiple commands to automate a specific job.</strong></p><p><strong>As you might know, you can run multiple commands from the shell by separating them with semicolons (;):</strong></p><blockquote><strong>ls; pwd</strong></blockquote><p><strong>The previous line is a mini bash script.</strong></p><p><strong>The first command runs, followed by the result of the second command.</strong></p><p><strong>Every keyword you type in bash scripting is actually a Linux binary (program), even the if statement, or else or while loops. All are Linux executables.</strong></p><p><strong>You can say that the shell is the glue that binds these commands together.</strong></p><h3>The bash command hierarchy</h3><p><strong>When working on the bash shell and when you are sitting comfortably at your prompt eagerly waiting to type a command, you will most likely feel that it is a simple matter of typing and hitting the <em>Enter </em>key. You should know better than to think this, as things are never quite as simple as we imagine.</strong></p><h3>Command type</h3><p><strong>For example, if we type and enter ls to list files, it is reasonable to think that we were running the command. It is possible, but we often will be running an alias. Aliases exist in memory as a shortcut to commands or commands with options; these aliases are used before we even check for the file. Bash’s built-in type command can come to our aid here. The type command will display the type of command for a given word entered at the command line. The types of command are listed as follows:</strong></p><ul><li><strong>Alias</strong></li><li><strong>Function</strong></li><li><strong>Shell built-in</strong></li><li><strong>Keyword File</strong></li></ul><p><strong>This list is also representative of the order in which they are searched. As we can see, it is not until the very end where we search for the executable file ls.</strong></p><h3>Command PATH</h3><p><strong>Linux will check for executables in the PATH environment only when the full or relative path to the program is supplied. In general, the current directory is not searched unless it is in the PATH. It is possible to include our current directory within the PATH by adding the directory to the PATH variable. This is shown in the following command example:</strong></p><blockquote><strong>$ export PATH=$PATH:</strong></blockquote><p><strong>This appends the current directory to the value of the PATH variable; each item in the PATH is separated using a colon. Now your PATH has been updated to include the current working directory and, each time you change directories, the scripts can be executed easily. In general, organizing scripts into a structured directory hierarchy is probably a great idea. Consider creating a subdirectory called bin within your home directory and add the scripts into that folder. Adding $HOME/bin to your PATH variable will enable you to find the scripts by name and without the file path.</strong></p><p><strong>The following command-line list will only create the directory if it does not already exist:</strong></p><blockquote><strong>$ test -d $HOME/bin || mkdir $HOME/bin</strong></blockquote><p><strong>Although the preceding command-line list is not strictly necessary, it does show that scripting in bash is not limited to the actual script, and we can use conditional statements and another syntax directly at the command line. From our viewpoint, we know that the preceding command will work whether you have the bin directory or not. The use of the $HOME variable ensures that the command will work without considering your current filesystem context.</strong></p><p><strong>As I work through the article, we will add scripts into the $HOME/bin directory so that they can be executed regardless of our working directory.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*mI39uGd1CtmzUz-mdbuFsQ.gif" /></figure><h3>Creating and executing scripts</h3><p><strong>With our editors primed and ready, we can now move quickly to creating and executing our scripts. You can use any compatible editor of your Linux System like Gedit, Vim.</strong></p><h3>Hello World!</h3><p><strong>As you know, it is almost obligatory, to begin with, a Hello World script and we will not disappoint as far as this is concerned. We will begin by creating a new script, $HOME/bin/hello1.sh. The contents of the file should read as in the following screenshot:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/232/1*3iMqDTwQgk1i2zIVEvbm5A.png" /></figure><p><strong>We hope that you haven’t struggled with this too much; it is just three lines, after all. We encourage you to run through the examples as you read to really help you instill the information with good hands-on practice.</strong></p><blockquote><strong>#!/bin/bash: Normally, this is always the first line of the script and is known as the shebang. The shebang starts with a comment, but the system still uses this line. A comment in a shell script has the # symbol. The shebang instructs the interpreter of the system to execute the script. We use bash for shell scripts, and we may use PHP or Perl for other scripts, as required. If we do not add this line, then the commands will be run within the current shell; it may cause issues if we run another shell.</strong></blockquote><blockquote><strong>echo “Hello World”: The echo command will be picked up in a built-in shell and can be used to write a standard output, STDOUT; this defaults to the screen. The information to print is enclosed in double quotes; there will be more on quotes later.</strong></blockquote><blockquote><strong>exit 0: The exit command is a built-in shell, and is used to leave or exit the script. The exit code is supplied as an integer argument. A value of anything other than 0 will indicate some type of error in the script’s execution.</strong></blockquote><h3>Executing the script</h3><p><strong>With the script saved in our PATH environment, it still will not execute as a standalone script. We will have to assign and execute permissions for the file, as needed. For a simple test, we can run the file directly with bash. The following command shows you how to do this:</strong></p><blockquote><strong>$ bash $HOME/bin/hello1.sh</strong></blockquote><p><strong>We should be rewarded with the Hello World text being displayed on our screens. This is not a long-term solution, as we need to have the script in the $HOME/bin directory, specifically, to make running the script easy from any location without typing the full path. We need to add in the execute permissions as shown in the following code:</strong></p><blockquote><strong>$ chmod +x $HOME/bin/hello1.sh</strong></blockquote><p><strong>We should now be able to run the script simply, as shown in the following screenshot:</strong></p><blockquote><strong>dj@lucifer ~ $ chmod +x $HOME/bin/hello1.sh</strong></blockquote><blockquote><strong>dj@lucifer ~ $ hello1.sh</strong></blockquote><blockquote><strong>Hello World</strong></blockquote><blockquote><strong>dj@lucifer ~ $</strong></blockquote><h3>Set Environmental Variable</h3><p><strong>Now if you want to use your bash file from a different folder or from a different terminal like the date command, then we have to add a PATH inside Environmental Variable.</strong></p><p><strong>$ echo $PATH — List all the path’s set inside your Linux System.</strong></p><p><strong>To add path : $ PATH = $PATH:&lt; Folder path where your bash file is stored &gt;</strong></p><p><strong>To make it executable: $ chmod +x hello1.sh</strong></p><p><strong>Now, see your file will act as a command.</strong></p><p><strong>$ hello1.sh</strong></p><p><strong>Hello World</strong></p><p><strong>To make this path persistent or permanent —</strong></p><p><strong>$ vim /root/.bashrc (Add your path inside this file to make it permanent.)</strong></p><p><strong>PATH = $PATH:&lt; Folder path where your bash file is stored &gt;</strong></p><h3>Hello Learners!</h3><p><strong>It is possible that we might need a little more substance in the script than a simple fixed message. Static message content does have its place, but we can make this script much more useful by building in some flexibility.</strong></p><h3>Running the script with arguments</h3><p><strong>We can run the script with arguments; after all, it’s a free world and Linux promotes your freedom to do what you want to do with the code. However, if the script does not make use of the arguments, then they will be silently ignored. The following command shows the script running with a single argument:</strong></p><p><strong>$0 — The name of the script itself, which is often used in usage statements.</strong></p><p><strong>$1 — A positional argument, which is the first argument passed to the script.</strong></p><p><strong>${10} — Where two or more digits are needed to represent the argument position. Brace brackets are used to delimit the variable name from any other content. Single value digits are expected.</strong></p><p><strong>$# — The argument count is especially useful when we need to set the number of arguments needed for correct script execution.</strong></p><p><strong>$@ — Refers to all arguments.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/211/1*KFCVS2ZY3lEJwLEk_yZUDA.png" /></figure><p><strong>Run the script now; we can provide an argument as shown in the following:</strong></p><blockquote><strong>$ hello2.sh Learner</strong></blockquote><p><strong>The output should now say, Hello Learner. If we do not provide an argument, then the variable will be empty and will just print Hello. You can refer to the following screenshot to see the execution argument and output:</strong></p><blockquote><strong>dj@lucifer ~ $ chmod +x $HOME/bin/hello2.sh</strong></blockquote><blockquote><strong>dj@lucifer ~ $ hello2.sh Learner</strong></blockquote><blockquote><strong>Hello Learner</strong></blockquote><blockquote><strong>dj@lucifer ~ $</strong></blockquote><p><strong>If we adjust the script to use $*, all the arguments will print. We will see Hello and then a list of all the supplied arguments. Edit the script and replace the echo line as follows:</strong></p><blockquote><strong>echo “Hello $*”</strong></blockquote><p><strong>This will execute the script with the following arguments:</strong></p><blockquote><strong>dj@lucifer ~ $ chmod +x $HOME/bin/hello2.sh</strong></blockquote><blockquote><strong>dj@lucifer ~ $ hello2.sh Learner’s this article is created by Dhyey Joshi</strong></blockquote><blockquote><strong>Hello Learner’s this article is created by Dhyey Joshi</strong></blockquote><blockquote><strong>dj@lucifer ~ $</strong></blockquote><p><strong>If we want to print Hello &lt;name&gt;, with each name on a separate line, we will need to wait a little until we cover looping structures. A for loop is a good way to achieve this.</strong></p><h3>Using the test shell built-in</h3><p><strong>It is probably time for us to pull over to the side of the scripting highway and look a little more at the command test. This is both a shell built-in and a file executable in its own right. Of course, we will have to hit the built-in command first, unless we specify the full path to the file.</strong></p><p><strong>When the test command is run without any expressions to evaluate, then the test will return false. So, if we run the test as shown in the following command, the exit status will be 1, even though no error output is shown:</strong></p><blockquote><strong>$ test</strong></blockquote><p><strong>The test command will always return either True or False or 0 or 1, respectively. The basic syntax of test is as follows:</strong></p><blockquote><strong>$ test EXPRESSION</strong></blockquote><p><strong>If we need to include multiple expressions, this can be done using AND or OR together, using the -a and -o options, respectively:</strong></p><blockquote><strong>$ test EXPRESSION -a EXPRESSION</strong></blockquote><blockquote><strong>$ test EXPRESSION -o EXPRESSION</strong></blockquote><p><strong>We can also write this as a shorthand version, replacing the test with square brackets to surround the expression, as shown in the following example:</strong></p><blockquote><strong>$ [ EXPRESSION ]</strong></blockquote><h3>Testing integers</h3><p><strong>As well as testing string values of bash scripts, we can test for integer values and whole numbers. Another way of testing the input of a script is to count the numbers of positional parameters and also test if the number is above 0:</strong></p><blockquote><strong>test $# -gt 0</strong></blockquote><p><strong>Or using the brackets, as follows:</strong></p><blockquote><strong>[ $# -gt 0 ]</strong></blockquote><p><strong>When in a relationship, the top positional parameters of the $# variable represent the number of parameters passed to the script.</strong></p><p><strong>There are many tests that can be done for numbers:</strong></p><ul><li><strong>number1 -eq number2: This checks if number1 is equal to number2.</strong></li><li><strong>number1 -ge number2: This checks if number1 is greater than or equal to number2.</strong></li><li><strong>number1 -gt number2: This checks if number1 is greater than number2.</strong></li><li><strong>number1 -le number2: This checks if number1 is smaller than or equal to number 2.</strong></li><li><strong>number1 -lt number2: This checks if number1 is smaller than number2.</strong></li><li><strong>number1 -ne number2: This checks if number1 is not equal to number2.</strong></li></ul><h3>Testing file types</h3><p><strong>While testing for values, we can test for the existence of a file or file type. For example, we may only want to delete a file if it is a symbolic link. We use this while compiling a kernel. The /usr/src/linux directory should be a symbolic link to the latest kernel source code. If we download a newer version before compiling the new kernel, we need to delete the existing link and create a new link. Just in case someone has created the /usr/src/linux directory, we can test if it has a link before removing it:</strong></p><blockquote><strong># [ -h /usr/src/linux ] &amp;&amp;rm /usr/src/linux</strong></blockquote><p><strong>The -h option tests that the file has a link. Other options include the following:</strong></p><ul><li><strong>-d: This shows that it’s a directory</strong></li><li><strong>-e: This shows that the file exists in any form</strong></li><li><strong>-x: This shows that the file is executable</strong></li><li><strong>-f: This shows that the file is a regular file</strong></li><li><strong>-r: This shows that the file is readable</strong></li><li><strong>-p: This shows that the file is a named pipe</strong></li><li><strong>-b: This shows that the file is a block device</strong></li><li><strong>file1 -nt file2: This checks if file1 is newer than file2</strong></li><li><strong>file1 -ot file2: This checks if file1 is older than file2</strong></li><li><strong>-O file: This checks if the logged-in user is the owner of the file</strong></li><li><strong>-c: This shows that the file is a character device</strong></li></ul><p><strong>More options do exist, so delve into the main pages as you need to. We will use different options throughout the article, and thus giving you practical and useful examples.</strong></p><h3>Creating conditional statements using If statement</h3><p><strong>As we have seen so far, it is possible to build simple conditions using command-line lists. These conditional statements can be written both with and without a test. As the complexity of the tasks increases, it becomes easier to create statements using if. This will certainly ease both the readability of the script and the logic layout. To a degree, it also matches the way in which we think and speak; it is semantic in our spoken language in the same way it is within the bash script.</strong></p><p><strong>Even though it will take up more than a single line in the script, with an</strong></p><p><strong>if statement we can achieve more and make the script more legible. That being said, let’s look at creating if conditions. The following is an example of a script using an if statement:</strong></p><blockquote><strong>#!/bin/bash</strong></blockquote><blockquote><strong># Welcome script to display a message to users on login — File name: hello3.sh</strong></blockquote><blockquote><strong># Author: Dhyey Joshi</strong></blockquote><blockquote><strong>if [ $# -lt 1 ] ; then</strong></blockquote><blockquote><strong>echo “Usage: $0 &lt;name&gt;”</strong></blockquote><blockquote><strong>exit 1</strong></blockquote><blockquote><strong>fi</strong></blockquote><blockquote><strong>echo “Hello $1”</strong></blockquote><blockquote><strong>exit 0</strong></blockquote><p><strong>Within the script, we can easily add in multiple statements to run when the condition is true. In our case, this includes exiting the script with an error indicated, as well as including the usage statement to assist the user. This ensures that we only display the hello message if we have supplied the name of the person to be welcomed.</strong></p><p><strong>We can view the script execution both with and without the argument in the following screenshot:</strong></p><blockquote><strong>dj@lucifer ~ $ hello3.sh</strong></blockquote><blockquote><strong>Usage: /HOME/bin/hello2.sh &lt;name&gt;</strong></blockquote><blockquote><strong>dj@lucifer ~ $ hello3.sh Dhyey</strong></blockquote><blockquote><strong>Hello Dhyey</strong></blockquote><blockquote><strong>dj@lucifer ~ $</strong></blockquote><p><strong>The following pseudocode shows the syntax of the if conditional statement:</strong></p><blockquote><strong>if condition; then</strong></blockquote><blockquote><strong>statement 1</strong></blockquote><blockquote><strong>statement 2</strong></blockquote><blockquote><strong>fi</strong></blockquote><p><strong>Indenting the code is not required, but it helps readability and is highly recommended. Adding the then statement to the same line as the if statement, again assists with the readability of the code, and the semicolon is required to separate the if from the then.</strong></p><h3>Extending if with else</h3><p><strong>When a script is required to continue regardless of the result of the if condition, it is often necessary to deal with both conditions of the evaluation, what to do when it is true as well as false. This is where we can make use of the else keyword. This allows the execution of one block of code when the condition is true and another when the condition is evaluated as false. The pseudocode for this is shown as follows:</strong></p><blockquote><strong>if condition; then</strong></blockquote><blockquote><strong>statement</strong></blockquote><blockquote><strong>else</strong></blockquote><blockquote><strong>statement</strong></blockquote><blockquote><strong>fi</strong></blockquote><h3>Test command with the If Statement</h3><p><strong>You have seen how to use the test command or the short version [ ]. This test returns zero (true) or non-zero (false).</strong></p><p><strong>You will see how to check the returned result using the if command.</strong></p><h3>Checking strings</h3><p><strong>Let’s see some examples to explain how if statements work:</strong></p><blockquote><strong>#!/bin/bash</strong></blockquote><blockquote><strong>if [ “dhyey” = “Dhyey” ] then</strong></blockquote><blockquote><strong>echo “Strings are identical”</strong></blockquote><blockquote><strong>else</strong></blockquote><blockquote><strong>echo “Strings are not identical”</strong></blockquote><blockquote><strong>fi</strong></blockquote><blockquote><strong>dj@lucifer ~ $ hello4.sh</strong></blockquote><blockquote><strong>Strings are not identical</strong></blockquote><blockquote><strong>dj@lucifer ~ $</strong></blockquote><h3><strong>Checking files and directories</strong></h3><p><strong>Similarly, you can check files and directories using the if statement.</strong></p><p><strong>Let’s look at an example:</strong></p><blockquote><strong>#!/bin/bash mydir=/home/mydir</strong></blockquote><blockquote><strong>if [ -d $mydir ] then</strong></blockquote><blockquote><strong>echo “Directory $mydir exists.”</strong></blockquote><blockquote><strong>else</strong></blockquote><blockquote><strong>echo “Directory $mydir not found.”</strong></blockquote><blockquote><strong>fi</strong></blockquote><p><strong>We used the -d test to check if the path is a directory.</strong></p><p><strong>The rest of the tests work the same way.</strong></p><h3>Checking numbers</h3><p><strong>Also, in the same way, we can check numbers using the test, and the if</strong></p><p><strong>commands.</strong></p><blockquote><strong>#!/bin/bash</strong></blockquote><blockquote><strong>if [ 12 -gt 10 ] then</strong></blockquote><blockquote><strong>echo “number1 is greater than number2”</strong></blockquote><blockquote><strong>else</strong></blockquote><blockquote><strong>echo “number1 is less than number2”</strong></blockquote><blockquote><strong>fi</strong></blockquote><blockquote><strong>dj@lucifer ~ $ hello5.sh</strong></blockquote><blockquote><strong>number1 is greater than number2</strong></blockquote><blockquote><strong>dj@lucifer ~ $</strong></blockquote><p><strong>As expected, 12 is greater than 10.</strong></p><p><strong>All other numeric tests work the same way.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*QtX9tWcVWv1YFYr_rFEjbA.gif" /></figure><h3><strong>UNIX / Linux: How to lock or disable a user account</strong></h3><h3>- Lock the password</h3><p><strong>To lock a user&#39;s account use the command usermod -L or passwd -l. Both the commands adds an exclamation mark (“!”) in the second field of the file /etc/passwd. For example :</strong></p><blockquote><strong># passwd -l &lt;username&gt; : It will lock the user account.</strong></blockquote><h3>Tar Command in Linux To Create Backups</h3><p><strong>The Linux “tar” stands for tape archive, which is used by a large number of Linux/Unix system administrators to deal with tape drives backup. The tar command used to rip a collection of files and directories into highly compressed archive files commonly called tarball or tar, gzip and bzip in Linux. The tar is the most widely used command to create compressed archive files and that can be moved easily from one disk to another disk or machine to machine.</strong></p><p><strong>The below example command will create a tar archive file &lt;filename&gt;-(date or time).tar for a directory /home/bin in the current working directory. See the example command in action.</strong></p><blockquote><strong>tar -cvf Jack-14-09-12.tar /home/bin/</strong></blockquote><p><strong>Let’s discuss each option that we have used in the above command for creating a tar archive file.</strong></p><ol><li><strong>c — Creates a new .tar archive file.</strong></li><li><strong>v — Verbosely show the .tar file progress.</strong></li><li><strong>f — File name type of the archive file.</strong></li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*jxvqIkOwcVPTqZRjFlELcA.gif" /></figure><h3>Let&#39;s just create a sample script using all the concepts we discussed earlier. The script will first ask you to add users then it will ask you for a password, if already exists then throw an error message + Authenticated users are locked + Creating backups of each user account in the tar file at the same time.</h3><h3><strong>Code :</strong></h3><p><strong>#! /usr/bin/bash</strong></p><p><strong>ts=$(date +%F)<br>read -p “Add User: Enter username: “ user<br>for j in $user<br>do<br> useradd $j<br> echo “Hello $j!!”<br> read -p “Enter Password ${j}: “ password<br> echo $password | passwd — stdin $j<br>done</strong></p><p><strong>read -p “Enter your existing username to lock and backup: “ user</strong></p><p><strong>if [ -z $user ]<br>then<br> echo “Username is not provided..”<br>else<br> if id $user &amp;&gt; /dev/null<br> then<br> passwd -l $user<br> Homedir=$(grep ^${user}: /etc/passwd | cut -d “:” -f6)</strong></p><p><strong>echo $Homedir<br> if [ -d “${Homedir}” ]<br> then<br> echo “ Home directory exists..”<br> tar -cvf ${user}-${ts}.tar $Homedir <br> echo “Backup files of user ${user} !!”<br> echo “Backup successfully created of user ${user} !!”<br> fi <br> exit 0<br> else <br> echo “Provided user $user not exists..”<br> fi <br>fi</strong></p><h3>Output :</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uFLKBl0Dd7p9nUoBmlH7Tw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/224/1*ZUbSRX2_VhFcTu4Fwkn-Sg.png" /></figure><h3>Real Use Case Of Scripting Used By Data Analyst To Create Data Model For Better Performance Of Website/Webapp</h3><h3>Suppose, You have a website/web app hosted on AWS and you are on your local machine but you want to know each and every detail of that website/web app like the total number of clients connected plus their IPs plus how much time they visit our page plus which page is visited more plus what type of errors they are facing in viewing our website/web app and many more …</h3><p><strong>Step1: Start the webserver — Launch website — Give public Ip of your website to the client so that they can connect and do whatever they wanted to do.</strong></p><blockquote><strong>Sample website I created on the cloud….</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bSP_N5qawhblxiR10D7xEw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9IDmOVL5mqUUulz9Ze_cvA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8nSchdTgcbush9CvI88hYQ.png" /></figure><p><strong>Step2: Go to /var/log/httpd — Open access_log file where all the activities regarding your website are written.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y1Pf_cJriOWb-Yu5zymPbQ.png" /></figure><p><strong>Step3: Use this command to count the number of hits made by clients on your website — # wc -l access_log</strong></p><p><strong>Step4: Now, suppose you want to handle everything from your local Linux Machine. Without going to an AWS account from a browser rather directly connect your local machine with your AWS account using CLI i.e. called “Remote Login”.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a1XjQrELcac1VSYaOdgNIQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3pdoCRUqfm_yRnx-fyhAXg.png" /></figure><p><strong>Log in successfully with your AWS account !!</strong></p><p><strong>Let’s learn one more new concept of shell scripting i.e. of “Awk Command”.</strong></p><h3>AWK Command</h3><p><strong>The awk command is used for text processing in Linux. Although the sed command is also used for text processing, it has some limitations, so the awk command becomes a handy option for text processing. It provides powerful control to the data.</strong></p><p><strong>The Awk is a powerful scripting language used for text scripting. It searches and replaces the texts and sorts, validates, and indexes the database.</strong></p><p><strong>It is one of the most widely used tools for the programmer, as they write the scaled-down effective program in the form of a statement to define the text patterns and designs.</strong></p><h3>Features of AWK command</h3><p><strong>Various features of the Awk command are as follows:</strong></p><ul><li><strong>It scans a file line by line.</strong></li><li><strong>It splits a file into multiple fields.</strong></li><li><strong>It compares the input text or a segment of a text file.</strong></li><li><strong>It performs various actions on a file like searching a specified text and more.</strong></li><li><strong>It formats the output lines.</strong></li><li><strong>It performs arithmetic and string operations.</strong></li><li><strong>It applies the conditions and loops on output.</strong></li><li><strong>It transforms the files and data into a specified structure.</strong></li><li><strong>It produces the format reports.</strong></li></ul><h4><strong>Syntax:</strong></h4><p><strong>The Awk command is used as follows:</strong></p><blockquote><strong># awk options ‘selection _criteria {action }’ input-file &gt; output-file</strong></blockquote><p><strong>The options can be:</strong></p><ul><li><strong>-f program files: It reads the source code of the script written on the awk command</strong></li><li><strong>-F fs: It is used as the input field separator.</strong></li></ul><h3>Built-in variables in AWK command</h3><p><strong>Awk command supports many built-in variables, which include $1, $2, and so on, that break the file content into individual segments.</strong></p><p><strong>NR: It is used to show the current count of the lines. The awk command performs action once for each line. These lines are said as records.</strong></p><p><strong>NF: It is used to count the number of fields within the current database.</strong></p><p><strong>FS: It is used to create a field separator character to divide fields into the input lines.</strong></p><p><strong>OFS: It is used to store the output field separator. It separates the output fields.</strong></p><p><strong>ORS: It is used to store the output record separator. It separates the output records. It prints the content of the ORS command automatically.</strong></p><h4>Features of Awk Command</h4><p><strong>Create a file then dump some random data inside it.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/618/1*pacor6xdXXoOAzkg9lyV6Q.png" /></figure><ul><li><strong>Print number of lines in a file.</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/705/1*OZw7kpYaVPQdlBMnuQPRdg.png" /></figure><ul><li><strong>Print number of lines and print all data of a file.</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/928/1*WYbAKPtZXfqc_-UtuoGlQw.png" /></figure><ul><li><strong>Count the number of fields or words per line in a file separated by spaces. Where [ 0 ] → Empty Space.</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/866/1*UukZ0SCX80G_EePlhSjDpg.png" /></figure><ul><li><strong>Print last field or word of a file.</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/874/1*0TkSnY8lX5nxDZncBx7KWQ.png" /></figure><ul><li><strong>Print second last field or word of a file.</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/725/1*9gRWXZtcRYJrJfcOmGmFAA.png" /></figure><ul><li><strong>Print the last line of a file.</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/716/1*NKJa3-6WNbViYBS9RatLBA.png" /></figure><ul><li><strong>Remove empty spaces from a file.</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/688/1*Jwh-xmUAaBJ-1Hh7OEJigg.png" /></figure><ul><li><strong>Use of ‘BEGIN’ keyword in Awk.</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/923/1*qQA3ei4XRxBTLzHqgGiJOg.png" /></figure><h4><strong>Creating and Deleting multiple files using Bash Shell Scripting Commands</strong></h4><ul><li><strong>Create multiple text files in one go, here we created 13 empty text files.</strong></li></ul><blockquote><strong>#touch {1..13}.txt</strong></blockquote><ul><li><strong>Delete multiple text files in one go, here we deleted 13 empty text files.</strong></li></ul><blockquote><strong>#rm -f {1..13}.txt</strong></blockquote><p><strong>You can also use for loop concept to print file names and any information you can generate using for loop. Example:</strong></p><blockquote><strong>#for i in {10..13}.txt; do ls -l $i ; done | awk ‘{print $6}’</strong></blockquote><p><strong>This command will go to the long list of files ranging from 10 to 13 text files and print the 6th field of the record. Many more things you can format using the Shell For Loop.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w-kdvuoPU31YaPzSgb9JVw.png" /></figure><h4>Find and Replace text within a file using the AWK command</h4><p><strong>AWK, being a text processing utility, is quite appropriate for such a task. It can do simple replacements and much more advanced ones based on regular expressions. It provides two functions: </strong><strong>sub() and </strong><strong>gsub(). The first one only replaces only the first occurrence, while the second - replaces occurrences in the whole string. For instance, if we have a string </strong><strong>one potato two potato , this would be the result:</strong></p><blockquote><strong>$ echo “one potato two potato” | awk ‘{gsub(/potato/,”banana”)}1’<br>one banana two banana</strong></blockquote><blockquote><strong>$ echo “one potato two potato” | awk ‘{sub(/potato/,”banana”)}1’ <br>one banana two potato</strong></blockquote><p><strong>AWK can take an input file as an argument, so doing the same things with </strong><strong>input.txt , would be easy:</strong></p><blockquote><strong>awk ‘{sub(/blue/,”azure”)}1’ input.txt</strong></blockquote><p><strong>Now let’s continue with an analysis of our website …</strong></p><p><strong>Step5: Use different features of the awk command to view your output or data in a specific structured order.</strong></p><ul><li><strong>To view only IPs of client from access_log file.</strong></li></ul><blockquote><strong># awk ‘{print $1}’ access_log</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SllVRYd35V9exdZ4mifcQw.png" /></figure><ul><li><strong>To grab unique IPs of clients from the access_log file that hits your website.</strong></li></ul><blockquote><strong># awk ‘{print $1}’ access_log | sort | uniq | wc -l</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R99HD6lf9TpM9rdV4X_4Wg.png" /></figure><ul><li><strong>To get the total count per IP of clients that hits your website.</strong></li></ul><blockquote><strong># awk ‘{print $1}’ access_log | sort | uniq -c | sort -n -k1</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pAZIDZ1GdSusBsJuo6zGQA.png" /></figure><ul><li><strong>To make an analysis of the data of your server per day or per year basis then use this command.</strong></li></ul><blockquote><strong># grep $(date +%e/%b/%G) access_log | awk ‘{print $1&quot; →” $4 }’ | uniq -c | sort -n -k1</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4MTHeX6WJdtxsNOAxZ48Cg.png" /></figure><p><strong>Now.. you can see like writing each and every command is a boring process so let&#39;s create a script containing all the commands and run that script remotely from the Linux system to the AWS account and get output back on your screen without even logging into your AWS account. This is the power of Bash Shell Scripting i.e. “Automation”.</strong></p><p><strong>Code:</strong></p><blockquote><strong>#! /usr/bin/bash<br># To view only IPs of client from access_log file.<br>awk ‘{print $1}’ /var/log/httpd/access_log<br># To grab unique IPs of clients from the access_log file that hits your website.<br>awk ‘{print $1}’ /var/log/httpd/access_log | sort | uniq -c | wc -l<br># To get the total count per IP of clients that hits your website.<br>awk ‘{print $1}’ /var/log/httpd/access_log | sort | uniq -c | sort -n -k1<br># To make an analysis of the data of your server per day or per year basis then use this command.<br>grep $(date +%e/%b/%G) /var/log/httpd/access_log | awk ‘{print $1&quot; →” $4 }’ | uniq -c | sort -n -k1<br># Show all the record containing 404 error plus show which page they visited with unique Ips and count how many times they visited this page in sorted manner <br>awk ‘$9==404{print $1 “ — -&gt;” $7}’ /var/log/httpd/access_log | sort | uniq -c | sort -n</strong></blockquote><p><strong>Running this script remotely in the AWS account and will give me output back on my screen (local machine):</strong></p><blockquote><strong># scp -i shell_scripting.pem web_analysis.sh ec2-user@65.2.152.14:/tmp<br>result: web_analysis.sh 100% 775 21.2KB/s 00:00 [Trasfering my web_analysis.sh Bash Script to my AWS account]</strong></blockquote><blockquote><strong>Running the Bash Script Remotely:</strong></blockquote><blockquote><strong># ssh -l ec2-user -i shell_scripting.pem 65.2.152.14 sudo bash /tmp/web_analysis.sh</strong></blockquote><p><strong>Output:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OjyuPbNReLz4fnPu9i52xw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Wp6dAjcIRdfx68YykkYQwA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HC5zS4g3zU83yxTobsHd1Q.png" /></figure><p><strong>All the analysis of the data in one go ….. this is the power of “BASH SHELL SCRIPTING”.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*O1Cl_ZRpZ2zYD8KjQdSyDw.gif" /></figure><h3><strong>What is the difference between a function and a command?</strong></h3><h4><strong><em>Differences from the perspective of shell:</em></strong></h4><p><strong>When a command is executed, the shell creates a new child process for the command and tells Operating System to execute the instructions found in the corresponding binary file. Generally, the shell waits for this process to finish and then the prompt returns for your next command (Except when you run a program in the background. This can be done by appending ‘&amp;’ to the command.).</strong></p><p><strong>But when a function is called, the shell simply reads the commands in the function and executes them. This may create many processes during the execution of the function. So you can define a function in shell-like bash to do some task that requires a few lines of commands to be executed on the shell. The function only lasts for a session of the shell process unless you write it in a file and import it every time. The function defined in one shell (Terminal) process will not be available in another shell process.</strong></p><p><strong>When you compile a C/C++ file and put it in /usr/bin it’s going to be a command. The use of a function in a shell script is to avoid copying the code and keeping the shell script neat.</strong></p><h3>How to create a function in shell i.e. “Shell Function”?</h3><h3>Creating Functions</h3><p><strong>To declare a function, simply use the following syntax −</strong></p><blockquote><strong>function_name () { <br> list of commands<br>}</strong></blockquote><p><strong>The name of your function is function_name, and that’s what you will use to call it from elsewhere in your scripts. The function name must be followed by parentheses, followed by a list of commands enclosed within braces.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mwg23jaKBrdBIGxWEt6FPQ.png" /><figcaption><strong>Difference Between Command v/s Function</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1019/1*kRSgLXVlwUqa60yl7EUkdA.png" /><figcaption><strong>You can set your own Success code / Exit code</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*s1nf5OxzAEf-U-HXgwkjIQ.png" /><figcaption><strong>You can remove the function using an unset keyword</strong></figcaption></figure><h3>UNIX / Linux Shell — The case…esac Statement</h3><p><strong>You can use multiple if…elif statements to perform a multiway branch. However, this is not always the best solution, especially when all of the branches depend on the value of a single variable.</strong></p><p><strong>Shell supports case…esac statement which handles exactly this situation, and it does so more efficiently than repeated if…elif statements.</strong></p><h4><strong>Syntax</strong></h4><blockquote><strong>case word in<br> pattern1)<br> Statement(s) to be executed if pattern1 matches<br> ;;<br> pattern2)<br> Statement(s) to be executed if pattern2 matches<br> ;;<br> pattern3)<br> Statement(s) to be executed if pattern3 matches<br> ;;<br> *)<br> Default condition to be executed<br> ;;<br>esac</strong></blockquote><p><strong>Here the string word is compared against every pattern until a match is found. The statement(s) following the matching pattern executes. If no matches are found, the case statement exits without performing any action.</strong></p><p><strong>There is no maximum number of patterns, but the minimum is one.</strong></p><p><strong>When the statement(s) part executes, the command ;; indicates that the program flow should jump to the end of the entire case statement</strong></p><p><strong>A good use for a case statement is the evaluation of command line arguments as follows −</strong></p><p><strong>Code:</strong></p><blockquote><strong>#!/bin/sh</strong></blockquote><blockquote><strong>option=”${1}” <br>case ${option} in <br> -f) FILE=”${2}” <br> echo “File name is $FILE”<br> ;; <br> -d) DIR=”${2}” <br> echo “Dir name is $DIR”<br> ;; <br> *) <br> echo “`basename ${0}`:usage: [-f file] | [-d directory]” <br> exit 1 # Command to come out of the program with status 1<br> ;; <br>esac</strong></blockquote><p><strong>Output:</strong></p><blockquote><strong>$./test.sh<br>test.sh: usage: [ -f filename ] | [ -d directory ]<br>$ ./test.sh -f index.htm<br>$ vi test.sh<br>$ ./test.sh -f index.htm<br>File name is index.htm<br>$ ./test.sh -d linux<br>Dir name is linux<br>$</strong></blockquote><h4><strong>Extra Information</strong></h4><p><strong>While creating a script you can also use different options with the first line of the script i.e. Using #!/usr/bin/env NAME makes the shell search for the first match of NAME in the $PATH environment variable. It can be useful if you aren’t aware of the absolute path or don’t want to search for it. If the shell scripts start with #!/bin/bash, they will always run with bash from /bin.</strong></p><blockquote><strong>#!/usr/bin/bash -x [ -x: It is used to print commands and their arguments in a sequential way (as they are executed). ]</strong></blockquote><blockquote><strong>#!/usr/bin/bash -e [ -e: It is used to exit immediately if a command exits with a non-zero status. ]</strong></blockquote><blockquote><strong>#!/usr/bin/bash -v [ -v: It is used to print shell input lines. ]</strong></blockquote><p><strong>and much more. A similar thing can be done using the built-in shell variable i.e. “ set [Option] ”</strong></p><h3>Linux Set Command</h3><p><strong>Linux set command is used to set and unset certain flags or settings within the shell environment. These flags and settings determine the behavior of a defined script and help in executing the tasks without facing any issue. The values of shell attributes and parameters can be changed or displayed by using the set command.</strong></p><h4><strong>Syntax:</strong></h4><blockquote><strong>set [options]</strong></blockquote><p><strong>Options: The supported options by the set command are as following:</strong></p><ul><li><strong>-a: It is used to mark variables that are modified or created for export.</strong></li><li><strong>-b: It is used to notify of job termination immediately.</strong></li><li><strong>-e: It is used to exit immediately if a command exits with a non-zero status.</strong></li><li><strong>-f: It is used to disable the file name generation (globbing).</strong></li><li><strong>-h: It is used to save the location of commands where they looked up.</strong></li><li><strong>-k: It is used to place all assignment arguments in the environment variable of a command, except those that precede the command name.</strong></li><li><strong>-m: It is used to enable Job control.</strong></li><li><strong>-n: It is used to read commands.</strong></li><li><strong>-o: It is used for option-name.</strong></li><li><strong>-p: It is used to disable the processing of the ‘$ENV’ file and import shell functions. It is turned on whenever the real and effective user ids do not match. Turning off this option may cause the working uid and gid to be set as the authorized uid and gid.</strong></li><li><strong>-t: It is used to exit from the command after executing one command.</strong></li><li><strong>-u: It is used to treat unset variables as an error when substituting.</strong></li><li><strong>-v: It is used to print shell input lines.</strong></li><li><strong>-x: It is used to print commands and their arguments in a sequential way (as they are executed).</strong></li><li><strong>-B: It is used to perform brace expansion by the Shell.</strong></li><li><strong>-C: It is used to disallow existing regular files to be overwritten by redirection of output.</strong></li><li><strong>-E: It is used if the ERR trap is inherited by the shell functions.</strong></li><li><strong>-H: It is used to enable style history substitution. By default, it is on when the shell is interactive.</strong></li><li><strong>-P: It is used if we do not want to follow symbolic links when executing commands.</strong></li><li><strong>-T: If this flag is set, the DEBUG trap is inherited by the shell functions.</strong></li></ul><p><strong>To turn on the debugging information:</strong></p><blockquote><strong>set -x</strong></blockquote><p><strong>To turn off the debugging information:</strong></p><blockquote><strong>set +x</strong></blockquote><p><strong>Disable Bash’s default behavior.</strong></p><p><strong>To disable the default behavior of Bash, execute the command as follows:</strong></p><blockquote><strong>set -C</strong></blockquote><p><strong>Stop a script immediately.</strong></p><p><strong>To stop a script immediately, execute the following command:</strong></p><blockquote><strong>set -e</strong></blockquote><h3><strong>Getting Help</strong></h3><p><strong>If you get stuck during the use of the set command, you can access the help documentation from your terminal. To access the help manual, execute the below command:</strong></p><blockquote><strong>set [-][-] help</strong></blockquote><p><strong><em>Hope you enjoyed the article!</em></strong></p><p><strong><em>I’d like to sign off with a big thank you. I have tried to cover everything about Bash Shell Scripting in a single article. Thank you for reading and for helping me grow as a technical writer.</em></strong></p><p><strong><em>Special thanks to Mr</em></strong><em>. </em><strong><em>Vimal Daga sir for explaining bash shell scripting soo well !!</em></strong></p><p><strong><em>With love,<br>Dhyey Joshi</em></strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=966c06ea0720" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Industry Use Cases for Kubernetes / Openshift from RedHat]]></title>
            <link>https://dhyey-joshi12.medium.com/industry-use-cases-for-kubernetes-openshift-from-redhat-9f52cbee36d8?source=rss-2ac64d1c310a------2</link>
            <guid isPermaLink="false">https://medium.com/p/9f52cbee36d8</guid>
            <category><![CDATA[redhat-linux]]></category>
            <category><![CDATA[red-hat-openshift]]></category>
            <category><![CDATA[openshift]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[vimal-daga]]></category>
            <dc:creator><![CDATA[Dhyey Joshi]]></dc:creator>
            <pubDate>Tue, 02 Mar 2021 06:03:59 GMT</pubDate>
            <atom:updated>2021-03-02T06:07:57.821Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jqC23BCFBHKR42kYOUqorw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/848/1*C4iPcBwIim8tt4iTS_iBAQ.jpeg" /><figcaption>Kubernetes vs Openshift</figcaption></figure><h3>Kubernetes, also known as K8s, is an open-source system for automating deployment, scaling, and management of containerized applications.</h3><h3>What is Kubernetes?</h3><p>Kubernetes is a portable, extensible, open-source platform for managing containerized workloads and services, that facilitates both declarative configuration and automation. It has a large, rapidly growing ecosystem. Kubernetes services, support, and tools are widely available.</p><p>The name Kubernetes originates from Greek, meaning helmsman or pilot. Google open-sourced the Kubernetes project in 2014. Kubernetes combines over 15 years of Google’s experience running production workloads at scale with best-of-breed ideas and practices from the community.</p><h3>Going back in time</h3><p>Let’s take a look at why Kubernetes is so useful by going back in time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WCzhGyl7HsHCCS9zXA7vyg.png" /></figure><p><strong>Traditional deployment era:</strong> Early on, organizations ran applications on physical servers. There was no way to define resource boundaries for applications in a physical server, and this caused resource allocation issues. For example, if multiple applications run on a physical server, there can be instances where one application would take up most of the resources, and as a result, the other applications would underperform. A solution for this would be to run each application on a different physical server. But this did not scale as resources were underutilized, and it was expensive for organizations to maintain many physical servers.</p><p><strong>Virtualized deployment era:</strong> As a solution, virtualization was introduced. It allows you to run multiple Virtual Machines (VMs) on a single physical server’s CPU. Virtualization allows applications to be isolated between VMs and provides a level of security as the information of one application cannot be freely accessed by another application.</p><p>Virtualization allows better utilization of resources in a physical server and allows better scalability because an application can be added or updated easily, reduces hardware costs, and much more. With virtualization, you can present a set of physical resources as a cluster of disposable virtual machines.</p><p>Each VM is a full machine running all the components, including its own operating system, on top of the virtualized hardware.</p><p><strong>Container deployment era:</strong> Containers are similar to VMs, but they have relaxed isolation properties to share the Operating System (OS) among the applications. Therefore, containers are considered lightweight. Similar to a VM, a container has its own filesystem, the share of CPU, memory, process space, and more. As they are decoupled from the underlying infrastructure, they are portable across clouds and OS distributions.</p><p>Containers have become popular because they provide extra benefits, such as:</p><ul><li>Agile application creation and deployment: Increased ease and efficiency of container image creation compared to VM image use.</li><li>Continuous development, integration, and deployment: Provides for reliable and frequent container image build and deployment with quick and efficient rollbacks (due to image immutability).</li><li>Dev and Ops separation of concerns: Create application container images at build/release time rather than deployment time, thereby decoupling applications from infrastructure.</li><li>Observability Not only surfaces OS-level information and metrics, but also application health and other signals.</li><li>Environmental consistency across development, testing, and production: Runs the same on a laptop as it does in the cloud.</li><li>Cloud and OS distribution portability: Runs on Ubuntu, RHEL, CoreOS, on-premises, on major public clouds, and anywhere else.</li><li>Application-centric management: Raises the level of abstraction from running an OS on virtual hardware to running an application on an OS using logical resources.</li><li>Loosely coupled, distributed, elastic, liberated micro-services: Applications are broken into smaller, independent pieces and can be deployed and managed dynamically — not a monolithic stack running on one big single-purpose machine.</li><li>Resource isolation: predictable application performance.</li><li>Resource utilization: High efficiency and density.</li></ul><h3>Why you need Kubernetes and what it can do</h3><p>Containers are a good way to bundle and run your applications. In a production environment, you need to manage the containers that run the applications and ensure that there is no downtime. For example, if a container goes down, another container needs to start. Wouldn’t it be easier if this behavior was handled by a system?</p><p>That’s how Kubernetes comes to the rescue! Kubernetes provides you with a framework to run distributed systems resiliently. It takes care of scaling and failover for your application, provides deployment patterns, and more. For example, Kubernetes can easily manage a canary deployment for your system.</p><p>Kubernetes provides you with:</p><ul><li><strong>Service discovery and load balancing</strong> Kubernetes can expose a container using the DNS name or using their own IP address. If traffic to a container is high, Kubernetes is able to load balance and distribute the network traffic so that the deployment is stable.</li><li><strong>Storage orchestration</strong> Kubernetes allows you to automatically mount a storage system of your choice, such as local storage, public cloud providers, and more.</li><li><strong>Automated rollouts and rollbacks</strong> You can describe the desired state for your deployed containers using Kubernetes, and it can change the actual state to the desired state at a controlled rate. For example, you can automate Kubernetes to create new containers for your deployment, remove existing containers and adopt all their resources to the new container.</li><li><strong>Automatic bin packing</strong> You provide Kubernetes with a cluster of nodes that it can use to run containerized tasks. You tell Kubernetes how much CPU and memory (RAM) each container needs. Kubernetes can fit containers onto your nodes to make the best use of your resources.</li><li><strong>Self-healing</strong> Kubernetes restarts containers that fail, replaces containers, kills containers that don’t respond to your user-defined health check, and doesn’t advertise them to clients until they are ready to serve.</li><li><strong>Secret and configuration management</strong> Kubernetes let you store and manage sensitive information, such as passwords, OAuth tokens, and SSH keys. You can deploy and update secrets and application configuration without rebuilding your container images, and without exposing secrets in your stack configuration.</li></ul><h3>What Kubernetes is not</h3><p>Kubernetes is not a traditional, all-inclusive PaaS (Platform as a Service) system. Since Kubernetes operates at the container level rather than at the hardware level, it provides some generally applicable features common to PaaS offerings, such as deployment, scaling, load balancing, and lets users integrate their logging, monitoring, and alerting solutions. However, Kubernetes is not monolithic, and these default solutions are optional and pluggable. Kubernetes provides the building blocks for building developer platforms but preserves user choice and flexibility where it is important.</p><p>Kubernetes:</p><ul><li>Does not limit the types of applications supported.</li><li>Do not deploy source code and do not build your application.</li><li>Does not provide application-level services, such as middleware (for example, message buses), data-processing frameworks (for example, Spark), databases (for example, MySQL), caches, nor cluster storage systems (for example, Ceph) as built-in services.</li><li>Does not dictate logging, monitoring, or alerting solutions.</li><li>Does not provide nor mandate a configuration language/system (for example, Jsonnet).</li><li>Does not provide nor adopt any comprehensive machine configuration, maintenance, management, or self-healing systems.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/280/1*naZlyi2VC7KsMkXFhBDCvg.png" /></figure><h3>What is OpenShift?</h3><p>OpenShift is a family of containerization software offerings created by open-source software provider Red Hat. According to the company, Kubernetes is the kernel of distributed systems, while OpenShift is the distribution. At its core, OpenShift is a cloud-based Kubernetes container platform that’s considered both containerization software and a platform-as-a-service (PaaS). It’s also partly built on Docker, another popular containerization platform. OpenShift offers consistent security, built-in monitoring, centralized policy management, and compatibility with Kubernetes container workloads. It’s fast, enables self-service provisioning, and integrates with a variety of tools. In other words, there’s no vendor lock-in. Previously known as Origin, the open-source platform, OpenShift OKD lets developers create, test, and deploy applications on the cloud. It also supports several programming languages, including Go, Node.js, Ruby, Python, PHP, Perl, and Java.</p><h3>Kubernetes Vs. OpenShift</h3><p>Both Kubernetes and OpenShift feature robust and scalable architecture that enables rapid and large-scale application development, deployment, and management. They both run on the Apache License 2.0. But that’s just about where the similarities end. Here are just a few of the many ways OpenShift and Kubernetes differ.</p><h3>Deployment</h3><p>Kubernetes offers more flexibility as an open-source framework and can be installed on almost any platform — like Microsoft Azure and AWS — as well as any Linux distribution, including Ubuntu and Debian. OpenShift, on the other hand, requires Red Hat’s proprietary Red Hat Enterprise Linux Atomic Host (RHELAH), Fedora, or CentOS. This narrows options for many businesses, especially if they’re not already using these platforms.</p><h3>Security</h3><p>OpenShift has stricter security policies. For instance, it is forbidden to run a container as a root. It also offers a secure-by-default option to enhance security. Kubernetes doesn’t come with built-in authentication or authorization capabilities, so developers must create bearer tokens and other authentication procedures manually.</p><h3>Support</h3><p>Kubernetes has a large active community of developers who continuously collaborate on refining the platform. It also offers support for multiple frameworks and languages. OpenShift has a much smaller support community that is limited primarily to Red Hat developers.</p><h3>Releases and Updates</h3><p>Kubernetes has an average of four releases each year, while OpenShift trails with around three. Similarly, Kubernetes supports several concurrent and simultaneous updates, while OpenShift DeploymentConfig does not.</p><h3>Networking</h3><p>Kubernetes lacks a networking solution but lets users employ third-party network plug-ins. OpenShift, on the other hand, has its out-of-the-box networking solution called Open vSwitch, which comes with three native plug-ins.</p><h3>Templates</h3><p>Kubernetes offers Helm templates that are easy to use and provide a generous amount of flexibility. OpenShift templates are nowhere near as flexible or user-friendly.</p><h3>Container Image Management</h3><p>OpenShift lets developers use Image Streams to manage container images, while Kubernetes doesn’t offer container image management features.</p><h3><strong>Kubernetes User Case Studies -</strong></h3><p><em>( Reference Link: </em><a href="https://kubernetes.io/case-studies/"><em>https://kubernetes.io/case-studies/</em></a><em> )</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/596/1*pKPC4gYhYwWghv4aJXdGgA.png" /></figure><h3>Openshift User Case Studies -</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*MDpZpyZoDg35RqXepuK2ng.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UJCW2TAEFNkb7x5D_q3x5Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*s4ZIJp_fzS2U7uZGxgFpgw.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r34HspkXivyTPI5jEnji_A.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*fo1hma5DWyRX4wqQbtYfGw.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*6wob1hPKQ60Vr_mjLdKUcg.png" /></figure><p><strong><em>Conclusion:</em></strong></p><p><em>Both Kubernetes and OpenShift enable you to easily deploy and manage containerized applications. However, they have some differences. Kubernetes installation is complex and often requires a third-party solution. OpenShift has a built-in Kubernetes platform, which makes the installation process easier, but limited to Red Hat Linux distributions.</em></p><p><em>Hopefully, this article helped you understand the main differences between OpenShift and Kubernetes. When choosing a platform, you should consider your skillset and specs. The goal is to design the pipeline that works best for you, so be sure to experiment and check out the solution before adding the tool to your workflow.</em></p><p><em>I hope you like the article. Reach me on my </em><a href="http://www.linkedin.com/in/dhyey-joshi12"><em>LinkedIn</em></a><em>.</em></p><p><em>Thank You.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9f52cbee36d8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How Top MNC Companies Use Artificial Intelligence In Practice]]></title>
            <link>https://dhyey-joshi12.medium.com/how-top-mnc-companies-use-artificial-intelligence-in-practice-ad70646b540f?source=rss-2ac64d1c310a------2</link>
            <guid isPermaLink="false">https://medium.com/p/ad70646b540f</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[computer-science]]></category>
            <category><![CDATA[robotics]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Dhyey Joshi]]></dc:creator>
            <pubDate>Tue, 20 Oct 2020 07:42:59 GMT</pubDate>
            <atom:updated>2020-10-20T07:56:41.841Z</atom:updated>
            <content:encoded><![CDATA[<p><em>All the world’s tech giants from Alibaba to Amazon are in a race to become the world’s leaders in artificial intelligence (AI). These companies are AI trailblazers and embrace AI to provide next-level products and services. Here are 10 of the best examples of how these companies are using artificial intelligence in practice.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*7jzuWpmCXulYkt1aJhNj3w.jpeg" /><figcaption>“AI doesn’t have to be evil to destroy humanity — if AI has a goal and humanity just happens to come in the way, it will destroy humanity as a matter of course without even thinking about it, no hard feelings.” — Elon Musk.</figcaption></figure><p>The 5 Best Examples Of How Companies Use Artificial Intelligence In Practice</p><p><strong>1. Alphabet — Google</strong></p><p>Alphabet is Google’s parent company. Waymo, the company’s self-driving technology division, began as a project at Google. Today, Waymo wants to bring self-driving technology to the world not only to move people around but to reduce the number of crashes. Its autonomous vehicles are currently shuttling riders around California in self-driving taxis. Right now, the company can’t charge fare and a human driver still sits behind the wheel during the pilot program. Google signaled its commitment to deep learning when it acquired DeepMind. Not only did the system learn how to play 49 different Atari games, but the AlphaGo program was also the first to beat a professional player at the game of Go. Another AI innovation from Google is Google Duplex. Using natural language processing, an AI voice interface can make phone calls and schedule appointments on your behalf. Learn even more about how Google is incorporating artificial intelligence and machine learning into operations.</p><p>Artificial intelligence (AI)’s importance in business is set to increase in 2020 with Google stepping up its focus on the technology after a decade filled with activity.</p><p>With more than 1 billion users, Google is one of the world’s biggest brands, and arguably the biggest key driver of modern technological advancement — only recently, Google announced their ‘Quantum Supremacy’ breakthrough, in which their quantum computer managed to solve a mathematical calculation in 3 minutes and 20 seconds — something that supercomputers are unable to complete in under 10,000 years.</p><p>The past decade alone has seen huge leaps in Google’s capabilities and focus on AI, with the arrival of Google Assistant in 2016 and the Neural Matching algorithm introduced in 2018, to deliver more diverse search results by analyzing language on a deeper level than previous algorithms. For the first time, Google could match words to concepts and figure out what a user wanted from a looser search.</p><p>Google’s commitment only looks set to grow, with the company’s co-founder, Larry Page, taking a particularly close interest in AI, revealing that: “Google will fulfill its mission only when its search engine is AI-complete.”</p><p>Google is definitely, absolutely committed to AI. Larry Page is committed, and it is viable in both short-term, by being able to out-analyze competition, and the long-term, the consequences of being the first creator of anything resembling true AI is vast.</p><blockquote>“AI is no different from the climate,” Pichai said. “You can’t get safety by having one country or a set of countries working on it. You need a global framework.”</blockquote><p><em>— Sundar Pichai, CEO of Google</em></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FAXgjEW2nA9I%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DAXgjEW2nA9I&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FAXgjEW2nA9I%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/6a19ba6294ae6e67b791a3fba58be30e/href">https://medium.com/media/6a19ba6294ae6e67b791a3fba58be30e/href</a></iframe><p><strong>2. Amazon</strong></p><p>Not only is Amazon in the artificial intelligence game with its digital voice assistant, Alexa, but artificial intelligence is also part of many aspects of its business. Another innovative way Amazon uses artificial intelligence is to ship things to you before you even think about buying it. They collect a lot of data about each person’s buying habits and have such confidence in how the data they collect helps them recommend items to its customers and now predict what they need even before they need it by using predictive analytics. In a time when many brick-and-mortar stores are struggling to figure out how to stay relevant, America’s largest e-tailer offers a new convenience store concept called Amazon Go. Unlike other stores, there is no checkout required. The stores have artificial intelligence technology that tracks what items you pick up and then automatically charges you for those items through the Amazon Go app on your phone. Since there is no checkout, you bring your own bags to fill up with items, and there are cameras watching your every move to identify every item you put in your bag to ultimately charge you for it.</p><h4>How AI transformed Amazon into an eCommerce powerhouse</h4><p>Amazon is an eCommerce powerhouse. In 2019, the giant retailer is expected to hit a 52% market share in the US and almost 14% worldwide.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/738/0*uEfTA95FIgcOUfGB.png" /></figure><p>From predicting how many customers might buy a new product to running Amazon Go, the cashier-less grocery store to assessing the quality of fruits and vegetables without smelling them for Amazon Fresh because humans are too slow and inconsistent.</p><p>Amazon’s AI capabilities allow the platform to provide customized recommendations. According to the latest reports, Amazon’s recommendation engine drives 35% of total sales.</p><p>With more than 100 million Alexa devices having been sold, Amazon’s smart speaker maintains a 70% market share in the US. In a statement in the last quarter of fiscal 2018, Jeff Bezos said Echo Dot was the best-selling item across all products on Amazon globally, and customers purchased millions of more devices from the Echo family compared to last year (source).</p><p>In 2018, Amazon developers improved Alexa’s ability to understand requests and answer questions by more than 20% through advances in machine learning. Also, the number of research scientists working on Alexa has more than doubled in the past year, Bezos noted.</p><p><em>The giant American e-commerce retailer builds everything on AI because AI solves shoppers’ problems which in turn supports Amazon’s customer-centric approach.</em></p><blockquote>“Autonomous weapons are ‘genuinely scary,’ robots won’t put us all out of work.”</blockquote><p>— Jeff Bezos, CEO of Amazon</p><p><strong>3. Apple</strong></p><p>Apple, one of the world’s largest technology companies, selling consumer electronics such as iPhones and Apple Watches, as well as computer software and online services. Apple uses artificial intelligence and machine learning in products like the iPhone, where it enables the FaceID feature, or in products like the AirPods, Apple Watch, or HomePod smart speakers, where it enables the smart assistant Siri. Apple is also growing its service offering and is using AI to recommend songs on Apple Music, help you find your photo in the iCloud, or navigate to your next meeting using Maps.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*X3rmPZRNGHzDf_vP.jpg" /><figcaption>Apple Acquires More AI Startups Than Any Other Tech Company</figcaption></figure><p>Unfortunately for Apple, it doesn’t seem like all this spending has resulted in an AI platform that dominates the market. According to the Voice Platform Impact Ranking, which factors in both consumer adoption and ecosystem support, Siri has fallen behind the competition. On a scale of -250 to 250, Alexa currently scores at 110.2 and Google scores at 97.3, while Siri sits at a mere 38.3 in front of Samsung’s Bixby.</p><p>So far, the Siri ecosystem has failed to expand beyond the iOS operating system. Meanwhile, Google Home and Amazon Echo devices feature robust support from any third parties. It would seem that money spent doesn’t necessarily equal long term growth or success, but Apple continues to make acquisitions in 2020.</p><blockquote>“We can achieve both great artificial intelligence and great privacy standards. It’s not only a possibility, it is a responsibility. In the pursuit of artificial intelligence, we should not sacrifice humanity, creativity, and ingenuity that define our human intelligence.”</blockquote><p><em>— Tim Cook, CEO of Apple</em></p><p><strong>4. Facebook</strong></p><p>One of the primary ways Facebook uses artificial intelligence and deep learning is to add structure to its unstructured data. They use DeepText, a text understanding engine, to automatically understand and interpret the content and emotional sentiment of the thousands of posts (in multiple languages) that its users publish every second. With DeepFace, the social media giant can automatically identify you in a photo that is shared on its platform. In fact, this technology is so good, it’s better at facial recognition than humans. The company also uses artificial intelligence to automatically catch and remove images that are posted on its site as revenge porn.</p><p>The social media giant along with its subsidiaries like Instagram and WhatsApp, has the biggest user-base in the country, with millions of users subscribing to its apps presently. Facebook, in the last few years, has strengthened its presence in the country with initiatives like the India Innovation Program, where it has tied-up with T-Hub to mentor startups that are capable of solving problems that are unique to India. Announcing the commencement of its second edition, Facebook stated that this year’s theme will focus on artificial intelligence, where startups working on the technology will be picked by the company.</p><p>It then stated the startups chosen for the cohort will focus on solving societal problems in the areas of Women and Girl Empowerment, Agriculture, Healthcare, Education, Climate Change, Resilient Cities, etc.</p><p>“With the second edition of the India Innovation Accelerator program, we want to help to build the ecosystem for AI in India and support startups with the resources and opportunities,” Satyajeet Singh, Head of Platform Partnerships, Facebook India and South Asia, said in a press statement.</p><p>Apart from the exclusive cohort program, in March, Facebook concluded its AI for India summit, where the social media platform discussed its AI strategy for India. At the sidelines of the summit, it announced that 100 scholarships will be awarded to students and developers to nurture their ideas for utilizing AI for social good, training through Facebook Hubs and exclusive AI hackathons for women will be organized to increase development in the field of AI and ML.</p><blockquote><em>“I think we should be very careful about artificial intelligence. If I had to guess at what our biggest existential threat is, I’d probably say that. So we need to be very careful.“</em></blockquote><p><em>— Mark Zuckerberg, CEO of Facebook</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*XagHoJpsEobN5NAh.jpg" /></figure><p><strong>5. Microsoft</strong></p><p>Artificial intelligence is a term that appears on Microsoft’s vision statement, which illustrates the company’s focus on having smart machines central to everything they do. They are incorporating intelligent capabilities to all its products and services, including Cortana, Skype, Bing, and Office 365, and are one of the world’s biggest AI as a Service (AIaaS) vendors.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*DNfSXhctlpOUkN3I.png" /></figure><p>Microsoft, one of the most valuable companies on this planet, is heavily focusing on AI as the go-to technology for their future success. In the following blog post, we will try to unpack</p><ol><li>whether the claim <strong>“bringing ai to everyone” </strong>can and should be taken literally, and</li><li>what the strategic considerations behind the positioning towards AI are.</li></ol><blockquote><em>“We want to pursue democratizing AI just like we pursued information at your fingertips”</em></blockquote><p><em>— Satya Nadella, CEO of Microsoft</em></p><p>we already gave an overview of the various cloud platforms claiming the topic of intelligent automation and process integration for themselves. After the first wave of digitization, where the focus was primarily on data storage, access options, and communication, the key technological component now is machine learning, which naturally benefits greatly from the enormous data volumes and immediate availability in the cloud and has made considerable progress in recent years.</p><p>In 2014, Microsoft CEO Satya Nadella announced the cloud-first strategy for the company to prevent another missed entry as seen before in the mobile business. Furthermore, the aim was to transform the company from an outdated single software license business for operating systems and productivity applications towards SaaS business models. Since then, market capitalization has roughly quadrupled within 6 years, making Microsoft one of the most valuable companies in the world. After Amazon Web Service, Microsoft’s own cloud platform Azure is the second-largest cloud provider with well-known partners and clients that contributed almost one-third of total sales in 2019 with around US$ 39 billion.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/732/0*aS9BayGqHvyEHtuU.png" /><figcaption>Market cap development of Microsoft from 2014 to date</figcaption></figure><h3><strong>SUMMARY</strong></h3><p>The market for artificial intelligence has exploded over the last decade. While platforms such as Amazon’s Alexa, Apple’s Siri, Google’s Assistant, and Microsoft’s Cortana have dominated the market, these companies have not been the source of that growth. Instead, stiff competition between the biggest corporations has led to aggressive acquisitions of AI startups.</p><p>According to data compiled by CB Insights, there have been a total of 635 artificial intelligence acquisitions between 2010 and September 2019. Those purchases also increased six times over from 2013 to 2018, with 166 acquisitions in 2018 alone — a 38 percent increase year over year. For companies like Apple, Google, and the rest, these purchases can almost always be linked back to a new product or feature.</p><p>Leading the charge is Apple, with 20 acquisitions since the 2010 purchase of Siri. As an example, the tech giant acquired Novauris Technologies and Voysis to improve speech recognition of its voice assistant platform. Apple has also bought companies including Perceptio and RealFace to develop its Apple ID technology.</p><p>Google follows with 14 acquisitions in the same time period, though it once led all tech companies in artificial intelligence acquisitions, between 2012 and 2016. Google grabbed DeepMind Technologies to develop applications for artificial intelligence. It also acquired companies including Api.ai and Superpod to strengthen Google Assistant’s capabilities.</p><p>Microsoft has bought 10 startups, Facebook has acquired eight, and Amazon purchased seven in total. Of the 489 companies to buy an AI startup over the last decade, 431 of them purchased only one company. That means these tech giants — and Apple, specifically — are dominating this AI arms race.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*feWTa-W6TL4HzzOk.jpg" /></figure><h3>CONCLUSION</h3><p>In business management, the trend is becoming more customer-centric, personalized, and data-driven. As technology continues to progress, current practices will continue to evolve. In the end, it is about creating real customer value. No matter how businesses achieve that, being open and adaptive to change is one step closer to staying competitive and relevant in the face of new challenges, whether brought by humans or technology.</p><p>Most business leaders expect significant changes from AI in the next five years. But according to AI industry experts, the more dramatic effects of AI may occur within 10 to 20 years. That means that just about any company today needs a plan with respect to AI — it’s up to you to decide which technology will keep you competitive. Keep yourself informed, choose a strong platform, and watch your business reap the benefits.</p><blockquote>These amazing videos will inspire you to learn Artificial Intelligence and Robotics.</blockquote><blockquote>How can we make a robot that is truly artificially intelligent?</blockquote><p>Note: They are not VFX, They are Real.</p><p>Must Watch !!</p><p>(5) Video: Horror Short Film “Slaughterbots” | ALTER !!!</p><p>(6) Video: Chappie: Switching Human’s Consciousness to Robot.</p><p>— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —(1)</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FTxobtWAFh8o%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DTxobtWAFh8o&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FTxobtWAFh8o%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/1f82fa340264700e586320777c42d96d/href">https://medium.com/media/1f82fa340264700e586320777c42d96d/href</a></iframe><p>(2)</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FAO4In7d6X-c%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DAO4In7d6X-c&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FAO4In7d6X-c%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/14ef9cb4598adc7bf226718d2104fd6b/href">https://medium.com/media/14ef9cb4598adc7bf226718d2104fd6b/href</a></iframe><p>(3)</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FRCXGpEmFbOw%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DRCXGpEmFbOw&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FRCXGpEmFbOw%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/e86c4871e94277a5ed1247c5d1ceec21/href">https://medium.com/media/e86c4871e94277a5ed1247c5d1ceec21/href</a></iframe><p>(4)</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FFg_JcKSHUtQ%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DFg_JcKSHUtQ&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FFg_JcKSHUtQ%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/debccec785a1683ef2cfc5c1a0db9b08/href">https://medium.com/media/debccec785a1683ef2cfc5c1a0db9b08/href</a></iframe><p>(5)</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F9fa9lVwHHqg%3Fstart%3D276%26feature%3Doembed%26start%3D276&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D9fa9lVwHHqg&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F9fa9lVwHHqg%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/f25a06d80143421e1430cca8483c7157/href">https://medium.com/media/f25a06d80143421e1430cca8483c7157/href</a></iframe><p>(6)</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FygtR0iOncA4%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DygtR0iOncA4&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FygtR0iOncA4%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/d2998baf9c9704acec057adbd1191591/href">https://medium.com/media/d2998baf9c9704acec057adbd1191591/href</a></iframe><p><em>Thank you !!</em></p><p><em>Like | Share | Comment</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ad70646b540f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What is a Big Data problem?]]></title>
            <link>https://dhyey-joshi12.medium.com/what-is-a-big-data-problem-da5d1c85f87d?source=rss-2ac64d1c310a------2</link>
            <guid isPermaLink="false">https://medium.com/p/da5d1c85f87d</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[makingindiafutureready]]></category>
            <category><![CDATA[hadoop]]></category>
            <dc:creator><![CDATA[Dhyey Joshi]]></dc:creator>
            <pubDate>Wed, 16 Sep 2020 13:24:11 GMT</pubDate>
            <atom:updated>2020-09-16T13:50:34.737Z</atom:updated>
            <content:encoded><![CDATA[<h3>What is Big Data Problem?</h3><p><em>!!Hello Everyone …!!</em></p><p><strong><em>Big data is a term that describes the large volume of data — both structured and unstructured — that inundates a business on a day-to-day basis. But it’s not the amount of data that’s important. It’s what organizations do with the data that matters. Big data can be analyzed for insights that lead to better decisions and strategic business moves.</em></strong></p><p><em>In order to understand ‘Big Data’, Let’s split the word ” BIG DATA “ in two parts, what do you get? .We can say “big data” refers to data that is so large, fast, or complex that it’s difficult or impossible to process using traditional methods.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EBAQhDuxjlIZpOXQ_aKKYw.jpeg" /><figcaption>“Processed data is information, Processed information is knowledge, Processed knowledge is Wisdom.”</figcaption></figure><p>In this video, you will understand the whole concept of big data in just a few mins.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FQvyQFXbgW2c%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DQvyQFXbgW2c&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FQvyQFXbgW2c%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/99788121583de5656853d632a541c959/href">https://medium.com/media/99788121583de5656853d632a541c959/href</a></iframe><h3>Characteristics Of Big Data Problem</h3><blockquote><strong>i) Volume </strong>— The name Big Data itself is related to a size that is enormous. The size of data plays a very crucial role in determining value out of data. Also, whether a particular data can actually be considered as a Big Data or not, is dependent upon the volume of data. Hence, <strong>‘Volume’</strong> is one characteristic that needs to be considered while dealing with Big Data.</blockquote><blockquote>This illustration will show you how much amount of data uploaded to <strong>the Internet</strong> in <strong>a</strong> single <strong>second</strong> is <strong>a</strong> staggering min of 24,000 gigabytes v/s How much your system can store this data !!</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/389/1*XPBq3gR0WVqUG-VjwrVSZw.gif" /><figcaption>Volume (Capacity, Size) →Big Data V/S Your System Storage</figcaption></figure><blockquote><strong>(iii) Velocity </strong>— The term <strong>‘velocity’</strong> refers to the speed of generation of data. How fast the data is generated and processed to meet the demands, determines the real potential in the data.</blockquote><blockquote>Big Data Velocity deals with the speed at which data flows in from sources like business processes, application logs, networks, and social media sites, sensors, Mobile devices, etc. The flow of data is massive and continuous.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/373/1*xBpvHKdS4uqfSfkF9SUOsQ.gif" /><figcaption><strong>Data transferring and receiving is far faster then you think!!</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/777/1*VAMRW1Sj2arK3zUPzduyEQ.gif" /><figcaption><strong>What your actual data transferring looks like!! (Bits form i.e 0 and 1)</strong></figcaption></figure><blockquote><strong>(iv) Variability </strong>— This refers to the inconsistency which can be shown by the data at times, thus hampering the process of being able to handle and manage the data effectively. (This means your data will not be the same every time.) and many more sub-problems are under big data.</blockquote><p><a href="https://www.internetlivestats.com/">Internet Live Stats - Internet Usage &amp; Social Media Statistics</a></p><blockquote><strong>Click the above website to see how much data is uploading , downloading and sharing right now on internet .</strong></blockquote><h3>Why Is Big Data Important?</h3><p><strong><em>The importance of big data does not revolve around how much data a company has but how a company utilizes the collected data. Every company uses data in its own way; the more efficiently a company uses its data, the more potential it has to grow. The company can take data from any source and analyze it to find answers which will enable:</em></strong></p><blockquote><strong><em>1. Cost reductions.</em></strong></blockquote><blockquote><strong><em>2. Time reductions.</em></strong></blockquote><blockquote><strong><em>3. New product development and optimized offerings.</em></strong></blockquote><blockquote><strong><em>4. Smart decision making.</em></strong></blockquote><h3>Key advantages of big data</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*ZWJhRBRVHmLZd4j1WMPBWg.png" /></figure><blockquote>Here’re the biggest advantages of using <strong>big data</strong>.</blockquote><blockquote><strong>Improved business processes: </strong>Probably the biggest advantage of <strong>big data</strong> is it helps businesses to gain a huge competitive advantage. Apart from being able to understand, as well as, target customers better, analyzing <strong>big data</strong> can result in the improvement and optimization of certain facets of business operations.</blockquote><blockquote><strong>Fraud detection:</strong> This advantage of using <strong>big data</strong> comes from the implementation of machine learning technologies. It helps banks and other financial institutions to detect frauds like fraudulent purchases with credit cards often before even the cardholder gets to know about it.</blockquote><blockquote><strong>Improved customer service:</strong> One of the most common goals among <strong>big data</strong> analytics programs is improving customer service. Today’s businesses capture a huge amount of information from different sources like customer relationship management (CRM) systems, social media together with other points of customer contact. By analyzing this massive amount of information they get to know about the tastes and preferences of a user. And with the help of <strong>big data</strong> technologies, they become able to create experiences that are more responsive, personal, and accurate than ever before.</blockquote><h3>Key disadvantages of big data</h3><blockquote>Despite the advantages of <strong>big data</strong>, it comes with some serious challenges that make its implementation difficult or risky. Here’re the biggest disadvantages.</blockquote><blockquote><strong>Privacy and security concerns: </strong>Probably the biggest disadvantage of <strong>big data</strong> is that it can make businesses a softer target for cyberattackers. Even giant businesses have experienced instances of massive data breaches. However, with the implementation of GDPR, businesses are increasingly trying to invest in processes, protocols, and infrastructure to be able to maintain <strong>big data</strong></blockquote><blockquote><strong>Need for technical expertise:</strong> Working with <strong>big data</strong> needs a great deal of technical proficiency and that’s one of the key reasons for which big data experts and data scientists belong to the highly paid and highly coveted group in the IT landscape. Training existing staff or hiring experts to handle <strong>big data</strong> can easily increase the cost of a business considerably.</blockquote><h3>WHAT IS DISTRIBUTED DATA STORAGE?</h3><p><strong><em>A distributed data store is a computer network where information is stored on more than one node, often in a replicated fashion. It is usually specifically used to refer to either a distributed database where users store information on a number of nodes, or a computer network in which users store information on a number of peer network nodes.</em></strong></p><p><strong><em>Using Distributed Data Storage, you can deliver any of the 3 types of storage, may it be block, file, or object.</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*t-EskMQxiYYDY9KWwpoXxw.gif" /><figcaption>The best example of distributing data into small Sub-Units …!! And this is how a huge data is distributed into small storages at the same time and speed(Velocity).</figcaption></figure><blockquote><strong><em>Distributed Data Storage</em></strong> is just an<strong> idea </strong>which is accomplished by software named as</blockquote><blockquote><strong>“ Hadoop “</strong></blockquote><blockquote><strong>What is Hadoop?</strong></blockquote><blockquote>Apache <strong>Hadoop</strong> is an open-source framework that is <strong>used</strong> to efficiently store and process large datasets ranging in size from gigabytes to petabytes of data. Instead of using one large computer to store and process the data, <strong>Hadoop</strong> allows clustering multiple computers to analyze massive datasets in parallel more quickly.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*kLaAM18QdsQ2ZfGMDO5bQg.gif" /><figcaption><strong><em>How Hadoop Works Internally — Inside Hadoop — DataFlow</em></strong></figcaption></figure><p><em>I hope this article is helpful to you !!!!!</em></p><p><em>Thank you Mr.Vimal Daga sir for this amazing Task!!</em></p><p><strong><em>Thanks for reading !!!</em></strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=da5d1c85f87d" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>