<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Alamin Musa Magaga on Medium]]></title>
        <description><![CDATA[Stories by Alamin Musa Magaga on Medium]]></description>
        <link>https://medium.com/@alaminmusamagaga?source=rss-d9bc7626f8c------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*3yrBwk2VHqeYyMApufCwmQ.jpeg</url>
            <title>Stories by Alamin Musa Magaga on Medium</title>
            <link>https://medium.com/@alaminmusamagaga?source=rss-d9bc7626f8c------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 08:04:11 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@alaminmusamagaga/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Predicting Heart DiseaseUsing Machine Learning]]></title>
            <link>https://alaminmusamagaga.medium.com/predicting-heart-diseaseusing-machine-learning-bfd334aee5a5?source=rss-d9bc7626f8c------2</link>
            <guid isPermaLink="false">https://medium.com/p/bfd334aee5a5</guid>
            <dc:creator><![CDATA[Alamin Musa Magaga]]></dc:creator>
            <pubDate>Fri, 09 Jun 2023 01:22:04 GMT</pubDate>
            <atom:updated>2023-06-09T01:32:30.278Z</atom:updated>
            <content:encoded><![CDATA[<h3>Predicting Heart Disease Using Machine Learning</h3><p>A Machine Learning Approach to Accurate Heart Disease Classification</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*CfFkUmqrM2k0CEi_" /><figcaption>Photo by <a href="https://unsplash.com/de/@jessedo81?utm_source=medium&amp;utm_medium=referral">jesse orrico</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Heart disease is a common and potentially life-threatening condition that affects a significant number of people worldwide. Early detection and accurate diagnosis of heart disease are crucial for effective treatment and prevention of complications. In recent years, machine learning techniques have shown promise in assisting healthcare professionals with heart disease classification tasks.</p><p>In this blog post, we will explore a machine-learning classification task for heart disease using a dataset containing various patient attributes. We will walk through the code implementation, explain the key steps involved, and discuss the results obtained. So, let’s get started!</p><h3>Table of Contents</h3><ol><li>Introduction to Heart Disease Classification</li><li>Understanding the Dataset</li><li>Exploratory Data Analysis (EDA)</li><li>Data Preprocessing</li><li>Feature Selection</li><li>Model Selection and Evaluation</li><li>Conclusion</li></ol><h3>1. Introduction to Heart Disease Classification</h3><p>In this section, we will provide a brief overview of heart disease classification and its significance in healthcare. We will discuss how machine learning techniques can be leveraged to develop predictive models for heart disease diagnosis.</p><h3>2. Understanding the Dataset</h3><p>Before diving into the implementation, it is crucial to understand the dataset we will be working with. We will examine the attributes present in the dataset and their meanings. Additionally, we will explore basic statistics and gain insights into the data.</p><pre>#import libraries<br>import pandas as pd<br>import matplotlib.pyplot as plt<br>import seaborn as sns<br>import warnings<br>warnings.filterwarnings(&#39;ignore&#39;)<br><br># Load the dataset<br>heart_data = pd.read_csv(&#39;heart.csv&#39;)<br><br># Display the first few rows of the dataset<br>print(heart_data.head())<br><br># Display basic statistics of the dataset<br>print(heart_data.describe())</pre><h3>3. Exploratory Data Analysis (EDA)</h3><p>EDA is a crucial step in any data analysis task. In this section, we will visualize the data, analyze the distribution of features, and investigate relationships between variables. EDA helps us understand the dataset better and identify any patterns or correlations.</p><pre># Visualize the distribution of the target variable<br>sns.countplot(x=&#39;target&#39;, data=heart_data)<br>plt.title(&#39;Distribution of Target Variable&#39;)<br>plt.show()<br><br># Analyze the relationship between variables<br>sns.heatmap(heart_data.corr(), annot=True, cmap=&#39;YlGnBu&#39;)<br>plt.title(&#39;Correlation Heatmap&#39;)<br>plt.show()</pre><h3>4. Data Preprocessing</h3><p>Data preprocessing plays a vital role in preparing the dataset for machine learning algorithms. In this section, we will handle missing values, perform feature scaling, and encode categorical variables. Preprocessing ensures that the data is in a suitable format for training our models.</p><pre># Handle missing values<br>heart_data = heart_data.dropna()<br><br># Perform feature scaling<br>from sklearn.preprocessing import StandardScaler<br><br>scaler = StandardScaler()<br>heart_data[[&#39;age&#39;, &#39;trestbps&#39;, &#39;chol&#39;, &#39;thalach&#39;, &#39;oldpeak&#39;]] = scaler.fit_transform(<br>    heart_data[[&#39;age&#39;, &#39;trestbps&#39;, &#39;chol&#39;, &#39;thalach&#39;, &#39;oldpeak&#39;]])<br><br># Encode categorical variables<br>heart_data = pd.get_dummies(heart_data, columns=[&#39;sex&#39;, &#39;cp&#39;, &#39;fbs&#39;, &#39;restecg&#39;, &#39;exang&#39;, &#39;slope&#39;, &#39;ca&#39;, &#39;thal&#39;])</pre><h3>5. Feature Selection</h3><p>Feature selection involves identifying the most relevant features that contribute significantly to the prediction task. In this section, we will employ various techniques to select the most informative features from our dataset. Feature selection helps in reducing dimensionality and improving model performance.</p><pre># Perform feature selection using Recursive Feature Elimination (RFE)<br>from sklearn.feature_selection import RFE<br>from sklearn.linear_model import LogisticRegression<br><br>X = heart_data.drop(&#39;target&#39;, axis=1)<br>y = heart_data[&#39;target&#39;]<br><br>estimator = LogisticRegression()<br>selector = RFE(estimator, n_features_to_select=8)<br>selector = selector.fit(X, y)<br><br>selected_features = X.columns[selector.support_]<br>X_selected = X[selected_features]</pre><h3>6. Model Selection and Evaluation</h3><p>Now comes the exciting part — building machine learning models for heart disease classification. We will explore different classification algorithms, including logistic regression, decision trees, and support vector machines (SVM). We will train these models, evaluate their performance, and compare the results.</p><pre># Split the data into training and testing sets<br>from sklearn.model_selection import train_test_split<br><br>X_train, X_test, y_train, y_test = train_test_split(X_selected, y, test_size=0.2, random_state=42)<br><br># Train and evaluate logistic regression model<br>from sklearn.linear_model import LogisticRegression<br>from sklearn.metrics import accuracy_score, classification_report, confusion_matrix<br><br>lr_model = LogisticRegression()<br>lr_model.fit(X_train, y_train)<br>y_pred = lr_model.predict(X_test)<br><br>print(&#39;Logistic Regression Model:&#39;)<br>print(&#39;Accuracy:&#39;, accuracy_score(y_test, y_pred))<br>print(&#39;Classification Report:&#39;)<br>print(classification_report(y_test, y_pred))<br>print(&#39;Confusion Matrix:&#39;)<br>print(confusion_matrix(y_test, y_pred))<br><br># Train and evaluate decision tree model<br>from sklearn.tree import DecisionTreeClassifier<br><br>dt_model = DecisionTreeClassifier()<br>dt_model.fit(X_train, y_train)<br>y_pred = dt_model.predict(X_test)<br><br>print(&#39;Decision Tree Model:&#39;)<br>print(&#39;Accuracy:&#39;, accuracy_score(y_test, y_pred))<br>print(&#39;Classification Report:&#39;)<br>print(classification_report(y_test, y_pred))<br>print(&#39;Confusion Matrix:&#39;)<br>print(confusion_matrix(y_test, y_pred))<br><br># Train and evaluate support vector machine (SVM) model<br>from sklearn.svm import SVC<br><br>svm_model = SVC()<br>svm_model.fit(X_train, y_train)<br>y_pred = svm_model.predict(X_test)<br><br>print(&#39;SVM Model:&#39;)<br>print(&#39;Accuracy:&#39;, accuracy_score(y_test, y_pred))<br>print(&#39;Classification Report:&#39;)<br>print(classification_report(y_test, y_pred))</pre><h3>7. Conclusion</h3><p>Finally, we will summarize our findings, discuss the limitations of our approach, and highlight potential areas for future improvement. We will conclude by emphasizing the role of machine learning in heart disease classification and its potential impact on healthcare.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bfd334aee5a5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Exploring Heart Disease Risk Factors through Data Visualization]]></title>
            <link>https://alaminmusamagaga.medium.com/exploring-heart-disease-risk-factors-through-data-visualization-957bd80ab07c?source=rss-d9bc7626f8c------2</link>
            <guid isPermaLink="false">https://medium.com/p/957bd80ab07c</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Alamin Musa Magaga]]></dc:creator>
            <pubDate>Sun, 14 May 2023 11:56:54 GMT</pubDate>
            <atom:updated>2023-05-14T12:06:10.042Z</atom:updated>
            <content:encoded><![CDATA[<p>Visualize the correlation and relationship between different Heart Disease Risk Factors</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*Q2lxHR88MXSTLqu6gJbJiA.jpeg" /></figure><p>Heart disease is a leading cause of death worldwide, and understanding its risk factors is crucial for prevention and management. In this blog post, we will delve into the heart disease dataset and utilize data visualization techniques to gain insights into the factors that contribute to heart disease. We will explore various visualizations to better understand the relationships between different variables and their impact on the target variable, which indicates the presence of heart disease.</p><p><strong>Dataset Overview:</strong></p><p>The heart disease dataset consists of 303 rows and 14 columns, containing information about various patient characteristics and heart disease diagnosis. Before diving into the visualizations, let’s load the dataset and take a look at its structure.</p><p><strong>Data Loading and Inspection:</strong></p><pre>import pandas as pd<br>import matplotlib.pyplot as plt<br>import seaborn as sns<br><br>df = pd.read_csv(&#39;heart.csv&#39;)<br>df.head()</pre><p>output</p><pre>Age Sex ChestPain RestBP       Chol  Fbs RestECG MaxHR ExAng Oldpeak Slope Ca Thal AHD<br>0   63      1     typical      145   233    1      2    150     0     2.3   3 0.0 fixed No<br>1   67      1     asymptomatic 160   286    0      2    108     1     1.5   2 3.0 normal Yes<br>2   67      1     asymptomatic 120   229    0      2    129     1     2.6   2 2.0 reversable Yes<br>3   37      1     nonanginal   130   250    0      0    187     0     3.5   3 0.0 normal No<br>4   41      0     nontypical   130   204    0      2    172     0     1.4   1 0.0 normal No</pre><p><strong>Visualization 1: Bar Plot</strong></p><p>One of the risk factors associated with heart disease is exercise-induced angina, which refers to chest pain experienced during physical activity. We can visualize the distribution of patients with and without exercise-induced angina using a bar plot.</p><pre><br>plt.figure(figsize=(8, 6))<br>sns.countplot(x=&#39;exang&#39;, data=df)<br>plt.title(&#39;Distribution of Exercise-Induced Angina&#39;)<br>plt.xlabel(&#39;Exercise-Induced Angina&#39;)<br>plt.ylabel(&#39;Count&#39;)<br>plt.show()</pre><p>output</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*I3IZr0-G1qlsmJbacW3Xpg.png" /></figure><p>The bar plot showcases the distribution of exercise-induced angina among the patients. The x-axis represents the presence (1) or absence (0) of exercise-induced angina, while the y-axis denotes the count of patients. This visualization helps us understand the proportion of patients with and without exercise-induced angina in the dataset.</p><p><strong>Visualization 2: Correlation Heatmap</strong></p><p>To identify relationships between different variables and heart disease, we can create a correlation heatmap. It provides a color-coded matrix that depicts the correlation coefficients between pairs of variables.</p><pre>plt.figure(figsize=(12, 10))<br>sns.heatmap(df.corr(), annot=True, cmap=&#39;coolwarm&#39;)<br>plt.title(&#39;Correlation Heatmap&#39;)<br>plt.show()</pre><p>output</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/659/1*FKi5nCvaUk8ukB4wIpCr-w.png" /></figure><p>The correlation heatmap displays the correlation coefficients between various variables in the dataset. Each cell represents the strength and direction of the correlation. Positive values indicate a positive correlation, while negative values denote a negative correlation. This visualization enables us to identify which factors have a significant impact on heart disease.</p><p><strong>Visualization 3: Box Plot</strong></p><p>Cholesterol levels are often associated with heart disease risk. We can visualize the distribution of cholesterol levels among patients with and without heart disease using a box plot.</p><pre>plt.figure(figsize=(8, 6))<br>sns.boxplot(x=&#39;Target&#39;, y=&#39;Chol&#39;, data=df)<br>plt.title(&#39;Cholesterol Levels by Heart Disease&#39;)<br>plt.xlabel(&#39;Heart Disease&#39;)<br>plt.ylabel(&#39;Cholesterol&#39;)<br>plt.xticks([0, 1], [&#39;No Disease&#39;, &#39;Disease&#39;])<br>plt.show()</pre><p>output</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*YpF_6JaSAUuGNTvJoKilrw.png" /></figure><p>The box plot represents the distribution of cholesterol levels (chol) based on the presence (1) or absence (0) of heart disease. The y-axis denotes the cholesterol values, while the x-axis represents the two categories: “No Disease” and “Disease.” This visualization helps us compare the cholesterol levels between patients with and without heart disease.</p><p><strong>Visualization 4: Scatter Plot</strong></p><p>The maximum heart rate achieved during cholesterol levels(Chol) is an essential parameter in assessing heart health. We can create a scatter plot to visualize the relationship between age and cholesterol levels(Chol).</p><pre>plt.figure(figsize=(8, 6))<br>plt.scatter(df[&#39;Age&#39;], df[&#39;Chol&#39;], alpha=0.8)<br>plt.title(&#39;Age vs. cholesterol levels&#39;)<br>plt.xlabel(&#39;Age&#39;)<br>plt.ylabel(&#39;cholesterol levels&#39;)<br>plt.show()</pre><p>output</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*gT6eNywLNtKBZONMLTxa7g.png" /></figure><p>The scatter plot showcases the relationship between a patient’s age and their cholesterol levels during exercise. Each point represents a patient, and the x-axis represents their age, while the y-axis represents their cholesterol levels. This visualization allows us to observe any patterns or trends between age and heart rate, which can provide insights into the impact of age on heart health.</p><p><strong>Visualization 4: Pie Chart</strong></p><p>To visualize the presence of heart disease within our dataset, we can use a pie chart. The code snippet below demonstrates how to create a pie chart to display the proportion of individuals with and without heart disease</p><pre>plt.figure(figsize=(8, 6))<br>heart_disease_counts = df[&#39;Target&#39;].value_counts()<br>plt.pie(heart_disease_counts.values, labels=heart_disease_counts.index, autopct=&#39;%1.1f%%&#39;, startangle=90)<br>plt.title(&#39;Presence of Heart Disease&#39;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/340/1*xiKfff6ZVeKL5UK9wd8ocQ.png" /></figure><p><strong>Conclusion:</strong></p><p>In this blog post, we explored the Heart Disease Dataset and utilized various visualizations to gain insights into heart disease risk factors. The visualizations highlighted the distribution of exercise-induced angina, the correlations between different variables, and the relationship between cholesterol levels and heart disease, and their relationship. By leveraging these visualizations, we can better understand the factors contributing to heart disease and potentially develop strategies for prevention and management.</p><p>you can find the visualization code <a href="https://github.com/alaminmagaga/Heart-Disease-Visualization">here</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=957bd80ab07c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Exploratory Data Analysis and Preprocessing | USA Housing]]></title>
            <link>https://alaminmusamagaga.medium.com/exploratory-data-analysis-and-preprocessing-usa-housing-f8fbc50c5822?source=rss-d9bc7626f8c------2</link>
            <guid isPermaLink="false">https://medium.com/p/f8fbc50c5822</guid>
            <dc:creator><![CDATA[Alamin Musa Magaga]]></dc:creator>
            <pubDate>Sat, 08 Apr 2023 14:05:40 GMT</pubDate>
            <atom:updated>2023-04-10T18:16:36.298Z</atom:updated>
            <content:encoded><![CDATA[<p>A simple guide to analyze, clean, and preprocess data</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BAWItoAaGj_Pg5dPLNos8g.jpeg" /></figure><p>Data Exploration is one of the most important and essential steps in any data driven or machine-learning project and plays a crucial role in understanding and uncovering insights from the dataset.</p><p>Exploratory Data Analysis (EDA) and preprocessing are two key components of data analysis that help to uncover insights, identify trends, and clean and prepare data for further analysis.</p><h4><strong>What is Exploratory Data Analysis (EDA)?</strong></h4><p>Exploratory Data Analysis (EDA) is a process of visually and statistically analyzing and summarizing data to gain insights and understand the underlying structure and patterns in the data. It involves examining the data from various angles, such as identifying the data distribution, detecting outliers, exploring relationships between variables, and visualizing data using plots and charts.</p><p>EDA helps in identifying potential issues with the data, such as missing values, inconsistent data, and anomalies, which can impact the quality and reliability of the analysis results.</p><p>In this blog, we will explore some effective techniques for conducting EDA and preprocessing in the context of the USA Housing dataset.</p><p>first, download the dataset from <a href="https://www.kaggle.com/datasets/kanths028/usa-housing">Kaggle</a></p><p><strong>Step 1: Import the necessary libraries</strong></p><p>Pandas are used for data exploration and manipulation such as cleaning, transformation, and aggregation of data. the numpy library is used for numerical computation and maths functions, and the matplotlib and seaborn library are used for data visualization and plotting graph</p><pre>import numpy as np<br>import pandas as pd<br>import matplotlib.pyplot as plt<br>import seaborn as sns</pre><p><strong>Step 2: Read the Dataset</strong></p><p>The line of code below reads the USA housing file and stores it in the pandas dataframe df and the df.head() displays the first five rows of the data.</p><pre>df=pd.read_csv(&#39;USA_Housing.csv&#39;)<br>df.head()</pre><p>output</p><pre>Avg. Area Income Avg. Area House Age Avg. Area Number of Rooms Avg. Area Number of Bedrooms Area Population Price Address<br>0 79545.458574 5.682861 7.009188 4.09 23086.800503 1.059034e+06 208 Michael Ferry Apt. 674\nLaurabury, NE 3701...<br>1 79248.642455 6.002900 6.730821 3.09 40173.072174 1.505891e+06 188 Johnson Views Suite 079\nLake Kathleen, CA...<br>2 61287.067179 5.865890 8.512727 5.13 36882.159400 1.058988e+06 9127 Elizabeth Stravenue\nDanieltown, WI 06482...<br>3 63345.240046 7.188236 5.586729 3.26 34310.242831 1.260617e+06 USS Barnett\nFPO AP 44820<br>4 59982.197226 5.040555 7.839388 4.23 26354.109472 6.309435e+05 USNS Raymond\nFPO AE 09386</pre><p>you can decide to display a higher number of rows by placing the number inside the parentheses “()” or you can display the last five rows by using df.tail()</p><p><strong>Step 3: Check the Shape of the Data</strong></p><p>the code below displays the number of rows and columns of the data</p><pre>df.shape</pre><p>output:</p><pre>(5000, 7)</pre><p>the output shows that there are 5000 rows and 7 columns in the dataset</p><p><strong>Step 4: Check information about the DataFrame</strong></p><p>This line of code below provides information about the DataFrame, including the number of non-null values, data types, and memory usage.</p><pre>df.info()</pre><p>output:</p><pre>&lt;class &#39;pandas.core.frame.DataFrame&#39;&gt;<br>RangeIndex: 5000 entries, 0 to 4999<br>Data columns (total 7 columns):<br> #   Column                        Non-Null Count  Dtype  <br>---  ------                        --------------  -----  <br> 0   Avg. Area Income              5000 non-null   float64<br> 1   Avg. Area House Age           5000 non-null   float64<br> 2   Avg. Area Number of Rooms     5000 non-null   float64<br> 3   Avg. Area Number of Bedrooms  5000 non-null   float64<br> 4   Area Population               5000 non-null   float64<br> 5   Price                         5000 non-null   float64<br> 6   Address                       5000 non-null   object <br>dtypes: float64(6), object(1)<br>memory usage: 273.6+ KB</pre><p>It gives a summary of the data structure, including the number of rows, number of columns, column names, and data types of each column. It also provides information about missing values in the DataFrame, which can help identify any data quality issues.</p><p><strong>Step 5: Check the Statistical Description of the columns</strong></p><p>The code below generates descriptive statistics of the numeric columns in the DataFrame. It provides a summary of key statistical measures such as count, mean, standard deviation, minimum, 25th percentile (Q1), median (50th percentile or Q2), 75th percentile (Q3), and maximum for each numeric column.</p><pre>df.describe()</pre><p>output:</p><pre><br>Avg. Area Income Avg. Area House Age Avg. Area Number of Rooms Avg. Area Number of Bedrooms Area Population Price<br>count 5000.000000 5000.000000 5000.000000 5000.000000 5000.000000 5.000000e+03<br>mean 68583.108984 5.977222 6.987792 3.981330 36163.516039 1.232073e+06<br>std 10657.991214 0.991456 1.005833 1.234137 9925.650114 3.531176e+05<br>min 17796.631190 2.644304 3.236194 2.000000 172.610686 1.593866e+04<br>25% 61480.562388 5.322283 6.299250 3.140000 29403.928702 9.975771e+05<br>50% 68804.286404 5.970429 7.002902 4.050000 36199.406689 1.232669e+06<br>75% 75783.338666 6.650808 7.665871 4.490000 42861.290769 1.471210e+06<br>max 107701.748378 9.519088 10.759588 6.500000 69621.713378 2.469066e+06</pre><p>This function is useful for getting a quick overview of the central tendency, dispersion, and distribution of the data.</p><p><strong>Step 6: Check for the missing values</strong></p><p>The code below displays the total number of missing values for all the columns in the DataFrame.</p><pre>df.isnull().sum()</pre><p>output:</p><pre>Avg. Area Income                0<br>Avg. Area House Age             0<br>Avg. Area Number of Rooms       0<br>Avg. Area Number of Bedrooms    0<br>Area Population                 0<br>Price                           0<br>Address                         0<br>dtype: int64</pre><p>The output shows that there are no missing values in any of the columns of the DataFrame.</p><p><strong>Step 7: Drop the Address column from the DataFrame</strong>.</p><p>the code below drops the ‘Address’ column from the DataFrame</p><pre>df=df.drop([&#39;Address&#39;],axis=&#39;columns&#39;)</pre><p>we can now display our data columns to see if the address column is dropped</p><pre>df.columns</pre><p>output:</p><pre>Index([&#39;Avg. Area Income&#39;, &#39;Avg. Area House Age&#39;, &#39;Avg. Area Number of Rooms&#39;,<br>       &#39;Avg. Area Number of Bedrooms&#39;, &#39;Area Population&#39;, &#39;Price&#39;],<br>      dtype=&#39;object&#39;)</pre><p><strong>Step 8: Create a new column in the DataFrame</strong></p><p>The line of code below creates a new column in the DataFrame called ‘Area Income Category’ based on the values in the ‘Avg. Area Income’ column, using the “qcut()” function from the Pandas library.</p><pre><br>df[&#39;Area Income Category&#39;] = pd.qcut(df[&#39;Avg. Area Income&#39;],<br>                                     3,<br>                                    labels=[&#39;Low&#39;,&#39;Medium&#39;,&#39;High&#39;]</pre><p>output:</p><pre>Avg. Area Income    Avg. Area House Age    Avg. Area Number of Rooms      Avg. Area Number of Bedrooms     Area Population      Price Area Income Category<br> 79545.458574           5.682861                7.009188                        4.09                              23086.800503  1.059034e+06        High<br> 79248.642455           6.002900                6.730821                        3.09                              40173.072174  1.505891e+06        High <br> 61287.067179           5.865890                8.512727                        5.13                              36882.159400  1.058988e+06        Low  <br> 63345.240046           7.188236                5.586729                        3.26                              34310.242831  1.260617e+06        Low<br>4 59982.197226          5.040555                7.839388                        4.23                              26354.109472  6.309435e+05        Low</pre><p>The new column in the DataFrame called ‘Area Income Category’ contains the labels &quot;Low,&quot; &quot;Medium,&quot; or &quot;High&quot; based on the quantile values of the ‘Avg. Area Income’ column.</p><p><strong>Step 9: Count the occurrences of Unique Values in the column</strong></p><p>The code counts the occurrences of each unique value in the “Area Income Category” column of the DataFrame using the “value_counts()” function in Pandas.</p><pre>df[&#39;Area Income Category&#39;].value_counts()</pre><p>output:</p><pre>Low       1667<br>High      1667<br>Medium    1666<br>Name: Area Income Category, dtype: int64</pre><p>This shows that “1667”, “1667”, and “1666” are the counts of occurrences for the values “Low”, “High”, and “Medium”, in the ‘Area Income Category’ column respectively.</p><p>we can also decide to look at any other column of the DataFrame</p><pre>df[&#39;Avg. Area Number of Bedrooms&#39;].value_counts()</pre><p>output:</p><pre>4.38    46<br>3.17    41<br>3.40    40<br>3.16    39<br>3.22    38<br>        ..<br>6.31     7<br>6.01     6<br>6.02     6<br>5.42     6<br>6.42     5<br>Name: Avg. Area Number of Bedrooms, Length: 255, dtype: int64</pre><p><strong>Step 10: creates a cross-tabulation</strong></p><p>The code creates a cross-tabulation (also known as a contingency table) of two variables, ‘Area Income Category’ and ‘Avg. Area Number of Bedrooms’, from the DataFrame and the ‘.T.head()’ at the end is used to transpose the resulting cross-tabulation and display the first few rows of the transposed table.</p><pre>pd.crosstab(df[&#39;Area Income Category&#39;],<br>            df[&#39;Avg. Area Number of Bedrooms&#39;]).T.head()</pre><p>output:</p><pre>Area Income Category           Low      Medium        High<br>Avg. Area Number of Bedrooms   <br>          2.00                  5         11           4<br>          2.01                  4          7           5<br>          2.02                  7          8           5<br>          2.03                  6          5           3<br>          2.04                  11        11           3</pre><p>The cross-tabulation is a tabular representation of the frequency of occurrences of different values in two or more variables. It can provide insights into the relationships between variables, allowing you to identify patterns, trends, and associations in the data.</p><p><strong>Step 11: Create a pairs plot of the DataFrame</strong></p><p>The code below creates the scatterplot matrix, or pairs plot, which is a matrix of scatterplots that shows scatterplots of all possible pairs of variables in the DataFrame using the Seaborn library.</p><pre>sns.pairplot(df)</pre><p>output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6nFBaVlmSDzX-IO1pI43OQ.png" /></figure><p><strong>Step 12: Create a heatmap of the correlation matrix</strong></p><p>The code below creates a heatmap of the correlation matrix for the DataFrame ‘df’ using the Seaborn library.</p><p>The heatmap is a graphical representation of the correlation matrix, where the correlation values between pairs of variables are represented by colors. High positive correlations are typically represented by warmer colors (e.g., red), while high negative correlations are represented by cooler colors (e.g., blue). No or low correlations are typically represented by neutral colors (e.g., white or gray).</p><pre>sns.heatmap(df.corr(),annot=True)</pre><p>output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/497/1*7PusNcLztpbIy0IaWVzVnw.png" /></figure><p>“df.corr()” is a method of the DataFrame ‘df’ that calculates the correlation coefficients between all pairs of numeric variables in the DataFrame. It returns a new DataFrame that contains the correlation values.</p><p><strong>Step 13: Create a countplot</strong></p><p>The code provided creates a countplot of the ‘Area Income Category’ column in the DataFrame ‘df’ using the Matplotlib and Seaborn libraries.</p><p>The countplot is a type of bar plot that displays the frequency of occurrences of each category in a categorical variable.</p><pre>fig =plt.figure(figsize=(10,5))<br>sns.countplot(df[&quot;Area Income Category&quot;],palette=&#39;Set1&#39;)<br>plt.title(&#39;Area Income Frequency&#39;)</pre><p>output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/618/1*nv0Nq6uiy42xbc2D2HCtyQ.png" /></figure><p><strong>Step 11: Create a distribution plot</strong></p><p>The code below creates a distribution plot (also known as a histogram) of the ‘Avg. Area Income’ column in the DataFrame using the Seaborn library.</p><p>The distribution plot (histogram) is a graphical representation of the distribution of a single variable.</p><pre>sns.distplot(df[&#39;Avg. Area Income&#39;])</pre><p>output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/403/1*5rDI6OW0TygqBbiaqldgyw.png" /></figure><h3><strong>conclusion</strong></h3><p>In conclusion, exploratory data analysis (EDA) and data preprocessing are critical steps in the data analysis process that involve understanding, cleaning, and transforming raw data to gain insights and make informed decisions. In this blog, we covered various techniques and methods using Python and popular libraries such as NumPy, Pandas, Matplotlib, and Seaborn for performing EDA and data preprocessing on a dataset.</p><p>We started by loading the dataset using Pandas and examining its basic information using the “info()” and “describe()” methods. We then performed data cleaning and preprocessing tasks such as handling missing values, dropping unnecessary columns, and converting data types to appropriate formats. We also explored techniques for data visualization using Matplotlib and Seaborn, including scatter plots, histograms, pair plots, heatmaps, and count plots, to gain insights into the distribution, relationships, and patterns in the data.</p><p>Additionally, we utilized advanced data manipulation techniques such as creating new columns, applying mathematical operations, and using cross-tabulations to further analyze the data.</p><p>By leveraging these EDA and data preprocessing techniques, we were able to gain a deeper understanding of the dataset and uncover meaningful insights that can guide decision-making in various domains, such as business, finance, marketing, healthcare, and more.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f8fbc50c5822" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding AI and Machine Learning: Explaining Complex Technologies in Simple Terms]]></title>
            <link>https://alaminmusamagaga.medium.com/understanding-ai-and-machine-learning-explaining-complex-technologies-in-simple-terms-f10a78690249?source=rss-d9bc7626f8c------2</link>
            <guid isPermaLink="false">https://medium.com/p/f10a78690249</guid>
            <dc:creator><![CDATA[Alamin Musa Magaga]]></dc:creator>
            <pubDate>Sat, 01 Apr 2023 14:49:26 GMT</pubDate>
            <atom:updated>2023-04-01T15:06:18.252Z</atom:updated>
            <content:encoded><![CDATA[<p>Demystifying the Complexity of AI and Machine Learning Concepts in Simple Explanation</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kgmImYxdhX01Yfk5qzkX-A.jpeg" /></figure><p>The recent advancements in AI technology have surprised many, particularly those who have little awareness of the impact of AI and its applications in our day-to-day activities. These applications include autocorrection and word suggestions in our chats with friends, content suggestions and recommendations on our social media platforms, and even facial recognition and ID on our mobile phones.</p><p>With all this advancement, it still takes a great deal of effort to explain this technology to someone with little or no technical expertise in AI.</p><p>Artificial Intelligence (AI) and Machine Learning (ML) are rapidly transforming various industries and our daily lives. These technologies enable machines to learn, adapt, and perform complex tasks that were once only possible by humans. In this blog post, we will explore AI and ML in simple terms so that even individuals with little or no technical expertise can understand. but before we dive into the main content let us look into the broader explanation of AI and Machine Learning</p><blockquote>Artificial intelligence (AI) is defined as “the ability of a computer or a robot controlled by a computer to do tasks that are usually done by humans because they require human intelligence and discernment” — (Britannica.com)</blockquote><p>These machines are designed to learn, reason, and self-correct without human intervention. One of the most common examples of AI is language translation, text-to-speech, facial recognition, recommender systems, and virtual assistants such as Siri and Alexa, which can perform tasks such as playing music, setting reminders, and answering questions.</p><p>On the other hand</p><blockquote>ML is a subset of AI that enables machines to learn from data and improve over time without being explicitly programmed. — (expert.ai)</blockquote><p>In other words, instead of being given a set of rules to follow, ML algorithms can learn from experience and improve their performance through trial and error. This allows them to adapt to new and unseen situations and make accurate predictions or decisions based on the data they have learned.</p><p>To help understand the concept of ML, let’s consider the example of</p><ul><li>A child learning to recognize a cat</li><li>A house-selling expert</li></ul><p><strong>A child learning to recognize a cat</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*mOAdFi5Wcdu53cbORR6LmA.jpeg" /></figure><p>imagine a child learning to identify and recognize a cat, at first the child may be shown different animals, pictures, or videos of the animals and still may not be able to distinguish between them, however with the guidance of parents and teachers the child will be able to understand some certain features that are common to cat and able to differentiate between cats and other animals</p><p>Similarly, in machine learning, a model requires data to understand and draw inferences from it. As more data is fed to the model, it becomes more accurate and can differentiate between different images.</p><p><strong>A house-selling expert</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*HCBLj_9q9ENRnOlyXtps3A.jpeg" /></figure><p>Another example of AI and Machine Learning(ML) is house price prediction. A house selling expert can accurately predict the price of a house based on various factors such as the number of rooms, square feet, location, and other features of the house. the house-selling experts were able to accurately predict the price of the house because of the experience they have selling many houses.</p><p>In machine learning terms, to build a regression task of house price prediction, we need a dataset that includes thousands of house information with input features such as the number of rooms, baths, toilets, kitchen, area, and the output (price). with more data, the model learns and becomes better</p><h4><strong>Conclusion</strong></h4><p>In conclusion, AI and ML are transforming the way we live, work, and interact with technology. These technologies have the potential to solve complex problems, increase efficiency, and improve our quality of life. It is essential to have a basic understanding of these technologies to fully appreciate their potential and to make informed decisions about their applications.</p><p><a href="https://alaminmusamagaga.medium.com/django-web-scraping-app-download-data-in-csv-pdf-and-json-formats-5374592c2b69">Django Web Scraping App: Download Data in CSV, PDF, and JSON Formats</a></p><p><a href="https://medium.com/@alaminmusamagaga/building-an-ai-powered-image-generation-application-with-django-and-openai-api-3e117a35face">https://medium.com/@alaminmusamagaga/building-an-ai-powered-image-generation-application-with-django-and-openai-api-3e117a35face</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f10a78690249" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building an AI-powered Image Generation Application with Django and OpenAI API]]></title>
            <link>https://medium.com/analytics-vidhya/building-an-ai-powered-image-generation-application-with-django-and-openai-api-3e117a35face?source=rss-d9bc7626f8c------2</link>
            <guid isPermaLink="false">https://medium.com/p/3e117a35face</guid>
            <dc:creator><![CDATA[Alamin Musa Magaga]]></dc:creator>
            <pubDate>Thu, 16 Mar 2023 20:39:11 GMT</pubDate>
            <atom:updated>2023-03-16T21:02:31.715Z</atom:updated>
            <content:encoded><![CDATA[<p>Integrate OpenAI API with Django to build an Image Generation App</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/716/1*aCL9Xj-D_Nu9l1B61Ej7ag.gif" /><figcaption>by alamin magaga</figcaption></figure><p>In this blog, we will explore how to build an image generation app that utilizes the powerful OpenAI GPT-3 model to generate images based on the user’s input text. We will be using Django, a popular Python web framework, to build the application.</p><p>Table of Contents:</p><ul><li>Introduction</li><li>Prerequisites</li><li>Setting Up the Django Project and OpenAI API</li><li>Creating the Image Generation App</li><li>Building the Views and Templates</li><li>Testing the Application</li><li>Conclusion</li></ul><p><strong>Introduction</strong></p><p>Image generation has become a popular application of machine learning, and OpenAI’s GPT-3 model has made it easier than ever to generate high-quality images based on textual input. In this tutorial, we will be building an image generation application that uses GPT-3 and Django to allow users to generate images based on their text input.</p><p><strong>Prerequisites:</strong></p><ul><li>Python 3. x</li><li>Django framework</li><li>OpenAI API key (for using GPT-3)</li></ul><p><strong>Setting Up the Django Project</strong></p><p>To start, we will create a new Django project using the following command:</p><pre>django-admin startproject image_generator</pre><p><strong>Creating the Image Generation App</strong></p><p>Next, we will create a Django app called “api” to hold all the image generation functionality of our project.</p><pre>python manage.py startproject api</pre><p>We will also add this app to our project’s INSTALLED_APPS list in the settings.py file.</p><pre># Application definition<br><br>INSTALLED_APPS = [<br>    &quot;django.contrib.admin&quot;,<br>    &quot;django.contrib.auth&quot;,<br>    &quot;django.contrib.contenttypes&quot;,<br>    &quot;django.contrib.sessions&quot;,<br>    &quot;django.contrib.messages&quot;,<br>    &quot;django.contrib.staticfiles&quot;,<br>    &quot;api&quot;,<br>]</pre><p>To obtain your OpenAI API key, please follow these steps:</p><ol><li>Go to the OpenAI website.</li><li>Click on the “View” button to access your API keys.</li><li>Create a new API key or access an existing one.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hm1mkf9jx_XE78I8frGv9Q.png" /></figure><p>To use OpenAI’s GPT-3 model, we need to also install the OpenAI Python library using the following command:</p><pre>pip install openai</pre><p>We will create a model for our image generation app to store generated images along with their associated text input. This will allow users to view their previously generated images.</p><p><strong>Building the Views and Templates</strong></p><p>In this process, we will develop views and templates for our image generation application to enable users to enter their text and generate images accordingly.</p><blockquote>views.py</blockquote><pre># views.py<br>import openai<br>from PIL import Image<br>import requests<br>from django.http import HttpResponse<br>from django.shortcuts import render<br><br><br>openai.api_key = &#39;Your API Key &#39;<br><br>import openai<br><br>def generate_image(request):<br>    text_input = request.GET.get(&#39;text_input&#39;)<br>    image_url = None<br>    if text_input is not None:<br>        if &#39;\n&#39; in text_input:<br>            prompt = text_input<br>        else:<br>            features_input = request.GET.get(&#39;features_input&#39;)<br>            if features_input is not None:<br>                prompt = f&quot;{text_input}\n{features_input}&quot;<br>            else:<br>                prompt = text_input<br>        response = openai.Image.create(<br>            prompt=prompt,<br>            n=1,<br>            size=&#39;512x512&#39;,<br>        )<br>        image_url = response[&#39;data&#39;][0][&#39;url&#39;]<br>    context = {&#39;image_url&#39;: image_url}<br>    return render(request, &#39;generate.html&#39;, context)</pre><p>The code in the views.pyis responsible for handling the logic behind the image generation process. It first imports the necessary modules and libraries, such as the OpenAI API key and the Django HttpResponse library.</p><p>It then defines a function named generate_image that retrieves the user input from the HTML form, sends it to the OpenAI API to generate an image, and returns a HttpResponse containing the generated image.</p><blockquote>urls.py</blockquote><pre><br>from django.urls import path<br>from . import views<br><br>urlpatterns = [<br>    path(&#39;&#39;, views.generate_image, name=&#39;generate_image&#39;),<br>]<br></pre><p>This urls.pyabove defines the URL patterns for the app in a Django project. It imports the path function from the ‘django.urls’ module and the ‘views’ module from the current package.</p><p>It then creates a list of URL patterns using the ‘urlpatterns’ variable. The single URL pattern defined in this code uses an empty string as the URL path and maps it to the ‘generate_image’ function in the ‘views’ module.</p><blockquote>templates/generate.html</blockquote><pre>&lt;!DOCTYPE html&gt;<br>&lt;html&gt;<br>&lt;head&gt;<br>    &lt;title&gt;Generate Image&lt;/title&gt;<br>    &lt;style&gt;<br>        * {<br>            box-sizing: border-box;<br>        }<br>        body {<br>            font-family: Arial, sans-serif;<br>            background-color: #eee;<br>            margin: 0;<br>            display: flex;<br>            justify-content: center;<br>            align-items: center;<br>            height: 100vh;<br>        }<br>        .chat {<br>            max-width: 600px;<br>            margin: 0 auto;<br>            background-color: #fff;<br>            border: 1px solid #ccc;<br>            box-shadow: 0 0 10px rgba(0, 0, 0, 0.1);<br>            height: 100%;<br>            display: flex;<br>            flex-direction: column;<br>        }<br>        .chat-header {<br>            background-color: #075E54;<br>            color: #fff;<br>            padding: 10px;<br>            display: flex;<br>            align-items: center;<br>            justify-content: space-between;<br>        }<br>        .chat-header h3 {<br>            margin: 0;<br>            font-size: 18px;<br>        }<br>        .chat-messages {<br>            flex: 1;<br>            overflow-y: scroll;<br>            padding: 10px;<br>        }<br>        .chat-message {<br>            display: flex;<br>            margin-bottom: 20px;<br>            max-width: 70%;<br>        }<br>        .chat-message img {<br>            max-width: 100%;<br>        }<br>        .chat-message-content {<br>            background-color: #DCF8C6;<br>            color: #000;<br>            border-radius: 10px;<br>            padding: 10px;<br>            margin-left: 10px;<br>            font-size: 16px;<br>            word-wrap: break-word;<br>            line-height: 1.5;<br>            box-shadow: 0 0 10px rgba(0, 0, 0, 0.1);<br>            position: relative;<br>        }<br>        .chat-message-content::before {<br>            content: &quot;&quot;;<br>            display: block;<br>            position: absolute;<br>            top: 50%;<br>            left: -10px;<br>            width: 0;<br>            height: 0;<br>            border-top: 10px solid transparent;<br>            border-bottom: 10px solid transparent;<br>            border-right: 10px solid #DCF8C6;<br>            transform: translateY(-50%);<br>        }<br>        .chat-input {<br>            display: flex;<br>            align-items: center;<br>            padding: 10px;<br>            background-color: #F4F4F4;<br>            border-top: 1px solid #ccc;<br>            position: fixed;<br>            bottom: 0;<br>            left: 0;<br>            width: 100%;<br>        }<br>       <br>        .chat-input button[type=&quot;submit&quot;] {<br>            padding: 10px 20px;<br>            background-color: #075E54;<br>            color: #fff;<br>            border: none;<br>            font-size: 16px;<br>            cursor: pointer;<br>            transition: background-color 0.3s ease-in-out;<br>        }<br>        .chat-input button[type=&quot;submit&quot;]:hover {<br>            background-color: #044C39;<br><br>        &lt;/style<br>    <br>&lt;/head&gt;<br>&lt;body&gt;<br>    <br>    &lt;div class=&quot;chatbot&quot;&gt;<br>        &lt;br&gt;&lt;br&gt;<br>        &lt;div class=&quot;messages&quot;&gt;<br>            &lt;div class=&quot;message&quot;&gt;<br>                &lt;p&gt;Hello, how can I help you today?&lt;/p&gt;<br>            &lt;/div&gt;<br>        &lt;/div&gt;<br>        &lt;form method=&quot;get&quot; action=&quot;{% url &#39;generate_image&#39; %}&quot;&gt;<br>            &lt;input type=&quot;text&quot; name=&quot;text_input&quot; placeholder=&quot;generate your image here...&quot; width=&quot;20px&quot;&gt;<br>            &lt;button &gt;Generate&lt;/button&gt;<br>        &lt;/form&gt;<br>        {% if image_url %}<br>        &lt;div class=&quot;messages&quot;&gt;<br>            &lt;div class=&quot;message&quot;&gt;<br>                &lt;p&gt;&lt;img src=&quot;{{ image_url }}&quot;&gt;&lt;/p&gt;<br>            &lt;/div&gt;<br>        &lt;/div&gt;<br>        {% endif %}<br>    &lt;/div&gt;<br>&lt;/body&gt;<br>&lt;/html&gt;</pre><p>The generate.htmlcode is the HTML file that creates the app interface with an input field and a “Generate” button. Users can enter text in the input field to generate an image, and the image will be displayed in the interface.</p><p>The interface is styled using CSS, with a background color, font style, and layout for the different elements in the template.</p><p><strong>Testing the Application</strong></p><p>to test our application, we will run the Django server and generate a few images based on a user input text.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SS4BtvXSCyglbdUcRnwQxQ.png" /></figure><p><strong>Conclusion</strong></p><p>In this tutorial, we have explored how to build an image generation application that uses OpenAI’s GPT-3 model to generate images based on a user text input. We have also covered the entire process, from setting up the Django project to creating the image generation app and building the views and templates. With this application, users can easily generate high-quality images based on their text input, opening up new possibilities in the world of image generation.</p><p>you can clone and download the complete project <a href="https://github.com/alaminmagaga/Image-Generator-with-OpenAI-API">here</a></p><ul><li><a href="https://alaminmusamagaga.medium.com/text-summarization-app-with-flask-and-spacy-3b9cdcf0afb5">Text Summarization App with Flask and Spacy</a></li><li><a href="https://alaminmusamagaga.medium.com/django-web-scraping-app-download-data-in-csv-pdf-and-json-formats-5374592c2b69">Django Web Scraping App: Download Data in CSV, PDF, and JSON Formats</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3e117a35face" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/building-an-ai-powered-image-generation-application-with-django-and-openai-api-3e117a35face">Building an AI-powered Image Generation Application with Django and OpenAI API</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Text Summarization App with Flask and Spacy]]></title>
            <link>https://medium.com/analytics-vidhya/text-summarization-app-with-flask-and-spacy-3b9cdcf0afb5?source=rss-d9bc7626f8c------2</link>
            <guid isPermaLink="false">https://medium.com/p/3b9cdcf0afb5</guid>
            <dc:creator><![CDATA[Alamin Musa Magaga]]></dc:creator>
            <pubDate>Thu, 16 Mar 2023 11:21:04 GMT</pubDate>
            <atom:updated>2023-03-26T09:13:33.181Z</atom:updated>
            <content:encoded><![CDATA[<h4>A Step-by-Step Guide to Creating Text Summarization App with Flask and Spacy</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/642/1*aadX29yMuMsbsCA2L6PAug.jpeg" /></figure><p>In today’s world, there is a huge abundance of information available to us. As a result, it can be challenging to go through all the information and extract the relevant information. That’s where text summarization comes in.</p><blockquote>Text summarization is the process of reducing and shortening large text documents and contents into shorter and concise versions while retaining the essential information.</blockquote><blockquote><em>In this tutorial, we will be creating a text summarization app using Flask and Spacy.</em></blockquote><p><strong>Understanding Text Summarization</strong></p><p>There are two types of text summarization: extractive and abstractive summarization. Extractive summarization involves the identifying of important sentences from the text and presenting them in a summarized format. On the other hand, abstractive summarization involves the creating new sentences to convey the essential information from the text. In this tutorial, we will focus on extractive summarization.</p><p><strong>SpaCy</strong></p><p>SpaCy is an advanced Natural Language Processing (NLP) library for Python, which is free and open-source. It is specifically designed for production use and enable the development of applications that can efficiently process and comprehend large amounts of text. SpaCy can be utilized for building natural language understanding systems or informations extraction, and it can also pre-process text for deep learning.</p><p><strong>Flask</strong></p><p>Python’s Flask web framework makes it simple and quick for developers to create web apps. It’s an easy-to-use framework that can be customized and is excellent for creating small to medium-sized applications. Numerous add-ons are available to further develop your app, and Flask comes with a ton of built-in tools like routing and troubleshooting. Flask is extremely adaptable for web development tasks because it integrates well with other Python libraries and frameworks.</p><p><strong>Requirement</strong></p><p>To create a text summarization app, we will be using the following tools and libraries:</p><ul><li>Python 3.7 or above</li><li>Flask</li><li>Spacy</li></ul><p>To install the libraries, Run the following commands in your terminal:</p><pre>pip install flask<br>pip install spacy<br>python -m spacy download en_core_web_sm</pre><p>After installation, further packages may be downloaded by utilizing the command below to access the Natural Language Toolkit resources.</p><pre>nltk.download()</pre><p>below is the project folder tree:<br>├── app.py<br>├── static/<br>│ ├── style.css<br>│ └── w3.css<br>└── templates/<br> ├── index.html<br> └── result.html</p><p><strong>Creating the Flask App</strong></p><p>Now, let’s create a new Flask app. Create a new file named app.py and add the following code:</p><blockquote>app.py</blockquote><pre><br>from flask import Flask, render_template, url_for, request<br>import time<br>import spacy<br>from spacy.lang.en.stop_words import STOP_WORDS<br>from string import punctuation<br>from heapq import nlargest<br>from bs4 import BeautifulSoup<br>from urllib.request import urlopen<br><br>nlp = spacy.load(&quot;en_core_web_sm&quot;)<br>app = Flask(__name__)<br><br><br>def txt_summarizer(raw_docx):<br>    stopwords = list(STOP_WORDS)<br>    raw_text = raw_docx<br>    docx = nlp(raw_text)<br><br>   <br>    word_frequency = {}<br>    for word in docx:<br>        if word.text not in stopwords:<br>            if word.text not in word_frequency.keys():<br>                word_frequency[word.text] = 1<br>            else:<br>                word_frequency[word.text] += 1<br>    maximum_frequncy = max(word_frequencies.values())<br>    for word in word_frequency.keys():<br>        word_frequency[word] = (word_frequency[word]/maximum_frequncy)<br>    <br>    <br>    sentence_list = [sentence for sentence in docx.sents]<br><br>    <br>    sentence_scores = {}<br>    for sent in sentence_list:<br>        for word in sent:<br>            if word.text.lower() in word_frequencies.keys():<br>                if len(sent.text.split(&#39; &#39;)) &lt; 30:<br>                    if sent not in sentence_scores.keys():<br>                        sentence_scores[sent] = word_frequencies[word.text.lower()]<br>                    else:<br>                        sentence_scores[sent] += word_frequencies[word.text.lower()]<br><br>    summarized_sentences = nlargest(7, sentence_scores, key=sentence_scores.get)<br>    final_sentences = [w.text for w in summarized_sentences]<br>    summary = &#39; &#39;.join(final_sentences)<br><br>    return summary<br><br><br>def readingT(mytext):<br>    total_words = len([token.text for token in nlp(mytext)])<br>    estimatedTime = total_words/200.0<br>    return estimatedTime<br><br>@app.route(&#39;/&#39;)<br>def index():<br>    return render_template(&#39;index.html&#39;)</pre><p>we have created a function that takes raw text as input, preprocesses it, and returns a summarized version of the text.</p><p>The txt_summarizerfunction takes a raw text as input and first removes any stop words, which are common words that do not carry much meaning such as ‘an’, ‘of’, ‘the’, and others. It then calculates the word frequency for each word in the text and normalizes these frequencies. Next, it splits the text into individual sentences and assigns a score to each sentence based on the frequency of its words. Finally, it selects the top N sentences with the highest scores and concatenates them to form a summarized version of the text.</p><p>The readingTfunction takes the tokenized text input using Spacy’s natural language processing library. The total number of words in the text is calculated by counting the tokens. The estimated reading time is calculated by dividing the total number of words in the text by 200 which is the typical reading speed of 200 words per minute is assumed.<br>The function then returns the estimated reading time in minutes.</p><p>the index() function simply renders an HTML template called index.html using the Flask render_template() function.</p><p><strong>Static folder</strong></p><p>The static folder includes the project CSS, JavaScript, and image files. The url_for() method is used to include and link the files with the HTML templates.<br>For example, you could use the following code to include the CSS and W3.CSS file in an HTML templates:</p><pre>&lt;link rel=&quot;stylesheet&quot; type=&quot;text/css&quot; href=&quot;{{ url_for(&#39;static&#39;, filename=&#39;style.css&#39;) }}&quot;&gt;<br>&lt;link rel=&quot;stylesheet&quot; href=&quot;{{ url_for(&#39;static&#39;, filename=&#39;w3.css&#39;) }}&quot;&gt;</pre><p><strong>Creating the templates</strong></p><p>the index.html template will be displayed as the homepage of the web application, which will allow users to input the text they want to summarize and result.html templates will display the summarized text along with the estimated reading time.</p><blockquote>templates/index.html</blockquote><pre>&lt;!-- templates/index.html --&gt;<br>&lt;!DOCTYPE html&gt;<br>&lt;html&gt;<br>&lt;head&gt;<br> &lt;title&gt;Summaryzer&lt;/title&gt;<br>&lt;meta name=&quot;viewport&quot; content=&quot;width=device-width, initial-scale=1&quot;&gt;<br>&lt;link rel=&quot;stylesheet&quot; href=&quot;https://cdnjs.cloudflare.com/ajax/libs/font-awesome/4.7.0/css/font-awesome.min.css&quot;&gt;<br>&lt;link rel=&quot;stylesheet&quot;  href=&quot;{{ url_for(&#39;static&#39;, filename=&#39;w3.css&#39;) }}&quot;&gt;<br><br>&lt;/style&gt;<br>&lt;/head&gt;<br>&lt;body&gt;<br>&lt;div class=&quot;w3-bar w3-teal w3-padding&quot;&gt;<br>  &lt;a href=&quot;{{url_for(&#39;index&#39;)}}&quot; class=&quot;w3-bar-item w3-button w3-padding &quot;&gt;Home&lt;/a&gt;<br>  <br>  &lt;a href=&quot;#C1&quot; class=&quot;w3-bar-item w3-button w3-right w3-padding&quot; &gt;About&lt;/a&gt;<br>  &lt;a href=&quot;#C1&quot; class=&quot;w3-bar-item w3-button w3-right w3-padding&quot; &gt;Contact&lt;/a&gt;<br>&lt;/div&gt;<br>&lt;br&gt;<br><br><br>&lt;!-- Start of Main Section --&gt;<br>&lt;div class=&quot;w3-container w3-center&quot;&gt;&lt;h3&gt;Enter your text Here&lt;/h3&gt;<br> &lt;form method=&quot;POST&quot; action=&quot;/process&quot;&gt;<br> &lt;textarea  name=&quot;input_text&quot; cols=&quot;5&quot; rows=&quot;10&quot;  required=&quot;true&quot; placeholder=&quot;Enter Text Here&quot; style=&quot;width:60%&quot;&gt;&lt;/textarea&gt;<br>            <br>         <br>         <br>                &lt;br&gt;<br>         &lt;button class=&quot;w3-btn w3-blue-grey&quot; type=&quot;reset&quot; value=&quot;reset&quot;&gt;Clear&lt;/button&gt;<br>          &lt;button class=&quot;w3-btn w3-teal&quot; type=&quot;submit&quot; value=&quot;reset&quot;&gt;Summarize&lt;/button&gt;<br>          <br>          <br>          &lt;div class=&quot;input-field&quot;&gt;<br>                    <br>               <br>        &lt;/div&gt;<br>&lt;/form&gt;&lt;/div&gt;<br>&lt;br&gt;&lt;br&gt;<br>&lt;div class=&quot;w3-container w3-center&quot;&gt;&lt;h3&gt;Enter your document link Here&lt;/h3&gt;<br>&lt;form method=&quot;POST&quot; action=&quot;/process_url&quot;&gt;<br>         &lt;input  type=&quot;text&quot; name=&quot;input_url&quot; placeholder=&quot;Enter URL Here&quot; required=&quot;true&quot;  style=&quot;width:50%&quot;&gt;<br>&lt;br&gt;&lt;br&gt;<br>          &lt;button class=&quot;w3-btn w3-blue-grey&quot; type=&quot;reset&quot; value=&quot;reset&quot;&gt;Clear&lt;/button&gt;<br>          &lt;button class=&quot;w3-btn w3-teal&quot; type=&quot;submit&quot; value=&quot;reset&quot;&gt;Summarize&lt;/button&gt;<br>&lt;/form&gt;<br><br>&lt;/div&gt;<br>&lt;br&gt;&lt;br&gt;<br><br>  <br>  &lt;/div&gt;<br>&lt;/div&gt;<br>&lt;div class=&quot;w3-container w3-teal&quot;&gt;<br><br>    &lt;div class=&quot;w3-row&quot;&gt;<br><br>  &lt;div class=&quot;w3-half w3-container&quot;&gt;<br>          &lt;h4 &gt;About the App&lt;/h4&gt;<br>          &lt;p&gt;the app was build with Flask framework,Sumy packages(Lex rank,Isa and luhn) and w3.css(modern css framework) for the application<br>.&lt;/p&gt;<br>Made by &lt;a  href=&quot;https://twitter.com/Alamin_Magaga&quot;&gt;Alamin M Magaga&lt;/a&gt;&lt;br/&gt;<br>      Founder&lt;a  href=&quot;https://twitter.com/Magtech_Dihub&quot;&gt; @Magtech Digital Hub&lt;/a&gt;<br><br><br>   &lt;/div&gt;<br>      <br>   &lt;div class=&quot;w3-half w3-container w3-center&quot;&gt;<br>          &lt;h4 id=&quot;C1&quot;&gt;Connect With Me&lt;/h4&gt;<br>          &lt;ul&gt;<br>           &lt;a href=&quot;https://web.facebook.com/alaminmusa.magaga&quot; target=&quot;_blank&quot; class=&quot;white-text&quot;&gt;<br>            &lt;i class=&quot;fa fa-facebook fa-4x&quot;&gt;&lt;/i&gt;<br>            <br>          &lt;/a&gt;<br>          &lt;a href=&quot;https://www.youtube.com/channel/UCy3KM0QxpZUTgKjXRVID6-A&quot; target=&quot;_blank&quot; &gt;<br>            &lt;i class=&quot;fa fa-youtube-square fa-4x&quot;&gt;&lt;/i&gt;<br>          <br>          &lt;/a&gt;<br>          <br>          &lt;a href=&quot;alaminmusamagaga.medium.com&quot; target=&quot;_blank&quot; class=&quot;white-text&quot;&gt;<br>            &lt;i class=&quot;fa fa-medium fa-4x&quot;&gt;&lt;/i&gt;<br>            <br>          &lt;/a&gt;<br>          <br>          &lt;a href=&quot;https://www.linkedin.com/in/alamin-musa-magaga-8b388118b&quot; target=&quot;_blank&quot; class=&quot;white-text&quot;&gt;<br>            &lt;i class=&quot;fa fa-linkedin fa-4x&quot;&gt;&lt;/i&gt;<br>          &lt;/a&gt;<br>          <br>          &lt;/a&gt;<br>           &lt;a href=&quot;https://github.com/alaminmagaga&quot; target=&quot;_blank&quot; class=&quot;white-text&quot;&gt;<br>            &lt;i class=&quot;fa fa-github-square fa-4x&quot;&gt;&lt;/i&gt;<br>          &lt;/a&gt;<br>          <br>          &lt;a href=&quot;https://twitter/Alamin_Magaga&quot; target=&quot;_blank&quot; class=&quot;white-text&quot;&gt;<br>            &lt;i class=&quot;fa fa-twitter fa-4x&quot;&gt;&lt;/i&gt;<br>        <br>          &lt;/a&gt;<br>         <br>          &lt;/ul&gt;<br>    &lt;/div&gt;<br><br>    <br>      <br>      <br>      &lt;/div&gt;<br>&lt;br&gt;<br>    &lt;/div&gt;<br>&lt;/body&gt;<br>&lt;/html&gt;</pre><p>The index.html above enables the user to enter text for summarization. The page is divided into two sections: the first section prompt the user to enter a text input, while the second enables the user to enter a URL of text content.</p><blockquote>templates/result.html</blockquote><pre>&lt;!-- templates/result.html --&gt;<br><br>&lt;!DOCTYPE html&gt;<br>&lt;html &gt;<br>&lt;head&gt;<br>  &lt;meta charset=&quot;UTF-8&quot;&gt;<br>  &lt;title&gt;prediction&lt;/title&gt;<br>  &lt;link rel=&quot;stylesheet&quot; href=&quot;https://cdnjs.cloudflare.com/ajax/libs/font-awesome/4.7.0/css/font-awesome.min.css&quot;&gt;<br><br>  &lt;link rel=&quot;stylesheet&quot;  href=&quot;{{ url_for(&#39;static&#39;, filename=&#39;w3.css&#39;) }}&quot;&gt;<br> &lt;style type=&quot;text/css&quot;&gt;<br><br>  &lt;/style&gt;<br>&lt;/head&gt;<br><br>&lt;body&gt;<br><br>&lt;div class=&#39;w3-padding w3-teal w3-center&#39;&gt;&lt;h1&gt;Text Summarization Result&lt;/h1&gt;&lt;/div&gt;<br>  <br>&lt;/form&gt;<br>&lt;/div&gt;<br><br>  <br>&lt;/div&gt;<br><br>&lt;div class=&quot;w3-container w3-padding-24&quot;&gt;<br><br>  <br>  <br>&lt;div class=&quot;w3-container w3-card&quot;&gt;<br> &lt;div class=&quot;w3-row&quot;&gt;<br><br>  &lt;div class=&quot;w3-half w3-container &quot;&gt;<br>    &lt;h5&gt;Original Text&lt;/h5&gt;<br>             &lt;p&gt;The Reading Time: {{ final_reading_time }} minute &lt;/p&gt;<br>             &lt;p &gt;{{ctext}}&lt;/p&gt;<br>            <br>&lt;/div&gt;<br><br>  &lt;div class=&quot;w3-half w3-container w3-light-grey&quot;&gt;<br>    &lt;h5&gt;Summarized Text&lt;/h5&gt;<br>              <br>              &lt;p&gt;Reading Time: &lt;span class=&quot;w3-text-teal&quot;&gt;{{ summary_reading_time }} minute &lt;/span&gt;&lt;/p&gt;<br>        <br>            &lt;p&gt;{{ final_summary }}&lt;/p&gt;<br>            <br>          <br>  &lt;/div&gt;<br>&lt;/div&gt;<br><br> &lt;/div&gt;<br><br>    &lt;/div&gt;<br><br> &lt;/div&gt;<br> &lt;div class=&quot;w3-center&quot;&gt;&lt;a href=&quot;{{url_for(&#39;index&#39;)}}&quot; style=&quot;text-decoration:none&quot;&gt;&lt;button class=&quot;w3-btn w3-teal w3-round-large&quot; style=&quot;width:10%;text-decoration:none;&quot; type=&quot;submit&quot; value=&quot;reset&quot;&gt;Back&lt;/button&gt;&lt;/a&gt;&lt;/div&gt;<br>&lt;br&gt;<br><br><br>&lt;div class=&quot;w3-container w3-teal&quot;&gt;<br><br>    &lt;div class=&quot;w3-row&quot;&gt;<br><br>  &lt;div class=&quot;w3-half w3-container&quot;&gt;<br>          &lt;h4 &gt;About the App&lt;/h4&gt;<br>          &lt;p&gt;the app was build with Flask framework,Sumy packages(Lex rank,Isa and luhn) and w3.css(modern css framework) for the application<br>.&lt;/p&gt;<br>Made by &lt;a  href=&quot;https://twitter.com/Alamin_Magaga&quot;&gt;Alamin M Magaga&lt;/a&gt;&lt;br/&gt;<br>      Founder&lt;a  href=&quot;https://twitter.com/Magtech_Dihub&quot;&gt; @Magtech Digital Hub&lt;/a&gt;<br><br><br>   &lt;/div&gt;<br>      <br>   &lt;div class=&quot;w3-half w3-container w3-center&quot;&gt;<br>          &lt;h4 id=&quot;C1&quot;&gt;Connect With Me&lt;/h4&gt;<br>          &lt;ul&gt;<br>           &lt;a href=&quot;https://web.facebook.com/alaminmusa.magaga&quot; target=&quot;_blank&quot; class=&quot;white-text&quot;&gt;<br>            &lt;i class=&quot;fa fa-facebook fa-4x&quot;&gt;&lt;/i&gt;<br>            <br>          &lt;/a&gt;<br>          &lt;a href=&quot;https://www.youtube.com/channel/UCy3KM0QxpZUTgKjXRVID6-A&quot; target=&quot;_blank&quot; &gt;<br>            &lt;i class=&quot;fa fa-youtube-square fa-4x&quot;&gt;&lt;/i&gt;<br>          <br>          &lt;/a&gt;<br>          <br>          &lt;a href=&quot;alaminmusamagaga.medium.com&quot; target=&quot;_blank&quot; class=&quot;white-text&quot;&gt;<br>            &lt;i class=&quot;fa fa-medium fa-4x&quot;&gt;&lt;/i&gt;<br>            <br>          &lt;/a&gt;<br>          <br>          &lt;a href=&quot;https://www.linkedin.com/in/alamin-musa-magaga-8b388118b&quot; target=&quot;_blank&quot; class=&quot;white-text&quot;&gt;<br>            &lt;i class=&quot;fa fa-linkedin fa-4x&quot;&gt;&lt;/i&gt;<br>          &lt;/a&gt;<br>          <br>          &lt;/a&gt;<br>           &lt;a href=&quot;https://github.com/alaminmagaga&quot; target=&quot;_blank&quot; class=&quot;white-text&quot;&gt;<br>            &lt;i class=&quot;fa fa-github-square fa-4x&quot;&gt;&lt;/i&gt;<br>          &lt;/a&gt;<br>          <br>          &lt;a href=&quot;https://twitter/Alamin_Magaga&quot; target=&quot;_blank&quot; class=&quot;white-text&quot;&gt;<br>            &lt;i class=&quot;fa fa-twitter fa-4x&quot;&gt;&lt;/i&gt;<br>        <br>          &lt;/a&gt;<br>         <br>          &lt;/ul&gt;<br>    &lt;/div&gt;<br><br>    <br>      <br>      <br>      &lt;/div&gt;<br>      &lt;br&gt;<br>    &lt;/div&gt;<br><br><br>&lt;/body&gt;<br>&lt;/html&gt;</pre><p>The template for displaying the outcome of a text summarization task is result.html, which is shown above. The result.html file is divided into two parts, the first part contains the original text while the second of which contains the summarized text. <br>Each section includes a heading, a paragraph for displaying the text, and a reading time for the text. The file also includes a button that allows the user to go back to the previous page.</p><p><strong>Testing the App</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CSbUGQ83W0SPQxvlJp-RSA.png" /></figure><p>To test the application, run the command below</p><pre>python app.py</pre><p>Next, open <a href="http://127.0.0.1:5000/">http://127.0.0.1:5000/</a> in your web browser to view the app.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UmDS2n1FI9T4T4iwSHdDHg.png" /></figure><p><strong>Conclusion</strong></p><p>Text summarization is an important aspect of natural language processing due to its ability to process large caches of information for various fields, including journalism, legal, scientific and research.</p><p>In this project, we used Flask framework, and SpaCy to build a text summarization web application. The application can summarize texts of varying lengths and return a summary with a reading time estimation. We also included a reset button that allows users to input a new text for summarization.</p><p>The project structure includes the Flask application file (app.py), two HTML templates (index.html and result.html), and a static folder that contains the W3.CSS file and a custom style.css file. We used Flask to handle HTTP requests and created HTML templates to render the application’s front end.</p><p>In conclusion, the text summarization web application we built using Flask, Spacy can summarize large texts quickly. It can be further improved by adding features such as language detection and better summarization algorithms.</p><p>you can download the complete project <a href="https://github.com/alaminmagaga/Text-summarizer-with-Spacy">here</a></p><ul><li><a href="https://alaminmusamagaga.medium.com/django-web-scraping-app-download-data-in-csv-pdf-and-json-formats-5374592c2b69">Django Web Scraping App: Download Data in CSV, PDF, and JSON Formats</a></li><li><a href="https://medium.com/analytics-vidhya/simple-way-to-create-a-machine-learning-app-with-flask-69a532663fd5">Simple Way To Create A Machine Learning Web App With Flask</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3b9cdcf0afb5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/text-summarization-app-with-flask-and-spacy-3b9cdcf0afb5">Text Summarization App with Flask and Spacy</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Django Web Scraping App: Download Data in CSV, PDF, and JSON Formats]]></title>
            <link>https://python.plainenglish.io/django-web-scraping-app-download-data-in-csv-pdf-and-json-formats-5374592c2b69?source=rss-d9bc7626f8c------2</link>
            <guid isPermaLink="false">https://medium.com/p/5374592c2b69</guid>
            <category><![CDATA[naturallanguageprocessing]]></category>
            <category><![CDATA[web-scraping]]></category>
            <dc:creator><![CDATA[Alamin Musa Magaga]]></dc:creator>
            <pubDate>Sun, 12 Mar 2023 10:13:18 GMT</pubDate>
            <atom:updated>2023-06-26T00:55:13.548Z</atom:updated>
            <content:encoded><![CDATA[<p>Learn how to use the beautifulsoup library and Django framework to build a web application that can extract data from internet sources.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/874/1*ZpjAD9j7RgW2MEfsjEm6nA.jpeg" /><figcaption>upwork</figcaption></figure><p>Table of Contents:</p><ol><li>Introduction</li><li>Django Framework</li><li>BeautifulSoup</li><li>Prerequisites</li><li>Building the Web Scraper</li><li>Testing the Web Scrapper</li><li>Conclusion</li></ol><h4><strong>Introduction</strong></h4><p>Web scraping is a powerful technique for extracting data from websites. It involves collection of information from websites and converting it into a structured format such as PDF,CSV, JSON or XML. In this tutorial, we will build a web scraping tool that downloads data from a website in CSV, PDF, and JSON formats using the Django web framework.</p><p>Web scraping refers to the process of extracting data from websites. Python, with its vast array of libraries, has made web scraping relatively easy, and in this article, we will learn how to build a web scraper with Django and BeautifulSoup.</p><h4><strong>Django Framework</strong></h4><p>Django is a high-level,secured, free, and open-source Python web framework that makes it easier for developers to build websites with greater flexibility.</p><p>Django is one of the most popular Python frameworks for web development because of its Compatibility, Infrastructure, Security, great documentation, and active community support.</p><p>Django is being used by top global industries such as Instagram, Spotify, Pinterest, Mozilla, and Eventbrite.</p><h4><strong>Beautifulsoup Libary</strong></h4><p>BeautifulSoup is a Python library that simplifies the extraction of data from HTML and XML files for coders. it is a powerful web scraping and parsing tool, it can assist you in extracting valuable information from websites.<br>BeautifulSoup, for example, can assist you in extracting product information such as the product name, price, description, and other pertinent information from the HTML of an online shopping website.</p><p><strong>Prerequisites</strong><br>Before starting with the tutorial, make sure you have the following installed:</p><ul><li>Python 3.x</li><li>Django 3.x</li><li>requests</li><li>beautifulsoup4</li><li>reportlab</li><li>xhtml2pdf</li></ul><p>You can install the libraries by running the command below:</p><pre>pip install &lt;library&gt;</pre><p><strong>Building the Web Scraper</strong></p><p>In this section, we will explore on how to build a web scraper with Django. The web scraper will allows users to enter a URL and extract data from the website. The application will then allow users to scrap data online and download the extracted data in different formats such as PDF,CSV and JSON.<br>To build the web scraper, we will need to do the following:<br>Create a new Django project and application<br>Add views and URLs to the Django application<br>Create HTML templates for the home page and result page</p><p><strong>Create a new Django project and application</strong></p><p>the first step is to use the command below to create a Django project</p><pre>django-admin startproject webscrapper</pre><p>This command will create a Django project named webscrapper. Now navigate to the project directory using the below command:</p><pre>cd webscrapper</pre><p>Next,create a new app called scraper to handle the web scraping functionality. To create the new app, run the following command:</p><pre>python manage.py startapp scraper</pre><p>Add scraper to the INSTALLED_APPS list in the settings.py file</p><pre>INSTALLED_APPS = [<br>    &#39;django.contrib.admin&#39;,<br>    &#39;django.contrib.auth&#39;,<br>    &#39;django.contrib.contenttypes&#39;,<br>    &#39;django.contrib.sessions&#39;,<br>    &#39;django.contrib.messages&#39;,<br>    &#39;django.contrib.staticfiles&#39;,<br>    &#39;scraper&#39;,<br>]</pre><p>To tell Django where to look for our HTML files, add the directory holding our templates to the TEMPLATES “DIRS” list in the settings file.</p><pre>TEMPLATES = [<br>    {<br>        &quot;BACKEND&quot;: &quot;django.template.backends.django.DjangoTemplates&quot;,<br>        &quot;DIRS&quot;: [&quot;templates&quot;],<br>        &quot;APP_DIRS&quot;: True,<br>        &quot;OPTIONS&quot;: {<br>            &quot;context_processors&quot;: [<br>                &quot;django.template.context_processors.debug&quot;,<br>                &quot;django.template.context_processors.request&quot;,<br>                &quot;django.contrib.auth.context_processors.auth&quot;,<br>                &quot;django.contrib.messages.context_processors.messages&quot;,<br>            ],<br>        },<br>    },<br>]</pre><p>We must append the “scraper” app’s url path to the main urls.py file in order for Django to recognize the app’s urls.</p><pre><br>from django.contrib import admin<br>from django.urls import path,include<br><br>urlpatterns = [<br>    path(&quot;admin/&quot;, admin.site.urls),<br>    path(&#39;&#39;,include(&#39;scraper.urls&#39;)),<br>]</pre><p><strong>Add views and URLs to the Django application</strong></p><p>We will use BeautifulSoup and Requests libraries to build the web scraper.The scraper utilize this libraries to extract data from the website. We also use the io, reportlab, csv, and json libraries to download the extracted data in different formats.</p><p>First, Let&#39;s start by building a simple scraper that extracts the title of a website. In the scraper/views.py file, add the following code:</p><blockquote>scrapper/views.py</blockquote><pre>from django.shortcuts import render<br>import requests<br>from bs4 import BeautifulSoup<br>from io import BytesIO<br>from reportlab.pdfgen import canvas<br>from django.http import HttpResponse<br>import csv<br>import json<br>from django.urls import path<br>from django.template.loader import get_template<br>from xhtml2pdf import pisa<br><br>def home(request):<br>    return render(request, &#39;home.html&#39;)<br><br>def scrape(request):<br>    if request.method == &#39;POST&#39;:<br>        url = request.POST.get(&#39;url&#39;)<br>        r = requests.get(url)<br>        soup = BeautifulSoup(r.content, &#39;lxml&#39;)<br>        # get the required data from soup object<br>        title = soup.title.string<br>        headings = []<br>        for heading in soup.find_all([&#39;h1&#39;, &#39;h2&#39;, &#39;h3&#39;, &#39;h4&#39;, &#39;h5&#39;, &#39;h6&#39;]):<br>            headings.append(heading.text.strip())<br>        paragraphs = []<br>        for paragraph in soup.find_all(&#39;p&#39;):<br>            paragraphs.append(paragraph.text.strip())<br>        context = {<br>            &#39;title&#39;: title,<br>            &#39;headings&#39;: headings,<br>            &#39;paragraphs&#39;: paragraphs,<br>        }<br>        request.session[&#39;scraped_data&#39;] = context<br>        return render(request, &#39;result.html&#39;, context)<br>    else:<br>        return render(request, &#39;home.html&#39;)<br><br>def download_file(request):<br>    if &#39;scraped_data&#39; in request.session:<br>        context = request.session[&#39;scraped_data&#39;]<br>        response = HttpResponse(content_type=&#39;&#39;)<br>        if request.POST[&#39;download_type&#39;] == &#39;pdf&#39;:<br>            template_path = &#39;result.html&#39;<br>            template = get_template(template_path)<br>            html = template.render(context)<br>            response = HttpResponse(content_type=&#39;application/pdf&#39;)<br>            filename = f&quot;{context[&#39;title&#39;]}.pdf&quot;<br>            response[&#39;Content-Disposition&#39;] = f&#39;attachment; filename=&quot;{filename}&quot;&#39;<br>            buffer = BytesIO()<br>            pisa_status = pisa.CreatePDF(html, dest=response, encoding=&#39;utf-8&#39;)<br>            if pisa_status.err:<br>                return HttpResponse(&#39;PDF generation failed&#39;)<br>        elif request.POST[&#39;download_type&#39;] == &#39;csv&#39;:<br>            response = HttpResponse(content_type=&#39;text/csv&#39;)<br>            filename = f&quot;{context[&#39;title&#39;]}.csv&quot;<br>            response[&#39;Content-Disposition&#39;] = f&#39;attachment; filename=&quot;{filename}&quot;&#39;<br>            writer = csv.writer(response)<br>            writer.writerow([&#39;Title&#39;, &#39;Headings&#39;, &#39;Paragraphs&#39;])<br>            rows = zip([context[&#39;title&#39;]], context[&#39;headings&#39;], context[&#39;paragraphs&#39;])<br>            for row in rows:<br>                writer.writerow(row)<br>        elif request.POST[&#39;download_type&#39;] == &#39;json&#39;:<br>            response = HttpResponse(content_type=&#39;application/json&#39;)<br>            filename = f&quot;{context[&#39;title&#39;]}.json&quot;<br>            response[&#39;Content-Disposition&#39;] = f&#39;attachment; filename=&quot;{filename}&quot;&#39;<br>            json.dump(context, response, indent=4)<br>        return response<br>    else:<br>        return render(request, &#39;home.html&#39;)</pre><p>In the code above, we import necessary libraries such as Django, Requests, BeautifulSoup, io, reportlab, csv, json, django.urls, django.template.loader and xhtml2pdf.</p><p>we then defines the three functions ; home(), scrape() and download_file(). The home() function renders the home.html template. The scrape() function will take a URL as an input, extract data from the website, and store the data in a context dictionary. The context dictionary will be stored in the user’s session and passed to the result.html template.</p><p>The download_file() function downloads the scraped data as either a PDF, CSV or JSON file depending on the user’s selection. The appropriate response is returned depending on the type of download.</p><p>Next, in the scraper app we need to make a fresh urls.py file. The urls.py file in a Django project is used to specify URL patterns for the application. It includes a list of paths that are mapped to various views in the application.</p><blockquote>scraper/urls.py</blockquote><pre>#scraper/urls.py<br>from django.urls import path<br>from . import views<br><br>urlpatterns = [<br>            path(&#39;&#39;, views.home, name=&#39;home&#39;),<br>            path(&#39;scrape/&#39;, views.scrape, name=&#39;scrape&#39;),<br>            path(&#39;download/&#39;, views.download_file, name=&#39;download_file&#39;),<br>        ]</pre><p>In this blog post, the urls.py file defines two paths — one for the home page and one for the scrape view. The home page path maps to the home view which simply renders the <strong>home.html </strong>template.</p><p>The scrape path maps to the scrape view which handles the web scraping and renders the <strong>result.html</strong> template. Additionally, the urls.py file defines a path for downloading the scraped data in different formats (PDF, CSV, and JSON) using the <strong>download_file</strong> view.</p><p>When a user submits the form on the result page, the <strong>download_file</strong> view is called with the download_type parameter indicating the format the user wants to download the data in. The urls.py file also includes the necessary imports for the views and the Django path function.</p><p><strong>Create HTML templates for the home page and result page</strong></p><p>To create the user interface for our web scraper application, we need to create HTML templates that will define the structure and layout of our pages. We can do this by creating two HTML files, namely <strong>home.htm</strong>l and <strong>result.html</strong>, and saving them in the “templates” folder of our Django project.</p><blockquote>home.html</blockquote><pre>&lt;!--templates/home.html--&gt;<br><br>{% load static %}<br>&lt;!DOCTYPE html&gt;<br>&lt;html&gt;<br>&lt;head&gt;<br> &lt;title&gt;Web Scraper&lt;/title&gt;<br> &lt;style&gt;<br>  body {<br>   margin: 0;<br>   padding: 0;<br>   font-family: Arial, sans-serif;<br>   background-color: #f2f2f2;<br>  }<br>  h1 {<br>   margin: 20px;<br>   text-align: center;<br>   font-size: 40px;<br>   color: #333;<br>  }<br>  form {<br>   display: flex;<br>   flex-direction: column;<br>   align-items: center;<br>   margin-top: 50px;<br>  }<br>  input[type=&quot;text&quot;] {<br>   padding: 10px;<br>   border: none;<br>   border-radius: 5px;<br>   box-shadow: 0 2px 5px rgba(0,0,0,0.2);<br>   margin-bottom: 20px;<br>   width: 400px;<br>  }<br>  button[type=&quot;submit&quot;] {<br>   padding: 10px 20px;<br>   background-color: #4CAF50;<br>   color: white;<br>   border: none;<br>   border-radius: 5px;<br>   cursor: pointer;<br>   box-shadow: 0 2px 5px rgba(0,0,0,0.2);<br>  }<br>  button[type=&quot;submit&quot;]:hover {<br>   background-color: #3e8e41;<br>  }<br> &lt;/style&gt;<br> &lt;link rel=&quot;stylesheet&quot; href=&quot;{% static &#39;w3.css&#39; %}&quot;&gt;<br>&lt;/head&gt;<br>&lt;body&gt;&lt;h1&gt;Web Scraper&lt;/h1&gt;<br> &lt;div class=&quot;w3-center&quot; &gt;&lt;img class=&quot;w3-round w3-hover-grayscale&quot; src=&quot;{% static &#39;ai.png&#39; %}&quot; width=&quot;50%&quot;&gt;&lt;/div&gt;<br> <br> &lt;form method=&quot;post&quot; action=&quot;{% url &#39;scrape&#39; %}&quot;&gt;<br>  {% csrf_token %}<br>  &lt;input type=&quot;text&quot; name=&quot;url&quot; placeholder=&quot;Enter URL here...&quot;&gt;<br>  &lt;button type=&quot;submit&quot;&gt;Scrape&lt;/button&gt;<br> &lt;/form&gt;<br><br> &lt;br&gt;<br><br>&lt;/body&gt;<br>&lt;/html&gt;</pre><p>This code below is the form that allows users to input a URL that will be scraped by the web scraper. The form starts with the form tag and specifies the HTTP method to be used as POST which is recommended when sending sensitive data like passwords or when there are side effects such as updating a database. The action attribute specifies where the form data will be sent after it is submitted. In this case, the data will be sent to the scrape URL defined in urls.py.</p><pre>&lt;form method=&quot;post&quot; action=&quot;{% url &#39;scrape&#39; %}&quot;&gt;<br>  {% csrf_token %}<br>  &lt;input type=&quot;text&quot; name=&quot;url&quot; placeholder=&quot;Enter URL here...&quot;&gt;<br>  &lt;button type=&quot;submit&quot;&gt;Scrape&lt;/button&gt;<br> &lt;/form&gt;</pre><p>Next, a security token is generated using the Django {% csrf_token %} template tag. This token is required for security reasons to prevent cross-site request forgery (CSRF) attacks.</p><p>Then an input field is created with the input tag, where users can enter the URL they want to scrape. The type attribute is set to &quot;text&quot; and the name attribute is set to &quot;url&quot;. The placeholder attribute provides an instructional text that appears in the input field.</p><p>Finally, a button is created with the button tag, and when the user clicks the &quot;Scrape&quot; button, the form is submitted and the entered URL is sent to the web scraper for processing.</p><p>On the other side, the <strong>result.html</strong> file will display the results of the web scraping process. It will show the user the scraped data in a well-formatted and easy-to-read manner. By creating these HTML templates, we can separate the presentation logic from the application logic, making our code more modular and maintainable.</p><blockquote>result.html</blockquote><pre>&lt;!--templates/result.html--&gt;<br><br>&lt;!DOCTYPE html&gt;<br>&lt;html&gt;<br>&lt;head&gt;<br> &lt;title&gt;Scraped Data&lt;/title&gt;<br> &lt;style&gt;<br>  body {<br>   font-family: Arial, sans-serif;<br>   background-color: #f2f2f2;<br>   margin: 0;<br>   padding: 0;<br>  }<br><br>  .container {<br>   max-width: 800px;<br>   margin: 0 auto;<br>   padding: 20px;<br>   background-color: #fff;<br>   box-shadow: 0 0 10px rgba(0, 0, 0, 0.2);<br>   overflow-y: scroll; <br>   height: 500px; <br>  }<br><br>  h1 {<br>   font-size: 36px;<br>   color: #333;<br>   margin-top: 0;<br>  }<br><br>  h2 {<br>   font-size: 24px;<br>   color: #333;<br>   margin-top: 30px;<br>   margin-bottom: 10px;<br>  }<br><br>  p {<br>   font-size: 16px;<br>   color: #666;<br>   line-height: 1.5;<br>   margin-top: 0;<br>  }<br><br> <br>  .button {<br>   background-color: #4CAF50;<br>   border: none;<br>   color: white;<br>   padding: 15px 32px;<br>   text-align: center;<br>   text-decoration: none;<br>   display: inline-block;<br>   font-size: 16px;<br>   margin: 4px 2px;<br>   cursor: pointer;<br>   border-radius: 5px;<br>   box-shadow: 0 2px 5px rgba(0,0,0,0.2);<br>  }<br><br> <br>  .button:hover {<br>   background-color: #3e8e41;<br>  }<br><br>  <br> &lt;/style&gt;<br> &lt;link rel=&quot;stylesheet&quot; href=&quot;https://www.w3schools.com/w3css/4/w3.css&quot;&gt;<br>&lt;/head&gt;<br>&lt;body&gt;<br> &lt;br&gt;&lt;br&gt;<br> &lt;div class=&quot;container&quot;&gt;<br>  &lt;h1&gt;{{ title }}&lt;/h1&gt;<br>  {% for heading in headings %}<br>   &lt;h2&gt;{{ heading }}&lt;/h2&gt;<br>  {% endfor %}<br>  {% for paragraph in paragraphs %}<br>   &lt;p&gt;{{ paragraph }}&lt;/p&gt;<br>  {% endfor %}<br>  <br> &lt;/div&gt;<br> <br> &lt;br&gt;&lt;br&gt;<br> <br>    <br> &lt;div class=&quot;w3-container&quot; style=&quot;margin:auto; width:50%; text-align:center;&quot;&gt;<br>  &lt;form method=&quot;POST&quot; action=&quot;{% url &#39;download_file&#39; %}&quot;&gt;<br>   {% csrf_token %}<br>   &lt;h3&gt;Download Option&lt;/h3&gt;<br>    <br>   &lt;select class=&quot;w3-select&quot; id=&quot;download_type&quot; name=&quot;download_type&quot;&gt;<br>     &lt;option value=&quot;&quot; disabled selected&gt;Choose your option&lt;/option&gt;<br>     &lt;option value=&quot;pdf&quot;&gt;PDF&lt;/option&gt;<br>     &lt;option value=&quot;csv&quot;&gt;CSV&lt;/option&gt;<br>     &lt;option value=&quot;json&quot;&gt;JSON&lt;/option&gt;<br>   &lt;/select&gt;<br>    &lt;br&gt;&lt;br&gt;<br>   &lt;p&gt;&lt;button type=&quot;submit&quot; class=&quot;w3-btn w3-teal&quot;&gt;Download&lt;/button&gt;&lt;/p&gt;<br><br>   &lt;a href=&quot;{% url &#39;home&#39; %}&quot;&gt;<br>    &lt;button class=&quot;w3-button w3-teal w3-center&quot; style=&quot;margin:auto&quot;&gt;Go Back&lt;/button&gt;<br>   &lt;/a&gt;<br>    <br>    &lt;/form&gt;<br> &lt;/div&gt;<br> &lt;br&gt;&lt;br&gt;<br>     <br>&lt;/body&gt;<br>&lt;/html&gt;</pre><p>The result HTML file contains placeholders that will be replaced by dynamic content using template tags. The “title” tag is replaced by the title of the scraped web page, the “headings” tag is replaced by the headings of the scraped content, and the “paragraphs” tag is replaced by the paragraphs of the scraped content.</p><pre>&lt;form method=&quot;POST&quot; action=&quot;{% url &#39;download_file&#39; %}&quot;&gt;<br>   {% csrf_token %}<br>   &lt;h3&gt;Download Option&lt;/h3&gt;<br>    <br>   &lt;select class=&quot;w3-select&quot; id=&quot;download_type&quot; name=&quot;download_type&quot;&gt;<br>     &lt;option value=&quot;&quot; disabled selected&gt;Choose your option&lt;/option&gt;<br>     &lt;option value=&quot;pdf&quot;&gt;PDF&lt;/option&gt;<br>     &lt;option value=&quot;csv&quot;&gt;CSV&lt;/option&gt;<br>     &lt;option value=&quot;json&quot;&gt;JSON&lt;/option&gt;<br>   &lt;/select&gt;<br>    &lt;br&gt;&lt;br&gt;<br>   &lt;p&gt;&lt;button type=&quot;submit&quot; class=&quot;w3-btn w3-teal&quot;&gt;Download&lt;/button&gt;&lt;/p&gt;<br><br>   &lt;a href=&quot;{% url &#39;home&#39; %}&quot;&gt;<br>    &lt;button class=&quot;w3-button w3-teal w3-center&quot; style=&quot;margin:auto&quot;&gt;Go Back&lt;/button&gt;<br>   &lt;/a&gt;<br>    <br>    &lt;/form&gt;</pre><p>The user can select the file format (PDF, CSV, or JSON) in which they want to download the data that was scraped using the download option that is included in the result.html. By selecting the “Go Back” button, the user can also return to the home screen. The code styles the container, headings, and buttons using CSS, and it also integrates the W3.CSS library to give the download button additional styling.</p><p><strong>Testing the App</strong></p><p>Testing the app is an important step before deploying it to production. This helps ensure that everything works as intended and catches any errors or bugs before it’s too late. To test our web scraping app, we first need to run it locally on our machine.</p><p>To run the app, we need to navigate to the project directory in the terminal and then run the command :</p><pre>python manage.py runserver</pre><p>This will start the server locally and we can access the app by visiting <a href="http://127.0.0.1:8000/">http://127.0.0.1:8000/</a> in our web browser.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Q3l9Z0KgZSb8X3AWoKTsQQ.png" /></figure><p>Once the app is running, we can enter a URL in the input field and click the “Scrape” button. This will scrape the website and display the content of the website on the result page. We can also choose to download the scraped data in PDF, CSV, or JSON format by selecting the desired option and clicking the “Download” button.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XrfjKcSSp76edcBhUWnEPg.png" /></figure><p>Testing the app thoroughly includes trying out various URLs, ensuring the data is scraped and displayed correctly on the result page, and testing the download functionality to make sure the data is downloaded in the desired format. By testing the app before deployment, we can catch any errors or issues and fix them before it goes live.</p><p>To follow along with this tutorial, you can download or clone the source code from the following GitHub repository: <a href="https://github.com/alaminmagaga/webscrapper">https://github.com/alaminmagaga/webscrapper</a>. Once you have downloaded the code, you can run it on your local machine by following the steps outlined in this post.</p><p><strong>Conclusion</strong></p><p>In conclusion, web scraping is a powerful tool that can be used to extract valuable data from websites. In this tutorial, we have seen how to build a simple web scraper using Python and Django. We started by setting up a Django project, defining the necessary models and views, and then implementing the web scraping logic. We also learned how to render the scraped data on a web page and how to allow users to download the scraped data in various formats. By following this tutorial, you should now have a basic understanding of how to build a web scraper using Python and Django, and you can build on this knowledge to create more complex and advanced web scraping projects.</p><ul><li><a href="https://medium.com/analytics-vidhya/simple-way-to-create-a-machine-learning-app-with-flask-69a532663fd5">Simple Way To Create A Machine Learning Web App With Flask</a></li><li><a href="https://alaminmusamagaga.medium.com/build-ussd-with-python-and-django-882ed03f789">Build USSD with Python and Django</a></li><li><a href="https://medium.com/analytics-vidhya/natural-language-processing-nlp-and-process-modeling-in-precision-medicine-a55fa9ec9818">Natural Language Processing(NLP) And Process Modeling In Precision Medicine</a></li></ul><p><em>More content at </em><a href="https://plainenglish.io/"><strong><em>PlainEnglish.io</em></strong></a><em>.</em></p><p><em>Sign up for our </em><a href="http://newsletter.plainenglish.io/"><strong><em>free weekly newsletter</em></strong></a><em>. Follow us on </em><a href="https://twitter.com/inPlainEngHQ"><strong><em>Twitter</em></strong></a>, <a href="https://www.linkedin.com/company/inplainenglish/"><strong><em>LinkedIn</em></strong></a><em>, </em><a href="https://www.youtube.com/channel/UCtipWUghju290NWcn8jhyAw"><strong><em>YouTube</em></strong></a><em>, and </em><a href="https://discord.gg/GtDtUAvyhW"><strong><em>Discord</em></strong></a><strong><em>.</em></strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5374592c2b69" width="1" height="1" alt=""><hr><p><a href="https://python.plainenglish.io/django-web-scraping-app-download-data-in-csv-pdf-and-json-formats-5374592c2b69">Django Web Scraping App: Download Data in CSV, PDF, and JSON Formats</a> was originally published in <a href="https://python.plainenglish.io/">Python in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Build USSD with Python and Django]]></title>
            <link>https://medium.com/analytics-vidhya/build-ussd-with-python-and-django-882ed03f789?source=rss-d9bc7626f8c------2</link>
            <guid isPermaLink="false">https://medium.com/p/882ed03f789</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[ussd]]></category>
            <category><![CDATA[fintech]]></category>
            <category><![CDATA[django]]></category>
            <dc:creator><![CDATA[Alamin Musa Magaga]]></dc:creator>
            <pubDate>Fri, 08 Jul 2022 00:20:50 GMT</pubDate>
            <atom:updated>2022-07-15T22:14:10.294Z</atom:updated>
            <content:encoded><![CDATA[<h4>How I build my USSD subscription service</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*FjChNpAlEQiLVdLI" /><figcaption>Photo by <a href="https://unsplash.com/@toshi1998?utm_source=medium&amp;utm_medium=referral">Ashutosh Singh Rana</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>In this tutorial, I will show how to build a USSD subscription service with python, Django, Ngrok, and Africastalking API. so let&#39;s get started</p><h3>USSD</h3><p>Unstructured Supplementary Service Data (USSD) is a Global System for Mobile Communications (<a href="https://www.techtarget.com/searchmobilecomputing/definition/GSM">GSM</a>) protocol that allows users without a smartphone or internet connection to subscribe to certain services, check account balances, transfer funds, and many others.USSD fosters a greater financial inclusion for the underbanked, people in rural areas, and those without a smartphone by allowing them to enjoy almost the same opportunities as those with smart devices in financial and transaction assessment.</p><p>The USSD gateway transmits the request to the USSD application, which then generates a response to the request and sends it back to the gateway. The gateway then delivers the response to the application, which in turn sends it to the user, who receives it on their mobile phone screen. This creates a real-time connection and enables speedier communication between the user and the client application.</p><p>The use of USSD is not only limited to financial transactions; it is also used in many other contexts, including voting, surveys, utility services, competitions, and promotions.</p><h3><strong>Django</strong></h3><p>Django is a highly secured, free, and open-source Python web framework that makes it easier for developers to build websites with greater flexibility.</p><p>Django is one of the most popular Python frameworks for web development because of its Compatibility, Infrastructure, Security, great documentation, and active community support.</p><p>Django is being used by top global industries such as Instagram, Spotify, Pinterest, Mozilla, and Eventbrite.</p><h3>Ngrok</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/682/1*PfpLHY2_Fvm31hTtjs4-4A.jpeg" /></figure><p>Ngrok is a cross-platform, that enables developers to share their localhost servers with people, systems, and cloud services<strong>.</strong></p><p>Ngrok creates a secure tunnel to our local machine and allows the local host server to be hosted on its subdomain with no need for the computer IP address or domain name.</p><p>It allows developers to share their local host server for clients and interesting parties to access projects run locally without deployment to a live server.</p><h3>Africastalking</h3><p>Africastalking is a platform that allows developers to employ strong SMS, USSD, Voice, Airtime, and Payments APIs to realize their ideas as they create and support scalable enterprises.</p><h4><strong>Step 1: Create A Django Project</strong></h4><p>open your command prompt and install Django</p><pre>pip install django==1.11</pre><p>Note: For this project, Django version 1.11 is Required.</p><p>Create a Django project after installation.</p><pre>django-admin startproject subscription</pre><p>Navigate to the project directory</p><pre>cd subscription</pre><p>Use your favorite editor or IDE to open the Subscription project; I’m using Visual Studio Code.</p><pre>code .</pre><h4><strong>Step 2: Create an Application</strong></h4><p>Create an application after opening the project by executing the following code in the Vs code terminal.</p><pre>python manage.py startapp demo</pre><p>Open the settings.py file in the subscription folder, and then add “demo” to the INSTALLED APPS list.</p><p>settings.py</p><pre># Application definition</pre><pre>INSTALLED_APPS = [</pre><pre>&#39;django.contrib.admin&#39;,</pre><pre>&#39;django.contrib.auth&#39;,</pre><pre>&#39;django.contrib.contenttypes&#39;,</pre><pre>&#39;django.contrib.sessions&#39;,</pre><pre>&#39;django.contrib.messages&#39;,</pre><pre>&#39;django.contrib.staticfiles&#39;,</pre><pre>&#39;demo&#39;, #demo added</pre><pre>]</pre><p>Also copy this, then paste it into the urls.py file.</p><p><strong>urls.py</strong></p><pre>from django.conf.urls import url, include</pre><pre>from django.contrib import admin<br></pre><pre>urlpatterns = [</pre><pre>url(https://medium.com/r&#39;^admin/&#39;,%20admin.site.urls),</pre><pre>url(https://medium.com/r&#39;&#39;,%20include(&#39;demo.urls&#39;)),</pre><pre>]</pre><p>Add this code to the views.py file in the demo folder as well.</p><p><strong>views.py</strong></p><pre>from django.views.decorators.csrf import csrf_exempt</pre><pre>from django.http import HttpResponse</pre><pre>@csrf_exempt</pre><pre>def index(request):<br>  if request.method == &#39;POST&#39;:<br>    session_id = request.POST.get(&#39;sessionId&#39;)<br>    service_code = request.POST.get(&#39;serviceCode&#39;)<br>    phone_number = request.POST.get(&#39;phoneNumber&#39;)<br>    text = request.POST.get(&#39;text&#39;)</pre><pre><br>    response = &quot;&quot;</pre><pre>    if text == &quot;&quot;:<br>      response = &quot;CON Welcome to our News subscription service \n&quot;<br>      response += &quot;1. Sport \n&quot;<br>      response += &quot;2. political \n<br>      response += &quot;3. Local  \n&quot;<br>      response += &quot;4. International&quot;</pre><pre>    elif text == &quot;1&quot;:<br>      response = &quot;CON Select Your Preferred Sport Plans \n&quot;<br>      response += &quot;1. Daily @ N100 \n&quot;<br>      response += &quot;2. Weekly @ N50 \n&quot;<br>      response += &quot;3. Monthly @ N25 &quot;</pre><pre>    elif text == &quot;1*1&quot;:<br>      response = &quot;CON You will be charged N100 for your Daily Sports news subscription \n&quot;<br>      response += &quot;1. Auto-Renew \n&quot;<br>      response += &quot;2. One-off Purchase \n&quot;<br>      </pre><pre>    elif text == &quot;1*1*1&quot;:<br>      response = &quot;END thank you for subscribing to our daily sport news plan \n&quot;<br>    elif text == &quot;1*1*2&quot;:<br>      response = &quot;END thank you for your one-off daily sport news plan \n&quot;</pre><pre>    elif text == &quot;1*2&quot;:<br>      response = &quot;CON You will be charged N50 for our weekly Sports news plan \n&quot;<br>      response += &quot;1. Auto-Renew \n&quot;<br>      response += &quot;2. One-off Purchase \n&quot;<br>     </pre><pre>elif text == &quot;1*2*1&quot;:<br>      response = &quot;END thank you for subscribing to our weekly sport news plan \n&quot;<br>    elif text == &quot;1*2*2&quot;:<br>      response = &quot;END thank you for your one-off weekly sport news plan \n&quot;</pre><pre>    elif text == &quot;1*3&quot;:<br>      response = &quot;CON You will be charged N25 for our monthly Sports news plan \n&quot;<br>      response += &quot;1. Auto-Renew \n&quot;<br>      response += &quot;2. One-off Purchase \n&quot;<br>     </pre><pre>    elif text == &quot;1*3*1&quot;:<br>      response = &quot;END thank you for subscribing to our monthly sport news plan \n&quot;<br>    elif text == &quot;1*3*2&quot;:<br>      response = &quot;END thank you for your one-off monthly sport news plan \n&quot;</pre><pre>return HttpResponse(response)</pre><p>the function index<strong> </strong>will receive the call response from your service code. This first line of code if request.method == ‘POST’ instructs Python to run the lines of code below if a POST is sent to the server when a user dials the service code. The next four lines of code are the defined variables.</p><pre> session_id = request.POST.get(‘sessionId’)<br> service_code = request.POST.get(‘serviceCode’)<br> phone_number = request.POST.get(‘phoneNumber’)<br> text = request.POST.get(‘text’)</pre><p>The line if text == “ “: means that the user has not sent any request after dialing the USSD code. the line of code response = “CON Welcome to our News subscription service \n” is the first message that appears on a User’s screen immediately after dialing a USSD code. According to Africastalking, every if or elif statement has a procedure that starts with the CON that appears in the response. response += “1. Sport \n”. is the first option that a user can reply while the response += “1. political \n” and response += “1. local \n” are the second and the third option that the user can reply to, you can create as many options as you like.</p><p>The elif text == “1” means that if a user replies with 1, do the following. If a user replies with 1, execute the following after the previous person replies with 1 which is indicated by the elif text == “1*1”, the elif text == “1*1*1” also means that if a user replies with 1, do the following after the user has initially replied twice to the first choices. the response = “END thank you for your one-off daily sport news plan \n”is the message that appears on a User screen since the User replied with 1. The END in the string is the last message after which the connection will be terminated. this principle is the same for elif text == “2” ,elif text == “2*1”,elif text == “2*1*1”and the rest of the elif text lines of code.</p><p>the last line of code return HttpResponse(response) allows a view function to return a response. we decorated our function with @csrf_exempt because the POST request is not from a form we have to explicitly inform Django to exempt this particular views function {% csrf_token %}which is generally used in forms.</p><p>Include this code below in the demo folder’s urls.py.py file.</p><p><strong>urls.py</strong></p><pre>from django.conf.urls import url</pre><pre>from . import views<br></pre><pre>urlpatterns = [</pre><pre>url(https://medium.com/r&#39;^r&#39;^$&#39;, views.index, name=&#39;index&#39;#39;,%20views.index,%20name=&#39;index&#39;)</pre><pre>]</pre><h4>Step 3:Create an Africastalking account</h4><p>Create an account on the Africastalking <a href="https://africastalking.com/">website</a> and log in, click on the Sandbox app.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oSJhmZhyF0DHGONDO7gRaA.png" /></figure><p>Navigate to the USSD in the dashboard and Click Create channel This will take you to a page where you will create your own USSD service code.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/998/1*R1z51rb8-ykZTAXkFq25HA.png" /></figure><p>Africastalking has a general shared Service Code *384#, and you are expected to add a unique code in the channel to differentiate your Service Code from others.</p><p>You will also be required to provide a USSD callback URL that is connected to our project. Since we are running our project locally, we will use Ngrok to provide a public URL that we will use later for accessing our local host</p><p>Run the Django server using the following command</p><pre>python manage.py runserver</pre><p>Go to the Ngrok <a href="https://ngrok.com/">website</a>, download the Ngrok zip file, unzip it, and then transfer the file to your computer’s /usr/local/bin PATH directory.</p><p>Run Ngrok in the terminal using the following command</p><pre>ngrok https 8000</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*h1cK4Oo2uI-pwdeSn02nzg.png" /></figure><p>open the settings.py file in the project and paste the Ngrok URL in ALLOWED_HOSTS.</p><pre>ALLOWED_HOSTS = [&#39;<a href="https://3121-197-210-53-7.eu.ngrok.io/">3121-197-210-53-7.eu.ngrok.io</a>&#39;]</pre><p>Use the command below to launch the Django server. Return to the africastalking<a href="https://africastalking.com/"> app</a> and enter the public URL for Ngrok in the callback URL field as well.</p><pre>python manage.py runserver</pre><p>Click on Launch simulator to test your new USSD app.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1008/1*SfMbpa8jbAEU5lZkgEl21A.png" /></figure><p>Note: Every time we run Ngrok, the URL changes, therefore you must also update the callback URL and the URL in the ALLOWED_HOSTS.</p><h4>Conclusion</h4><p><strong><em>Congratulations</em></strong>, we have successfully created a simple USSD subscription app with python, Django, Ngrok, and africastalking API.</p><p>you can download the complete project from the<a href="https://github.com/alaminmagaga/USSD-News-Subscribtion"> Link</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=882ed03f789" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/build-ussd-with-python-and-django-882ed03f789">Build USSD with Python and Django</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Text Summarization App with Flask, Sumy, and W3.CSS]]></title>
            <link>https://medium.com/analytics-vidhya/text-summarization-app-with-flask-and-sumy-92212bd05705?source=rss-d9bc7626f8c------2</link>
            <guid isPermaLink="false">https://medium.com/p/92212bd05705</guid>
            <category><![CDATA[naturallanguageprocessing]]></category>
            <category><![CDATA[web-app-development]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[flask]]></category>
            <dc:creator><![CDATA[Alamin Musa Magaga]]></dc:creator>
            <pubDate>Wed, 15 Sep 2021 01:55:43 GMT</pubDate>
            <atom:updated>2022-02-02T12:12:45.792Z</atom:updated>
            <content:encoded><![CDATA[<p>advance text/document summarization web app with Sumy package.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*B3WXIQHrQWZGTpy0" /><figcaption>Photo by <a href="https://unsplash.com/@claudiotesta?utm_source=medium&amp;utm_medium=referral">Claudio Testa</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Today there is a huge collection of information, documents, and blogs, but some of them contain highly repetitive, obscure, and discursive texts giving the reader more time to read, analyze and draw meaningful inferences from those texts.</p><p>the advancement of AI and natural language processing tools provide the state-of-art requirement for easier text preprocessing and summarization techniques.</p><p>text summarization is used in different fields and devices from our text editors to our mobile phones to shortened long sentences into a more concise format.</p><blockquote>Text summarization is the process of making texts or documents more simplified and concise for easier understanding and use</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*qnhaBBDap4bHFY-cd_aiTQ.jpeg" /></figure><p>In this tutorial, I will show a step-by-step approach to creating a functional text summarization app.</p><h3>Project Content</h3><ul><li>Sumy package that was used in the app</li><li>building our app</li><li>testing and running the app</li><li>Conclusion</li></ul><h4>Sumy package</h4><p>Sumy is a special package for performing text summarization tasks and offers remarkable tools such as LexRank, heuristic method(Luhn), Latent Semantic Analysis method (LSA) which we are going to use and incorporate with our text summarization application.</p><p>To get started with this package, run the script below in the command prompt, but if you have already installed it you can skip to the next stage.</p><pre>pip install sumy</pre><h4>Building the app</h4><p>the app development consists of two stages:<br>1-Frontend<br>2-Backend<br>we are going to build our front end with HTML and W3.CSS(modern CSS framework) and the backend with Flask framework.</p><p><strong>Front end</strong><br>Here, we are going to create two HTML files; the <strong><em>index.html</em></strong> and <strong><em>result.html</em></strong>, the <strong><em>index.html</em></strong><em> </em>is the homepage that contains the input of the application, the <strong><em>result.html</em></strong> output the summarized text while the static folder contains the W3.CSS file for styling the app.</p><p><strong>index.html</strong></p><pre>&lt;!DOCTYPE html&gt;<br>&lt;html&gt;<br>&lt;head&gt;<br> &lt;title&gt;Summaryzer&lt;/title&gt;<br>&lt;meta name=&quot;viewport&quot; content=&quot;width=device-width, initial-scale=1&quot;&gt;<br>&lt;link rel=&quot;stylesheet&quot; href=&quot;<a href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/4.7.0/css/font-awesome.min.css">https://cdnjs.cloudflare.com/ajax/libs/font-awesome/4.7.0/css/font-awesome.min.css</a>&quot;&gt;</pre><pre><strong>&lt;!-- linking to the external styling file--&gt;</strong><br>&lt;link rel=&quot;stylesheet&quot;  href=&quot;{{ url_for(&#39;static&#39;, filename=&#39;w3.css&#39;) }}&quot;&gt;</pre><pre>&lt;/style&gt;<br>&lt;/head&gt;<br>&lt;body&gt;</pre><pre><strong>&lt;!-- Navigation Bar--&gt;</strong><br>&lt;div class=&quot;w3-bar w3-teal w3-padding&quot;&gt;<br>  &lt;a href=&quot;{{url_for(&#39;index&#39;)}}&quot; class=&quot;w3-bar-item w3-button w3-padding &quot;&gt;Home&lt;/a&gt;<br>  <br>  &lt;a href=&quot;#C1&quot; class=&quot;w3-bar-item w3-button w3-right w3-padding&quot; &gt;About&lt;/a&gt;<br>  &lt;a href=&quot;#C1&quot; class=&quot;w3-bar-item w3-button w3-right w3-padding&quot; &gt;Contact&lt;/a&gt;<br>&lt;/div&gt;<br>&lt;br&gt;</pre><pre><strong>&lt;!-- Start of Main Section --&gt;</strong><br>&lt;div class=&quot;w3-container w3-center&quot;&gt;&lt;h3&gt;Enter your text Here&lt;/h3&gt;</pre><pre><strong>&lt;!-- text input--&gt;</strong><br> &lt;form method=&quot;POST&quot; action=&quot;/process&quot;&gt;<br> &lt;textarea  name=&quot;input_text&quot; cols=&quot;5&quot; rows=&quot;10&quot;  required=&quot;true&quot; placeholder=&quot;Enter Text Here&quot; style=&quot;width:60%&quot;&gt;&lt;/textarea&gt;<br>            <br>         &lt;br/&gt;</pre><pre>         <strong>&lt;!-- select the model choice--&gt;<br>         </strong>&lt;p&gt;&lt;h4&gt;Select your Algorithm&lt;/h4&gt;&lt;/p&gt;<br>         &lt;select class=&quot;w3-select w3-border w3-round-large&quot;  name=&quot;model_choice&quot; style=&quot;width:40%&quot;&gt;<br>                <br>                <br>  &lt;option value=&quot;default&quot;  selected&gt;Default&lt;/option&gt;<br>  &lt;option value=&quot;lex_summarizer&quot;&gt;Lex Summarizer&lt;/option&gt;<br>  &lt;option value=&quot;luhn_summarizer&quot;&gt;Luhn Summarizer&lt;/option&gt;<br>   &lt;option value=&quot;isa_summarizer&quot;&gt;Isa Summarizer&lt;/option&gt;<br>                    <br>  &lt;/select&gt;<br>   &lt;br&gt;<br>   &lt;br&gt;<br>&lt;button class=&quot;w3-btn w3-blue-grey&quot; type=&quot;reset&quot; value=&quot;reset&quot;&gt;Clear&lt;/button&gt;<br>&lt;button class=&quot;w3-btn w3-teal&quot; type=&quot;submit&quot; value=&quot;reset&quot;&gt;Summarize&lt;/button&gt;<br>   <br>  &lt;/div&gt;<br>&lt;/form&gt;<br>&lt;/div&gt;<br>&lt;br&gt;&lt;br&gt;</pre><pre><strong>&lt;!-- url link of the text document--&gt;</strong><br>&lt;div class=&quot;w3-container w3-center&quot;&gt;&lt;h3&gt;Enter your URL link Here&lt;/h3&gt;<br>&lt;form method=&quot;POST&quot; action=&quot;/process_url&quot;&gt;<br>&lt;input  type=&quot;text&quot; name=&quot;input_url&quot; placeholder=&quot;Enter URL Here&quot; required=&quot;true&quot;  style=&quot;width:50%&quot;&gt;<br>&lt;br&gt;&lt;br&gt;<br>  &lt;button class=&quot;w3-btn w3-blue-grey&quot; type=&quot;reset&quot; value=&quot;reset&quot;&gt;Clear&lt;/button&gt;<br>  &lt;button class=&quot;w3-btn w3-teal&quot; type=&quot;submit&quot; value=&quot;reset&quot;&gt;Summarize&lt;/button&gt;<br>&lt;/form&gt;</pre><pre>&lt;/div&gt;<br>&lt;br&gt;&lt;br&gt;</pre><pre>&lt;/div&gt;<br>&lt;/div&gt;</pre><pre><strong>&lt;!-- footer --&gt;</strong><br>&lt;div class=&quot;w3-container w3-teal&quot;&gt;</pre><pre>&lt;div class=&quot;w3-row&quot;&gt;</pre><pre>&lt;div class=&quot;w3-half w3-container&quot;&gt;<br>          &lt;h4 &gt;About the App&lt;/h4&gt;<br>          &lt;p&gt;the app was build with Flask framework,Sumy packages(Lex rank,Isa and luhn) and w3.css(modern css framework) for the application<br>.&lt;/p&gt;<br>Made by &lt;a  href=&quot;<a href="https://twitter.com/Alamin_Magaga">https://twitter.com/Alamin_Magaga</a>&quot;&gt;Alamin M Magaga&lt;/a&gt;&lt;br/&gt;<br>      Founder&lt;a  href=&quot;<a href="https://twitter.com/Magtech_Dihub">https://twitter.com/Magtech_Dihub</a>&quot;&gt; <a href="http://twitter.com/Magtech">@Magtech</a> Digital Hub&lt;/a&gt;</pre><pre>&lt;/div&gt;<br>      <br>   &lt;div class=&quot;w3-half w3-container w3-center&quot;&gt;<br>          &lt;h4 id=&quot;C1&quot;&gt;Connect With Me&lt;/h4&gt;<br>          &lt;ul&gt;<br>           &lt;a href=&quot;<a href="https://web.facebook.com/alaminmusa.magaga">https://web.facebook.com/alaminmusa.magaga</a>&quot; target=&quot;_blank&quot; class=&quot;white-text&quot;&gt;<br>            &lt;i class=&quot;fa fa-facebook fa-4x&quot;&gt;&lt;/i&gt;<br>            <br>          &lt;/a&gt;<br>          &lt;a href=&quot;<a href="https://www.youtube.com/channel/UCy3KM0QxpZUTgKjXRVID6-A">https://www.youtube.com/channel/UCy3KM0QxpZUTgKjXRVID6-A</a>&quot; target=&quot;_blank&quot; &gt;<br>            &lt;i class=&quot;fa fa-youtube-square fa-4x&quot;&gt;&lt;/i&gt;<br>          <br>          &lt;/a&gt;<br>          <br>          &lt;a href=&quot;alaminmusamagaga.medium.com&quot; target=&quot;_blank&quot; class=&quot;white-text&quot;&gt;<br>            &lt;i class=&quot;fa fa-medium fa-4x&quot;&gt;&lt;/i&gt;<br>            <br>          &lt;/a&gt;<br>          <br>          &lt;a href=&quot;<a href="https://www.linkedin.com/in/alamin-musa-magaga-8b388118b">https://www.linkedin.com/in/alamin-musa-magaga-8b388118b</a>&quot; target=&quot;_blank&quot; class=&quot;white-text&quot;&gt;<br>            &lt;i class=&quot;fa fa-linkedin fa-4x&quot;&gt;&lt;/i&gt;<br>          &lt;/a&gt;<br>          <br>          &lt;/a&gt;<br>           &lt;a href=&quot;<a href="https://github.com/alaminmagaga">https://github.com/alaminmagaga</a>&quot; target=&quot;_blank&quot; class=&quot;white-text&quot;&gt;<br>            &lt;i class=&quot;fa fa-github-square fa-4x&quot;&gt;&lt;/i&gt;<br>          &lt;/a&gt;<br>          <br>          &lt;a href=&quot;<a href="https://twitter/Alamin_Magaga">https://twitter/Alamin_Magaga</a>&quot; target=&quot;_blank&quot; class=&quot;white-text&quot;&gt;<br>            &lt;i class=&quot;fa fa-twitter fa-4x&quot;&gt;&lt;/i&gt;<br>        <br>          &lt;/a&gt;<br>         <br>          &lt;/ul&gt;<br>    &lt;/div&gt;</pre><pre>&lt;/div&gt;<br>&lt;br&gt;<br>    &lt;/div&gt;<br>&lt;/body&gt;<br>&lt;/html&gt;</pre><p>we style the index.html file externally by linking it to a styling file below</p><pre>&lt;link rel=”stylesheet” href=”{{ url_for(‘static’, filename=’w3.css’) }}”&gt;</pre><p>the form action send the text input data to a function<em> </em>in the <strong><em>app.py</em></strong> file.</p><pre>&lt;form method=”POST” action=”/process”&gt;</pre><p>the text area is the user input of the application.</p><pre>&lt;textarea name=”rawtext” cols=”5&quot; rows=”10&quot; required=”true” placeholder=”Enter Text Here” style=”width:60%”&gt;&lt;/textarea&gt;</pre><p>for the button types reset and submit, the latter is used for resetting back to the initial stage and the former is for submission of the input</p><pre>&lt;button class=”w3-btn w3-blue-grey” type=”reset” value=”reset”&gt;Clear&lt;/button&gt;<br>&lt;button class=”w3-btn w3-blue” type=”submit” value=”reset”&gt;Summarize&lt;/button&gt;</pre><p><strong>result.html</strong></p><p>the result.html file contains the variables that output the summarized text for the app.</p><pre>&lt;!DOCTYPE html&gt;<br>&lt;html &gt;<br>&lt;head&gt;<br>  &lt;meta charset=&quot;UTF-8&quot;&gt;<br>  &lt;title&gt;prediction&lt;/title&gt;</pre><pre><strong>&lt;!-- linking the index.html file to the styling folder--&gt;</strong><br>&lt;link rel=&quot;stylesheet&quot;  href=&quot;{{ url_for(&#39;static&#39;, filename=&#39;w3.css&#39;) }}&quot;&gt;</pre><pre> &lt;style type=&quot;text/css&quot;&gt;</pre><pre>&lt;/style&gt;<br>&lt;/head&gt;</pre><pre>&lt;body&gt;</pre><pre>&lt;div class=&#39;w3-padding w3-teal w3-center&#39;&gt;&lt;h1&gt;Text Summarization Result&lt;/h1&gt;&lt;/div&gt;<br>  <br>&lt;/form&gt;<br>&lt;/div&gt;</pre><pre>&lt;/div&gt;</pre><pre>&lt;div class=&quot;w3-container w3-padding-24&quot;&gt;</pre><pre>&lt;div class=&quot;w3-container w3-card&quot;&gt;<br> &lt;div class=&quot;w3-row&quot;&gt;</pre><pre>&lt;div class=&quot;w3-half w3-container &quot;&gt;<br>    &lt;h5&gt;Original Text&lt;/h5&gt;<br>             &lt;p&gt;Reading Time: {{ final_reading_time }} minute &lt;/p&gt;<br>             &lt;p &gt;{{ctext}}&lt;/p&gt;<br>            <br>&lt;/div&gt;</pre><pre>&lt;div class=&quot;w3-half w3-container w3-light-grey&quot;&gt;<br>    &lt;h5&gt;Summarized Text&lt;/h5&gt;<br>              <br>              &lt;p&gt;Reading Time: &lt;span class=&quot;w3-text-teal&quot;&gt;{{ summary_reading_time }} minute &lt;/span&gt;&lt;/p&gt;<br>        <br>            &lt;p&gt;{{ final_summary }}&lt;/p&gt;<br>            <br>          <br>  &lt;/div&gt;<br>&lt;/div&gt;<br>&lt;/div&gt;</pre><pre>&lt;/div&gt;</pre><pre>&lt;/div&gt;</pre><pre><strong>&lt;!-- back button--&gt;</strong><br> &lt;div class=&quot;w3-center&quot;&gt;<br>&lt;a href=&quot;{{url_for(&#39;index&#39;)}}&quot; style=&quot;text-decoration:none&quot;&gt;<br>&lt;button class=&quot;w3-btn w3-teal w3-round-large&quot; style=&quot;width:10%;text-decoration:none;&quot; type=&quot;submit&quot; value=&quot;reset&quot;&gt;Back&lt;/button&gt;&lt;/a&gt;&lt;/div&gt;<br>&lt;br&gt;</pre><pre>&lt;/div&gt;</pre><pre>&lt;/body&gt;<br>&lt;/html&gt;</pre><p>this script below shows variables of the total reading time variable of the original text and the input text variable of the application</p><pre>&lt;p&gt;Reading Time: {{ final_reading_time }} min &lt;/p&gt;<br> &lt;p &gt;{{ctext}}&lt;/p&gt;</pre><p>this script below also indicate the variables for the total reading time of the summarized text and summarized text of the application</p><pre>&lt;p&gt;Reading Time: {{ summary_reading_time }} min &lt;/p&gt;<br> &lt;/div&gt;<br> &lt;p&gt;{{ final_summary }}&lt;/p&gt;</pre><h4>Backend</h4><p>flask framework provides simplified access to create machine learning and data science app with flexibility, tools, and documentation that makes it easier and proficient in testing, running and debugging codes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/354/1*jG4o69c0wvmKjQxYHuo2DQ.png" /><figcaption>Photo by Monocubed</figcaption></figure><p>we are going to create an <strong><em>app.py</em></strong> file by using the necessary python libraries and tools such as flask, Sumy packages to write a code that will summarize texts, sentences, and documents into more concise and precise texts.</p><p><strong>app.py</strong></p><pre>#code<br>from flask import Flask,render_template,url_for,request<br>import time<br>import spacy<br>from sumy.parsers.plaintext import PlaintextParser<br>from sumy.nlp.tokenizers import Tokenizer<br>from sumy.summarizers.lex_rank import LexRankSummarizer<br>from sumy.summarizers.luhn import LuhnSummarizer<br>from sumy.summarizers.lsa import LsaSummarizer<br>from bs4 import BeautifulSoup<br>from urllib.request import urlopen,Request</pre><pre>nlp = spacy.load(&quot;en_core_web_sm&quot;)</pre><pre>app = Flask(__name__)</pre><pre>def lex_summary(docx):<br>   parser = PlaintextParser.from_string(docx,Tokenizer(&quot;english&quot;))<br>   lex_summarizer = LexRankSummarizer()<br>   summary = lex_summarizer(parser.document,3)<br>   summary_list = [str(sentence) for sentence in summary]<br>   result = &#39; &#39;.join(summary_list)<br>   return result</pre><pre>def luhn_summary(docx):<br>   parser = PlaintextParser.from_string(docx,Tokenizer(&quot;english&quot;))<br>   summarizer_luhn = LuhnSummarizer()<br>   summary_1 =summarizer_luhn(parser.document,3)<br>   summary_list = [str(sentence) for sentence in summary_1]<br>   result = &#39; &#39;.join(summary_list)<br>   return result</pre><pre>def isa_summary(docx):<br>   parser = PlaintextParser.from_string(docx,Tokenizer(&quot;english&quot;))<br>   summarizer_lsa = LsaSummarizer()<br>   summary_2 =summarizer_lsa(parser.document,3)<br>   summary_list = [str(sentence) for sentence in summary_2]<br>   result = &#39; &#39;.join(summary_list)<br>   return result</pre><pre># Reading Time<br>def readingTime(mytext):<br>   total_words = len([ token.text for token in nlp(mytext)])<br>   estimatedTime = total_words/200.0<br>   return estimatedTime</pre><pre><a href="http://twitter.com/app">@app</a>.route(&#39;/&#39;)<br>def index():<br>   return render_template(&#39;index.html&#39;)</pre><pre><a href="http://twitter.com/app">@app</a>.route(&#39;/process&#39;,methods=[&#39;GET&#39;,&#39;POST&#39;])<br>def process():<br>    start = time.time()<br>    if request.method == &#39;POST&#39;:<br>        input_text = request.form[&#39;input_text&#39;]<br>        model_choice = request.form[&#39;model_choice&#39;]<br>        final_reading_time = readingTime(input_text)<br>        if model_choice == &#39;default&#39;:<br>            final_summary = lex_summary(input_text)<br>        elif model_choice == &#39;lex_summarizer&#39;:<br>            final_summary = lex_summary(input_text)<br>        elif model_choice == &#39;luhn_summarizer&#39;:<br>            final_summary= luhn_summary(input_text)<br>        elif model_choice == &#39;isa_summarizer&#39;:<br>            final_summary= isa_summary(input_text)<br>    summary_reading_time = readingTime(final_summary)<br>    end = time.time()<br>    final_time = end-start<br>    return render_template(&#39;result.html&#39;,ctext=input_text,final_reading_time=final_reading_time,summary_reading_time=summary_reading_time,final_summary=final_summary,model_selected=model_choice)</pre><pre>def get_text(url):<br>   reqt = Request(url,headers={&#39;User-Agent&#39; : &quot;Magic Browser&quot;})<br>   page = urlopen(reqt)<br>   soup = BeautifulSoup(page)<br>   fetched_text = &#39; &#39;.join(map(lambda p:p.text,soup.find_all(&#39;p&#39;)))<br>   return fetched_text</pre><pre><a href="http://twitter.com/app">@app</a>.route(&#39;/process_url&#39;,methods=[&#39;GET&#39;,&#39;POST&#39;])<br>def process_url():<br> start = time.time()<br> if request.method == &#39;POST&#39;:<br>    input_url = request.form[&#39;input_url&#39;]<br>    raw_text = get_text(input_url)<br>    final_reading_time = readingTime(raw_text)<br>    final_summary = lex_summary(raw_text)<br> summary_reading_time = readingTime(final_summary)<br> end = time.time()<br> final_time = end-start<br> return render_template(&#39;result.html&#39;,ctext=raw_text,<br>                        final_summary=final_summary,<br>                        final_time=final_time,<br>                        final_reading_time=final_reading_time,<br>                        summary_reading_time=summary_reading_time)</pre><pre>if __name__ == &#39;__main__&#39;:<br> app.run(debug=True)</pre><p><strong>Step 1: Import the necessary libraries</strong></p><pre><br>from flask import Flask,render_template,url_for,request<br>import time<br>from sumy.parsers.plaintext import PlaintextParser<br>from sumy.nlp.tokenizers import Tokenizer<br>from sumy.summarizers.lex_rank import LexRankSummarizer<br>from sumy.summarizers.luhn import LuhnSummarizer<br>from sumy.summarizers.lsa import LsaSummarizer</pre><p><strong>Step 2: Create a function that will summarize the text with three different Sumy Algorithms</strong></p><pre>def lex_summary(docx):<br>   parser = PlaintextParser.from_string(docx,Tokenizer(&quot;english&quot;))<br>   lex_summarizer = LexRankSummarizer()<br>   summary = lex_summarizer(parser.document,3)<br>   summary_list = [str(sentence) for sentence in summary]<br>   result = &#39; &#39;.join(summary_list)<br>   return result</pre><pre>def luhn_summary(docx):<br>   parser = PlaintextParser.from_string(docx,Tokenizer(&quot;english&quot;))<br>   summarizer_luhn = LuhnSummarizer()<br>   summary_1 =summarizer_luhn(parser.document,3)<br>   summary_list = [str(sentence) for sentence in summary_1]<br>   result = &#39; &#39;.join(summary_list)<br>   return result</pre><pre>def isa_summary(docx):<br>   parser = PlaintextParser.from_string(docx,Tokenizer(&quot;english&quot;))<br>   summarizer_lsa = LsaSummarizer()<br>   summary_2 =summarizer_lsa(parser.document,3)<br>   summary_list = [str(sentence) for sentence in summary_2]<br>   result = &#39; &#39;.join(summary_list)<br>   return result</pre><p>the code above will summarize text with the use of three Sumy algorithms; LexRank, heuristic method(Luhn), Latent Semantic Analysis method (LSA)</p><p><strong>Step 3: Calculate the total reading time</strong></p><pre><br>def readingTime(mytext):<br> total_words = len([ token.text for token in nlp(mytext)])<br> estimatedTime = total_words/200.0<br> return estimatedTime</pre><p>this function calculates the estimated time taken required to read the text document</p><p><strong>Step 4. Defining a function for returning the summarized text</strong></p><pre><strong>&lt;!-- this will render the template folder to index.html as the homepage --&gt;</strong><br><a href="http://twitter.com/app">@app</a>.route(&#39;/&#39;)<br>def index():<br>   return render_template(&#39;index.html&#39;)</pre><pre><strong>&lt;!-- recieve text input and summaize it with the chosen algorithm --&gt;<br></strong><a href="http://twitter.com/app">@app</a>.route(&#39;/process&#39;,methods=[&#39;GET&#39;,&#39;POST&#39;])<br>def process():<br>    start = time.time()<br>    if request.method == &#39;POST&#39;:<br>        input_text = request.form[&#39;input_text&#39;]<br>        model_choice = request.form[&#39;model_choice&#39;]<br>        final_reading_time = readingTime(input_text)<br>        if model_choice == &#39;default&#39;:<br>            final_summary = lex_summary(input_text)<br>        elif model_choice == &#39;lex_summarizer&#39;:<br>            final_summary = lex_summary(input_text)<br>        elif model_choice == &#39;luhn_summarizer&#39;:<br>            final_summary= luhn_summary(input_text)<br>        elif model_choice == &#39;isa_summarizer&#39;:<br>            final_summary= isa_summary(input_text)<br>    summary_reading_time = readingTime(final_summary)<br>    end = time.time()<br>    final_time = end-start<br>    return render_template(&#39;result.html&#39;,ctext=input_text,final_reading_time=final_reading_time,summary_reading_time=summary_reading_time,final_summary=final_summary,model_selected=model_choice)</pre><p>the function<em> </em><strong><em>process</em></strong> will take both the text input from the form and also the choice of the algorithm that will be used for the summarization and returns the summarized version of the text with the estimated reading time in the <strong><em>result.html</em></strong> file.</p><p><strong>step 5: Create a function that will fetch texts document from the URL</strong></p><pre>def get_text(url):<br>  reqt = Request(url,headers={&#39;User-Agent&#39; : &quot;Magic Browser&quot;})<br>  page = urlopen(reqt)<br>  soup = BeautifulSoup(page)<br>  fetched_text = ‘ ‘.join(map(lambda p:p.text,soup.find_all(‘p’)))<br>  return fetched_text</pre><p><strong>Step 6: Create a function that returns the summarized text of the text URL</strong></p><pre><a href="http://twitter.com/app">@app</a>.route(&#39;/process_url&#39;,methods=[&#39;GET&#39;,&#39;POST&#39;])<br>def process_url():<br> start = time.time()<br> if request.method == &#39;POST&#39;:<br>    input_url = request.form[&#39;input_url&#39;]<br>    raw_text = get_text(input_url)<br>    final_reading_time = readingTime(raw_text)<br>    final_summary = lex_summary(raw_text)<br> summary_reading_time = readingTime(final_summary)<br> end = time.time()<br> final_time = end-start<br> return render_template(&#39;result.html&#39;,ctext=raw_text,<br>                        final_summary=final_summary,<br>                        final_time=final_time,<br>                        final_reading_time=final_reading_time,<br>                        summary_reading_time=summary_reading_time)</pre><p>the function <strong><em>get_text</em></strong> will fetch text document from the link URL while the function<strong> process<em>_url</em> </strong>will take the input, summarize it with the chosen model, and then returns the summarized text and the estimated reading time in the <strong>result.html</strong> file.</p><h4>Testing and Running the App</h4><p>navigate to the project directory from your command or anaconda prompt and run the command below:</p><pre>python app.py</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*Va_i9p9yCjQsC54gJdsA3A.jpeg" /></figure><p>copy and paste this URL:<a href="http://127.0.0.1:5000/">http://127.0.0.1:5000/</a> in your browser to access the app.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zCajME920Tn2DtEHbZUmig.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gfbzLkI0EAlqJSmWB3FO2Q.jpeg" /><figcaption>photo by magtech digital hub</figcaption></figure><h4><strong>4-Conclusion</strong></h4><p><strong><em>Congratulations</em></strong>, at the end of this project we have created three Sumy algorithms that will summarize texts document and we also incorporate w3.css(modern CSS framework) with the app to make it more presentable and interactive.</p><p>you can download the complete project from the <a href="https://github.com/alaminmagaga/text-summarization-web-app-with-flask-and-sumy-package">Link</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=92212bd05705" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/text-summarization-app-with-flask-and-sumy-92212bd05705">Text Summarization App with Flask, Sumy, and W3.CSS</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How To Deal With Class Imbalance In Dataset]]></title>
            <link>https://medium.com/analytics-vidhya/how-to-deal-with-class-imbalance-in-dataset-a8e1c957602b?source=rss-d9bc7626f8c------2</link>
            <guid isPermaLink="false">https://medium.com/p/a8e1c957602b</guid>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Alamin Musa Magaga]]></dc:creator>
            <pubDate>Wed, 19 May 2021 16:29:43 GMT</pubDate>
            <atom:updated>2023-10-02T23:21:22.502Z</atom:updated>
            <content:encoded><![CDATA[<h3>A simple techniques to deal with the imbalance of data</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ZIbr628-RfVOv2xG" /><figcaption>Photo by <a href="https://unsplash.com/@shotsbywolf?utm_source=medium&amp;utm_medium=referral">Geert Pieters</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><strong>What is class imbalance</strong></p><p>class imbalance is the unequal distribution and variation of data in machine learning task where one class tend to have more values than the other classes or distributions. the class imbalance can occur as a result of undiverse and biased resampling of data which is due to the collection of data from a single geographical area, or as a result of the nature of the problem such as fraudulent transactions which tend to have a biased distribution of data between the classes because of less occurrence and prevalence of fraud in the overall transactions. imbalance occurs also because of the presence of minority class such as race, ethnicity, or tribe in the data set.</p><p>so let us start by uploading our dataset and then printing the value counts of the <strong>Sex </strong>column.</p><pre>df=pd.read_csv(&#39;titanic_train.csv&#39;)<br>df.Sex.value_counts()</pre><p>Output [1]:</p><pre>male      577<br>female    314<br>Name: Sex, dtype: int64</pre><p>in the output above, the male class has more values than the female class due to an imbalance of data.</p><p><strong>Dealing with class imbalanced in dataset</strong><br> 1-Over sampling the minority class<br> 2- Under-sampling the majority class</p><p><strong>Oversampling the minority class</strong><br>in oversampling, we created more synthetic data for the minority class so that the minority class will have more data to match the majority class, this is recommendable if there are fewer data to work with.</p><p>In [ 1]:</p><pre>from sklearn.utils import resample<br>Male=df[df.Sex==&#39;male&#39;]<br>Female=df[df.Sex==&#39;female&#39;]</pre><pre>female_upsampled = resample(Female,<br> replace = True,<br> n_samples = len(Male), # Match the number  with the majority class<br> random_state=20)</pre><pre>df = pd.concat([Male, female_upsampled])</pre><p>let us now see the column</p><pre>df.Sex.value_counts()</pre><p>Output [2]:</p><pre>female    577<br>male      577<br>Name: Sex, dtype: int64</pre><p>now, both the number of female and male class are equal.</p><p><strong>Under-sampling majority class</strong><br>in this sampling method, we remove some data from the majority class to match the minority class, but this may affect the performance of the model if there are fewer data to work with.</p><pre>male_downsampled = resample(Male,<br> replace = True, # Sample with replacement<br> n_samples = len(Female), # Match number with the minority class<br> random_state=20)</pre><pre>df = pd.concat([women, men_downsampled])</pre><p>let us now also see the <strong>Sex</strong> column</p><pre>df.Sex.value_counts()</pre><p>Output [2]:</p><pre>female    314<br>male      314<br>Name: Sex, dtype: int64</pre><p><strong>Conclusion</strong></p><p>the oversampling and undersampling method is one of the very common and effective techniques in dealing with an imbalance of data, but there are other techniques that can be used such as synthetic minority oversampling technique(SMOTH).</p><p><a href="https://alaminmusamagaga.medium.com/simple-way-to-create-a-machine-learning-app-with-flask-69a532663fd5">https://alaminmusamagaga.medium.com/simple-way-to-create-a-machine-learning-app-with-flask-69a532663fd5</a></p><p><a href="https://medium.com/analytics-vidhya/natural-language-processing-nlp-and-process-modeling-in-precision-medicine-a55fa9ec9818">https://medium.com/analytics-vidhya/natural-language-processing-nlp-and-process-modeling-in-precision-medicine-a55fa9ec9818</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a8e1c957602b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/how-to-deal-with-class-imbalance-in-dataset-a8e1c957602b">How To Deal With Class Imbalance In Dataset</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>