Dataset

We will be using two datasets, train data and test data. You can download the data set using this link: https://github.com/urvashithakre/Flight-Price-Prediction-Using-Machine-Learning . The train data comprises of 10683 rows and 11 attributes whereas the test data has 2671 rows.

Let’s start with the coding!

1. Lets import the necessary libraries first.

2. Import training data and display the first five rows to take the overall view of the data.

3. We will check whether the data contains any null values using isna() and sum() method and then drop the NaN values using dropna() method so that there are no discrepancies in our data by which we can predict precisely.

3. Now that our data is clean with no NaN values we will have a look at the datatypes.

It can be observed that Date_of_Journey , Dep_Time and Arrival_Time have been assigned as object by default. We will convert this datatype into timestamp to use this column for prediction.

4. Separate the column “Date_of_Journey” into “journey_day” and “journey_month” to help our machine learning model understand and use the column for prediction. After doing so we will delete the “Date_of_Journey” column which will no longer be useful to us.

5. Now we will be dealing with the “ Dep_time” and Arrival_Time” feature because the machine learning model won’t be able to understand what the time is or what day it is. We will be fetching minutes and hours from both these columns as follows:

6. Now we will be processing the “Duration” column as in some cases there is no Hour(hr) term or minutes (m) term in the colum. We will be using the split function and then append ‘0h’ or ‘0m’ wherever needed.

7. Let’s separate the ‘Duration’ attribute into ‘Duration_hours’ and ‘Duration_mins’ using the apply function.

8. To extract categorical data or continuous features we will iterate every column in the dataset and whosoever has the datatype object we will consider that column as categorical data.

9. Now we will handle the categorical data and basically perform Feature Encoding because machine learning works only on numerical data.

To begin with, first we will plot a boxplot

We can come up with a conclusion that Jet airways has the highest price whereas other airlines had almost similar median with minimal fluctuations.

Airlines with 1 or 2 stops has many outliers and hence their price varies. On the contrary price for airline with 4 stops is not fluctuating.

10. Here we will be changing the ‘Airline’ feature into integer format using one hot encoding

All the features are converted to integer values using get_dummies function.

We will apply the get_dummies function on the ‘Source’ and ‘Destination’ column as well.

11. The ‘Route’ columns mainly tell us that how many cities they have taken to reach from source to destination . So We split the Route column on the basis of the seperator i.e(→) to extract the information .Regarding the ‘Nan’ values we replace those ‘Nan’ values with ‘None’.

12. To get rid of high dimensionality issue we will be using label encoder class.

13. Dealing with ‘Total_Stops’ attribute and assign 1 for ‘1 stop’, 2 for ‘2 Stops’ and so on.

14. Concatenating categorical dataframe with all the dataframes that we have defined earlier. For this we will be using concat() from pandas library. Also, we will be dropping the unnecessary columns present in our data.

15. Checking for outliers in ‘price’ column by plotting some plots such as subplot, distplot and boxplot.

Median plays a very significant role wherever we have outliers. So we will use ‘where’ function to replace all the columns with price>4000 with the median value.

16. Separating independent and dependent features.

Now that all our data is numerical after label encoding so we split the data into test and train and drop the price column from the test set because we have to predict the price with our test data set

Feature selection is nothing but to find the best feature that contributes the most and that has good relationship with the target values. The main reason to apply feature selection is to select important feature so that we don’t face the problem of multiple dimensions.

We will pass the independent and dependent data to the mutual_info_classif() function.

Sorting values according to importance by which we will get to know which column contributes the most and which attribute contributes the least.

17. Splitting the data into train and test data. We taken 80% data for training and remaining 20% for testing.

18. Let us now define a function to predict the Training score, Predictions, r2 score, Mean absolute error(MAE), Mean square error(MSE) and Root mean square error (RMSE).

Similarly we can test different methods like LinearRegression(), DecisionTreeRegressor() , KNeighborsRegressor etc.

19. Cross validating our model. Here we will be using RandomizedSearchCV

Predicting the values for X_test by plotting a distribution plot

We have achieved approximately 83% accuracy but before when we used RandomForest the accuracy was around 80%, that’s the outcome of model hyperparameter tuning. You can see the full code in the jupyter notebook uploaded in this Github Link .

Please let me know your thoughts about this article and do comment if you face any issues.

Flight Price Prediction Using Machine Learning was originally published in Geek Culture on Medium, where people are continuing the conversation by highlighting and responding to this story.

Recognizing handwritten text is a problem that can be traced back to the first automatic machines that needed to recognize individual characters in handwritten documents. Think about, for example, the ZIP codes on letters at the post office and the automation needed to recognize these five digits. Perfect recognition of these codes is necessary in order to sort mail automatically and efficiently. Here we are going to analyze the digits data-set of the Sci-Kit learn library using Jupyter Notebook.

First we begin with importing the required libraries.

Now we load the Digits dataset into the notebook. After loading the data we can read lots of information about the datasets by calling the DESCR attribute.

The images of the handwritten digits are contained in a digits.images array. Each element of this array is an image that is represented by an 8x8 matrix of numerical values that correspond to a grayscale from white, with a value of 0, to black, with the value 15.

Let’s visually check the contents of this result using the matplotlib library.

The numerical values represented by images, i.e. the targets, are contained in the digit.targets array.

This dataset contains 1,797 elements, and so we can consider the first 1,791 as a training set and will use the last six as a validation set. Here we can see in detail these six handwritten digits by using the matplotlib library:

Now we train the svc estimator that we defined earlier.

Now we have to test our estimator, making it interpret the six digits of the validation set and then compare them with the actual digits as follows:

From the above output we can see that the targeted and predicted values are the same. Hence estimator is able to recognize the handwritten digits, interpreting correctly all six digits of the validation set.

Let’s predict the accuracy of our model using KNN classifier.

Conclusion:

Thus we can see how easy it is to import a dataset, build a model using Scikit-Learn, train the model, make predictions with it, and finding the accuracy of our prediction(which in our case is 98.33%).

One type of data that’s easier to find on the net is Weather data. Many sites provide historical data on many meteorological parameters such as pressure, temperature, humidity, wind speed, visibility, etc.

We are going to analyze Weather data-set. You can find the date-set from the mentioned link(https://www.kaggle.com/muthuj7/weather-dataset). Our main aim is to perform data cleaning ,data normalizing ,testing the hypothesis and deriving appropriate insights. So lets begin!

1. Importing necessary libraries such as Pandas , Numpy, Matplotlib and Seaborn.

2. To load the data set using Pandas library. Here I have displayed the first 5 rows.

3. Cleaning the Dataset. Our main aim is to analyze the change in temperature and humidity so we focus on important factors only. Here we Convert Date to a proper format and then resample the data.

4. Plotting a line plot for Temperature vs Humidity from the year 2006 to 2016.

From the above plot we can say that the humidity remains almost constant lying between 0 to 5. The peaks are between 20 to 25 and lowest dip lies below -5.

5. Seaborn’s lmplot is a 2D scatterplot with an optional overlaid regression line. This is useful for comparing numeric variables. Here we plot Apparent temperature (C ) vs Humidity.

6. Now we create a bar plot analyzing the Apparent temperature and Humidity with specified years.

From the above insights it can be seen that the Apparent Temperature is lowest between 2011 to 2012 ie. below -8(C) whereas its highest between 2015 to 2016. Humidity is ranging between 0 to 2 throughout.

7. Plotting temperature and humidity variation for the month of May over the time.

8. Now we we will plot a line plot for better understanding by analyzing the peaks and dips in the graphs with respect to humidity

9. Scatter plot

Conclusion:

Over the years we can see that Global Warming is significantly increasing which in turn is deteriorating the weather conditions and ecosystems. Hence from this analysis we have gathered useful insights which say that over the past 10 years the temperature is consistently increasing while in the humidity also there is a slight rise in some years but mostly remained same throughout. Hence we can conclude that Global Warming is indeed having a lot of adverse effects on the weather conditions.