<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Kalicharan Baksi on Medium]]></title>
        <description><![CDATA[Stories by Kalicharan Baksi on Medium]]></description>
        <link>https://medium.com/@kalicharanbaksi?source=rss-ae3df9e9e945------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*-BQwi9fdBUFIL6qw.jpg</url>
            <title>Stories by Kalicharan Baksi on Medium</title>
            <link>https://medium.com/@kalicharanbaksi?source=rss-ae3df9e9e945------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 02:06:44 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@kalicharanbaksi/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Traffic Sign Detection using YOLO]]></title>
            <link>https://kalicharanbaksi.medium.com/traffic-sign-detection-using-yolo-7e92ec75f787?source=rss-ae3df9e9e945------2</link>
            <guid isPermaLink="false">https://medium.com/p/7e92ec75f787</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[yolov3]]></category>
            <category><![CDATA[object-detection]]></category>
            <dc:creator><![CDATA[Kalicharan Baksi]]></dc:creator>
            <pubDate>Sat, 29 May 2021 10:17:59 GMT</pubDate>
            <atom:updated>2021-05-31T19:29:22.092Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1BmmPdE1UJFW4eh1p4zV2Q.jpeg" /></figure><p>We will be discussing the problem statement from exploratory data analysis to pipeline deployment.</p><h3>Table Of Content:-</h3><ol><li>Introduction</li><li>Business problem</li><li>Mapping to DL problem</li><li>Understanding the Data</li><li>EDA</li><li>Preprocessing of Data</li><li>Understanding YOLO</li><li>Model</li><li>Results</li><li>Post-training Quantization</li><li>Deployment of Model</li><li>Future work</li><li>Profile</li><li>References</li></ol><h3>1.Introduction:-</h3><p>Traffic sign detection is a challenging real-world problem of high industrial relevance. Even autonomous vehicle companies are recently working on upgrading their traffic lights and stop signs detecting techniques. The objective of this case study is to detect traffic signs and classify them.</p><h3>2.Business problem:-</h3><p>The objective is to detect traffic signs that are present inside an image. There is a latency requirement for this problem. we need to build a solution such that the model should be able to detect the signs within seconds.</p><h3>3.Mapping to DL problem:-</h3><p>We will use object detection models to detect the traffic signs in this problem statement. We will use IOU as an evaluation metric here. It helps us benchmark the accuracy of our model predictions. Using it, we can figure out how well does our predicted bounding box overlaps with the ground truth bounding box. The higher the IoU, the better the performance.</p><h3>4.Understanding The Data:-</h3><p>The dataset consists of 900 images in ppm format. The dataset also contains a text file that is in CSV format and consists of the ground truth for all traffic signs in the images. The ground truth file contains the image filename, bounding box coordinates which contain the left, top, right, and bottom coordinates of traffic signs present in the image, and the class id of the traffic sign.</p><h3>5.EDA:-</h3><p>We will convert the txt file into pandas dataframe.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ecadef9e3de721e8d8e8fc3596614ac7/href">https://medium.com/media/ecadef9e3de721e8d8e8fc3596614ac7/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/267/1*wF1cZ2RyFSPR-0C5OdILng.png" /><figcaption>csv file</figcaption></figure><p>There are a total of 600 images in training data and a total of 852 rows in the dataset. Multiple rows for an image refer to multiple traffic signs in the same image.</p><p>Let&#39;s visualize the ppm image file and check its size.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/785209fcf90ddf2c9e44e3d944b396f2/href">https://medium.com/media/785209fcf90ddf2c9e44e3d944b396f2/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/351/1*f6bfFIerwhyOuxhO_nKRNg.png" /></figure><p>Now we will check the distribution of different class IDs in the dataset.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/20ec822ec06d81272ea56b143504a5b8/href">https://medium.com/media/20ec822ec06d81272ea56b143504a5b8/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/893/1*UTQM109WpzTFsdcf3KEVTA.png" /><figcaption>distribution of sign ids</figcaption></figure><p>The class id 1,2,10,12,13,38 present high in number where as 0,19,24,27,31,37 are fewer in number.</p><p>Now we will categorize 43 classes into 4 classes. ( prohibitory, mandatory, danger, other).</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9f29b24b3c0b47978695d6709aa2563f/href">https://medium.com/media/9f29b24b3c0b47978695d6709aa2563f/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/330/1*7XyyXbq6ntcu7faljCXAoA.png" /><figcaption>updated csv file</figcaption></figure><p>Now again we will check the distribution of these newly created classes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/614/1*1FYbpqcCZezQfapI1x9F5Q.png" /><figcaption>distribution of class</figcaption></figure><h3>6.Data Preprocessing:-</h3><p>We will create a file where we will store the png images from the ppm files.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bb36af53a0991b33d24fa158f842b969/href">https://medium.com/media/bb36af53a0991b33d24fa158f842b969/href</a></iframe><p>Here we removed those images which don&#39;t contain any traffic sign.</p><p>We will now create an annotation file for each image which will be stored in XML format.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8799114eca88dc3049c52482e8076d28/href">https://medium.com/media/8799114eca88dc3049c52482e8076d28/href</a></iframe><p>Now we will create TFRecord files which will be used while training our YOLO model. The TFRecord format is a simple format for storing a sequence of binary records. We will use the XML files to generate the TFRecord files. we will split the images into train TFRecord and test TFRecord files.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f609fa32abc9e9b43b5fcdc37983f61b/href">https://medium.com/media/f609fa32abc9e9b43b5fcdc37983f61b/href</a></iframe><h3>7.Understanding YOLO:-</h3><p>We will use YOLOV3 for training our custom object detection model. It is a single-shot detector that also runs quite fast and predicts the output in real-time.</p><p>Let&#39;s understand some terminology related to YOLO.</p><ol><li>Grid cell:-</li></ol><p>YOLOv3 devices the images into three granularity levels such as 52*52,26*26,13*13 grid cells.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/317/1*c4_x6l4XjqIxVhs5DqviPQ.jpeg" /></figure><p>2.Anchor box or bounding box:-</p><p>YOLO can work well for multiple objects where each object is associated with one grid cell. But when one grid cell contains a center point of two different objects then we can use bounding boxes to allow one grid cell to detect multiple objects. For each anchor boxes, three values are associated with it. The location of the bounding box(x,y,w,h), the objectness score, class probability which is a vector of size equal to the number of classes. Here we are predicting the 4+1+4 size of vector for each anchor box.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/586/0*0AIUCVwvAVewWOp6.png" /></figure><p>3.Non-maximal supression:-</p><p>Yolo uses NMS to only keep the best bounding box. The First step in NMS is to remove all the predicted bounding boxes that have a detection probability less than a given NMS threshold.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*iYor7cT9hxlhiT3X.png" /></figure><p>YOLO architecture:-</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*GV6dZrfuD1OtmHZz.png" /></figure><p>YOLO v3 uses a variant of Darknet, which originally has a 53 layer network on IMageNet<strong>.</strong>For the task of detection, 53 more layers are stacked onto it, giving us a 106 layer fully convolutional underlying architecture for YOLO v3.In YOLO v3, the detection is done by applying 1 x 1 detection kernels on feature maps of three different sizes at three different places in the network.The shape of detection kernel is 1 x 1 x (B x (5 + C)). Here B is the number of bounding boxes a cell on the feature map can predict, ‘5’ is for the 4 bounding box attributes and one object confidence and C is the no. of classes.YOLO v3 uses binary cross-entropy<strong> </strong>for calculating the classification loss for each label while object confidence and class predictions are predicted through logistic regression.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/317/0*55zH7gKbuPkdiTbm.png" /><figcaption>Darknet-53</figcaption></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9d3b94a0073b249b456b2920e9f9cc9f/href">https://medium.com/media/9d3b94a0073b249b456b2920e9f9cc9f/href</a></iframe><p>YOLO loss:-</p><p>The YOLO loss for each box prediction is comprised of the following terms-</p><ol><li>Coordinate loss — due to a box prediction not exactly covering an object,</li><li>Objectness loss — due to a wrong box-object IoU prediction,</li><li>Classification loss — due to deviations from predicting ‘1’ for the correct classes and ‘0’ for all the other classes for the object in that box.</li></ol><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ce2ea2106c1d8ea587867d6a8db3b849/href">https://medium.com/media/ce2ea2106c1d8ea587867d6a8db3b849/href</a></iframe><p>More can be read about YOLO <a href="https://afrozchakure.medium.com/yolov3-you-only-look-once-12de76ad74d5">here</a>.</p><h3>8.Modelling:-</h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ba72208d111b8a884be7d267e3c1acb1/href">https://medium.com/media/ba72208d111b8a884be7d267e3c1acb1/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/685/1*8rsVV1ShAthmIG8zZOac5w.png" /><figcaption>model summary</figcaption></figure><p>The training is done by using the train TFrecord file and the validation TFrecord file is used for validation.</p><p>different hyperparameters used need to be passed for training.</p><ol><li>learning rate-0.001</li><li>threshold value-0.4</li><li>image size-416*416*3</li></ol><p>The training was done till validation loss reached 15.76. Training code can be referred from my <a href="https://github.com/kalicharanbaksi/german-traffic-sign-detection-yolo">Github repository</a>.</p><h3>9.Results:-</h3><p>Now we will test our model’s output.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/89c34c81d7512013f4e7c29237f7b0da/href">https://medium.com/media/89c34c81d7512013f4e7c29237f7b0da/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rZpnxbPVFbem5XRmizZo_Q.jpeg" /><figcaption>prohibitory sign detection</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ccvE2EuUjTr4XwNmABgN0Q.jpeg" /><figcaption>danger and prohibitory sign detection</figcaption></figure><h3>10.Post-training Quantization:-</h3><p>Post-training quantization is a conversion technique that can reduce model size while also improving CPU and hardware accelerator latency, with little degradation in model accuracy. You can quantize an already-trained float TensorFlow model when you convert it to TensorFlow Lite format using the <a href="https://www.tensorflow.org/lite/convert/">TensorFlow Lite Converter</a>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d524baf98cb113b2bc799b137d2aaaa8/href">https://medium.com/media/d524baf98cb113b2bc799b137d2aaaa8/href</a></iframe><p>Above code convert model to TFlite format.After quantization model size has been reduced to 50MB from 240MB.It&#39;s a 4x size reduction.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7586612aa88eea2de4690d7f858e3302/href">https://medium.com/media/7586612aa88eea2de4690d7f858e3302/href</a></iframe><p>The above piece of code will load the TFlite weight and predict the output.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*19HClhr94dlv4_cIWrxkgA.jpeg" /><figcaption>prohibitory sign detection by TFlite model</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1BmmPdE1UJFW4eh1p4zV2Q.jpeg" /><figcaption>danger sign detection by TFlite model</figcaption></figure><p>As we can see there is no major difference in the output of both the TFlite model and the original model.</p><h3>11.Deployment of model:-</h3><p>We will use Streamlit for deploying our DL model. Here we will display the predicted image with a bounding box and then we will crop the detected images from the predicted image and display those separately.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f169651c7e8224155690bfe2d2667fd8/href">https://medium.com/media/f169651c7e8224155690bfe2d2667fd8/href</a></iframe><p>The above code will crop the detected sign from the image.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8b7260cb1736c5f9f87a078d6566054a/href">https://medium.com/media/8b7260cb1736c5f9f87a078d6566054a/href</a></iframe><p>The above code snippet is a sample used for the deployment of the model. The output of the webpage looks like this.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F24tQ2kxGCz8%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D24tQ2kxGCz8&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F24tQ2kxGCz8%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/5929778f7641823ee34a04ff4e159ff2/href">https://medium.com/media/5929778f7641823ee34a04ff4e159ff2/href</a></iframe><p>Some images of deployment:-</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/731/1*K-DrEXzD1f6LDo9UwrvzEA.png" /><figcaption>part-1</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/824/1*Rref_4GDl3EMyEPvap3mOg.png" /><figcaption>part-2</figcaption></figure><h3>12.Future Work:-</h3><ol><li>we can use increase the dataset size by using some augmentation technique and retrain the model.</li><li>we can do some post-training data analysis on which images the model did not perform well and do some retraining for those images with more data points.</li></ol><h3>13.Profile:-</h3><p>If u have any query feel free to contact me on <a href="https://www.linkedin.com/in/kali-charan-baksi-721653190/">Linkedin</a>. You can find my full project <a href="https://github.com/kalicharanbaksi/german-traffic-sign-detection-yolo">here</a>.</p><h3>14.References:-</h3><ol><li><a href="https://www.appliedaicourse.com/">https://www.appliedaicourse.com/</a></li><li><a href="https://machinelearningmastery.com/how-to-perform-object-detection-with-yolov3-in-keras/">https://machinelearningmastery.com/how-to-perform-object-detection-with-yolov3-in-keras/</a></li><li><a href="https://github.com/zzh8829/yolov3-tf2">https://github.com/zzh8829/yolov3-tf2</a></li><li><a href="https://medium.com/@anirudhsr97/german-traffic-signs-detection-using-yolov3-ab7b974fca4e">https://medium.com/@anirudhsr97/german-traffic-signs-detection-using-yolov3-ab7b974fca4e</a></li><li><a href="https://afrozchakure.medium.com/yolov3-you-only-look-once-12de76ad74d5">https://afrozchakure.medium.com/yolov3-you-only-look-once-12de76ad74d5</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7e92ec75f787" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Wikipedia Web Traffic Time Series Forecasting]]></title>
            <link>https://kalicharanbaksi.medium.com/wikipedia-web-traffic-time-series-forecasting-ee0ad354a6a7?source=rss-ae3df9e9e945------2</link>
            <guid isPermaLink="false">https://medium.com/p/ee0ad354a6a7</guid>
            <dc:creator><![CDATA[Kalicharan Baksi]]></dc:creator>
            <pubDate>Sat, 13 Feb 2021 08:43:37 GMT</pubDate>
            <atom:updated>2021-02-15T05:55:29.042Z</atom:updated>
            <content:encoded><![CDATA[<p>We will be discussing the problem statement from exploratory data analysis to pipeline deployment.</p><h3>Table of Content:-</h3><ol><li>Introduction</li><li>Business problem</li><li>Mapping to ML/DL problem</li><li>understanding the data</li><li>Existing approaches</li><li>EDA</li><li>feature engineering</li><li>ML model</li><li>Dl model</li><li>Result</li><li>deployment</li><li>future work</li><li>Profile</li><li>References</li></ol><h3>1.Introduction:-</h3><p>The problem statement is about forecasting future web traffic for 145k Wikipedia article. Dataset has 145k time series data on the views of different Wikipedia pages starting from July, 1st, 2015 up until December 31st, 2016. Each row in the dataset is time series of daily page views of a specific article. For each time series, the name of the article is given along with some additional data are given such as the type of traffic that these time series represent (all, mobile, desktop, spider), language, etc.</p><h3>2.Business problem:-</h3><p>The objective is to predict the page visit as close as possible. There is a latency requirement for this problem. we need to build a solution such that I should be able to predict a date within seconds.</p><h3>3.Mapping to ML/dL problem:-</h3><p>This problem statement can be considered as a regression problem. we are going to use SMAPE as evaluation metrics while we use ML models but while training the DL model we will change the loss to MAE and data points are log1p transformed.SMAPE:-the evaluation metrics used in the problem statement is SMAPE(Symmetric mean absolute percentage error).it is the absolute difference between forecast value and actual value divided by half of the sum of absolute values of forecast and actual value. This value is again divided by the number of data points multiplied by 100.</p><h3>3.understanding the data:-</h3><p>Train.csv contains about 145k rows each of which represents a different Wikipedia page and it has 804 columns. The first column contains the name of the page which includes the language of the Wikipedia page, type of access, agent type. other columns represent page visits in different dates.key.csv gives the mapping between the page names and the shortened Id column used for prediction.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*M2s7rZarA_AA8y8S1kyjfg.png" /></figure><h3>5.Existing approaches:-</h3><h4>5.1.Statistical model:-</h4><p>ARIMA is a statistical model. There are three components in the ARIMA model. The first is autoregressive, the second is Integrated and the third is moving average. this model can perform pretty well with seasonal data as it has an inbuilt algorithm to remove seasonality and trends.</p><h4>5.2.Kalman filter:-</h4><p>In the Kalman filter, there are two ways to smoothen the data, data which was accessed by the spider was smoothed by using the Fibonacci median of medians, and the non-spider data was smoothed using Kalman smoothing. The predictions were made using Kalman filters</p><h3>6.EDA:-</h3><p>First, we will fill the missing values with zeros. There are 145k Wikipedia pages are present in the dataset so it&#39;s impossible to examine these pages individually. So we will divide these pages into different categories to do the analysis.</p><p>First, we will extract different information present in the page name such as language, access type, agent, and project.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8a78c3a23ddf1f17152781b31848959e/href">https://medium.com/media/8a78c3a23ddf1f17152781b31848959e/href</a></iframe><p>now we will check the number of articles in each language.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/473/1*4Z7auKpa_nfJKqRzMnlRCg.png" /></figure><p>now we will generate the mean of each day language-wise and examine the data. The results are as follows.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1bSDsqudZLYdiwDGBY4IaA.png" /></figure><p>From the above chart, it can be seen that English language wiki pages have higher traffic than all other languages. The Spanish language has the second-highest everyday mean page visit. Around august of 2016, there is a spike in visits for Russian pages and English pages. Spanish page is very much periodic in nature.</p><p>now we will check the number of articles of each project type.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/447/1*OCsKbvIuduR0KGNBH1IlqQ.png" /></figure><p>More number of Wikipedia pages are given which means more traffic on Wikipedia pages. Wikimedia pages and MediaWiki pages have less traffic.</p><p>Now we will generate the mean of each day Project wise and examine the data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*i2rpOqlLf-KgFElis14n3w.png" /></figure><p>Wikipedia pages have higher mean traffic. The traffic spike that we have seen in the language plot was on the Wikipedia page other project types did not contribute to the spike. Wikimedia pages got traffic spike in January 2017 and September 2017. MediaWiki pages have got unexpectedly high traffic in May 2017.</p><p>now we will check the number of articles of each access type.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/453/1*ZKYnopSZia8dp37RUM31ww.png" /></figure><p>The all-access type has higher traffic compared to the other two categories of access. mobile-web and desktop bar size looks almost similar but mobile-web has slightly higher traffic.</p><p>Now we will generate the mean of each day access type-wise and examine the data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Gvh0eQkq0tX4TK4nZ96lTg.png" /></figure><p>desktop access has the highest mean traffic and mobile web has the lowest mean traffic. a similar trend as language around august 2016 is seen here as well.</p><p>now we will check the number of articles of each agent type.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/439/1*pQcBzMaWSCiCvoV5EiE1Kg.png" /></figure><p>All-agent type has a higher number of pages than spider agent.</p><p>Now we will generate the mean of each day agent type-wise and examine the data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Fsp7MXPSKJs702iWrD9f9Q.png" /></figure><p>all-agent has higher mean traffic as compared to spider.</p><p>we will plot the autocorrelation plot for each language page.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/522ecdbe3c06c74a43a420cdb0404c92/href">https://medium.com/media/522ecdbe3c06c74a43a420cdb0404c92/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/684/1*1Oq3m8FegLQuP33mdVHjlA.png" /></figure><p>here I have shown only for the English page but the above code will generate plots for all the languages. From these autocorrelation plots, it can be observed that lag day7 has a high correlation in most of the language pages.</p><p>now we will generate Fourier transform for each language page.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b0a64c076ee9546ab097acfb2b35d011/href">https://medium.com/media/b0a64c076ee9546ab097acfb2b35d011/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/911/1*evIKZtQC_Fyk_rL5Bpy01w.png" /></figure><p>For the English language, the plot has been shown here. The left plot is for the time domain and the right plot is for the frequency domain. From these Fourier transformations, we can conclude that there are some peak values in 120 days,230 days, and 340 days for all the language except Russian and media pages.</p><h3>7.Feature Engineering:-</h3><p>we will be created features for the last 15 days of the dataset. First of all, we will extract the language, access type, agent, and project. These are important as we have seen in the data exploration.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bf4fbe109b0c0a3816ed6e19932ce304/href">https://medium.com/media/bf4fbe109b0c0a3816ed6e19932ce304/href</a></iframe><p>As seen in the correlation plot there is some high correlation at 7th,14th, and 21st days. so we will be creating features based on that.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0874448ea7e74238a8a7daffccde7b76/href">https://medium.com/media/0874448ea7e74238a8a7daffccde7b76/href</a></iframe><p>now we will create features such as yearly trend, monthly trend, quarterly trend.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/86fd65f6c3c3f7d34477fc5f3ee01dfc/href">https://medium.com/media/86fd65f6c3c3f7d34477fc5f3ee01dfc/href</a></iframe><p>now we will create a weekday feature as this might be a key feature.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9c4b1ad999ad867e72cdfd75cc7e0aa6/href">https://medium.com/media/9c4b1ad999ad867e72cdfd75cc7e0aa6/href</a></iframe><p>now we will create some rolling window features with window sizes like 7 and 3. The rolling window features such as mean, median, max, min, the standard deviation will be created here.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/18ed20754227f6282b39d71426e73cd1/href">https://medium.com/media/18ed20754227f6282b39d71426e73cd1/href</a></iframe><p>now we will make the data ready for use in the model such as making the categorical data encoded and normalize the numerical data.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0471ba1964ee3c4f7a862b331eb6f9ac/href">https://medium.com/media/0471ba1964ee3c4f7a862b331eb6f9ac/href</a></iframe><p>now we will generate the target variables for training.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/be629585a1074a3701033174d0bcb872/href">https://medium.com/media/be629585a1074a3701033174d0bcb872/href</a></iframe><p>now we will define our evaluation metrics. As there is no inbuilt function in Python for SMAPE we have to write out a custom loss function.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e5abe8bee4839356e745b81f6f7c7a1c/href">https://medium.com/media/e5abe8bee4839356e745b81f6f7c7a1c/href</a></iframe><p>now we will split our dataset into train and test with a test size of 33%.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/05113196e5363c4fbe4ca5b7ca5d9b32/href">https://medium.com/media/05113196e5363c4fbe4ca5b7ca5d9b32/href</a></iframe><h3>8.ML model:-</h3><p>first, we will use a linear regression model.</p><h4>8.1.Linear Regression:-</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d77c9a74a76b4f4135745f56d0dfa485/href">https://medium.com/media/d77c9a74a76b4f4135745f56d0dfa485/href</a></iframe><p>linear regression is not able to perform well with this score in test data we will get a very poor result in Kaggle. so we have to use some ensemble model.</p><h4>8.2.XgBoost:-</h4><p>in GBDT we will do some hyperparameter tuning to get better results.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e7a105b38ad4151556015652aaf834f0/href">https://medium.com/media/e7a105b38ad4151556015652aaf834f0/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/461/1*iy3JHXemk5D7sJvDr37zfQ.png" /><figcaption>the output of the above code looks like this</figcaption></figure><p>after hyperparameter tuning, we get max-depth as 15 and n-estimator as 500.now we will train the model using these parameters.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2b1e8df6969a53c891cc90f496ad30ed/href">https://medium.com/media/2b1e8df6969a53c891cc90f496ad30ed/href</a></iframe><p>this score seems good so we will try to predict for Kaggle using this model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/922/1*cHVE-ZRH2LgWBqU0YOrQ4w.png" /></figure><p>the reason for getting this bad score as we are using the predicted data for feature engineering of the future days. This increases the error exponentially. so we have to make a model such that it does not depend on the predicted data.</p><h3>9.DL model:-</h3><p>As ML models are vastly dependent on feature engineering we will use the DL model for forecasting.</p><h4>9.1.LSTM+CNN:-</h4><p>we will use the features that are extracted from the page name such as language, agent type, and access type.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fef4b162658d36367cadc094e4d74b59/href">https://medium.com/media/fef4b162658d36367cadc094e4d74b59/href</a></iframe><p>now we will use the last 264 days for model training. we will use 200 days as part of training and 64 days as a target. The reason for making the output 64 is as we are predicting 64 days in the future for Kaggle as well. we will convert our data into log1p format and train it with MAE loss as it is similar to SMAPE on original data.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/60a1b3c13a43bbaa98aa1252882265f0/href">https://medium.com/media/60a1b3c13a43bbaa98aa1252882265f0/href</a></iframe><p>now we will encode the categorical features into one-hot encoding. and split the data.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b083f896b99ed2e69a87d277d95bc8c4/href">https://medium.com/media/b083f896b99ed2e69a87d277d95bc8c4/href</a></iframe><p>now we will create our model for training. we will use an LSTM layer for the sequential data of 200 days and CNN days for one-hot encoded data. we will concatenate all the output pass this into several Dense layers.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2a09c77fb2840b0dbebb1a103003f382/href">https://medium.com/media/2a09c77fb2840b0dbebb1a103003f382/href</a></iframe><p>after training model seems to perform well, so let&#39;s predict for Kaggle using this model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/922/1*Z5pkyMDAOtqroSnDvtJL8g.png" /></figure><p>this score is almost close to the top 10% on the leader board, so we have to use some other model to improve it.</p><h4>9.2.LSTM + Embedding-layer:-</h4><p>In this model, we will encode the categorical features using a label encoder and then pass them to an embedding layer. For the sequential data, we will use the LSTM layer.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/13952469792be6bcda859b543c721372/href">https://medium.com/media/13952469792be6bcda859b543c721372/href</a></iframe><p>this model took a few extra epochs than the LSTM+CNN model. The training scores are quite similar.so let&#39;s predict for Kaggle using this model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/938/1*X4z7KtekDiIcSl1fckKZIA.png" /></figure><p>the Kaggle score is around 8% on the leader board. which is quite good for a simple LSTM model.</p><h3>10.Result:-</h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e3849711934946e4a8b0eddfc7d35191/href">https://medium.com/media/e3849711934946e4a8b0eddfc7d35191/href</a></iframe><h3>11.Deployment:-</h3><p>now we will create a pipeline for our model where we will use the pre-trained model. we will use only 800 pages for model deployment as we are using Heroku for the slug size is limited.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b4fd1e55804a3cbb8069e01125c018ce/href">https://medium.com/media/b4fd1e55804a3cbb8069e01125c018ce/href</a></iframe><p>the output of the pipeline looks like this.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3bce65f6fe2dcc78ecfc6d5da0ce5fd7/href">https://medium.com/media/3bce65f6fe2dcc78ecfc6d5da0ce5fd7/href</a></iframe><p>now we will write code to create a flask API and two HTML pages. One for taking data input and the other to show results.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cb9c83bfb9bde90be9b1e4ed08f74db9/href">https://medium.com/media/cb9c83bfb9bde90be9b1e4ed08f74db9/href</a></iframe><p>here we used Heroku for model deployment. Heroku takes two extra files named Procfile with no extension and requirements.txt for importing libraries used in deployment. There is an option for auto-deployment which will connect to the git repo and create an app link. <a href="https://webtrafficforecasting.herokuapp.com/">Here</a> is my app.</p><p>App result looks like this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/505/1*5OvjyNmU4KmZ_tH6RboywA.png" /><figcaption>input of app</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/250/1*9uIT42iX-Vvh5Ttc-x_ggA.png" /><figcaption>result of app</figcaption></figure><h3>12.Future work:-</h3><ol><li>we can use some advanced encoder-decoder models with attention mechanism so that we can use all 800 days of data to predict better results.</li><li>ML models are performing quite well in the test phase where we use real data for feature engineering but as we use predicted data for feature engineering result is getting bad. we can use a hack around, we will use LSTM results to get close to the real data, and then we will perform feature engineering on those data from XgBoost models. This might perform well.</li></ol><h3>13.Profile:-</h3><p>If u have any query feel free to contact me on <a href="https://www.linkedin.com/in/kali-charan-baksi-721653190/">Linkedin</a>. You can find my full project <a href="https://github.com/kalicharanbaksi/wikipedia-web-traffic-forecasting">Here</a>.</p><h3>14.References:-</h3><ol><li><a href="https://www.kaggle.com/c/web-traffic-time-series-forecasting/discussion/39395">https://www.kaggle.com/c/web-traffic-time-series-forecasting/discussion/39395</a></li><li><a href="https://www.kaggle.com/c/web-traffic-time-series-forecasting/discussion/43795">https://www.kaggle.com/c/web-traffic-time-series-forecasting/discussion/43795</a></li><li><a href="https://www.kaggle.com/c/web-traffic-time-series-forecasting/discussion/39370">https://www.kaggle.com/c/web-traffic-time-series-forecasting/discussion/39370</a></li><li><a href="https://medium.com/">https://medium.com/</a> ivshao53/web-traffic-time-series-Drediction-usino-arima-Istmhttps://medium.com/ ivshao53/web-traffic-time-series-Drediction-usino-arima-Istm-7ef3911845ae</li><li><a href="https://towardsdatascience.com/predicting-web-traffic-on-wikipedia-44004d6b152e">https://towardsdatascience.com/predicting-web-traffic-on-wikipedia-44004d6b152e</a></li><li><a href="https://www.appliedaicourse.com/">https://www.appliedaicourse.com/</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ee0ad354a6a7" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>